{ "eid": "btn-sign-in" } ``` rather than reasoning from fragile CSS selectors or repeatedly scanning a large DOM tree. This makes browser use more predictable for agents because observation and action are connected through a stable semantic contract. --- ## What it is Agent Web Interface is: - An MCP server for browser automation - A semantic observation layer over Puppeteer and CDP - A compact page representation for LLM agents - A stable `eid`-based action interface - A toolset for navigation, interaction, forms, screenshots, canvas, readability, and network inspection Agent Web Interface is not: - A replacement for Puppeteer - A general-purpose browser - A visual testing framework - A scraping framework - A CAPTCHA or anti-bot bypass tool Puppeteer and CDP remain the execution layer. Agent Web Interface changes what the agent sees and how it decides what to do next. --- ## How it works At a high level: 1. The agent calls a browser tool through MCP. 2. Agent Web Interface controls Chrome through Puppeteer and CDP. 3. The current page is reduced into semantic regions, readable content, and actionable elements. 4. The agent receives a compact snapshot instead of a raw browser dump. 5. The agent acts using stable element IDs. 6. Agent Web Interface waits for the page to stabilize and returns the updated state. This keeps browser lifecycle, page representation, and action execution separated. ```text AI Agent ↓ MCP Agent Web Interface ↓ semantic snapshots + stable eids Puppeteer / Chrome DevTools Protocol ↓ Chrome / Chromium ``` --- ## Example agent loop A typical browser-agent loop looks like this: 1. The agent calls `navigate` with a URL. 2. Agent Web Interface returns a compact page snapshot. 3. The agent calls `find` to locate a semantic element, such as a “Sign in” button or an email field. 4. The agent calls `click`, `type`, `select`, or `press` using the returned `eid`. 5. Agent Web Interface waits for the page to stabilize and returns the updated snapshot. 6. The agent continues from the changed page state instead of re-reading the entire DOM. This gives the model a browser interaction loop based on semantic state transitions rather than raw page internals. --- ## Example user journey The following example shows how an agent might use Agent Web Interface inside a dashboard-style web app. Task: > Find the failed payment from `client@example.com`, open it, add an internal note saying “Customer contacted. Waiting for bank confirmation.”, and confirm the note was saved. ### 1. Agent navigates to the payments dashboard Tool call: ```json { "tool": "navigate", "input": { "url": "https://dashboard.example.com/payments" } } ``` Tool response: ```xml Home Payments Customers Reports Payments Create payment Export Search payments Status Date ₹12,500 succeeded nadeem@example.com Jun 6 ₹8,999 failed client@example.com Jun 6 ₹2,400 refunded test@example.com Jun 5 ``` The agent does not need to inspect a DOM table. It sees the relevant row directly as semantic state. ### 2. Agent opens the failed payment Tool call: ```json { "tool": "click", "input": { "eid": "payment-row-2" } } ``` Tool response: ```xml Payment ₹8,999 Failed Customer client@example.com Payment method UPI Failure reason Bank declined transaction Timeline Payment created Jun 6, 10:42 AM Payment failed Jun 6, 10:43 AM Internal notes Add an internal note Save note Refund Retry payment Copy payment ID ``` Because this is a navigation, the response is a new baseline. The previous row IDs are no longer assumed valid. ### 3. Agent types the internal note Tool call: ```json { "tool": "type", "input": { "eid": "inp-internal-note", "text": "Customer contacted. Waiting for bank confirmation.", "clear": true } } ``` Tool response: ```xml Add an internal note ``` The agent receives only the changed field, not the whole page again. ### 4. Agent saves the note Tool call: ```json { "tool": "click", "input": { "eid": "btn-save-note" } } ``` Tool response: ```xml Note saved. Note saved. Customer contacted. Waiting for bank confirmation. Save note Note saved. ``` The agent can now conclude that the note was saved because both signals are present: 1. A status appeared: `Note saved.` 2. A new note row appeared with the submitted text. ### 5. Agent final answer to the user ```text Done. I opened the failed ₹8,999 payment for client@example.com, added the internal note, and confirmed that the dashboard showed “Note saved.” ``` This journey demonstrates the core interaction model: ```text navigate → baseline state click row → new page baseline type note → small diff click save → diff + observation + confirmation state ``` The agent does not receive raw DOM, brittle selectors, or a massive accessibility tree. It receives a compact state transition after every browser action. --- ## Tool response format Most browser interaction tools return a compact XML state response. The response is designed for LLM agents, not for humans reading browser internals. It favors stable semantic IDs, short tags, page regions, and incremental diffs over full DOM dumps. A typical response contains: - ``: current page state, title, URL, and step number - ``: viewport, scroll position, and active interaction layer - `` or ``: whether this is a full page state or an incremental update - ``: important transient UI changes such as dialogs, alerts, and toasts - ``: grouped actionable elements by semantic page region - Short element tags such as ``, ``, ``, ``, ``, and `` Agents should use the `id` attributes, not CSS selectors, when performing follow-up actions. ### Baseline response This is the shape after `navigate` when the agent has no previous state. ```xml Acme Docs Email Password Remember me Sign in ``` ### Diff response For same-page interactions, Agent Web Interface returns only the meaningful change. ```xml Email ``` ### Validation error Readable mutations are rendered inline inside ``. ```xml Password Sign in ``` ### Modal response Overlay layers such as modals, popovers, and drawers show the complete active overlay so the agent can reason about available actions. ```xml Delete product? Cancel Delete ``` ### Trimmed large region When a page has many repeated elements, large regions may be trimmed. The agent can call `find` with the region to retrieve more. ```xml Invoice 1001 Invoice 1002 Invoice 1003 Invoice 1004 Invoice 1005 Invoice 1048 Invoice 1049 Invoice 1050 ``` --- ## Real-world response examples These examples are illustrative, but shaped according to the actual response contract: ``, ``, `` or ``, optional ``, and region-grouped short element tags. ### GitHub-style issue page ```xml GitHub Search or jump to... Pull requests Issues Create new... Code Issues Pull requests Actions Checkout flow fails on Safari Open nadeem Edit Copy link nadeem commented Safari users cannot complete checkout after selecting Apple Pay. Add reaction Leave a comment Comment Close issue Assignees Labels Projects Milestone ``` ### Linear-style issue list ```xml Inbox My issues Views Roadmaps Projects Active issues New issue Display options Filter In Progress CORE-61 Operations console CORE-62 GitHub Actions CI pipeline Todo CORE-47 KYB Step 2 CORE-63 Fix Biome config drift ``` ### Payments dashboard ```xml Home Payments Customers Products Reports Payments Create payment Export Search payments Status Date ₹12,500 succeeded nadeem@example.com Jun 6 ₹8,999 failed client@example.com Jun 6 ₹2,400 refunded test@example.com Jun 5 ``` --- ## Tool surface Agent Web Interface exposes MCP tools across the main phases of browser use. ### Session - `list_pages` - `close_page` ### Navigation - `navigate` - `go_back` - `go_forward` - `reload` ### Observation - `snapshot` - `find` - `get_element` - `screenshot` ### Interaction - `click` - `type` - `press` - `select` - `hover` - `scroll_to` - `scroll` - `drag` - `wheel` ### Forms - `get_form` - `get_field` ### Canvas - `inspect_canvas` ### Content - `read_page` ### Network - `list_network_calls` - `search_network_calls` --- ## Quickstart Run the interactive installer — it auto-detects which AI tools you have installed and registers the MCP server and agent skill in one step: ```bash npx agent-web-interface install ``` Then ask your AI to use the browser: ```text Open https://example.com and summarize the main actions available to a user. ``` ### Target a specific harness ```bash # Claude Code (also installs the agent skill) npx agent-web-interface install --harness claude-code # Cursor npx agent-web-interface install --harness cursor # VS Code npx agent-web-interface install --harness vscode # Claude Desktop (MCP only — no skill placement) npx agent-web-interface install --harness claude-desktop # Multiple at once npx agent-web-interface install --harness cursor,vscode # All detected harnesses npx agent-web-interface install --harness all ``` ### Install flags | Flag | Description | | -------------------------- | -------------------------------------------------------------------------------------------------- | | `--harness ` | Target harness(es): `claude-code`, `cursor`, `vscode`, `claude-desktop`, `all`, or comma-separated | | `--scope project\|user` | Where to write the config (default: `project`) | | `--global` | Alias for `--scope global` | | `--project` | Alias for `--scope project` | | `--browser-mode ` | Browser mode: `auto` (default), `user`, `persistent`, `isolated` | | `--headless` | Launch Chrome in headless mode | | `--cdp-url ` | Connect to an existing Chrome DevTools Protocol endpoint | | `--pin ` | Register an exact version instead of `@latest` | | `--dry-run` | Preview changes without writing files | | `--yes` | Skip interactive prompts (non-TTY mode) | ### Check installation status ```bash npx agent-web-interface doctor ``` Prints a per-harness status table showing whether the MCP server is registered and whether the agent skill is installed. ### Skill-only installation (advanced) The [`agent-web-interface`](skills/agent-web-interface/SKILL.md) skill can also be installed independently — without the MCP server — using [`npx skills`](https://github.com/vercel-labs/skills): ```bash npx skills add lespaceman/agent-web-interface ``` This copies only the skill into your agent's `skills/` directory. You still need to register the MCP server separately (the `install` command above does both). `npx skills` is intentionally kept as a supported alternative for skill-only workflows — see [ADR-0003](docs/adr/0003-keep-npx-skills-alongside-installer.md). --- ## Manual setup (Claude Desktop / Cursor / VS Code) If you prefer to edit config files manually, add the server under the appropriate key: ```json { "mcpServers": { "agent-web-interface": { "command": "npx", "args": ["agent-web-interface@latest"] } } }

Agent Web Interface

Installation