{ "url": "https://example.com/page", "title": "Page Title", "page_map": { "headings": [{ "level": 1, "text": "...", "id": "...", "selector": "...", "char_count": 0, "preview": "..." }], "landmarks": [{ "tag": "nav", "role": "navigation", "id": "...", "selector": "...", "text_preview": "..." }], "links": [{ "text": "...", "href": "...", "selector": "..." }], "regions": [{ "kind": "Main", "label": "main panel", "handle": "@r1", "selector": "main", "visible": true, "children": [] }], "active_dialog": null, "meta": { "title": "...", "url": "...", "description": "..." }, "truncated_links": false, "truncated_forms": false, "truncated_landmarks": false, "truncated_regions": false, "truncated_controls": false } } ``` **Diff page_state**, returned on subsequent interactions on the same URL, showing only what changed: ```json { "url": "https://example.com/page", "title": "Page Title", "changed": true, "changes": { "added_headings": [], "removed_headings": [], "added_links": [], "removed_links": [], "added_landmarks": [], "removed_landmarks": [], "added_interactive": [{ "selector": "...", "tag": "...", "text": "..." }], "removed_interactive": [], "modified_interactive": [{ "selector": "...", "tag": "...", "text": "...", "state_changes": { "aria_expanded": "true" } }] } } ``` If nothing changed, `{ "url": "...", "title": "...", "changed": false }` is returned. On bridge failure, `{ "url": "unknown", "title": "unknown", "page_map": null }`. The embedded `page_map` preserves `regions` and `active_dialog` but omits the verbose `forms`, `interactive`, and `controls` arrays to keep interaction responses concise — call the `page_map` tool to retrieve those. Caps: max 50 links, 20 landmarks per page_state. ### Sub-Agent Parallelism The agent can fork child agents to crawl multiple pages concurrently. Each child gets its own browser tab, step budget, and independent state. | Setting | Default | Description | |---------|---------|-------------| | `max_concurrent_per_parent` | 5 | Max children running in parallel per parent | | `max_fork_depth` | 3 | Max nesting depth (agents forking agents) | | `max_total_agents` | 10 | Global cap across all parents | | `fork_child_max_steps` | 15 | Step budget per child agent | | `fork_wait_timeout_secs` | 60 | Timeout waiting for sub-agents | #### URL Claiming Before a child agent is spawned, its scope is registered in a shared claim registry. This prevents two sibling agents from crawling the same URL simultaneously. **Rules:** - **First-claimer-wins**, if a second agent tries to claim a URL already claimed by a sibling, the fork fails immediately with a conflict message naming the owner agent. The parent LLM sees this conflict and can adjust scope. - **Three scope types:** `SinglePage` (exact URL), `UrlList` (all-or-nothing batch), `UrlPattern` (regex, checked for overlap with all existing claims). - **RAII lifetime**, a claim is held for the life of the child. When the child finishes, is cancelled, or its parent aborts setup, the claim is released automatically and the URL becomes available again. - **Cross-type checking**, an exact URL conflicts with any already-claimed regex that matches it, and a new regex conflicts with any already-claimed exact URL it would match. - **Intra-list deduplication**, if the same URL appears twice in a `UrlList`, it is silently deduplicated (the LLM sometimes produces duplicates). Claiming is **automatic**, it happens inside `fork` before the child starts, not inside `navigate`. The agent does not call it explicitly. ### Smart Fetch Routing Every `navigate` call goes through a two-tier fetch router: 1. **HTTP first** — fast reqwest-based fetch (30s timeout, follows up to 10 redirects). 2. **Auto-escalation** — if any of the following are detected, the request is transparently replayed in a headless browser: - HTTP 403, 429, or 503 responses - JS framework markers: `__next_data__`, `__nuxt`, `__vue`, `ng-app`, `_react`, `data-reactroot` - Auth redirects: URLs containing `/login`, `/signin`, `/auth`, `/oauth`, `accounts.google.com` - Short response body (< 500 chars) with a `

Category

Provider

Auth

Env Var

Popular

Anthropic

API key

ANTHROPIC_API_KEY

OpenAI

API key

OPENAI_API_KEY

Google Gemini

API key

GEMINI_API_KEY

DeepSeek

API key

DEEPSEEK_API_KEY

Enterprise

Amazon Bedrock

AWS SigV4

AWS_ACCESS_KEY_ID

Azure OpenAI

Azure API key

AZURE_OPENAI_API_KEY

Google Vertex AI

GCP service account

GOOGLE_APPLICATION_CREDENTIALS

GitHub Copilot

Device OAuth

—

SAP AI Core

API key

SAP_AI_CORE_API_KEY

GitLab Duo

GitLab token

GITLAB_TOKEN

OSS Hosting

Groq

API key

GROQ_API_KEY

Cerebras

API key

CEREBRAS_API_KEY

DeepInfra

API key

DEEPINFRA_API_KEY

Together AI

API key

TOGETHER_API_KEY

Mistral AI

API key

MISTRAL_API_KEY

Specialized

Perplexity

API key

PERPLEXITY_API_KEY

xAI (Grok)

API key

XAI_API_KEY

Cohere

API key

COHERE_API_KEY

Alibaba (DashScope)

API key

DASHSCOPE_API_KEY

Gateways

OpenRouter

API key

OPENROUTER_API_KEY

Vercel AI

API key

VERCEL_API_KEY

Cloudflare Workers AI

API token

CLOUDFLARE_API_TOKEN

Cloudflare AI Gateway

API token

CLOUDFLARE_API_TOKEN

Other

Venice AI

API key

VENICE_API_KEY

Custom (OpenAI-compatible)

API key (optional)

—

` | This creates a `credentials.json` entry with `auth_method: "api_key"` and your `base_url`. Leave the API key blank if your server doesn't require one. Models use the `provider/model-id` format: `anthropic/claude-sonnet-4-6`, `openai/gpt-4o`, `amazon-bedrock/anthropic.claude-sonnet-4-6-20250514-v1:0`, etc. #### Providers With Non-Standard Auth **GitHub Copilot, device code flow:** ```bash acrawl auth copilot ``` 1. acrawl prints a URL (`https://github.com/login/device`) and an 8-character user code, and attempts to open your browser automatically. 2. Paste the code at the GitHub page and authorize the app. 3. acrawl polls GitHub until authorization completes, then exchanges the GitHub token for a short-lived Copilot API token, which is stored in `credentials.json`. No API key is needed, the entire flow is interactive. If authorization succeeds, credentials are stored automatically. **GitLab Duo, API key:** ```bash acrawl auth gitlab ``` Prompts for a GitLab Personal Access Token (PAT). Paste your token and press Enter. No browser redirect required. Note: GitLab Duo does not support tool calling. **Amazon Bedrock, AWS credentials:** ```bash acrawl auth amazon-bedrock ``` Prompts for AWS Access Key ID, AWS Secret Access Key, and region (default: `us-east-1`). These are stored directly in `credentials.json` and used to sign requests with AWS SigV4. **Azure OpenAI:** ```bash acrawl auth azure ``` Prompts for Resource Name (e.g. `myresource`), Deployment Name (e.g. `gpt-4o`), and API key. ### Interactive TUI The default interface is a full terminal UI with: - **Markdown rendering** with syntax highlighting and streaming output - **Slash command overlay** — type `/` to see all commands with Tab completion - **Model picker** — `/model` opens a searchable list grouped by provider category - **Auth modal** — `/auth` walks through provider setup interactively - **Session header** — shows current model, session ID, cost, and context usage in real time - **Debug mode** — `/debug` toggles raw tool call input/output in the transcript - **Reasoning effort** — `Ctrl+T` cycles through high/medium/low for reasoning models (o3, o4-mini) **Keybindings:** | Key | Action | |-----|--------| | `Enter` | Submit prompt | | `Shift+Enter` / `Ctrl+J` | Insert newline | | `PageUp` / `PageDown` | Scroll transcript | | `Ctrl+T` | Cycle reasoning effort | | `Ctrl+C` | Interrupt task (busy) or exit (idle) | | `Esc` `Esc` | Interrupt task (double-tap while busy) | | `Tab` | Auto-complete slash command | Running `acrawl` without a TTY on stdout (e.g. piped or redirected) exits with an error pointing at `acrawl prompt` for one-shot use and `acrawl --resume` for session maintenance. ### Session Management - **Auto-save** — sessions are saved automatically on exit. - **Resume** — `--resume session.json` reloads a conversation. Resume-safe slash commands (`/status`, `/compact`, `/cost`, `/config`, `/version`, `/export`, `/help`, `/clear`) can be appended to the command line. - **Export** — `/export [file]` writes a human-readable markdown transcript. - **Auto-compaction**, when cumulative input tokens exceed the threshold (default 200K), acrawl compacts the session: - **Preserved verbatim:** the most recent ~80K tokens of messages (always at least 2 messages), with tool call pairs kept intact, no `ToolResult` is ever left without its matching `ToolUse`. - **Preserved as metadata:** message counts, deduplicated tool list, last 3 user requests (160 chars each), pending work items (inferred from messages containing "todo", "next", "pending", "follow up", "remaining"), up to 10 key URLs, and the most recent non-empty message as "current work". - **Tool output pruning:** tool outputs older than the innermost 40K-token window are truncated to 2,000 chars with a `[… output truncated from N chars]` marker before the preserved window is calculated. - **Summary generation:** template-based by default (no LLM call). Opt in to LLM summarization via `compaction_llm_summarization: true` in `settings.json`, this sends the removed messages to the model and uses its output as the summary, with a fallback to the template if the LLM fails. - **Continuation prompt:** the compacted session prepends a system message instructing the agent to resume directly without recapping or asking questions. - **Browser state is unaffected**, compaction only modifies message history; the current browser tab, URL, cookies, and page state are unchanged. - **Multiple sessions** — `/session list` to browse, `/session switch ` to switch. ### Tool Allowlist Use `--allowedTools` to restrict which tools the agent can invoke (comma-separated, flag is repeatable): ``` acrawl prompt "scrape titles" --allowedTools navigate,read_content,screenshot ``` Omit `--allowedTools` to allow all 42 tools. Useful for locking down a crawl to read-only tools or excluding `fork`/`wait_for_subagents` when sub-agent parallelism is not desired. ### MCP Extensibility acrawl supports [Model Context Protocol](https://modelcontextprotocol.io) servers as a **client**, allowing you to extend the agent with custom tools. MCP tools are namespaced as `server_name__tool_name` and available alongside the built-in 42. Supported transports: **stdio**, **SSE**, **HTTP**, **WebSocket**. ### MCP Server (expose acrawl as a tool) `acrawl mcp` starts a built-in MCP server that exposes acrawl's browser automation capabilities to external agents like Claude Code, Cursor, VS Code, Zed, JetBrains, TRAE, Gemini CLI, or any MCP-compatible client. The server provides **39 tools** in three modes: **Direct browser tools (31)** — fine-grained control for clients that orchestrate themselves: `navigate`, `click`, `click_at`, `fill_form`, `page_map`, `read_content`, `screenshot`, `go_back`, `scroll`, `wait`, `select_option`, `execute_js`, `hover`, `press_key`, `switch_tab`, `list_resources`, `save_file`, `set_device`, `refresh`, `list_network_activity`, `inspect_request`, `list_page_logs`, `inspect_log`, `list_websocket_activity`, `inspect_websocket`, `get_page_performance`, `inspect_cookies`, `inspect_storage`, `measure_coverage`, `audit_accessibility`, `intercept_network` **Script tools (7)** — deterministic multi-step automation without per-step LLM calls: `run_script`, `save_script`, `list_scripts`, `read_script`, `wait_for_scripts`, `script_status`, `cancel_script` **Autonomous agent (1)** — delegate a full crawl task: - **`run_goal`** — Execute a high-level crawl goal autonomously. The agent plans, navigates, and extracts data using its own LLM loop. Requires `~/.acrawl/credentials.json` configured with a model. **Transport:** stdio only (no SSE / HTTP / WebSocket in this release). #### Quick install ```bash acrawl mcp install ``` Interactive installer that auto-detects your IDEs, lets you toggle which to configure (Space to select, Enter to confirm), and writes the correct config for each. Supports global (user-level) and project-level scopes. Supported clients: **Claude Code**, **Claude Desktop**, **Cursor**, **Windsurf**, **VS Code (Copilot)**, **OpenCode**, **Zed**, **TRAE**, **JetBrains IDEs**, **Gemini CLI**, **Qwen Code**, **Codex CLI**, **Hermes**, **OpenClaw**, **Goose**, **Crush**, **Aider**. #### Manual configuration If you prefer to configure manually, add this to your IDE's MCP config file:

IDE	Config file	Configuration
Claude Code	`.mcp.json` (project) `~/.claude.json` (user)	```json { "mcpServers": { "acrawl": { "command": "acrawl", "args": ["mcp"] } } }

acrawl

Installation