` (or first H1 if missing). | | `crawled_at` | string (ISO 8601) | UTC timestamp of when the page was fetched. | | `citation` | string | Pre-formatted academic-style citation including access date. | | `tool` | string | Always `"markcrawl"`. Helps when merging output from multiple crawlers. | | `text` | string | Clean Markdown content (nav/footer/scripts stripped). | | `downloads` | array (optional) | Present when `download_types` is set; one entry per saved binary: `{url, path, size_bytes, content_type}`. | | `images` | array (optional) | Present when `--download-images` is set; lists saved image paths. | | `screenshot` | string (optional) | Present when `--screenshot` is set; relative path to the PNG/JPEG capture. | ## Common Recipes Runnable examples for the most common patterns: - **Single-page scrapes** — including JS-rendered pages (React, Vue, YouTube) - **Whole-site crawls** — docs, blogs, subsections; resume interrupted runs - **URL filtering** — `--exclude-path`, `--include-path`, `--dry-run`, smart sampling - **Extraction backends** — BS4 (default), trafilatura, ensemble, ReaderLM-v2 - **Binary downloads** — images, PDFs (with pre-fetch filter callbacks), DOCX - **Screenshots** — full-page or cropped, PNG or JPEG - **End-to-end use cases** — competitive analysis, RAG chatbot, API-docs → code-gen Full recipes with copy-paste commands and expected outputs: **[docs/RECIPES.md](docs/RECIPES.md)**. <details> <summary>How it compares to other crawlers — decision matrix</summary> ### Pick this tool when… | If you need… | Use… | Why | |---|---|---| | Clean Markdown for LLM/RAG ingestion, run locally, no API keys | **MarkCrawl** | Default install bundles local embedder ($0 API spend); strips nav/scripts; produces JSONL with citations out of the box | | A hosted scraping API (no infra to run) | **FireCrawl** | SaaS option; pay-per-call; outsources crawling entirely | | AI-native crawling with built-in LLM extraction | **Crawl4AI** | Deeper LLM-extraction primitives; built-in Playwright | | Massive distributed crawling (millions of pages, custom pipelines) | **Scrapy** | Battle-tested framework; rich plugin ecosystem; spider architecture | | JavaScript-heavy automation without framework overhead | **Playwright** (direct) | Lower-level control over browser automation | | Sites behind login/auth or aggressive bot protection | **None of the above** (build custom) | See [When NOT to use MarkCrawl](#when-not-to-use-markcrawl); same constraints apply to most public crawlers | ### Feature comparison | | MarkCrawl | FireCrawl | Crawl4AI | Scrapy | |---|---|---|---|---| | License | MIT | AGPL-3.0 | Apache-2.0 | BSD-3 | | Install | `pip install markcrawl` | SaaS or self-host | pip + Playwright | pip + framework | | Output | Markdown + JSONL | Markdown + JSON | Markdown | Custom pipelines | | JS rendering | Optional (`--render-js`) | Built-in | Built-in | Plugin | | LLM extraction | Optional add-on | Via API | Built-in | None | | Local-only operation | ✅ | ❌ (SaaS) | ✅ | ✅ | | Citations + timestamps in output | ✅ | Partial | ❌ | Manual | | Best for | Single-site crawl → clean Markdown | Hosted scraping API | AI-native crawling | Large-scale distributed | MarkCrawl's niche is **focused-scope RAG ingestion** — narrow crawls of docs/blogs/product sites that produce LLM-ready Markdown with minimal junk. For broader scope or bigger scale, the other tools above are stronger choices. ### Benchmark results (6 tools, May 2026) **Speed:** scrapy+md is fastest (5.0 pages/sec), markcrawl at 2.7. Playwright-based tools average 1.4-2.1 pages/sec. **Output cleanliness:** markcrawl has the lowest nav pollution (53 words vs 500+ for others) — less junk in your embeddings. **RAG answer quality:** markcrawl scores 3.77/5 on answer quality with the fewest chunks (27,193 total, 2.2x fewer than the most), keeping embedding costs low. | Tool | Chunks/page | Answer Quality (/5) | Annual cost (100K pages, 1K queries/day) | |---|---|---|---| | **markcrawl** | **18.7** | **3.77** | **$4,505** | | scrapy+md | 31.7 | 3.68 | $5,464 | | crawl4ai | 16.8 | 4.72 | $6,960 | | colly+md | 40.6 | 4.36 | $7,213 | | playwright | 39.0 | 4.48 | $7,320 | | crawlee | 40.5 | 4.68 | $7,467 | Full benchmark data: [docs/BENCHMARKS.md](docs/BENCHMARKS.md) | Methodology: [llm-crawler-benchmarks](https://github.com/AIMLPM/llm-crawler-benchmarks) > **Methodology caveat (numbers as of bench v1.4, 2026-05-11):** the v1.4 leaderboard sourced test queries from a single high-coverage crawler's output. The bench is actively being updated in v1.5 to source queries from each site's own sitemap independent of any crawler ([release notes](https://github.com/AIMLPM/llm-crawler-benchmarks/blob/main/docs/V14_RELEASE_NOTES.md)). Numbers above are single-trial; multi-trial measurement is on the v1.5.1 roadmap. Treat individual rankings as point-in-time signal, not steady-state. </details> ## Installation ```bash pip install markcrawl # Core crawler + chunker + local embedder # (no API keys required for embedding) ``` Optional add-ons (tasks beyond the crawl-and-embed core): ```bash pip install markcrawl[js] # + JavaScript rendering (Playwright) pip install markcrawl[extract] # + LLM extraction (OpenAI, Claude, Gemini, Grok) pip install markcrawl[upload] # + Supabase upload integration pip install markcrawl[mcp] # + MCP server for AI agents pip install markcrawl[langchain] # + LangChain tool wrappers pip install markcrawl[all] # Everything ``` For Playwright, also run `playwright install chromium` after installing. **Lean install** (skip the local-embedder dep stack — you'll need an `OPENAI_API_KEY` and pass `embedding_model="text-embedding-3-small"` for any embedding work): ```bash pip install --no-deps markcrawl beautifulsoup4 lxml markdownify requests certifi tenacity ``` <details> <summary>Install from source (for development)</summary> ```bash git clone https://github.com/AIMLPM/markcrawl.git cd markcrawl python -m venv .venv source .venv/bin/activate pip install -e ".[all]" ``` </details> ## Crawling ```bash markcrawl --base https://www.example.com --out ./output --show-progress ``` Add flags as needed: ```bash markcrawl \ --base https://www.example.com \ --out ./output \ --include-subdomains \ # crawl sub.example.com too --render-js \ # render JavaScript (React, Vue, etc.) --concurrency 5 \ # fetch 5 pages in parallel --proxy http://proxy:8080 \ # route through a proxy --max-pages 200 \ # stop after 200 pages --format markdown \ # or "text" for plain text --show-progress ``` Resume an interrupted crawl: ```bash markcrawl --base https://www.example.com --out ./output --resume --show-progress ``` ### Output Each page becomes a `.md` file with a citation header: ```markdown # Getting Started > URL: https://docs.example.com/getting-started > Crawled: April 04, 2026 > Citation: Getting Started. docs.example.com. Available at: https://docs.example.com/getting-started [Accessed April 04, 2026]. Welcome to the platform. This guide walks you through installation... ``` Navigation, footer, cookie banners, and scripts are stripped. Only the main content remains. <details> <summary>All crawler CLI arguments</summary> | Argument | Description | |---|---| | `--base` | Base site URL to crawl | | `--out` | Output directory | | `--format` | `markdown` or `text` (default: `markdown`) | | `--show-progress` | Print progress and crawl events | | `--render-js` | Render JavaScript with Playwright before extracting | | `--concurrency` | Pages to fetch in parallel (default: `1`) | | `--proxy` | HTTP/HTTPS proxy URL | | `--resume` | Resume from saved state | | `--include-subdomains` | Include subdomains under the base domain | | `--max-pages` | Max pages to save; `0` = unlimited (default: `500`) | | `--delay` | Minimum delay between requests in seconds (default: `0`, adaptive throttle adjusts automatically) | | `--timeout` | Per-request timeout in seconds (default: `15`) | | `--min-words` | Skip pages with fewer words (default: `20`) | | `--user-agent` | Override the default user agent | | `--use-sitemap` / `--no-sitemap` | Enable/disable sitemap discovery. Use `--no-sitemap` when you want to scrape a specific page or subsection — without it, large sites (YouTube, GitHub) may discover thousands of unrelated pages via their sitemap | | `--exclude-path` | Glob pattern to exclude URL paths (e.g. `'/job/*'`). Can be repeated | | `--include-path` | Glob pattern to include URL paths (e.g. `'/blog/*'`). Only matching paths are crawled. Can be repeated | | `--dry-run` | Discover URLs (via sitemap/links) and print them without fetching content | | `--smart-sample` | Auto-detect templated URL patterns and sample from large clusters instead of crawling every page | | `--sample-size` | Pages to sample per templated cluster (default: `5`, used with `--smart-sample`) | | `--sample-threshold` | Clusters larger than this are sampled (default: `20`, used with `--smart-sample`) | | `--auto-resume` | Automatically resume if saved state exists, otherwise start fresh | | `--cross-dedup` | Skip pages already seen in previous crawls to the same output directory | | `--prioritize-links` | Score discovered links by predicted content yield — crawl high-value pages first | | `--extractor` | Content extraction backend: `default`, `trafilatura`, `ensemble`, or `readerlm` | | `--download-images` | Download images from the content area to `assets/` and use local paths in Markdown | | `--min-image-size` | Minimum image file size in bytes to keep (default: `5000`). Smaller images are skipped | | `--i18n-filter` | Skip URLs under locale path segments (`/fr/`, `/de-DE/`, `/zh-Hans/`, ...) — generic, no per-domain config | | `--title-at-top` | Prepend `# {title}` to the `text` field of every JSONL row when not already present — top-MRR RAG recipe | </details> ## Optional: structured extraction If you need structured data (not just text), the extraction add-on uses an LLM to pull specific fields from each page. ```bash pip install markcrawl[extract] markcrawl-extract \ --jsonl ./output/pages.jsonl \ --fields company_name pricing features \ --show-progress ``` Auto-discover fields across multiple crawled sites: ```bash markcrawl-extract \ --jsonl ./comp1/pages.jsonl ./comp2/pages.jsonl ./comp3/pages.jsonl \ --auto-fields \ --context "competitor pricing analysis" \ --show-progress ``` Supports OpenAI, Anthropic (Claude), Google Gemini, and xAI (Grok) via `--provider`. <details> <summary>Extraction details</summary> ### Provider and model selection ```bash markcrawl-extract --jsonl ... --fields pricing --provider openai # default markcrawl-extract --jsonl ... --fields pricing --provider anthropic # Claude markcrawl-extract --jsonl ... --fields pricing --provider gemini # Gemini markcrawl-extract --jsonl ... --fields pricing --provider grok # Grok markcrawl-extract --jsonl ... --fields pricing --model gpt-4o # override model ``` | Provider | API key env var | Default model | |---|---|---| | OpenAI | `OPENAI_API_KEY` | `gpt-4o-mini` | | Anthropic | `ANTHROPIC_API_KEY` | `claude-sonnet-4-20250514` | | Google Gemini | `GEMINI_API_KEY` | `gemini-2.0-flash` | | xAI (Grok) | `XAI_API_KEY` | `grok-3-mini-fast` | ### All extraction CLI arguments | Argument | Description | |---|---| | `--jsonl` | Path(s) to `pages.jsonl` — pass multiple for cross-site analysis | | `--fields` | Field names to extract (space-separated) | | `--auto-fields` | Auto-discover fields by sampling pages | | `--context` | Describe your goal for auto-discovery | | `--sample-size` | Pages to sample for auto-discovery (default: `3`) | | `--provider` | `openai`, `anthropic`, `gemini`, or `grok` | | `--model` | Override the default model | | `--output` | Output path (default: `extracted.jsonl`) | | `--delay` | Delay between LLM calls in seconds (default: `0.25`) | | `--show-progress` | Print progress | ### Output format Extracted rows include LLM attribution: ```json { "url": "https://competitor.com/pricing", "citation": "Pricing. competitor.com. Available at: ... [Accessed April 04, 2026].", "pricing_tiers": "Starter ($29/mo), Pro ($99/mo), Enterprise (contact sales)", "extracted_by": "gpt-4o-mini (openai)", "extraction_note": "Field values were extracted by an LLM and may be interpreted, not verbatim." } ``` </details> ## Optional: Supabase vector search (RAG) Chunk pages, generate embeddings, and upload to Supabase with pgvector: ```bash pip install markcrawl[upload] markcrawl --base https://docs.example.com --out ./output --show-progress markcrawl-upload --jsonl ./output/pages.jsonl --show-progress ``` Requires `SUPABASE_URL`, `SUPABASE_KEY`, and `OPENAI_API_KEY`. See **[docs/SUPABASE.md](docs/SUPABASE.md)** for table setup, query examples, and recommendations. ## Optional: agent integrations MarkCrawl includes integrations for AI agents. Each is an optional add-on. <details> <summary>MCP Server (Claude Desktop, Cursor, Windsurf)</summary> ```bash pip install markcrawl[mcp] ``` ```json { "mcpServers": { "markcrawl": { "command": "python", "args": ["-m", "markcrawl.mcp

{ "url": "https://quotes.toscrape.com/", "title": "Quotes to Scrape", "crawled_at": "2026-04-04T12:30:00Z", "citation": "Quotes to Scrape. quotes.toscrape.com. Available at: https://quotes.toscrape.com/ [Accessed April 04, 2026].", "tool": "markcrawl", "text": "# Quotes to Scrape\n\n> "The world as we have created it is a process of our thinking..." — Albert Einstein\n\nTags: change, deep-thoughts, thinking, world..." } ``` **Schema** — every page in `pages.jsonl` has these fields: | Field | Type | Description | |---|---|---| | `url` | string | Original URL fetched. | | `title` | string | Page title from `` (or first H1 if missing). | | `crawled_at` | string (ISO 8601) | UTC timestamp of when the page was fetched. | | `citation` | string | Pre-formatted academic-style citation including access date. | | `tool` | string | Always `"markcrawl"`. Helps when merging output from multiple crawlers. | | `text` | string | Clean Markdown content (nav/footer/scripts stripped). | | `downloads` | array (optional) | Present when `download_types` is set; one entry per saved binary: `{url, path, size_bytes, content_type}`. | | `images` | array (optional) | Present when `--download-images` is set; lists saved image paths. | | `screenshot` | string (optional) | Present when `--screenshot` is set; relative path to the PNG/JPEG capture. | ## Common Recipes Runnable examples for the most common patterns: - **Single-page scrapes** — including JS-rendered pages (React, Vue, YouTube) - **Whole-site crawls** — docs, blogs, subsections; resume interrupted runs - **URL filtering** — `--exclude-path`, `--include-path`, `--dry-run`, smart sampling - **Extraction backends** — BS4 (default), trafilatura, ensemble, ReaderLM-v2 - **Binary downloads** — images, PDFs (with pre-fetch filter callbacks), DOCX - **Screenshots** — full-page or cropped, PNG or JPEG - **End-to-end use cases** — competitive analysis, RAG chatbot, API-docs → code-gen Full recipes with copy-paste commands and expected outputs: **[docs/RECIPES.md](docs/RECIPES.md)**. <details> <summary>How it compares to other crawlers — decision matrix</summary> ### Pick this tool when… | If you need… | Use… | Why | |---|---|---| | Clean Markdown for LLM/RAG ingestion, run locally, no API keys | **MarkCrawl** | Default install bundles local embedder ($0 API spend); strips nav/scripts; produces JSONL with citations out of the box | | A hosted scraping API (no infra to run) | **FireCrawl** | SaaS option; pay-per-call; outsources crawling entirely | | AI-native crawling with built-in LLM extraction | **Crawl4AI** | Deeper LLM-extraction primitives; built-in Playwright | | Massive distributed crawling (millions of pages, custom pipelines) | **Scrapy** | Battle-tested framework; rich plugin ecosystem; spider architecture | | JavaScript-heavy automation without framework overhead | **Playwright** (direct) | Lower-level control over browser automation | | Sites behind login/auth or aggressive bot protection | **None of the above** (build custom) | See [When NOT to use MarkCrawl](#when-not-to-use-markcrawl); same constraints apply to most public crawlers | ### Feature comparison | | MarkCrawl | FireCrawl | Crawl4AI | Scrapy | |---|---|---|---|---| | License | MIT | AGPL-3.0 | Apache-2.0 | BSD-3 | | Install | `pip install markcrawl` | SaaS or self-host | pip + Playwright | pip + framework | | Output | Markdown + JSONL | Markdown + JSON | Markdown | Custom pipelines | | JS rendering | Optional (`--render-js`) | Built-in | Built-in | Plugin | | LLM extraction | Optional add-on | Via API | Built-in | None | | Local-only operation | ✅ | ❌ (SaaS) | ✅ | ✅ | | Citations + timestamps in output | ✅ | Partial | ❌ | Manual | | Best for | Single-site crawl → clean Markdown | Hosted scraping API | AI-native crawling | Large-scale distributed | MarkCrawl's niche is **focused-scope RAG ingestion** — narrow crawls of docs/blogs/product sites that produce LLM-ready Markdown with minimal junk. For broader scope or bigger scale, the other tools above are stronger choices. ### Benchmark results (6 tools, May 2026) **Speed:** scrapy+md is fastest (5.0 pages/sec), markcrawl at 2.7. Playwright-based tools average 1.4-2.1 pages/sec. **Output cleanliness:** markcrawl has the lowest nav pollution (53 words vs 500+ for others) — less junk in your embeddings. **RAG answer quality:** markcrawl scores 3.77/5 on answer quality with the fewest chunks (27,193 total, 2.2x fewer than the most), keeping embedding costs low. | Tool | Chunks/page | Answer Quality (/5) | Annual cost (100K pages, 1K queries/day) | |---|---|---|---| | **markcrawl** | **18.7** | **3.77** | **$4,505** | | scrapy+md | 31.7 | 3.68 | $5,464 | | crawl4ai | 16.8 | 4.72 | $6,960 | | colly+md | 40.6 | 4.36 | $7,213 | | playwright | 39.0 | 4.48 | $7,320 | | crawlee | 40.5 | 4.68 | $7,467 | Full benchmark data: [docs/BENCHMARKS.md](docs/BENCHMARKS.md) | Methodology: [llm-crawler-benchmarks](https://github.com/AIMLPM/llm-crawler-benchmarks) > **Methodology caveat (numbers as of bench v1.4, 2026-05-11):** the v1.4 leaderboard sourced test queries from a single high-coverage crawler's output. The bench is actively being updated in v1.5 to source queries from each site's own sitemap independent of any crawler ([release notes](https://github.com/AIMLPM/llm-crawler-benchmarks/blob/main/docs/V14_RELEASE_NOTES.md)). Numbers above are single-trial; multi-trial measurement is on the v1.5.1 roadmap. Treat individual rankings as point-in-time signal, not steady-state. </details> ## Installation ```bash pip install markcrawl # Core crawler + chunker + local embedder # (no API keys required for embedding) ``` Optional add-ons (tasks beyond the crawl-and-embed core): ```bash pip install markcrawl[js] # + JavaScript rendering (Playwright) pip install markcrawl[extract] # + LLM extraction (OpenAI, Claude, Gemini, Grok) pip install markcrawl[upload] # + Supabase upload integration pip install markcrawl[mcp] # + MCP server for AI agents pip install markcrawl[langchain] # + LangChain tool wrappers pip install markcrawl[all] # Everything ``` For Playwright, also run `playwright install chromium` after installing. **Lean install** (skip the local-embedder dep stack — you'll need an `OPENAI_API_KEY` and pass `embedding_model="text-embedding-3-small"` for any embedding work): ```bash pip install --no-deps markcrawl beautifulsoup4 lxml markdownify requests certifi tenacity ``` <details> <summary>Install from source (for development)</summary> ```bash git clone https://github.com/AIMLPM/markcrawl.git cd markcrawl python -m venv .venv source .venv/bin/activate pip install -e ".[all]" ``` </details> ## Crawling ```bash markcrawl --base https://www.example.com --out ./output --show-progress ``` Add flags as needed: ```bash markcrawl \ --base https://www.example.com \ --out ./output \ --include-subdomains \ # crawl sub.example.com too --render-js \ # render JavaScript (React, Vue, etc.) --concurrency 5 \ # fetch 5 pages in parallel --proxy http://proxy:8080 \ # route through a proxy --max-pages 200 \ # stop after 200 pages --format markdown \ # or "text" for plain text --show-progress ``` Resume an interrupted crawl: ```bash markcrawl --base https://www.example.com --out ./output --resume --show-progress ``` ### Output Each page becomes a `.md` file with a citation header: ```markdown # Getting Started > URL: https://docs.example.com/getting-started > Crawled: April 04, 2026 > Citation: Getting Started. docs.example.com. Available at: https://docs.example.com/getting-started [Accessed April 04, 2026]. Welcome to the platform. This guide walks you through installation... ``` Navigation, footer, cookie banners, and scripts are stripped. Only the main content remains. <details> <summary>All crawler CLI arguments</summary> | Argument | Description | |---|---| | `--base` | Base site URL to crawl | | `--out` | Output directory | | `--format` | `markdown` or `text` (default: `markdown`) | | `--show-progress` | Print progress and crawl events | | `--render-js` | Render JavaScript with Playwright before extracting | | `--concurrency` | Pages to fetch in parallel (default: `1`) | | `--proxy` | HTTP/HTTPS proxy URL | | `--resume` | Resume from saved state | | `--include-subdomains` | Include subdomains under the base domain | | `--max-pages` | Max pages to save; `0` = unlimited (default: `500`) | | `--delay` | Minimum delay between requests in seconds (default: `0`, adaptive throttle adjusts automatically) | | `--timeout` | Per-request timeout in seconds (default: `15`) | | `--min-words` | Skip pages with fewer words (default: `20`) | | `--user-agent` | Override the default user agent | | `--use-sitemap` / `--no-sitemap` | Enable/disable sitemap discovery. Use `--no-sitemap` when you want to scrape a specific page or subsection — without it, large sites (YouTube, GitHub) may discover thousands of unrelated pages via their sitemap | | `--exclude-path` | Glob pattern to exclude URL paths (e.g. `'/job/*'`). Can be repeated | | `--include-path` | Glob pattern to include URL paths (e.g. `'/blog/*'`). Only matching paths are crawled. Can be repeated | | `--dry-run` | Discover URLs (via sitemap/links) and print them without fetching content | | `--smart-sample` | Auto-detect templated URL patterns and sample from large clusters instead of crawling every page | | `--sample-size` | Pages to sample per templated cluster (default: `5`, used with `--smart-sample`) | | `--sample-threshold` | Clusters larger than this are sampled (default: `20`, used with `--smart-sample`) | | `--auto-resume` | Automatically resume if saved state exists, otherwise start fresh | | `--cross-dedup` | Skip pages already seen in previous crawls to the same output directory | | `--prioritize-links` | Score discovered links by predicted content yield — crawl high-value pages first | | `--extractor` | Content extraction backend: `default`, `trafilatura`, `ensemble`, or `readerlm` | | `--download-images` | Download images from the content area to `assets/` and use local paths in Markdown | | `--min-image-size` | Minimum image file size in bytes to keep (default: `5000`). Smaller images are skipped | | `--i18n-filter` | Skip URLs under locale path segments (`/fr/`, `/de-DE/`, `/zh-Hans/`, ...) — generic, no per-domain config | | `--title-at-top` | Prepend `# {title}` to the `text` field of every JSONL row when not already present — top-MRR RAG recipe | </details> ## Optional: structured extraction If you need structured data (not just text), the extraction add-on uses an LLM to pull specific fields from each page. ```bash pip install markcrawl[extract] markcrawl-extract \ --jsonl ./output/pages.jsonl \ --fields company_name pricing features \ --show-progress ``` Auto-discover fields across multiple crawled sites: ```bash markcrawl-extract \ --jsonl ./comp1/pages.jsonl ./comp2/pages.jsonl ./comp3/pages.jsonl \ --auto-fields \ --context "competitor pricing analysis" \ --show-progress ``` Supports OpenAI, Anthropic (Claude), Google Gemini, and xAI (Grok) via `--provider`. <details> <summary>Extraction details</summary> ### Provider and model selection ```bash markcrawl-extract --jsonl ... --fields pricing --provider openai # default markcrawl-extract --jsonl ... --fields pricing --provider anthropic # Claude markcrawl-extract --jsonl ... --fields pricing --provider gemini # Gemini markcrawl-extract --jsonl ... --fields pricing --provider grok # Grok markcrawl-extract --jsonl ... --fields pricing --model gpt-4o # override model ``` | Provider | API key env var | Default model | |---|---|---| | OpenAI | `OPENAI_API_KEY` | `gpt-4o-mini` | | Anthropic | `ANTHROPIC_API_KEY` | `claude-sonnet-4-20250514` | | Google Gemini | `GEMINI_API_KEY` | `gemini-2.0-flash` | | xAI (Grok) | `XAI_API_KEY` | `grok-3-mini-fast` | ### All extraction CLI arguments | Argument | Description | |---|---| | `--jsonl` | Path(s) to `pages.jsonl` — pass multiple for cross-site analysis | | `--fields` | Field names to extract (space-separated) | | `--auto-fields` | Auto-discover fields by sampling pages | | `--context` | Describe your goal for auto-discovery | | `--sample-size` | Pages to sample for auto-discovery (default: `3`) | | `--provider` | `openai`, `anthropic`, `gemini`, or `grok` | | `--model` | Override the default model | | `--output` | Output path (default: `extracted.jsonl`) | | `--delay` | Delay between LLM calls in seconds (default: `0.25`) | | `--show-progress` | Print progress | ### Output format Extracted rows include LLM attribution: ```json { "url": "https://competitor.com/pricing", "citation": "Pricing. competitor.com. Available at: ... [Accessed April 04, 2026].", "pricing_tiers": "Starter ($29/mo), Pro ($99/mo), Enterprise (contact sales)", "extracted_by": "gpt-4o-mini (openai)", "extraction_note": "Field values were extracted by an LLM and may be interpreted, not verbatim." } ``` </details> ## Optional: Supabase vector search (RAG) Chunk pages, generate embeddings, and upload to Supabase with pgvector: ```bash pip install markcrawl[upload] markcrawl --base https://docs.example.com --out ./output --show-progress markcrawl-upload --jsonl ./output/pages.jsonl --show-progress ``` Requires `SUPABASE_URL`, `SUPABASE_KEY`, and `OPENAI_API_KEY`. See **[docs/SUPABASE.md](docs/SUPABASE.md)** for table setup, query examples, and recommendations. ## Optional: agent integrations MarkCrawl includes integrations for AI agents. Each is an optional add-on. <details> <summary>MCP Server (Claude Desktop, Cursor, Windsurf)</summary> ```bash pip install markcrawl[mcp] ``` ```json { "mcpServers": { "markcrawl": { "command": "python", "args": ["-m", "markcrawl.mcp_server"] } } }</article></body></html>