🎯

firecrawl-automation

🎯Skill
composiohq/awesome-claude-skills2 Installs66,868Last Updated May 22, 2026

Installation

$npx skills add composiohq/awesome-claude-skills --skill firecrawl-automation

Run it in a terminal; it asks which AI coding tools to install it for (Claude Code, Codex, Cursor and more).

Uses skills.sh Β· Node.js 22.20+

After installing

Ask for related work and the agent uses it on its own, or call it yourself.

Run it (Claude Code)/Firecrawl Automation
Install by hand

Copy the whole skill folder from GitHub to the place below. Copying only SKILL.md can leave out files the skill uses.

Open the skill folder β†’
  • This project .claude/skills/firecrawl-automation/
  • All projects ~/.claude/skills/firecrawl-automation/

These paths are for Claude Code. For other tools, the command above puts it in the right place.

Automate web crawling and data extraction with Firecrawl -- scrape pages, crawl sites, extract structured data, batch scrape URLs, and map website structures through the Composio Firecrawl integration.

Firecrawl Automation

Run Firecrawl web crawling and extraction directly from Claude Code. Scrape individual pages, crawl entire sites, extract structured data with AI, batch process URL lists, and map website structures without leaving your terminal.

Toolkit docs: composio.dev/toolkits/firecrawl


Setup

  1. Add the Composio MCP server to your configuration:
    https://rube.app/mcp
    
  2. Connect your Firecrawl account when prompted. The agent will provide an authentication link.
  3. Be mindful of credit consumption -- scope your crawls tightly and test on small URL sets before scaling.

Core Workflows

1. Scrape a Single Page

Fetch content from a URL in multiple formats with optional browser actions for dynamic pages.

Tool: FIRECRAWL_SCRAPE

Key parameters:

  • url (required) -- fully qualified URL to scrape
  • formats -- output formats: markdown (default), html, rawHtml, links, screenshot, json
  • onlyMainContent (default true) -- extract main content only, excluding nav/footer/ads
  • waitFor -- milliseconds to wait for JS rendering (default 0)
  • timeout -- max wait in ms (default 30000)
  • actions -- browser actions before scraping (click, write, wait, press, scroll)
  • includeTags / excludeTags -- filter by HTML tags
  • jsonOptions -- for structured extraction with schema and/or prompt

Example prompt: "Scrape the main content from https://example.com/pricing as markdown"


2. Crawl an Entire Site

Discover and scrape multiple pages from a website with configurable depth, path filters, and concurrency.

Tool: FIRECRAWL_CRAWL_V2

Key parameters:

  • url (required) -- starting URL for the crawl
  • limit (default 10) -- max pages to crawl
  • maxDiscoveryDepth -- depth limit from the root page
  • includePaths / excludePaths -- regex patterns for URL paths
  • allowSubdomains -- include subdomains (default false)
  • crawlEntireDomain -- follow sibling/parent links, not just children (default false)
  • sitemap -- include (default), skip, or only
  • prompt -- natural language to auto-configure crawler settings
  • scrapeOptions_formats -- output format for each page
  • scrapeOptions_onlyMainContent -- main content extraction per page

Example prompt: "Crawl the docs section of firecrawl.dev, max 50 pages, only paths matching docs"


3. Extract Structured Data

Extract structured JSON data from web pages using AI with a natural language prompt or JSON schema.

Tool: FIRECRAWL_EXTRACT

Key parameters:

  • urls (required) -- array of URLs to extract from (max 10 in beta). Supports wildcards like https://example.com/blog/*
  • prompt -- natural language description of what to extract
  • schema -- JSON Schema defining the desired output structure
  • enable_web_search -- allow crawling links outside initial domains (default false)

At least one of prompt or schema must be provided.

Check extraction status with FIRECRAWL_EXTRACT_GET using the returned job id.

Example prompt: "Extract company name, pricing tiers, and feature lists from https://example.com/pricing"


4. Batch Scrape Multiple URLs

Scrape many URLs concurrently with shared configuration for efficient bulk data collection.

Tool: FIRECRAWL_BATCH_SCRAPE

Key parameters:

  • urls (required) -- array of URLs to scrape
  • formats -- output format for all pages (default markdown)
  • onlyMainContent (default true) -- main content extraction
  • maxConcurrency -- parallel scrape limit
  • ignoreInvalidURLs (default true) -- skip bad URLs instead of failing the batch
  • location -- geolocation settings with country code
  • actions -- browser actions applied to each page
  • blockAds (default true) -- block advertisements

Example prompt: "Batch scrape these 20 product page URLs as markdown with ad blocking"


5. Map Website Structure

Discover all URLs on a website from a starting URL, useful for planning crawls or auditing site structure.

Tool: FIRECRAWL_MAP_MULTIPLE_URLS_BASED_ON_OPTIONS

Key parameters:

  • url (required) -- starting URL (must be https:// or http://)
  • search -- guide URL discovery toward specific page types
  • limit (default 5000, max 100000) -- max URLs to return
  • includeSubdomains (default true) -- include subdomains
  • ignoreQueryParameters (default true) -- dedupe URLs differing only by query params
  • sitemap -- include, skip, or only

Example prompt: "Map all URLs on docs.example.com, focusing on API reference pages"


6. Monitor and Manage Crawl Jobs

Track crawl progress, retrieve results, and cancel runaway jobs.

Tools: FIRECRAWL_CRAWL_GET, FIRECRAWL_GET_THE_STATUS_OF_A_CRAWL_JOB, FIRECRAWL_CANCEL_A_CRAWL_JOB

  • FIRECRAWL_CRAWL_GET -- get status, progress, credits used, and crawled page data
  • FIRECRAWL_CANCEL_A_CRAWL_JOB -- stop an active or queued crawl

Both require the crawl job id (UUID) returned when the crawl was initiated.

Example prompt: "Check the status of crawl job 019b0806-b7a1-7652-94c1-e865b5d2e89a"


Known Pitfalls

  • Rate limiting: Firecrawl can trigger "Rate limit exceeded" errors (429). Prefer FIRECRAWL_BATCH_SCRAPE over many individual FIRECRAWL_SCRAPE calls, and implement backoff on 429/5xx responses.
  • Credit consumption: FIRECRAWL_EXTRACT can fail with "Insufficient credits." Scope tightly and avoid broad homepage URLs that yield sparse fields. Test on small URL sets first.
  • Nested error responses: Per-page failures may be nested in response.data.code (e.g., SCRAPE_DNS_RESOLUTION_ERROR) even when the outer API call succeeds. Always validate inner status/error fields.
  • JS-heavy pages: Non-rendered fetches may miss key content. Use waitFor (e.g., 1000-5000ms) for dynamic pages, or configure scrapeOptions_actions to interact with the page before scraping.
  • Extraction schema precision: Vague or shifting schemas/prompts produce noisy, inconsistent output. Freeze your schema and test on a small sample before scaling to many URLs.
  • Crawl jobs are async: FIRECRAWL_CRAWL_V2 returns immediately with a job ID. Use FIRECRAWL_CRAWL_GET to poll for results. Cancel stuck crawls with FIRECRAWL_CANCEL_A_CRAWL_JOB to avoid wasting credits.
  • Extract job polling: FIRECRAWL_EXTRACT is also async for larger jobs. Retrieve final output with FIRECRAWL_EXTRACT_GET.
  • URL batching for extract: Keep extract URL batches small (~10 URLs) to avoid 429 rate limit errors.
  • Deeply nested responses: Results are often nested under data.data or deeper. Inspect the returned shape rather than assuming flat keys.

Quick Reference

Tool SlugDescription
FIRECRAWL_SCRAPEScrape a single URL with format/action options
FIRECRAWL_CRAWL_V2Crawl a website with depth/path control
FIRECRAWL_EXTRACTExtract structured data with AI prompt/schema
FIRECRAWL_BATCH_SCRAPEBatch scrape multiple URLs concurrently
FIRECRAWL_MAP_MULTIPLE_URLS_BASED_ON_OPTIONSDiscover/map all URLs on a site
FIRECRAWL_CRAWL_GETGet crawl job status and results
FIRECRAWL_GET_THE_STATUS_OF_A_CRAWL_JOBCheck crawl job progress
FIRECRAWL_CANCEL_A_CRAWL_JOBCancel an active crawl job
FIRECRAWL_EXTRACT_GETGet extraction job status and results
FIRECRAWL_CRAWL_PARAMS_PREVIEWPreview crawl parameters before starting
FIRECRAWL_SEARCHWeb search + scrape top results

Powered by Composio

More from this repository10

πŸ”Œ
meeting-insights-analyzerπŸ”ŒPlugin

A Claude Code plugin that analyzes meeting notes and transcripts to extract key insights, action items, and decisions using Composio app integrations.

πŸ”Œ
video-downloaderπŸ”ŒPlugin

A Claude Code plugin from the Awesome Claude Skills collection that downloads videos from YouTube and other platforms for offline viewing, editing, or archival, with support for various formats and quality options.

πŸ”Œ
content-research-writerπŸ”ŒPlugin

A Claude Code plugin that assists in writing high-quality content by conducting research, adding citations, improving hooks, and providing section-by-section feedback.

πŸ”Œ
file-organizerπŸ”ŒPlugin

A Claude Code plugin that automatically organizes and categorizes files in your workspace using Composio integrations, helping maintain a clean project structure.

πŸ”Œ
artifacts-builderπŸ”ŒPlugin

A Claude Code plugin providing tools for creating elaborate, multi-component HTML artifacts using React, Tailwind CSS, and shadcn/ui in claude.ai.

πŸ”Œ
raffle-winner-pickerπŸ”ŒPlugin

A Claude Code skill that randomly selects winners from participant lists, spreadsheets, or Google Sheets for giveaways and contests using cryptographically secure randomness.

πŸ”Œ
zoho-crm-automationπŸ”ŒPlugin

A Claude Code plugin that automates Zoho CRM workflows including lead management, contact updates, deal tracking, account operations, and custom module interactions through Composio integration.

πŸ”Œ
brand-guidelinesπŸ”ŒPlugin

A Claude Code plugin that enforces brand consistency by checking code and content against your brand guidelines using Composio integrations.

πŸ”Œ
helpdesk-automationπŸ”ŒPlugin

A pre-built workflow skill for automating Help Scout via Composio, enabling Claude to manage conversations, customers, mailboxes, and tags through structured tool sequences and parameter guidance.

πŸ”Œ
changelog-generatorπŸ”ŒPlugin

A Claude Code plugin that automatically creates user-facing changelogs from git commits by analyzing commit history and transforming technical commits into customer-friendly release notes.