🎯

apify-automation

🎯Skill
composiohq/awesome-claude-skills8 Installs66,152Last Updated May 22, 2026

Installation

$npx skills add composiohq/awesome-claude-skills --skill apify-automation

Run it in a terminal; it asks which AI coding tools to install it for (Claude Code, Codex, Cursor and more).

Uses skills.sh Β· Node.js 22.20+

After installing

Ask for related work and the agent uses it on its own, or call it yourself.

Run it (Claude Code)/Apify Automation
Install by hand

Copy the whole skill folder from GitHub to the place below. Copying only SKILL.md can leave out files the skill uses.

Open the skill folder β†’
  • This project .claude/skills/apify-automation/
  • All projects ~/.claude/skills/apify-automation/

These paths are for Claude Code. For other tools, the command above puts it in the right place.

Automate web scraping and data extraction with Apify -- run Actors, manage datasets, create reusable tasks, and retrieve crawl results through the Composio Apify integration.

Apify Automation

Run Apify web scraping Actors and manage datasets directly from Claude Code. Execute crawlers synchronously or asynchronously, retrieve structured data, create reusable tasks, and inspect run logs without leaving your terminal.

Toolkit docs: composio.dev/toolkits/apify


Setup

  1. Add the Composio MCP server to your configuration:
    https://rube.app/mcp
    
  2. Connect your Apify account when prompted. The agent will provide an authentication link.
  3. Browse available Actors at apify.com/store. Each Actor has its own unique input schema -- always check the Actor's documentation before running.

Core Workflows

1. Run an Actor Synchronously and Get Results

Execute an Actor and immediately retrieve its dataset items in a single call. Best for quick scraping jobs.

Tool: APIFY_RUN_ACTOR_SYNC_GET_DATASET_ITEMS

Key parameters:

  • actorId (required) -- Actor ID in format username/actor-name (e.g., compass/crawler-google-places)
  • input -- JSON input object matching the Actor's schema. Each Actor has unique field names -- check apify.com/store for the exact schema.
  • limit -- max items to return
  • offset -- skip items for pagination
  • format -- json (default), csv, jsonl, html, xlsx, xml
  • timeout -- run timeout in seconds
  • waitForFinish -- max wait time (0-300 seconds)
  • fields -- comma-separated list of fields to include
  • omit -- comma-separated list of fields to exclude

Example prompt: "Run the Google Places scraper for 'restaurants in New York' and return the first 50 results"


2. Run an Actor Asynchronously

Trigger an Actor run without waiting for completion. Use for long-running scraping jobs.

Tool: APIFY_RUN_ACTOR

Key parameters:

  • actorId (required) -- Actor slug or ID
  • body -- JSON input object for the Actor
  • memory -- memory limit in MB (must be power of 2, minimum 128)
  • timeout -- run timeout in seconds
  • maxItems -- cap on returned items
  • build -- specific build tag (e.g., latest, beta)

Follow up with APIFY_GET_DATASET_ITEMS to retrieve results using the run's datasetId.

Example prompt: "Start the web scraper Actor for example.com asynchronously with 1024MB memory"


3. Retrieve Dataset Items

Fetch data from a specific dataset with pagination, field selection, and filtering.

Tool: APIFY_GET_DATASET_ITEMS

Key parameters:

  • datasetId (required) -- dataset identifier
  • limit (default/max 1000) -- items per page
  • offset (default 0) -- pagination offset
  • format -- json (recommended), csv, xlsx
  • fields -- include only specific fields
  • omit -- exclude specific fields
  • clean -- remove Apify-specific metadata
  • desc -- reverse order (newest first)

Example prompt: "Get the first 500 items from dataset myDatasetId in JSON format"


4. Inspect Actor Details

View Actor metadata, input schema, and configuration before running it.

Tool: APIFY_GET_ACTOR

Key parameters:

  • actorId (required) -- Actor ID in format username/actor-name or hex ID

Example prompt: "Show me the details and input schema for the apify/web-scraper Actor"


5. Create Reusable Tasks

Configure reusable Actor tasks with preset inputs for recurring scraping jobs.

Tool: APIFY_CREATE_TASK

Configure a task once, then trigger it repeatedly with consistent input parameters. Useful for scheduled or recurring data collection workflows.

Example prompt: "Create an Apify task for the Google Search scraper with default query 'AI startups' and US location"


6. Manage Runs and Datasets

List Actor runs, browse datasets, and inspect run details for monitoring and debugging.

Tools: APIFY_GET_LIST_OF_RUNS, APIFY_DATASETS_GET, APIFY_DATASET_GET, APIFY_GET_LOG

For listing runs:

  • Filter by Actor and optionally by status
  • Get datasetId from run details for data retrieval

For dataset management:

  • APIFY_DATASETS_GET -- list all your datasets with pagination
  • APIFY_DATASET_GET -- get metadata for a specific dataset

For debugging:

  • APIFY_GET_LOG -- retrieve execution logs for a run or build

Example prompt: "List the last 10 runs for the web scraper Actor and show logs for the most recent one"


Known Pitfalls

  • Actor input schemas vary wildly: Every Actor has its own unique input fields. Generic field names like queries or search_terms will be rejected. Always check the Actor's page on apify.com/store for exact field names (e.g., searchStringsArray for Google Maps, startUrls for web scrapers).
  • URL format requirements: Always include the full protocol (https:// or http://) in URLs. Many Actors require URLs as objects with a url property: {"startUrls": [{"url": "https://example.com"}]}.
  • Dataset pagination cap: APIFY_GET_DATASET_ITEMS has a max limit of 1000 per call. For large datasets, loop with offset to collect all items.
  • Enum values are lowercase: Most Actors expect lowercase enum values (e.g., relevance not RELEVANCE, all not ALL).
  • Sync timeout at 5 minutes: APIFY_RUN_ACTOR_SYNC_GET_DATASET_ITEMS has a maximum waitForFinish of 300 seconds. For longer runs, use APIFY_RUN_ACTOR (async) and poll with APIFY_GET_DATASET_ITEMS.
  • Data volume costs: Large datasets can be expensive to fetch. Prefer moderate limits and incremental processing to avoid timeouts or memory pressure.
  • JSON format recommended: While CSV/XLSX formats are available, JSON is the most reliable for automated processing. Avoid CSV/XLSX for downstream automation.

Quick Reference

Tool SlugDescription
APIFY_RUN_ACTOR_SYNC_GET_DATASET_ITEMSRun Actor synchronously and get results immediately
APIFY_RUN_ACTORRun Actor asynchronously (trigger and return)
APIFY_RUN_ACTOR_SYNCRun Actor synchronously, return output record
APIFY_GET_ACTORGet Actor metadata and input schema
APIFY_GET_DATASET_ITEMSRetrieve items from a dataset (paginated)
APIFY_DATASET_GETGet dataset metadata (item count, etc.)
APIFY_DATASETS_GETList all user datasets
APIFY_CREATE_TASKCreate a reusable Actor task
APIFY_GET_TASK_INPUTInspect a task's stored input
APIFY_GET_LIST_OF_RUNSList runs for an Actor
APIFY_GET_LOGGet execution logs for a run

Powered by Composio

More from this repository10

πŸ”Œ
meeting-insights-analyzerπŸ”ŒPlugin

A Claude Code plugin that analyzes meeting notes and transcripts to extract key insights, action items, and decisions using Composio app integrations.

πŸ”Œ
video-downloaderπŸ”ŒPlugin

A Claude Code plugin from the Awesome Claude Skills collection that downloads videos from YouTube and other platforms for offline viewing, editing, or archival, with support for various formats and quality options.

πŸ”Œ
content-research-writerπŸ”ŒPlugin

A Claude Code plugin that assists in writing high-quality content by conducting research, adding citations, improving hooks, and providing section-by-section feedback.

πŸ”Œ
file-organizerπŸ”ŒPlugin

A Claude Code plugin that automatically organizes and categorizes files in your workspace using Composio integrations, helping maintain a clean project structure.

πŸ”Œ
artifacts-builderπŸ”ŒPlugin

A Claude Code plugin providing tools for creating elaborate, multi-component HTML artifacts using React, Tailwind CSS, and shadcn/ui in claude.ai.

πŸ”Œ
raffle-winner-pickerπŸ”ŒPlugin

A Claude Code skill that randomly selects winners from participant lists, spreadsheets, or Google Sheets for giveaways and contests using cryptographically secure randomness.

πŸ”Œ
zoho-crm-automationπŸ”ŒPlugin

A Claude Code plugin that automates Zoho CRM workflows including lead management, contact updates, deal tracking, account operations, and custom module interactions through Composio integration.

πŸ”Œ
brand-guidelinesπŸ”ŒPlugin

A Claude Code plugin that enforces brand consistency by checking code and content against your brand guidelines using Composio integrations.

πŸ”Œ
helpdesk-automationπŸ”ŒPlugin

A pre-built workflow skill for automating Help Scout via Composio, enabling Claude to manage conversations, customers, mailboxes, and tags through structured tool sequences and parameter guidance.

πŸ”Œ
changelog-generatorπŸ”ŒPlugin

A Claude Code plugin that automatically creates user-facing changelogs from git commits by analyzing commit history and transforming technical commits into customer-friendly release notes.