scrapfly-automation
by ComposioHQscrapfly-automation helps agents run Scrapfly web scraping tasks through Composio Rube MCP by discovering current tools, checking the Scrapfly connection, and avoiding stale schemas.
This skill scores 68/100, which means it is acceptable for directory listing but should be presented as a lightweight MCP workflow guide rather than a complete Scrapfly automation package. Directory users get enough clarity to know when to install it and how an agent should begin, but should expect runtime tool discovery to supply most task-specific detail.
- Valid skill frontmatter clearly identifies the trigger domain: automating Scrapfly tasks through Rube MCP.
- Prerequisites and setup steps are explicit, including the need for Rube MCP, an active Scrapfly connection, and use of RUBE_MANAGE_CONNECTIONS.
- Strong operational guardrail to call RUBE_SEARCH_TOOLS first so agents retrieve current tool slugs, schemas, execution plans, and pitfalls before acting.
- No support scripts, references, or examples beyond SKILL.md, so execution depends heavily on Rube MCP's returned schemas at runtime.
- The excerpted workflow is mostly generic discovery/connection scaffolding rather than specific Scrapfly task recipes, which may leave users guessing for complex scraping use cases.
Overview of scrapfly-automation skill
What scrapfly-automation does
The scrapfly-automation skill helps an AI agent run Scrapfly-related web scraping tasks through Composio’s Rube MCP connection. Its main value is not a built-in scraper script; it is an operating pattern for discovering the current Scrapfly tool schema, checking authentication, and executing the right Rube tool calls safely instead of guessing parameters from memory.
Best fit for Scrapfly and MCP users
This skill is a good fit if you already use Claude or another MCP-capable client, want agent-assisted workflows for Scrapfly, and prefer tool-backed execution over hand-written scraping prompts. It is especially useful for users who need to scrape pages through Scrapfly while relying on Composio/Rube to expose the available Scrapfly actions, inputs, and connection status.
Key adoption requirement
The most important prerequisite is an active Rube MCP setup. The skill requires the rube MCP server and expects RUBE_SEARCH_TOOLS and RUBE_MANAGE_CONNECTIONS to be available. You also need an active Scrapfly connection in Rube. If those pieces are not configured, the skill can explain the workflow but cannot actually execute Scrapfly operations.
What makes it different from a generic scraping prompt
A generic prompt may invent Scrapfly parameters, skip authentication checks, or assume stale API schemas. The differentiator in scrapfly-automation is the instruction to search tools first every time. That matters because Rube returns current tool slugs, input schemas, recommended execution plans, and pitfalls before the agent attempts a scrape.
How to Use scrapfly-automation skill
scrapfly-automation install and setup path
Install the skill from the Composio skills repository:
npx skills add ComposioHQ/awesome-claude-skills --skill scrapfly-automation
Then add Rube MCP to your client configuration using:
https://rube.app/mcp
After that, verify that RUBE_SEARCH_TOOLS responds. Use RUBE_MANAGE_CONNECTIONS with toolkit scrapfly to check whether the Scrapfly connection is ACTIVE. If Rube returns an authorization link, complete that flow before asking the agent to run scraping tasks.
Inputs the skill needs from you
For reliable scrapfly-automation usage, give the agent the scraping goal, target URL or URL pattern, required fields, output format, and operational constraints. A weak request is: “Scrape this site with Scrapfly.” A stronger request is: “Use Scrapfly via Rube to fetch https://example.com/products, extract product name, price, availability, and canonical URL, return JSON, avoid login-only pages, and first discover the current Scrapfly tool schema before execution.”
This helps the agent map your intent to the current Rube tools instead of making unsupported assumptions.
Recommended workflow for first run
Start by asking the agent to inspect the skill’s SKILL.md, then follow its workflow: discover tools, check connection, execute the selected Scrapfly action, and review the result. The key call pattern is to run RUBE_SEARCH_TOOLS with your specific use case, not just a broad “Scrapfly operations” query. For example, “fetch a rendered page,” “extract structured data from a URL,” or “scrape multiple product pages” will produce more relevant schemas and plans.
Files to read before trusting automation
This skill currently ships as a single SKILL.md under composio-skills/scrapfly-automation. Read that file first because there are no helper scripts, rules folders, reference examples, or local metadata files to clarify edge cases. The absence of support files is not a blocker, but it means your prompt and the live Rube tool discovery step carry more of the quality burden.
scrapfly-automation skill FAQ
Is scrapfly-automation for Web Scraping beginners?
It can help beginners who already have MCP configured, but it is not a standalone scraping tutorial. You should understand the difference between a target URL, extraction fields, rendered content, pagination, and output format. The skill reduces tool-call guesswork, but it does not remove the need to describe the scraping task clearly.
Does it include Scrapfly API code?
No. The skill does not provide a ready-made Python or JavaScript scraper. It guides an agent to use Scrapfly through Composio’s Rube MCP tools. If you want direct SDK code, you may need to ask the agent to generate it separately after inspecting the current Scrapfly toolkit documentation and your project requirements.
When should I not use this skill?
Do not use scrapfly-automation if you cannot enable Rube MCP, cannot create a Scrapfly connection through Rube, or need a fully offline workflow. It is also a poor fit for vague scraping requests where the target pages, required fields, and acceptable output are unknown.
How does it compare with ordinary prompts?
Ordinary prompts are fine for planning or explaining Scrapfly concepts. The scrapfly-automation skill is better when you want the agent to execute tool-backed operations because it forces schema discovery and connection validation before action. That reduces failures caused by outdated assumptions.
How to Improve scrapfly-automation skill
Improve prompts before execution
The fastest way to improve scrapfly-automation results is to provide a task brief with concrete fields: target URLs, desired data shape, pagination rules, rendering needs, success criteria, and what to skip. Add constraints such as “return only JSON,” “limit to 10 pages for the test,” or “do not retry more than twice” when cost, speed, or safety matters.
Use tool discovery as a quality gate
Do not let the agent jump straight into a Scrapfly action. Ask it to show the relevant result from RUBE_SEARCH_TOOLS, identify the chosen tool slug, summarize required inputs, and mention any returned pitfalls before execution. This creates a checkpoint where you can catch mismatched tools or missing parameters early.
Common failure modes to watch
Most failures will come from inactive connections, underspecified scraping goals, stale assumptions about tool schemas, or expecting the skill to bypass site restrictions automatically. If the first run fails, ask the agent to separate the problem into connection status, tool schema mismatch, request parameters, target page behavior, and extraction logic.
Iterate after the first output
Treat the first run as a calibration pass. Review whether the returned content contains the fields you need, whether rendered content was required, and whether the output shape is stable enough for downstream use. Then refine the prompt with examples of correct and incorrect rows, missing fields, or pages that should be excluded. This turns the scrapfly-automation guide into a repeatable workflow rather than a one-off scraping attempt.