How to Use the Octoparse MCP in CrewAI

Q: How do CrewAI agents authenticate with Octoparse?

The crew's admin agent calls gettoken to get an OAuth 2.0 access token. This token is shared across the crew's memory so all agents can access tools like listtasks and starttask.

Q: Can I restrict agents to specific scraping folders?

Yes. Use listtaskgroups to fetch folder IDs. You can instruct your CrewAI agents to only interact with tasks belonging to those specific group IDs.

Q: How do CrewAI agents handle scraping failures in Octoparse?

The monitoring agent checks the crawler using gettaskstatus. If the task fails, the agent can call stoptask, adjust the parameters via updatetaskparams, and restart it.

Q: Can my CrewAI agents download thousands of records from Octoparse safely?

Yes. The gettaskdata tool forces offset pagination with a 1000-row limit. This keeps the memory usage low, allowing your agents to process large datasets without crashing their environment.

Deploy autonomous agent crews to run, monitor, and harvest Octoparse scrapers with this MCP Server.

See Vinkius in Action

Works with every AI agent you already use

…and any MCP-compatible client

MCP Servers - Free for Subscribers

Connect Octoparse MCP to CrewAI

Create your Vinkius account to connect Octoparse to CrewAI and route execution through our secure gateway. The platform manages server hosting, runtime updates, and security layers. Configuration requires no manual server provisioning.

GDPR Free for Subscribers

Setup Octoparse with CrewAI

Ask AI about this MCP

ChatGPT

Claude

Perplexity

Coordinate multi-agent scraping teams using CrewAI

The `start_task` tool lets your CrewAI researcher agent trigger scrapers while a separate analyst agent monitors progress. These agents collaborate in real-time, sharing state information through their common memory pool. The monitoring agent polls `get_task_status` to track execution. If a crawler finishes, it alerts the analyst agent to begin downloading the data, keeping your entire extraction pipeline running autonomously.

Let agents optimize crawler parameters on the fly

The `update_task_params` tool lets your CrewAI moderator agent inject new target URLs or updated keywords into the crawler. This happens dynamically without you ever opening the desktop app. After the parameters are updated, the researcher agent restarts the crawler using `start_task`. This closed-loop system allows your agents to self-correct and complete extractions even when the initial parameters fail.

Harvest and clean cloud data with specialized agents

The `get_task_data` tool retrieves records in structured batches of up to 1000 rows using offset-based pagination. This MCP server lets your CrewAI data agent process these batches sequentially, cleaning and formatting the data as it arrives. Once the agent confirms the data is saved, it executes `mark_data_exported` to clear the cloud queue. If the crew needs to reset the environment for a new run, it calls `clear_task_data` to remove all traces.

Setup guide

Set up Octoparse MCP in CrewAI

Prerequisites

Python 3.10+ installed
crewai package (pip install crewai)
Active Vinkius subscription with a valid endpoint token

1

Install CrewAI
Run pip install crewai to install the framework. MCP support is built-in via the mcps parameter.
2

Add the MCP URL to your agent
Pass your Vinkius endpoint directly to the mcps list. Replace [YOUR_TOKEN_HERE] with your token from cloud.vinkius.com. CrewAI handles tool discovery and caching automatically.
3

Kick off your crew
Create a Crew with your agent and tasks. Call crew.kickoff() — the agent will automatically invoke Octoparse tools as needed.

crew.py

from crewai import Agent, Task, Crew

agent = Agent(
    role="Octoparse Analyst",
    goal="Access and analyze Octoparse data via MCP.",
    backstory="Expert analyst with direct Octoparse access.",
    mcps=[
        "https://edge.vinkius.com/[YOUR_TOKEN_HERE]/mcp"
    ],
)

task = Task(
    description="List recent Octoparse transactions",
    agent=agent,
    expected_output="A summary of recent activity",
)

crew = Crew(agents=[agent], tasks=[task])
result = crew.kickoff()
print(result)

Get your connection token →

Prerequisites

Python 3.10+ installed
crewai + crewai-tools packages
Active Vinkius subscription with a valid endpoint token

1

Install dependencies
Run pip install crewai crewai-tools. The MCPServerAdapter handles lifecycle management and tool conversion.
2

Connect with MCPServerAdapter
Use MCPServerAdapter as a context manager with SseServerParameters pointing to your Vinkius endpoint. The adapter automatically manages connection lifecycle.
3

Assign tools and run
Pass the returned mcp_tools to your agent's tools parameter. The adapter converts MCP tools to native BaseTool objects compatible with all CrewAI agents.

crew.py

from crewai import Agent, Task, Crew
from crewai_tools import MCPServerAdapter
from mcp import SseServerParameters

server_params = SseServerParameters(
    url="https://edge.vinkius.com/[YOUR_TOKEN_HERE]/mcp"
)

with MCPServerAdapter(server_params) as mcp_tools:
    agent = Agent(
        role="Octoparse Analyst",
        goal="Access and analyze Octoparse data via MCP.",
        backstory="Expert analyst with direct Octoparse access.",
        tools=mcp_tools,
    )

    task = Task(
        description="List recent Octoparse transactions",
        agent=agent,
        expected_output="A summary of recent activity",
    )

    crew = Crew(agents=[agent], tasks=[task])
    result = crew.kickoff()
    print(result)

Get your connection token →

Independent Platform Disclaimer: Vinkius is an independent platform and is not affiliated with, endorsed by, sponsored by, verified by, or otherwise authorized by Octoparse. All third-party trademarks, logos, and brand names are the property of their respective owners. Their use on this website is strictly for informational purposes to identify service compatibility and interoperability.

Why Choose Vinkius

Vinkius connects your tools to AI with real-time monitoring and automatic cost savings — all from one dashboard.

Connect Octoparse now

Real-time monitoring

Live

visibility into every interaction

Connect your favorite tools to your AI and see exactly what's happening — every request, every response, in real time.

Built-in savings

60%

lower AI costs

Vinkius compresses data between your apps and your AI automatically. Lower bills every month — no configuration required.

Single dashboard

One

place for every integration

Every tool your AI connects to, managed from a single screen. One account, complete control.

Common questions about Octoparse MCP in CrewAI

The crew's admin agent calls `get_token` to get an OAuth 2.0 access token. This token is shared across the crew's memory so all agents can access tools like `list_tasks` and `start_task`.

Yes. Use `list_task_groups` to fetch folder IDs. You can instruct your CrewAI agents to only interact with tasks belonging to those specific group IDs.

The monitoring agent checks the crawler using `get_task_status`. If the task fails, the agent can call `stop_task`, adjust the parameters via `update_task_params`, and restart it.

Yes. The `get_task_data` tool forces offset pagination with a 1000-row limit. This keeps the memory usage low, allowing your agents to process large datasets without crashing their environment.

Yes. Vinkius runs the MCP Server in an isolated V8 sandbox. Your credentials, OAuth tokens, and scraped payloads are never stored on Vinkius disks, and you can programmatically clear data using `clear_task_data` after ingestion.

Use it with your favorite AI tools

Connect this server to Cursor, Claude, VS Code, and more.

OpenAI Agents SDK sdk-python

Google ADK sdk-python

Pydantic AI sdk-python

Vercel AI SDK sdk-typescript