How to Use the Web Scraper MCP in CrewAI

Q: Does the Web Scraper support crawling with CrewAI?

Yes. The crawl tool lets a specialized agent map out up to 10 connected pages starting from a given URL, helping your crew build an internal site index collaboratively.

Q: Can I extract links using the Web Scraper with CrewAI?

You can use listlinks to gather all hyperlinks from a source page. This lets your crew build site maps and identify related content for follow-up research.

Q: What is the best way to scrape documentation sites with Web Scraper and CrewAI?

Use the read tool, as it employs Readability. This ensures you get clean Markdown of the main article content, which is exactly what your analyzing agents need.

Autonomous web data research with CrewAI’s collaborative MCP Server tools.

See Vinkius in Action

Works with every AI agent you already use

…and any MCP-compatible client

MCP Servers - Free for Subscribers

Connect Web Scraper MCP to CrewAI

Create your Vinkius account to connect Web Scraper to CrewAI and route execution through our secure gateway. The platform manages server hosting, runtime updates, and security layers. Configuration requires no manual server provisioning.

GDPR Free for Subscribers

Setup Web Scraper with CrewAI

Ask AI about this MCP

ChatGPT

Claude

Perplexity

Extracting Deep Metadata via Web Scraper

The `extract` tool gives your agents structured metadata like title, description, Open Graph tags, and the author. Your crew can analyze this data to compare site health across multiple documents. This means Agent A researches the structure while Agent B analyzes the key fields.

Reading Complex Web Pages

The `read` tool fetches any public web page and cleans it into Markdown. It's optimized for articles, blogs, or documentation, letting your team focus on the actual content. This prevents your specialized agents from getting lost in formatting noise.

Parallel Content Fetching

Use `batch_read` to pull multiple web pages in parallel. By limiting it to 10 URLs per batch, you keep the response size manageable for your crew's shared memory. You can assign one agent to manage a list of endpoints and another to process the results concurrently.

Setup guide

Set up Web Scraper MCP in CrewAI

Prerequisites

Python 3.10+ installed
crewai package (pip install crewai)
Active Vinkius subscription with a valid endpoint token

1

Install CrewAI
Run pip install crewai to install the framework. MCP support is built-in via the mcps parameter.
2

Add the MCP URL to your agent
Pass your Vinkius endpoint directly to the mcps list. Replace [YOUR_TOKEN_HERE] with your token from cloud.vinkius.com. CrewAI handles tool discovery and caching automatically.
3

Kick off your crew
Create a Crew with your agent and tasks. Call crew.kickoff() — the agent will automatically invoke Web Scraper tools as needed.

crew.py

from crewai import Agent, Task, Crew

agent = Agent(
    role="Web Scraper Analyst",
    goal="Access and analyze Web Scraper data via MCP.",
    backstory="Expert analyst with direct Web Scraper access.",
    mcps=[
        "https://edge.vinkius.com/[YOUR_TOKEN_HERE]/mcp"
    ],
)

task = Task(
    description="List recent Web Scraper transactions",
    agent=agent,
    expected_output="A summary of recent activity",
)

crew = Crew(agents=[agent], tasks=[task])
result = crew.kickoff()
print(result)

Get your connection token →

Prerequisites

Python 3.10+ installed
crewai + crewai-tools packages
Active Vinkius subscription with a valid endpoint token

1

Install dependencies
Run pip install crewai crewai-tools. The MCPServerAdapter handles lifecycle management and tool conversion.
2

Connect with MCPServerAdapter
Use MCPServerAdapter as a context manager with SseServerParameters pointing to your Vinkius endpoint. The adapter automatically manages connection lifecycle.
3

Assign tools and run
Pass the returned mcp_tools to your agent's tools parameter. The adapter converts MCP tools to native BaseTool objects compatible with all CrewAI agents.

crew.py

from crewai import Agent, Task, Crew
from crewai_tools import MCPServerAdapter
from mcp import SseServerParameters

server_params = SseServerParameters(
    url="https://edge.vinkius.com/[YOUR_TOKEN_HERE]/mcp"
)

with MCPServerAdapter(server_params) as mcp_tools:
    agent = Agent(
        role="Web Scraper Analyst",
        goal="Access and analyze Web Scraper data via MCP.",
        backstory="Expert analyst with direct Web Scraper access.",
        tools=mcp_tools,
    )

    task = Task(
        description="List recent Web Scraper transactions",
        agent=agent,
        expected_output="A summary of recent activity",
    )

    crew = Crew(agents=[agent], tasks=[task])
    result = crew.kickoff()
    print(result)

Get your connection token →

Independent Platform Disclaimer: Vinkius is an independent platform and is not affiliated with, endorsed by, sponsored by, verified by, or otherwise authorized by Web Scraper. All third-party trademarks, logos, and brand names are the property of their respective owners. Their use on this website is strictly for informational purposes to identify service compatibility and interoperability.

Why Choose Vinkius

Vinkius connects your tools to AI with real-time monitoring and automatic cost savings — all from one dashboard.

Connect Web Scraper now

Real-time monitoring

Live

visibility into every interaction

Connect your favorite tools to your AI and see exactly what's happening — every request, every response, in real time.

Built-in savings

60%

lower AI costs

Vinkius compresses data between your apps and your AI automatically. Lower bills every month — no configuration required.

Single dashboard

One

place for every integration

Every tool your AI connects to, managed from a single screen. One account, complete control.

Common questions about Web Scraper MCP in CrewAI

You pass the MCP server's tools directly into your crew setup. One agent can specialize in research using `read`, while another analyzes the output metadata from `extract`.

Yes. The `crawl` tool lets a specialized agent map out up to 10 connected pages starting from a given URL, helping your crew build an internal site index collaboratively.

The server deals exclusively with public web content and structured metadata (title, description, OG tags). No private user or backend system data is ever accessed.

You can use `list_links` to gather all hyperlinks from a source page. This lets your crew build site maps and identify related content for follow-up research.

Use the `read` tool, as it employs Readability. This ensures you get clean Markdown of the main article content, which is exactly what your analyzing agents need.

Use it with your favorite AI tools

Connect this server to Cursor, Claude, VS Code, and more.

OpenAI Agents SDK sdk-python

Google ADK sdk-python

Pydantic AI sdk-python

Vercel AI SDK sdk-typescript