How to Use the URL Metadata Scraper MCP in CrewAI

Q: How does the URL Metadata Scraper fit into CrewAI's multi-agent setup?

The scrapeurl tool acts as a crucial data input for your crew. One agent runs it, and then other specialized agents consume the structured metadata to perform their analysis or actions.

Run autonomous operations: specialized agents collaborate using the URL Metadata Scraper with CrewAI.

See Vinkius in Action

Works with every AI agent you already use

…and any MCP-compatible client

MCP Servers - Free for Subscribers

Connect URL Metadata Scraper MCP to CrewAI

Create your Vinkius account to connect URL Metadata Scraper to CrewAI and route execution through our secure gateway. The platform manages server hosting, runtime updates, and security layers. Configuration requires no manual server provisioning.

GDPR Free for Subscribers

Setup URL Metadata Scraper with CrewAI

Ask AI about this MCP

ChatGPT

Claude

Perplexity

Specialized Agent Collaboration

You assign specialized roles in your crew. For example, an 'SEO Researcher' agent uses `scrape_url` to gather metadata, and a separate 'Content Analyst' agent then analyzes the returned Title and Description. This role-based specialization ensures that each piece of data gets analyzed by the right expert agent.

Shared Memory for Metadata

The structured output from `scrape_url` is written to shared memory. This lets a monitor agent watch the process and report findings based on metadata scraped earlier in the sequence. No agent loses track of critical information because everything feeds into one central, accessible memory.

Autonomous Operations Triggering

Build an autonomous pipeline where Agent A scrapes a list of URLs. If the metadata for any URL shows missing descriptions, a moderator agent automatically triggers an escalation task. This means your crew handles monitoring and response without needing human intervention at every step.

Setup guide

Set up URL Metadata Scraper MCP in CrewAI

Prerequisites

Python 3.10+ installed
crewai package (pip install crewai)
Active Vinkius subscription with a valid endpoint token

1

Install CrewAI
Run pip install crewai to install the framework. MCP support is built-in via the mcps parameter.
2

Add the MCP URL to your agent
Pass your Vinkius endpoint directly to the mcps list. Replace [YOUR_TOKEN_HERE] with your token from cloud.vinkius.com. CrewAI handles tool discovery and caching automatically.
3

Kick off your crew
Create a Crew with your agent and tasks. Call crew.kickoff() — the agent will automatically invoke URL Metadata Scraper tools as needed.

crew.py

from crewai import Agent, Task, Crew

agent = Agent(
    role="URL Metadata Scraper Analyst",
    goal="Access and analyze URL Metadata Scraper data via MCP.",
    backstory="Expert analyst with direct URL Metadata Scraper access.",
    mcps=[
        "https://edge.vinkius.com/[YOUR_TOKEN_HERE]/mcp"
    ],
)

task = Task(
    description="List recent URL Metadata Scraper transactions",
    agent=agent,
    expected_output="A summary of recent activity",
)

crew = Crew(agents=[agent], tasks=[task])
result = crew.kickoff()
print(result)

Get your connection token →

Prerequisites

Python 3.10+ installed
crewai + crewai-tools packages
Active Vinkius subscription with a valid endpoint token

1

Install dependencies
Run pip install crewai crewai-tools. The MCPServerAdapter handles lifecycle management and tool conversion.
2

Connect with MCPServerAdapter
Use MCPServerAdapter as a context manager with SseServerParameters pointing to your Vinkius endpoint. The adapter automatically manages connection lifecycle.
3

Assign tools and run
Pass the returned mcp_tools to your agent's tools parameter. The adapter converts MCP tools to native BaseTool objects compatible with all CrewAI agents.

crew.py

from crewai import Agent, Task, Crew
from crewai_tools import MCPServerAdapter
from mcp import SseServerParameters

server_params = SseServerParameters(
    url="https://edge.vinkius.com/[YOUR_TOKEN_HERE]/mcp"
)

with MCPServerAdapter(server_params) as mcp_tools:
    agent = Agent(
        role="URL Metadata Scraper Analyst",
        goal="Access and analyze URL Metadata Scraper data via MCP.",
        backstory="Expert analyst with direct URL Metadata Scraper access.",
        tools=mcp_tools,
    )

    task = Task(
        description="List recent URL Metadata Scraper transactions",
        agent=agent,
        expected_output="A summary of recent activity",
    )

    crew = Crew(agents=[agent], tasks=[task])
    result = crew.kickoff()
    print(result)

Get your connection token →

Independent Platform Disclaimer: Vinkius is an independent platform and is not affiliated with, endorsed by, sponsored by, verified by, or otherwise authorized by Cheerio Scraper. All third-party trademarks, logos, and brand names are the property of their respective owners. Their use on this website is strictly for informational purposes to identify service compatibility and interoperability.

Why Choose Vinkius

Vinkius connects your tools to AI with real-time monitoring and automatic cost savings — all from one dashboard.

Connect URL Metadata Scraper now

Real-time monitoring

Live

visibility into every interaction

Connect your favorite tools to your AI and see exactly what's happening — every request, every response, in real time.

Built-in savings

60%

lower AI costs

Vinkius compresses data between your apps and your AI automatically. Lower bills every month — no configuration required.

Single dashboard

One

place for every integration

Every tool your AI connects to, managed from a single screen. One account, complete control.

Common questions about URL Metadata Scraper MCP in CrewAI

The `scrape_url` tool acts as a crucial data input for your crew. One agent runs it, and then other specialized agents consume the structured metadata to perform their analysis or actions.

Yes. You can assign a dedicated 'Researcher' agent that uses `scrape_url` to gather initial OpenGraph data from multiple sources, which then feeds into other analytical agents.

It handles Title, Description, and Image URLs. This structured trio of data points lets your autonomous crew perform comprehensive web content validation tasks.

Yes. The tool always returns clean, structured JSON metadata for each URL passed to it. This consistency allows your crew's shared memory to reliably store and process findings.

This server pulls Title, Description, and Image links from a given URL. Your agents use this clean metadata to perform tasks like bulk content auditing or competitive analysis.

Use it with your favorite AI tools

Connect this server to Cursor, Claude, VS Code, and more.

OpenAI Agents SDK sdk-python

Google ADK sdk-python

Pydantic AI sdk-python

Vercel AI SDK sdk-typescript