How to Use the Web Scraper MCP in OpenAI Agents SDK

Q: How does the Web Scraper work with OpenAI Agents SDK for reading content?

The read tool pulls full articles into clean Markdown. It uses a Reader View approach to ensure you get only the main body text, which is ideal for feeding documents back into your agent.

Q: Can I use the Web Scraper in OpenAI Agents SDK to check site structure?

Yeah, the listlinks tool handles that. It pulls all internal hyperlinks found on a page, but only those sharing the source page's hostname. This keeps your link map focused.

Q: Does Web Scraper handle metadata extraction with OpenAI Agents SDK?

Absolutely. The extract tool gives you structured data like titles, descriptions, and Open Graph tags. It lets you verify the page's core identity without reading all 10 pages.

Q: What is the limit when using Web Scraper with OpenAI Agents SDK?

The crawl tool limits how many pages it explores to a maximum of 10. This keeps the response size manageable and prevents your agent from getting lost in an infinite web deep dive.

Fetch content and structure data with the Web Scraper for OpenAI Agents SDK.

See Vinkius in Action

Works with every AI agent you already use

…and any MCP-compatible client

MCP Servers - Free for Subscribers

Connect Web Scraper MCP to OpenAI Agents SDK

Create your Vinkius account to connect Web Scraper to OpenAI Agents SDK and route execution through our secure gateway. The platform manages server hosting, runtime updates, and security layers. Configuration requires no manual server provisioning.

GDPR Free for Subscribers

Setup Web Scraper with OpenAI Agents SDK

Ask AI about this MCP

ChatGPT

Claude

Perplexity

Extract Metadata with the MCP Server

The `extract` tool pulls structured metadata from any page. You get title, description, OG tags, and more without needing to download the full content. It's perfect for indexing large amounts of web data quickly. Use this when you need specific facts—like checking which links are canonical or finding the author—instead of dumping massive text blocks.

Deep Content Retrieval with OpenAI Agents SDK

Need the full article body? The `read` tool uses a Reader View approach to pull out clean Markdown content. This works great for blogs, documentation, and Wikipedia pages. It strips away all the noise—ads, sidebars, navigation—leaving you with pure text that your agents can actually process. It’s way better than just dumping raw HTML.

Batch Data Fetching

Don't hit URLs one by one. The `batch_read` tool fetches up to 10 web pages simultaneously. This is how you quickly gather data from a list of articles or reports. Running these requests in parallel saves huge amounts of time compared to sequential fetching, making your agent workflow much faster and more efficient.

Setup guide

Set up Web Scraper MCP in OpenAI Agents SDK

Prerequisites

Python 3.10+ installed
openai-agents package (pip install openai-agents)
Active Vinkius subscription with a valid endpoint token

1

Install the SDK

Run pip install openai-agents to install the OpenAI Agents SDK. The MCP integration is built-in — no extra dependencies needed.
2

Connect via SSE transport

Use MCPServerSse with your Vinkius endpoint URL. Replace [YOUR_TOKEN_HERE] with your token from cloud.vinkius.com. The SDK auto-discovers all Web Scraper tools at runtime.
3

Create your Agent

Pass the MCP to Agent(mcp_servers=[server]). The agent receives Web Scraper tools as native definitions — JSON schemas resolve automatically.
4

Run the agent

Call Runner.run(agent, prompt) to execute. The agent invokes the appropriate Web Scraper tools and returns structured results. Copy the full example on the right to get started.

agent.py

import asyncio
from agents import Agent, Runner
from agents.mcp import MCPServerSse

async def main():
    async with MCPServerSse(
        url="https://edge.vinkius.com/[YOUR_TOKEN_HERE]/mcp"
    ) as server:
        agent = Agent(
            name="Web Scraper Agent",
            instructions="You have access to Web Scraper tools.",
            mcp_servers=[server],
        )
        result = await Runner.run(agent, "List recent transactions")
        print(result.final_output)

asyncio.run(main())

Get your connection token →

Independent Platform Disclaimer: Vinkius is an independent platform and is not affiliated with, endorsed by, sponsored by, verified by, or otherwise authorized by Web Scraper. All third-party trademarks, logos, and brand names are the property of their respective owners. Their use on this website is strictly for informational purposes to identify service compatibility and interoperability.

Why Choose Vinkius

Vinkius connects your tools to AI with real-time monitoring and automatic cost savings — all from one dashboard.

Connect Web Scraper now

Real-time monitoring

Live

visibility into every interaction

Connect your favorite tools to your AI and see exactly what's happening — every request, every response, in real time.

Built-in savings

60%

lower AI costs

Vinkius compresses data between your apps and your AI automatically. Lower bills every month — no configuration required.

Single dashboard

One

place for every integration

Every tool your AI connects to, managed from a single screen. One account, complete control.

Common questions about Web Scraper MCP in OpenAI Agents SDK

The `read` tool pulls full articles into clean Markdown. It uses a Reader View approach to ensure you get only the main body text, which is ideal for feeding documents back into your agent.

Yeah, the `list_links` tool handles that. It pulls all internal hyperlinks found on a page, but only those sharing the source page's hostname. This keeps your link map focused.

Absolutely. The `extract` tool gives you structured data like titles, descriptions, and Open Graph tags. It lets you verify the page's core identity without reading all 10 pages.

The `crawl` tool limits how many pages it explores to a maximum of 10. This keeps the response size manageable and prevents your agent from getting lost in an infinite web deep dive.

This server touches structured metadata, including titles, descriptions, Open Graph tags, language codes, and canonical URLs. It's all about the page's identity markers.

Use it with your favorite AI tools

Connect this server to Cursor, Claude, VS Code, and more.

OpenAI Agents SDK sdk-python

Google ADK sdk-python

Pydantic AI sdk-python

Vercel AI SDK sdk-typescript