How to Use the Crawlbase MCP in OpenAI Agents SDK

Q: How do I get my OpenAI Agent to scrape Amazon with Crawlbase?

Your agent just needs to call the scrapeamazon tool with the product URL. The Crawlbase MCP Server handles the request, bypasses blocks, and returns the page content. The SDK auto-discovers the tool, so there's no extra config needed.

Q: Can I handle JavaScript-heavy sites using Crawlbase and the OpenAI Agents SDK?

Yes, that's what the scrapejsrendered tool is for. Your agent can invoke it when a standard HTML scrape isn't enough. The server runs a headless browser for you.

Q: What's the setup for the Crawlbase MCP Server with an OpenAI Agent?

It's quick. Just pip install openai-agents, then instantiate MCPServerStreamableHttp with your Vinkius endpoint URL. Pass that server object to your Agent constructor and the tools become available automatically.

Build production web scraping agents with the full safety and tracing of the OpenAI Agents SDK.

See Vinkius in Action

Works with every AI agent you already use

…and any MCP-compatible client

MCP Servers - Free for Subscribers

Connect Crawlbase MCP to OpenAI Agents SDK

Create your Vinkius account to connect Crawlbase to OpenAI Agents SDK and route execution through our secure gateway. The platform manages server hosting, runtime updates, and security layers. Configuration requires no manual server provisioning.

GDPR Free for Subscribers

Setup Crawlbase with OpenAI Agents SDK

Ask AI about this MCP

ChatGPT

Claude

Perplexity

Pull Data from Walled Gardens

This toolset gives your agent direct access to pages that are notoriously hard to scrape. Use `scrape_amazon` to pull product details or `scrape_facebook` to get public page info. Your agent doesn't need to know about proxies or request headers; it just calls the tool and gets the data. Because you're using the OpenAI Agents SDK, you get built-in guardrails. You can define which agents are allowed to scrape which sites, and every action is logged in your OpenAI dashboard. It’s the difference between a prototype and a production system you can actually trust.

Scrape JavaScript-Heavy Pages with Your MCP Server

The `scrape_js_rendered` tool runs a full headless browser to get content from single-page apps and other dynamic sites. If a simple `scrape_html` call returns a loading spinner, this is the tool you need. Your agent gets the final, rendered HTML without any complex setup on your end. This is where the SDK's tracing shines. If a scrape fails, you can see the exact tool input and output in your trace logs. It makes debugging an agent that interacts with unpredictable websites much less painful.

Get Visual Proof and Handle Edge Cases

Sometimes you need to see what the scraper sees. The `get_screenshot_link` tool gives your agent a URL to a snapshot of the page, which is perfect for visual debugging or archival. You'll know exactly what the page looked like when it was scraped. For sites that require specific proxy configurations, `custom_scrape` lets your agent provision a request payload with custom settings. The OpenAI SDK lets you build specialized agents that hand off tasks. One agent can identify a tricky site, then pass it to another agent equipped to use this tool for a successful scrape.

Setup guide

Set up Crawlbase MCP in OpenAI Agents SDK

Prerequisites

Python 3.10+ installed
openai-agents package (pip install openai-agents)
Active Vinkius subscription with a valid endpoint token

1

Install the SDK

Run pip install openai-agents to install the OpenAI Agents SDK. The MCP integration is built-in — no extra dependencies needed.
2

Connect via SSE transport

Use MCPServerSse with your Vinkius endpoint URL. Replace [YOUR_TOKEN_HERE] with your token from cloud.vinkius.com. The SDK auto-discovers all Crawlbase tools at runtime.
3

Create your Agent

Pass the MCP to Agent(mcp_servers=[server]). The agent receives Crawlbase tools as native definitions — JSON schemas resolve automatically.
4

Run the agent

Call Runner.run(agent, prompt) to execute. The agent invokes the appropriate Crawlbase tools and returns structured results. Copy the full example on the right to get started.

agent.py

import asyncio
from agents import Agent, Runner
from agents.mcp import MCPServerSse

async def main():
    async with MCPServerSse(
        url="https://edge.vinkius.com/[YOUR_TOKEN_HERE]/mcp"
    ) as server:
        agent = Agent(
            name="Crawlbase Agent",
            instructions="You have access to Crawlbase tools.",
            mcp_servers=[server],
        )
        result = await Runner.run(agent, "List recent transactions")
        print(result.final_output)

asyncio.run(main())

Get your connection token →

Independent Platform Disclaimer: Vinkius is an independent platform and is not affiliated with, endorsed by, sponsored by, verified by, or otherwise authorized by Crawlbase. All third-party trademarks, logos, and brand names are the property of their respective owners. Their use on this website is strictly for informational purposes to identify service compatibility and interoperability.

Why Choose Vinkius

Vinkius connects your tools to AI with real-time monitoring and automatic cost savings — all from one dashboard.

Connect Crawlbase now

Real-time monitoring

Live

visibility into every interaction

Connect your favorite tools to your AI and see exactly what's happening — every request, every response, in real time.

Built-in savings

60%

lower AI costs

Vinkius compresses data between your apps and your AI automatically. Lower bills every month — no configuration required.

Single dashboard

One

place for every integration

Every tool your AI connects to, managed from a single screen. One account, complete control.

Common questions about Crawlbase MCP in OpenAI Agents SDK

Your agent just needs to call the `scrape_amazon` tool with the product URL. The Crawlbase MCP Server handles the request, bypasses blocks, and returns the page content. The SDK auto-discovers the tool, so there's no extra config needed.

Yes, that's what the `scrape_js_rendered` tool is for. Your agent can invoke it when a standard HTML scrape isn't enough. The server runs a headless browser for you.

It's quick. Just `pip install openai-agents`, then instantiate `MCPServerStreamableHttp` with your Vinkius endpoint URL. Pass that server object to your Agent constructor and the tools become available automatically.

Crawlbase manages a large pool of datacenter and other proxies. When your agent calls a scrape tool, the request is routed through this infrastructure, which deals with IP rotation and CAPTCHA solving for you. Your agent just sees a successful response.

The server only processes the target URLs your agent provides. It fetches the resulting web page content—like HTML, JSON, or a screenshot—and passes it back. Vinkius runs each MCP Server in an ephemeral sandbox, so no request data is ever stored.

Use it with your favorite AI tools

Connect this server to Cursor, Claude, VS Code, and more.

OpenAI Agents SDK sdk-python

Google ADK sdk-python

Pydantic AI sdk-python

Vercel AI SDK sdk-typescript