How to Use the Web Scraper MCP in Vercel AI SDK

Q: How do I use the Web Scraper with Vercel AI SDK?

You pass the MCP tool to your generateText or streamText call. When the user prompts for data, your agent calls the tool, and you'll see the results stream back into the UI in real time.

Q: Does the Web Scraper support crawling with Vercel AI SDK?

Yes. The crawl tool lets your agent start at one URL and systematically navigate up to 10 pages from there. This is how you build out documentation site indexes or research a small topic deep dive.

Q: Can I extract links using the Web Scraper with Vercel AI SDK?

You can use listlinks to grab all hyperlinks from a page, and you also have extract which gives you a count of internal link structures. Both are useful for site mapping.

Real-time web data streaming for Vercel AI SDK frontends.

See Vinkius in Action

Works with every AI agent you already use

…and any MCP-compatible client

MCP Servers - Free for Subscribers

Connect Web Scraper MCP to Vercel AI SDK

Create your Vinkius account to connect Web Scraper to Vercel AI SDK and route execution through our secure gateway. The platform manages server hosting, runtime updates, and security layers. Configuration requires no manual server provisioning.

GDPR Free for Subscribers

Setup Web Scraper with Vercel AI SDK

Ask AI about this MCP

ChatGPT

Claude

Perplexity

Read Full Article Content

Use the `read` tool to fetch any public page and get its full content in clean Markdown. It uses Readability, so it works best when pulling articles, documentation pages, or blogs. This means you don't just get raw HTML; you get the main body text, ready for immediate display right in your React component.

Structured Metadata Extraction

Need specific data points? The `extract` tool pulls structured metadata like titles, descriptions, Open Graph tags, and author info. You'll also get the language code, canonical URL, and a count of internal links. This is perfect for building product listings or search result cards that need precise data fields without pulling entire pages.

Batch Fetching Multiple URLs

The `batch_read` tool fetches several web pages at the same time, handling up to 10 URLs in a single go. This is fast and efficient for scraping multiple results quickly. You can't just hit one endpoint; this capability lets your agent handle bulk data collection across different sources simultaneously.

Setup guide

Set up Web Scraper MCP in Vercel AI SDK

Prerequisites

Node.js 18+ and a TypeScript project
ai + @modelcontextprotocol/sdk packages
Active Vinkius subscription with a valid endpoint token

1

Install dependencies

Run npm install ai @modelcontextprotocol/sdk plus your preferred model provider (e.g. @ai-sdk/openai).
2

Create the Streamable HTTP transport

Use StreamableHTTPClientTransport with your Vinkius endpoint URL. Replace [YOUR_TOKEN_HERE] with your token from cloud.vinkius.com.
3

Discover and use tools

Call mcpClient.tools() to auto-discover all Web Scraper tools. Pass them directly to generateText() or streamText() — no manual schema definitions needed.
4

Works with any model provider

Swap openai("gpt-4o") for any AI SDK provider — Anthropic, Google, Mistral. The MCP tools work identically across all supported models.

index.ts

import { experimental_createMCPClient as createMCPClient } from "ai";
import { StreamableHTTPClientTransport } from "@modelcontextprotocol/sdk/client/streamableHttp";
import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";

const transport = new StreamableHTTPClientTransport(
  new URL("https://edge.vinkius.com/[YOUR_TOKEN_HERE]/mcp")
);

const mcpClient = await createMCPClient({ transport });
const tools = await mcpClient.tools();

const { text } = await generateText({
  model: openai("gpt-4o"),
  tools,
  prompt: "List recent Web Scraper transactions",
});

console.log(text);
await mcpClient.close();

Get your connection token →

Independent Platform Disclaimer: Vinkius is an independent platform and is not affiliated with, endorsed by, sponsored by, verified by, or otherwise authorized by Web Scraper. All third-party trademarks, logos, and brand names are the property of their respective owners. Their use on this website is strictly for informational purposes to identify service compatibility and interoperability.

Why Choose Vinkius

Vinkius connects your tools to AI with real-time monitoring and automatic cost savings — all from one dashboard.

Connect Web Scraper now

Real-time monitoring

Live

visibility into every interaction

Connect your favorite tools to your AI and see exactly what's happening — every request, every response, in real time.

Built-in savings

60%

lower AI costs

Vinkius compresses data between your apps and your AI automatically. Lower bills every month — no configuration required.

Single dashboard

One

place for every integration

Every tool your AI connects to, managed from a single screen. One account, complete control.

Common questions about Web Scraper MCP in Vercel AI SDK

You pass the MCP tool to your `generateText` or `streamText` call. When the user prompts for data, your agent calls the tool, and you'll see the results stream back into the UI in real time.

Yes. The `crawl` tool lets your agent start at one URL and systematically navigate up to 10 pages from there. This is how you build out documentation site indexes or research a small topic deep dive.

This server primarily touches publicly available textual content and structured metadata (title, description, OG tags). It doesn't access private user accounts or sensitive backend data.

You can use `list_links` to grab all hyperlinks from a page, and you also have `extract` which gives you a count of internal link structures. Both are useful for site mapping.

It's really good for articles and blogs because it uses Readability, which strips out junk formatting. However, remember that `read` is designed for main body text, not a full site map.

Use it with your favorite AI tools

Connect this server to Cursor, Claude, VS Code, and more.

OpenAI Agents SDK sdk-python

Google ADK sdk-python

Pydantic AI sdk-python

Vercel AI SDK sdk-typescript