How to Use the Web Scraper MCP in Mastra AI

Q: Can the Web Scraper crawl sites using Mastra AI?

Yes. The crawl tool lets your agent start from one URL and map out up to 10 connected pages, which is perfect for deeply analyzing a small section of a site's documentation.

Q: Can I extract links using the Web Scraper with Mastra AI?

You can use listlinks to grab all hyperlinks from a page. For site structure analysis, you might also want the link count provided by the extract tool.

Q: Is the Web Scraper reliable for full-site data collection with Mastra AI?

The crawl function limits output to 10 pages per crawl, and batchread handles batches of 10. For massive sites, you'll need a multi-step process managed by your workflow engine.

Building complex web data workflows with Mastra AI's MCP Server.

See Vinkius in Action

Works with every AI agent you already use

…and any MCP-compatible client

MCP Servers - Free for Subscribers

Connect Web Scraper MCP to Mastra AI

Create your Vinkius account to connect Web Scraper to Mastra AI and route execution through our secure gateway. The platform manages server hosting, runtime updates, and security layers. Configuration requires no manual server provisioning.

GDPR Free for Subscribers

Setup Web Scraper with Mastra AI

Ask AI about this MCP

ChatGPT

Claude

Perplexity

Gathering Site Metadata via MCP Server

The `extract` tool pulls core metadata points: title, description, OG tags, author, and canonical links. You get a clean object containing everything from the page's header structure. This is great for building structured databases or ensuring your multi-step workflow always captures accurate site information.

Reading Clean Web Pages

Use `read` to fetch any public web page and clean it up into Markdown. The tool focuses on the main article content, which works best for documentation or blog posts. This means your workflow doesn't choke on unnecessary navigation menus or ads; you just get the core text.

Concurrent URL Processing

Need to check a dozen links fast? The `batch_read` tool fetches multiple web pages in parallel, handling up to 10 URLs per batch. This saves time when your workflow requires checking many sources. You just feed it the list of endpoints and let the automatic retries handle any hiccups.

Setup guide

Set up Web Scraper MCP in Mastra AI

Prerequisites

Node.js 18+ and a TypeScript project
@mastra/mcp + @mastra/core packages
Active Vinkius subscription with a valid endpoint token

1

Install dependencies
Run npm install @mastra/mcp @mastra/core plus your preferred model provider (e.g. @ai-sdk/openai).
2

Configure the MCPClient
Create an MCPClient with your Vinkius endpoint as a URL object. Replace [YOUR_TOKEN_HERE] with your token from cloud.vinkius.com.
3

Discover and inject tools
Call mcpClient.listTools() and spread the result into your agent's tools object. All Web Scraper tools become native Mastra tools.
4

Run with any model
Swap openai("gpt-4o") for any AI SDK-compatible provider. Call agent.generate() and the agent routes tool calls through MCP automatically.

agent.ts

import { MCPClient } from "@mastra/mcp";
import { Agent } from "@mastra/core/agent";
import { openai } from "@ai-sdk/openai";

const mcpClient = new MCPClient({
  id: "web-scraper-mcp-client",
  servers: {
    "web-scraper-mcp": {
      url: new URL(
        "https://edge.vinkius.com/[YOUR_TOKEN_HERE]/mcp"
      ),
    },
  },
});

const agent = new Agent({
  name: "Web Scraper Agent",
  model: openai("gpt-4o"),
  instructions: "You have access to Web Scraper tools.",
  tools: {
    ...(await mcpClient.listTools()),
  },
});

const result = await agent.generate(
  "List recent Web Scraper transactions"
);
console.log(result.text);

Get your connection token →

Independent Platform Disclaimer: Vinkius is an independent platform and is not affiliated with, endorsed by, sponsored by, verified by, or otherwise authorized by Web Scraper. All third-party trademarks, logos, and brand names are the property of their respective owners. Their use on this website is strictly for informational purposes to identify service compatibility and interoperability.

Why Choose Vinkius

Vinkius connects your tools to AI with real-time monitoring and automatic cost savings — all from one dashboard.

Connect Web Scraper now

Real-time monitoring

Live

visibility into every interaction

Connect your favorite tools to your AI and see exactly what's happening — every request, every response, in real time.

Built-in savings

60%

lower AI costs

Vinkius compresses data between your apps and your AI automatically. Lower bills every month — no configuration required.

Single dashboard

One

place for every integration

Every tool your AI connects to, managed from a single screen. One account, complete control.

Common questions about Web Scraper MCP in Mastra AI

You integrate the MCP tool into your complex workflows. If one step fails (like a network timeout), Mastra automatically retries, so you don't lose data just because of a hiccup.

Yes. The `crawl` tool lets your agent start from one URL and map out up to 10 connected pages, which is perfect for deeply analyzing a small section of a site's documentation.

It handles publicly accessible content and structured metadata (title, description, OG tags). It never touches private user credentials or secured API endpoints.

You can use `list_links` to grab all hyperlinks from a page. For site structure analysis, you might also want the link count provided by the `extract` tool.

The `crawl` function limits output to 10 pages per crawl, and `batch_read` handles batches of 10. For massive sites, you'll need a multi-step process managed by your workflow engine.

Use it with your favorite AI tools

Connect this server to Cursor, Claude, VS Code, and more.

OpenAI Agents SDK sdk-python

Google ADK sdk-python

Pydantic AI sdk-python

Vercel AI SDK sdk-typescript