How to Use the Web Scraper MCP in Claude

Q: Does Web Scraper handle large sites with Claude Desktop?

The crawl tool is limited to 10 pages, which keeps the response size manageable for fast processing. If you need a deeper map of a site, you'll have to run smaller crawls or use another category of tools.

Q: What if I need Web Scraper metadata for a whole list of pages with Claude Desktop?

Use batchread to grab the required information from multiple URLs in parallel. This lets you quickly gather titles and descriptions without running 10 separate commands.

Get Live Web Data Into Your Local Codebase with Claude Desktop.

See Vinkius in Action

Works with every AI agent you already use

…and any MCP-compatible client

MCP Servers - Free for Subscribers

Connect Web Scraper MCP to Claude Desktop

Create your Vinkius account to connect Web Scraper to Claude Desktop and route execution through our secure gateway. The platform manages server hosting, runtime updates, and security layers. Configuration requires no manual server provisioning.

GDPR Free for Subscribers

Setup Web Scraper with Claude Desktop

Ask AI about this MCP

ChatGPT

Claude

Perplexity

Run `read` for Clean Article Content

Need the full text from a blog post or documentation page? The `read` tool fetches any public URL and cleans it up, giving you Markdown. It’s perfect for grabbing main article content without all the site's junk. This works best when feeding source material into your agent. You can use this to quickly pull down knowledge bases or academic papers directly from a link, letting your AI client process clean text.

Extract Structured Metadata Fast

Don't just read the page; know what it is. The `extract` tool pulls metadata—like the title, description, and Open Graph tags—from a single URL. This gives your agent context about the source before it even reads the content. It’s useful for building directories or checking data consistency. If you just need to validate if a page is meant for public consumption, this tool gets those structured details fast.

Crawl Sites and Gather Links

Got a starting URL but don't know where the good stuff is? Use `crawl` to systematically explore a website up to 10 pages deep. The system handles the navigation, returning multiple pages for your agent to process. Need internal links? The `list_links` tool finds all hyperlinks on a page, making it easy for your AI client to map out site structure or find related documentation.

Setup guide

Set up Web Scraper MCP in Claude Web or Desktop

1

Open Claude Settings

Go to claude.ai, click your profile icon, then navigate to Customize → Connectors.
2

Add Custom Connector

Click the "+" button and select Add custom connector. Paste your Vinkius endpoint URL: https://edge.vinkius.com/[YOUR_TOKEN_HERE]/mcp Replace [YOUR_TOKEN_HERE] with your token from cloud.vinkius.com. For OAuth-protected servers, expand Advanced settings to add credentials.
3

Start a conversation

Open a new chat. The Web Scraper MCP tools are available immediately — no restart needed.

Endpoint URL

https://edge.vinkius.com/[YOUR_TOKEN_HERE]/mcp

No configuration file needed — paste the URL directly in the Claude web interface.

Available on Free (1 connector), Pro, Max, Team, and Enterprise plans.

Prerequisites

Claude Desktop installed (macOS or Windows)
Active Vinkius subscription with a valid endpoint token

1

Open Claude Desktop Settings

Click the menu icon at the top-left corner, go to Settings → Developer → Edit Config. This opens claude_desktop_config.json in your default text editor.
2

Paste the Web Scraper MCP configuration

Copy the JSON snippet on the right into the mcpServers object. Replace [YOUR_TOKEN_HERE] with your endpoint token from cloud.vinkius.com.
3

Restart Claude Desktop

Close and reopen the application. Claude needs a full restart to load new MCPs — refreshing a conversation is not enough.
4

Verify the connection

Open a new conversation. Click the 🔌 icon at the bottom of the message input. You should see tools listed under web-scraper-mcp.

json

{
  "mcpServers": {
    "web-scraper-mcp": {
      "url": "https://edge.vinkius.com/[YOUR_TOKEN_HERE]/mcp"
    }
  }
}

Get your connection token →

Independent Platform Disclaimer: Vinkius is an independent platform and is not affiliated with, endorsed by, sponsored by, verified by, or otherwise authorized by Web Scraper. All third-party trademarks, logos, and brand names are the property of their respective owners. Their use on this website is strictly for informational purposes to identify service compatibility and interoperability.

Why Choose Vinkius

Vinkius connects your tools to AI with real-time monitoring and automatic cost savings — all from one dashboard.

Connect Web Scraper now

Real-time monitoring

Live

visibility into every interaction

Connect your favorite tools to your AI and see exactly what's happening — every request, every response, in real time.

Built-in savings

60%

lower AI costs

Vinkius compresses data between your apps and your AI automatically. Lower bills every month — no configuration required.

Single dashboard

One

place for every integration

Every tool your AI connects to, managed from a single screen. One account, complete control.

Common questions about Web Scraper MCP in Claude Desktop

You simply invoke the tool name in your chat, just like calling any other function. Claude's agent recognizes that `read` is available and executes the fetch request using the URL you provide.

The `crawl` tool is limited to 10 pages, which keeps the response size manageable for fast processing. If you need a deeper map of a site, you'll have to run smaller crawls or use another category of tools.

Yep. Since it’s running on your machine via stdio, the resulting Markdown text is treated like any other piece of local data. Your agent can then write to files or process that content immediately.

Use `batch_read` to grab the required information from multiple URLs in parallel. This lets you quickly gather titles and descriptions without running 10 separate commands.

This server fetches public web content, primarily text (Markdown) and structured metadata like `title`, `description`, and `lang`. It only interacts with publicly visible information.

Use it with your favorite AI tools

Connect this server to Cursor, Claude, VS Code, and more.

OpenAI Agents SDK sdk-python

Google ADK sdk-python

Pydantic AI sdk-python

Vercel AI SDK sdk-typescript