How to Use the Web Scraper MCP in Claude Code

Q: How do I scrape structured data with Web Scraper in Claude Code?

Run the extract tool. It returns a clean set of metadata like title, description, and OG tags that you can directly pipe into your shell script or JSON output file. This is much cleaner than parsing HTML.

Q: What if I need to scrape multiple URLs at once with Web Scraper in Claude Code?

Use the batchread tool. It takes up to 10 distinct URLs and fetches the full, clean Markdown content for all of them simultaneously. This makes your CI/CD data ingestion jobs run much faster.

Q: Does Web Scraper handle non-article web pages for Claude Code?

Yes. While read works best on articles, it can pull the main content from any public page into clean Markdown format. If you're only concerned with links or metadata, stick to extract and listlinks.

Q: What kind of SEO metadata does Web Scraper provide for Claude Code?

The extract tool provides all the necessary structural elements: title, description, OG tags, language code, and author. This data is perfect for scripts that validate or generate required meta tags across a deployment.

Automate site discovery and content fetching for CI/CD pipelines with Claude Code and Web Scraper MCP Server.

See Vinkius in Action

Works with every AI agent you already use

…and any MCP-compatible client

MCP Servers - Free for Subscribers

Connect Web Scraper MCP to Claude Code

Create your Vinkius account to connect Web Scraper to Claude Code and route execution through our secure gateway. The platform manages server hosting, runtime updates, and security layers. Configuration requires no manual server provisioning.

GDPR Free for Subscribers

Setup Web Scraper with Claude Code

Ask AI about this MCP

ChatGPT

Claude

Perplexity

Map documentation sites for automation.

Start your automated crawl using `crawl`. This tool maps out a website, starting from one URL and handling up to 10 pages. It's perfect for CI/CD jobs that need to validate site structure or discover linked assets. You can refine this discovery process by running `list_links`, which pulls every internal hyperlink found on a page, keeping the focus strictly within the source hostname.

Fetch content in bulk for pipelines.

Use the `batch_read` tool to fetch up to 10 public web pages simultaneously. This is ideal when your pipeline needs to ingest multiple documentation updates or blog posts at once, returning clean Markdown. Alternatively, if you need the full text from a single page, run `read`. This strips out all the junk HTML and gives you pure Markdown, which is easy to pipe into other scripting tools.

Get metadata for asset validation.

The `extract` tool pulls key structured data points from a page: title, description, OG tags, language code, author, and canonical links. This allows your script to validate that required metadata fields are present across the site. When you need to know what's on the page without fetching all the text, `extract` is faster than using the full-page read operation.

Setup guide

Set up Web Scraper MCP in Claude Code

Prerequisites

Claude Code CLI installed (npm install -g @anthropic-ai/claude-code)
Active Vinkius subscription with a valid endpoint token

1

Run the add command

Open your terminal and run the command shown on the right. Replace [YOUR_TOKEN_HERE] with your endpoint token from cloud.vinkius.com. Use --scope user to make it available across all projects.
2

Verify the connection

Start a Claude Code session and type /mcp to list connected servers. You should see web-scraper-mcp with a green status indicator.
3

Start using tools

Ask Claude Code something like "Check my latest Web Scraper transactions." It will automatically discover and invoke the available Web Scraper tools.

Terminal

claude mcp add --transport http web-scraper-mcp https://edge.vinkius.com/[YOUR_TOKEN_HERE]/mcp

Get your connection token →

Prerequisites

Claude Code CLI installed
Active Vinkius subscription with a valid endpoint token

1

Open the config file

Create or edit .mcp.json in your project root for project-level scope, or ~/.claude.json for user-level scope.
2

Add the Web Scraper MCP

Paste the JSON snippet shown on the right into the mcpServers object. Replace [YOUR_TOKEN_HERE] with your endpoint token from cloud.vinkius.com.
3

Restart Claude Code

Start a new Claude Code session. Type /mcp to confirm the server is connected. The tools will be automatically available in your conversation.

.mcp.json

{
  "mcpServers": {
    "web-scraper-mcp": {
      "type": "url",
      "url": "https://edge.vinkius.com/[YOUR_TOKEN_HERE]/mcp"
    }
  }
}

Get your connection token →

Independent Platform Disclaimer: Vinkius is an independent platform and is not affiliated with, endorsed by, sponsored by, verified by, or otherwise authorized by Web Scraper. All third-party trademarks, logos, and brand names are the property of their respective owners. Their use on this website is strictly for informational purposes to identify service compatibility and interoperability.

Why Choose Vinkius

Vinkius connects your tools to AI with real-time monitoring and automatic cost savings — all from one dashboard.

Connect Web Scraper now

Real-time monitoring

Live

visibility into every interaction

Connect your favorite tools to your AI and see exactly what's happening — every request, every response, in real time.

Built-in savings

60%

lower AI costs

Vinkius compresses data between your apps and your AI automatically. Lower bills every month — no configuration required.

Single dashboard

One

place for every integration

Every tool your AI connects to, managed from a single screen. One account, complete control.

Common questions about Web Scraper MCP in Claude Code

Run the `extract` tool. It returns a clean set of metadata like title, description, and OG tags that you can directly pipe into your shell script or JSON output file. This is much cleaner than parsing HTML.

Use the `batch_read` tool. It takes up to 10 distinct URLs and fetches the full, clean Markdown content for all of them simultaneously. This makes your CI/CD data ingestion jobs run much faster.

Yes. While `read` works best on articles, it can pull the main content from any public page into clean Markdown format. If you're only concerned with links or metadata, stick to `extract` and `list_links`.

The `extract` tool provides all the necessary structural elements: title, description, OG tags, language code, and author. This data is perfect for scripts that validate or generate required meta tags across a deployment.

This MCP Server only accesses public web page content and metadata. Since you're running this headless, it can only read what is publicly available via HTTP requests; it never touches private user accounts or protected internal systems.

Use it with your favorite AI tools

Connect this server to Cursor, Claude, VS Code, and more.

OpenAI Agents SDK sdk-python

Google ADK sdk-python

Pydantic AI sdk-python

Vercel AI SDK sdk-typescript