# Web Scraper MCP for AI Agents AI Agent Connect

> Web Scraper lets your AI agent read live web pages, crawl documentation, and pull clean Markdown content. It turns messy websites into structured text for your agent to process, summarizing articles, auditing SEO tags, and parsing technical docs without the noise of ads or navbars.

## Overview
- **Category:** developer-tools
- **Price:** Free
- **Endpoint:** https://edge.vinkius.com/vk_preview_YmTaAx3VdeRzHJqckaOgMTVnBzECnxTL8z7QeMgb/ai-agent-connect
- **Tags:** web-crawling, markdown-conversion, data-extraction, reader-view, content-parsing, url-fetching

## Description

You're trying to get your agent to understand a specific library or summarize a news thread, but it's guessing based on outdated training data. This Connector changes that by giving your agent eyes on the live web. Instead of hallucinating, your agent can actually see the current state of a documentation hub or a blog post. It strips away the junk—the headers, the ads, and the sidebar noise—leaving just the core content in a format your agent loves: clean Markdown. If you need to compare multiple sources, you can have it pull several links at once or follow a path through a wiki to gather a full picture. It's a way to move from "I think the docs say this" to "I just read the docs and here's the answer." You'll find this works great when you're managing the Vinkius catalog to find and deploy the right tools for specific web-heavy tasks. It takes the friction out of research. Instead of you manually copying and pasting text into your prompt, you just provide the URL and let the agent do the heavy lifting. This is about accuracy. When your agent can pull the latest API specs or read the most recent news articles, the quality of its output improves instantly. It turns the web into a structured library that your agent can navigate on your behalf.

## Tools

### batch_read
Pulls up to 10 different URLs in parallel. Use this to compare multiple articles or summarize several sources at once.

### read
Fetches a public web page and returns its full content as clean Markdown using Mozilla Readability logic. This is perfect for getting the core text from blogs or Wikipedia without the junk.

### extract
Pulls structured metadata like titles, OG tags, and descriptions from a page. It's the fastest way to audit a site's SEO without downloading the full body.

### list_links
Grabs every hyperlink from a webpage. Use this to map out a site's structure or audit outbound links.

### read_json
Fetches and parses JSON data from a URL. This lets your agent interact with live API responses directly.

### extract_jsonld
Pulls JSON-LD blocks from a webpage. It's great for getting structured schema data from the web.

### crawl
Navigates a website starting from a specific URL. It goes up to 10 pages deep to gather a broader picture of a documentation hub.

## Prompt Examples

**Prompt:** 
```
Read https://en.wikipedia.org/wiki/Artificial_intelligence and summarize its history.
```

**Response:** 
```
I've fetched the Wikipedia page. The history of AI spans back to antiquity with myths of artificial beings, but the formal field was founded in 1956 at Dartmouth College.

**Key Milestones:**
* **1956:** Dartmouth Workshop
* **AI Winters:** Periods of reduced funding and interest.
* **Modern Era:** Deep learning revolution fueled by datasets.

Would you like me to dive deeper into any specific era?
```

**Prompt:** 
```
Extract the links from https://news.ycombinator.com/
```

**Response:** 
```
I've extracted the outbound links from Hacker News. The site currently points to several primary sources, including:

* **github.com**
* **nytimes.com**
* **theverge.com**

It also contains numerous internal links to user profiles and comment threads.
```

**Prompt:** 
```
Compare these two links: url1.com and url2.com
```

**Response:** 
```
I've analyzed both URLs simultaneously. Here is how they compare:

| Feature | URL 1 | URL 2 |
| :--- | :--- | :--- |
| **Architecture** | React-first | HTML-first |
| **Rendering** | Component styling | Server-rendered |
| **Hydration** | Client-side focus | Minimal hydration |

Both aim for performance but use opposite strategies for the DOM.
```

## Capabilities

### Convert web pages to Markdown
Use the read tool to get clean content from blogs or Wikipedia without ads and sidebars.

### Crawl multi-page sites
Use the crawl tool to follow links up to 10 levels deep on a documentation hub.

### Fetch multiple URLs at once
Use the batch_read tool to pull several links in parallel for faster comparison.

### Pull SEO metadata
Use the extract tool to grab titles, OG tags, and descriptions without downloading the body.

### Parse structured JSON data
Use the read_json tool to fetch and parse live API responses directly into your chat.

### Extract JSON-LD blocks
Use the extract_jsonld tool to pull structured schema data from a webpage.

### List all outbound hyperlinks
Use the list_links tool to map out a site's structure or audit all outbound links.

## Use Cases

### Learning a new library
A developer points the agent to a new library's docs and uses crawl to let it learn the full API structure before writing code.

### Synthesizing research
A researcher provides 5 Wikipedia links and uses batch_read to get a synthesized summary of all of them at once.

### SEO link auditing
An SEO specialist uses extract to grab meta tags and list_links to see where a competitor's site points.

### Parsing live API data
A data analyst provides a public endpoint and uses read_json to have the agent parse and explain the live response.

## Benefits

- Stop hallucinations by giving your agent access to real-time articles instead of relying on outdated training data.
- Save time on manual copy-pasting by using the read tool to pull entire blog posts or docs into your chat.
- Analyze entire websites quickly by using crawl to navigate up to 10 levels deep on a documentation hub.
- Compare multiple sources simultaneously using batch_read to see different perspectives on a single topic.
- Get structured data instantly with extract to audit SEO titles, descriptions, and OG tags without reading the whole page.
- Handle technical data easily by using read_json to parse API responses directly into your conversation.

## How It Works

The bottom line is your agent gets real-time web access without you having to copy-paste a single paragraph.

1. Subscribe to the Web Scraper MCP on Vinkius.
2. Paste a URL into your chat and tell your agent to read or crawl it.
3. Get clean Markdown or structured data back in your chat window.

## Frequently Asked Questions

**Can the Web Scraper MCP read paywalled content?**
No, this tool only accesses publicly available web pages. It cannot bypass logins, paywalls, or subscription screens.

**How many pages can the Web Scraper MCP crawl at once?**
The crawl tool allows your agent to navigate up to 10 pages deep from a starting URL, which is usually enough to capture a full documentation set.

**Does the Web Scraper MCP require an API key?**
No, there are no API keys or authentication requirements needed to use this Connector.

**Can I use the Web Scraper MCP to get clean text from a blog?**
Yes, it uses Mozilla Readability logic to strip out ads, navbars, and boilerplate, giving your agent just the core article content.

**Can the Web Scraper MCP handle JSON data?**
Yes, the read_json tool allows your agent to fetch and parse structured JSON data from any URL that returns it.

**How does the Web Scraper MCP handle messy websites?**
It automatically cleans the content by converting it to Markdown and removing common web clutter like sidebars and navigation menus.

**Can it read documentation sites that are split into multiple pages?**
Yes! You can use the `crawl` tool. For example: 'Crawl the getting started guide at https://example.com/docs'. The agent will fetch the starting page and automatically follow inner links to gather up to 10 pages of context.

**How does it handle ads and cluttered websites?**
The `read` tool uses the same underlying technology as Firefox's 'Reader View' (@mozilla/readability). It intelligently strips out standard website boilerplate—like navbars, sidebars, footers, and ads—leaving only the title and the clean main article text converted to Markdown.

**Is there a limit on how many URLs I can batch process?**
Yes, to ensure conversational AI latency remains reasonable, the `batch_read` tool accepts a maximum of 10 URLs in a single request. All 10 URLs are fetched simultaneously in parallel for maximum speed.