How to Use the Web Scraper MCP in LangChain

Q: Can I crawl documentation sites with Web Scraper and LangChain?

Yeah, absolutely. Use the crawl tool to explore a site starting from a seed URL, keeping responses manageable by limiting it to 10 pages. The resulting links can then feed into other tools within your LangChain application.

Q: What if I need to find related pages on a site using Web Scraper and LangChain?

The listlinks tool pulls all hyperlinks that share the same hostname as the source page. This lets your LangChain agent identify internal links, helping it build a map of connected documentation or content.

Q: Is Web Scraper good for batch processing when using LangChain?

It's excellent. The batchread tool fetches multiple URLs in parallel, maxing out at 10 per batch. This makes your entire workflow faster and keeps the overhead low when passing results into a chain.

Web Scraper | Build multi-step reasoning chains with LangChain.

See Vinkius in Action

Works with every AI agent you already use

…and any MCP-compatible client

MCP Servers - Free for Subscribers

Connect Web Scraper MCP to LangChain

Create your Vinkius account to connect Web Scraper to LangChain and route execution through our secure gateway. The platform manages server hosting, runtime updates, and security layers. Configuration requires no manual server provisioning.

GDPR Free for Subscribers

Setup Web Scraper with LangChain

Ask AI about this MCP

ChatGPT

Claude

Perplexity

Bulk Read Content in Parallel

Need to pull content from several URLs at once? The `batch_read` tool fetches up to 10 pages simultaneously. This lets your agent gather bulk data quickly, making it perfect for scraping a list of related articles or documentation endpoints. It's designed for speed. Your LangChain agent can process multiple inputs in parallel using this single call, drastically cutting down the time needed when building large-scale information pipelines.

Structured Metadata Extraction

The `extract` tool pulls specific data points like title, description, Open Graph tags, and author without having to download the full page content. This is ideal for indexing a site's meta-data quickly or building search results. When your LangChain agent encounters hundreds of links, it doesn't need to read everything. It just needs the structured metadata—the `extract` tool gets you those key points right away.

Deep Article Content Retrieval

If you're dealing with blogs or academic papers, use the `read` tool. It uses a Reader View process to strip out junk—ads, navigation, sidebars—and gives you clean Markdown of the main article. This means your agent only sees what matters. This output is perfect for feeding directly into a LangChain chain because it’s already cleaned up and structured as readable text.

Setup guide

Set up Web Scraper MCP in LangChain

Prerequisites

Python 3.10+ installed
langchain-mcp-adapters + langgraph packages
Active Vinkius subscription with a valid endpoint token

1

Install dependencies
Run pip install langchain-mcp-adapters langgraph langchain-openai. The MCP adapters package converts MCP tools into native LangChain BaseTool objects.
2

Connect via HTTP transport
Use MultiServerMCPClient with "transport": "http" pointing to your Vinkius endpoint. Replace [YOUR_TOKEN_HERE] with your token from cloud.vinkius.com.
3

Create a ReAct agent
Pass the discovered tools to create_react_agent() from LangGraph. The agent automatically routes Web Scraper tool calls through the MCP protocol.
4

Run with any LLM
Swap ChatOpenAI for ChatAnthropic, ChatGoogleGenerativeAI, or any LangChain-compatible model. The MCP tools work identically across all providers.

agent.py

from langchain_mcp_adapters.client import MultiServerMCPClient
from langgraph.prebuilt import create_react_agent
from langchain_openai import ChatOpenAI

async with MultiServerMCPClient({
    "web-scraper-mcp": {
        "transport": "http",
        "url": "https://edge.vinkius.com/[YOUR_TOKEN_HERE]/mcp",
    }
}) as client:
    tools = client.get_tools()

    agent = create_react_agent(
        ChatOpenAI(model="gpt-4o"),
        tools,
    )
    result = await agent.ainvoke({
        "messages": "List recent Web Scraper transactions"
    })
    print(result["messages"][-1].content)

Get your connection token →

Independent Platform Disclaimer: Vinkius is an independent platform and is not affiliated with, endorsed by, sponsored by, verified by, or otherwise authorized by Web Scraper. All third-party trademarks, logos, and brand names are the property of their respective owners. Their use on this website is strictly for informational purposes to identify service compatibility and interoperability.

Why Choose Vinkius

Vinkius connects your tools to AI with real-time monitoring and automatic cost savings — all from one dashboard.

Connect Web Scraper now

Real-time monitoring

Live

visibility into every interaction

Connect your favorite tools to your AI and see exactly what's happening — every request, every response, in real time.

Built-in savings

60%

lower AI costs

Vinkius compresses data between your apps and your AI automatically. Lower bills every month — no configuration required.

Single dashboard

One

place for every integration

Every tool your AI connects to, managed from a single screen. One account, complete control.

Common questions about Web Scraper MCP in LangChain

You use the Web Scraper's tools to gather raw data, then pass that output into a multi-step reasoning chain. Your agent decides if it needs `read` for an article or `extract` for metadata based on the prompt.

Yeah, absolutely. Use the `crawl` tool to explore a site starting from a seed URL, keeping responses manageable by limiting it to 10 pages. The resulting links can then feed into other tools within your LangChain application.

The server primarily touches public web page content, which is text and metadata (like titles or descriptions). It doesn't handle personal identifiers like emails or passwords. You always control the scope of what your agent reads.

The `list_links` tool pulls all hyperlinks that share the same hostname as the source page. This lets your LangChain agent identify internal links, helping it build a map of connected documentation or content.

It's excellent. The `batch_read` tool fetches multiple URLs in parallel, maxing out at 10 per batch. This makes your entire workflow faster and keeps the overhead low when passing results into a chain.

Use it with your favorite AI tools

Connect this server to Cursor, Claude, VS Code, and more.

OpenAI Agents SDK sdk-python

Google ADK sdk-python

Pydantic AI sdk-python

Vercel AI SDK sdk-typescript