# Diffbot MCP for AI Agents AI Agent Connect

> Diffbot MCP turns messy websites into clean, structured data. Use your AI agent to pull product specs, news articles, job listings, and forum discussions without writing a single line of scraping code. It handles the heavy lifting of identifying what's important on a page so you can get straight to the data.

## Overview
- **Category:** ai-frontier
- **Price:** Free
- **Endpoint:** https://edge.vinkius.com/vk_preview_FiT28RroNiKkMcafLWUuAoeV2pbdaAk9JBpgecYE/ai-agent-connect
- **Tags:** web-scraping, data-extraction, computer-vision, structured-data, e-commerce-intelligence, sentiment-analysis, content-parsing

## Description

You spend half your day wrestling with web pages that aren't built for humans to copy-paste from. You want the product price, but you get the header, the footer, and three ads. This Connector changes that by letting your AI agent handle the extraction for you. Instead of building custom scrapers or trying to find the right CSS selector, you just point your agent at a URL. It figures out the structure and hands you back clean JSON. Whether you're pulling a list of job openings from a career site or grabbing every review from a forum, it does the heavy lifting of identifying the actual content. It's one of the easiest ways to add high-quality web scraping to your workflow via the Vinkius catalog. You can grab images, video metadata, and even sentiment from discussions without the headache of maintenance. It's about moving from how do I get this data to what do I want to do with this data.

## Tools

### extract_product
Grabs SKU, brand, price, and specs from e-commerce listings.

### analyze_page
Automatically classifies a URL as a product, article, or list and returns structured JSON.

### extract_article
Pulls clean text and metadata from news or blog posts while skipping comments if requested.

### extract_custom_api
Executes your specific rulesets to pull data that doesn't fit standard categories.

### extract_discussion
Grabs forum threads, user reviews, and comments for sentiment analysis.

### extract_event
Pulls schedules and specific details from event listing pages.

### extract_image
Identifies and extracts the primary images from a webpage.

### extract_job
Pulls job titles, employers, and salary info from recruitment pages.

### extract_list
Scrapes search results or directory items into a clean array of links and titles.

### extract_video
Pulls video content and metadata from media heavy pages.

## Prompt Examples

**Prompt:** 
```
Look at this page and tell me the price and SKU for the main product: [URL]
```

**Response:** 
```
I've analyzed the page and found the following product details:

| Attribute | Value |
| :--- | :--- |
| **Product Name** | Noise-Cancelling Headphones |
| **Price** | $299.00 |
| **SKU** | NC-700-BLK |
| **Availability** | In Stock |

I also found 15 detailed specifications if you need them.
```

**Prompt:** 
```
Find all the job listings on this career page and list the titles and salaries.
```

**Response:** 
```
I've scanned the careers page and found these openings:

* **Senior Software Engineer** - $150k - $180k
* **Product Manager** - $140k - $160k
* **Data Scientist** - $130k - $155k

Would you like me to find more details on any of these roles?
```

**Prompt:** 
```
Extract the main news articles from this blog and summarize the top three.
```

**Response:** 
```
I've pulled the latest articles from the blog. Here are the top three summaries:

1. **AI Trends 2024**: A look at how LLMs are moving into edge computing.
2. **New Model Release**: Details on the latest weights and performance benchmarks.
3. **Partnership News**: A new collaboration between major tech firms for infrastructure.

I can provide the full text for any of these.
```

## Capabilities

### Identify page types automatically
The AI determines if a page is a product, article, or list and returns structured JSON.

### Pull clean text from news
Extract content from blogs and news sites while stripping away the clutter.

### Grab e-commerce product specs
Capture SKU, pricing, and brand details from any retail website.

### Collect forum threads
Pull reviews and discussion comments into a structured format for analysis.

### Scrape job postings
Retrieve job titles, employers, and salary info from recruitment pages.

### Extract media metadata
Get primary images and video details from media heavy pages.

### Run custom extraction rules
Bridge your existing Diffbot dashboard rules to your AI agent for specific needs.

## Use Cases

### Competitor Pricing
A retail manager asks for a table of prices for a specific shoe across five different websites. The agent uses extract_product to build the list.

### Job Market Trends
A recruiter asks the agent to find all Senior Engineer roles in Austin with salary info. The agent uses extract_job to pull the data.

### Sentiment Tracking
A brand manager wants to see the last 20 reviews for a product to see if people are complaining about a bug. The agent uses extract_discussion.

### News Aggregation
A journalist wants to pull all blog posts from a competitor's site from the last 30 days into a summary. The agent uses extract_article.

## Benefits

- Stop writing scrapers. Use analyze_page to let the AI figure out the structure for you.
- Get clean content. extract_article removes the clutter so you only see the text that matters.
- Monitor competitors. Use extract_product to track pricing and SKUs across different retailers.
- Analyze sentiment. extract_discussion pulls forum comments so you can see what people are actually saying.
- Gather job data. Use extract_job to map out salary trends and hiring patterns in your industry.
- Automate custom rules. Bridge your existing Diffbot dashboard rules to your agent with extract_custom_api.

## How It Works

The bottom line is you get structured data from any website using plain English commands.

1. Subscribe to the Diffbot MCP on Vinkius.
2. Add your Diffbot Developer Token to your AI client settings.
3. Give your agent a URL and ask for the specific data you need.

## Frequently Asked Questions

**Can the Diffbot MCP scrape websites that don't have a standard layout?**
Yes. It uses machine learning to identify content regardless of the site's design, so you don't have to worry about different layouts.

**Does the Diffbot MCP work for e-commerce product research?**
It is built specifically for that. You can easily pull prices, brands, and SKUs from retail websites to build your own product databases.

**Can I use the Diffbot MCP to monitor my competitors?**
Yes, it's a great tool for tracking pricing and product changes across different web boundaries without manual checking.

**How does the Diffbot MCP handle forum comments and reviews?**
It pulls threads and reviews into a structured format, making it easy for your agent to analyze sentiment or summarize user feedback.

**Can the Diffbot MCP extract job postings for market research?**
Yes. It maps titles, employers, and salaries so you can track hiring trends and salary benchmarks in your industry.

**Is the Diffbot MCP good for gathering news from multiple sources?**
It's excellent for this. It handles articles and blogs, making it easy to aggregate content and summaries from various news sites.

**Can my agent automatically identify what kind of page a URL points to?**
Yes. Use the 'analyze_page' tool. Diffbot uses ML to classify the URL as an article, product, image, video, or list, and returns the appropriate structured JSON payload automatically.

**How do I extract only the main text from a blog post without comments?**
Use the 'extract_article' tool and set the 'discussion' parameter to 'false'. The agent will retrieve the clean text and HTML body while explicitly ignoring any forum threads or review blocks on the page.

**Can I use custom extraction rules I've defined in my Diffbot dashboard?**
Absolutely. Use the 'extract_custom_api' tool. Provide your trained 'api_name' and the target URL. Diffbot will extract the data according to your specific structural ruleset natively.