• RESEARCH
  • DATA
  • LOGS
  • HUMAN

How to pull news articles with your AI agent

Connect Diffbot to your AI client once. From then on, your agent can scrape specific URLs and turn messy web pages into structured text for your research.

Ask AI about this page

Short answer

How can I get my AI to read news articles?

Your AI uses Diffbot to strip away the junk like ads and navigation menus, leaving only the core article text. This gives your agent clean data to summarize or analyze immediately. You get structured content ready for your specific research needs.

Where the data lands.

Clean content for your research.

The AI processes the web content and delivers it to these specific outcomes.

RESEARCH

Clean text summaries

The AI pulls the main body of a news story. You get a concise breakdown of the facts without reading through sidebars.

DATA

Structured article text

The tool removes HTML clutter. Your agent receives pure text that is easy to parse for sentiment or entity extraction.

LOGS

Research archives

You can have your AI save the extracted text into a document or sheet. This builds a searchable library of news you've processed.

HUMAN

Fact checking

The AI provides the source text alongside its summary. You review the original content to ensure the agent didn't miss any nuance.

The workflow

What your AI does when a URL arrives.

The AI handles the heavy lifting of web scraping so you don't have to copy and paste manually.

  1. Identify the content

    The agent looks at the provided link to determine if it's a news story or a blog post.

    analyze_page

  2. Strip the noise

    The AI removes ads, popups, and navigation elements from the page.

    extract_article

  3. Isolate the text

    The agent pulls the headline and the main body text into a clean format.

    extract_article

  4. Prepare for use

    The cleaned data is handed back to your agent to start the actual research task.

    analyze_page

Try it

Copy these to start.

Use these prompts once you've connected Diffbot to your agent.

Starting points

These are opening lines. Swap out the bracketed info for your actual news links or target files.

Diffbot Connector

You're all set. Choose your MCP client and follow the setup instructions.

Connector linkhttps://edge.vinkius.com/vk_preview_FiT28RroNiKkMcafLWUuAoeV2pbdaAk9JBpgecYE/mcp

Claude Desktop

Follow the steps below to connect in seconds.

  1. 1In Claude Desktop, open Settings → Connectors.
  2. 2Click “Add custom connector” and paste the connector link above as the remote MCP server URL.
  3. 3Click Add and start a new chat — Diffbot capabilities are ready to use.
Configuration · claude_desktop_config.jsonCopy
{
  "mcpServers": {
    "diffbot-mcp": {
      "url": "https://edge.vinkius.com/vk_preview_FiT28RroNiKkMcafLWUuAoeV2pbdaAk9JBpgecYE/mcp"
    }
  }
}
Copy into chat04

  • Extract the main text from this news article: [URL]

  • Summarize the key points from this blog post: [URL]

  • Read the article at [URL] and tell me who the main subjects are.

  • Pull the text from [URL] and save a summary to my research doc.

  • Claude
  • ChatGPT
  • Cursor
  • VS Code
  • Windsurf
  • Claude Code
  • JetBrains
  • Cline

Start here

Connect Diffbot once, then ask.

Just click the link to connect. Your credentials stay encrypted and private. Once it's active, you can start feeding URLs to your agent immediately.

Connect Diffbot to your AI

FAQ

How this task behaves.

  • 01

    Can the AI browse the live web on its own?

    No. The AI only accesses specific pages when you provide a URL or ask it to use the Diffbot tools.

  • 02

    Will the AI change the original news website?

    No. The AI only reads and extracts data. It cannot edit, delete, or interact with the actual website.

  • 03

    Does it work on sites behind a login?

    No. The tools can only access publicly available news articles and blog posts.

  • 04

    Can the AI download images from the articles?

    No. The current tools are designed to extract text and article structure, not media files.

  • 05

    What happens if a page is just a bunch of ads?

    The extraction tool will attempt to find the main content, but if no article text exists, it will return an empty result.

  • More questions about Diffbot? See everything the Diffbot Connector can do

Connect Diffbot to Claude, Cursor, ChatGPT & more