# Firecrawl MCP for AI Agents AI Agent Connect

> Firecrawl turns any website into clean Markdown for your AI agents. It handles the messy parts of web scraping like JavaScript rendering and anti-bot protections, so your agent gets structured data instead of a wall of HTML. It's built for high-quality data extraction at scale.

## Overview
- **Category:** friends-mcp
- **Price:** Free
- **Endpoint:** https://edge.vinkius.com/vk_preview_H3lslwPhMfSWLHyTzOylzN5WvZ9uNMEMQ6eYZqT4/ai-agent-connect
- **Tags:** markdown-conversion, llm-data, web-crawling, data-extraction, structured-data, dynamic-content

## Description

When you need your AI to know what's happening on the web, you usually run into a wall of messy HTML, pop-ups, and bot blockers. It's frustrating to build a pipeline that breaks every time a site updates its layout or adds a new cookie banner. This Connector fixes that by handling the heavy lifting of web navigation and content extraction. Instead of your agent struggling to parse a page, it gets a clean Markdown file that it can actually understand and use immediately. You can grab a single article, crawl an entire documentation site to build a knowledge base, or search the web for specific facts and get the results back in a structured format. It's a huge shift from manually scraping data to having your agent just know the web. If you're looking for a reliable way to feed live data into your projects, finding this in the Vinkius catalog makes it easy to get started without worrying about the underlying infrastructure. You just provide an API key and your agent suddenly has a window into the live internet, ready to do the research or data gathering you need.

## Tools

### search_web
Perform a Google-like search and get the content of the top results back in one step.

### crawl_site
Recursively visit every page on a domain to build a full content index for your agent.

### map_site
See every URL on a website to understand the structure before you decide what to extract.

### scrape_page
Get a single URL's content as clean Markdown while handling all the JS rendering and pop-ups for you.

## Prompt Examples

**Prompt:** 
```
Find the latest news on renewable energy from the last 24 hours and summarize the top 3.
```

**Response:** 
```
**Latest Renewable Energy News**

| Source | Headline | Summary |
| :--- | :--- | :--- |
| TechDaily | Solar Breakthrough | New perovskite cells hit 30% efficiency |
| GreenNews | Wind Farm Expansion | 500MW project approved in North Sea |
| EnergyHub | Battery Storage | Solid-state tech reaches milestone |

*Summary: The focus remains on high-efficiency solar and large-scale wind infrastructure.*
```

**Prompt:** 
```
Look at the documentation for the new API and tell me how to authenticate.
```

**Response:** 
```
**Authentication Guide**

To authenticate with the new API, you need to follow these steps:

1. **Generate a Key**: Go to the developer dashboard to create your API key.
2. **Header Setup**: Include the key in your request header as `X-API-Key`.
3. **Example Request**:
   ```bash
   curl -H "X-API-Key: YOUR_KEY" https://api.example.com/v1/data
   ```

*Note: Keys expire every 90 days.*
```

**Prompt:** 
```
Map out the entire blog section of this site so I can see how many posts they have.
```

**Response:** 
```
**Site Map: Blog Section**

I found 42 URLs in the blog section:
- /blog/getting-started
- /blog/advanced-tips
- /blog/case-study-1
- /blog/case-study-2
... and 38 others.

**Total Posts Found:** 42
```

## Capabilities

### Scrape pages into Markdown
Extract clean content from any URL while handling JavaScript and pop-ups.

### Crawl entire websites
Recursively visit every page on a domain to build a full content index.

### Map site architecture
Discover all URLs on a website to understand the structure before scraping.

### Search and extract web content
Perform a web search and get the content of the top results in one step.

### Bypass bot protections
Automatically handle cookies, banners, and anti-bot measures.

## Use Cases

### Building a RAG system for company docs
An engineer asks the agent to index the help center, and it recursively crawls the entire site to gather all the content for the vector database.

### Competitive price monitoring
A researcher asks for the latest prices on a competitor's site, and the agent scrapes the page to get the data in a clean format.

### Automated news summaries
A user wants a daily summary of a niche topic, so the agent searches the web to find and read the latest articles in one go.

### Mapping out a new lead list
A sales lead wants to see all the pages on a target company's site, so the agent maps the site to find every available URL.

## Benefits

- Stop worrying about anti-bot blocks. The system handles proxies and headers automatically so your agent doesn't get blocked.
- Get data in a format your agent actually likes. It converts messy HTML into clean Markdown, saving you from token waste.
- Index entire documentation libraries quickly. You can grab every page on a site to build a complete knowledge base.
- See the full picture of a website first. You can discover all hidden links so you don't miss important data.
- Combine searching and scraping into one step. Your agent can find and read the best results in a single turn.

## How It Works

The bottom line is you get clean, usable web data without the headache of manual scraping or bot blocking.

1. Connect your Firecrawl API key to the Connector.
2. Tell your agent which URL or search query to target.
3. Receive clean Markdown content or a list of discovered URLs.

## Frequently Asked Questions

**Can Firecrawl help my agent bypass bot protections?**
Yes, it's built to handle those issues automatically so your agent doesn't get blocked by common anti-bot measures.

**Does Firecrawl convert websites to Markdown?**
It does, which makes it much easier for your agent to read the content without the noise of HTML tags and scripts.

**Can I use Firecrawl to crawl an entire website at once?**
Yes, you can use it to recursively visit every page on a domain to build a full content index for your agent.

**How does Firecrawl handle JavaScript-heavy websites?**
It renders the JavaScript first, so your agent sees the actual content as it appears to a human user.

**Is Firecrawl good for building RAG systems?**
It's a great choice because it provides clean, structured data that is perfect for feeding into a vector database.

**Can Firecrawl search the web for me?**
It can perform a web search and then extract the content from the top results in one single step.

**How does Firecrawl pricing work?**
Firecrawl uses a credit-based system. You get 500 free lifetime credits to start (no credit card required). Base cost is 1 credit per page scraped. Advanced features like JSON extraction (+4 credits) or enhanced mode (+4 credits) consume additional credits per page. Paid plans start at $16/month with 3,000 monthly credits.

**Can Firecrawl handle JavaScript-heavy websites?**
Yes! Firecrawl renders pages in a full browser environment before extracting content — this means it handles React, Next.js, Angular, and any other JavaScript framework. It also automatically bypasses common anti-bot protections, removes cookie consent banners, and waits for dynamic content to load before extraction.

**What formats does Firecrawl return?**
Firecrawl can return content in multiple formats: Markdown (default and most popular for LLM consumption), HTML, raw HTML, structured JSON (with LLM-powered extraction), screenshots, links, and page metadata. You can request multiple formats in a single call.