# Firecrawl MCP for AI Agents AI Agent Connect > Firecrawl lets you scrape entire websites and turn them into clean LLM-ready markdown with a single request that handles JavaScript rendering. It supports recursive crawling to build full knowledge bases and site mapping to understand complex web structures. ## Overview - **Category:** friends-mcp - **Price:** Free - **Endpoint:** https://edge.vinkius.com/vk_preview_VUpOJRFVTRGTwbzAMeVqIv4t7KoCrITMFS6MmDGP/ai-agent-connect - **Tags:** data-extraction, markdown-conversion, rag-pipelines, recursive-crawling, llm-ready, web-data ## Description This web scraping MCP lets you take full control of web data acquisition by connecting your account to your AI agent. You can scrape any single URL and get back clean markdown that's already stripped of headers, footers, and other boilerplate content. It handles the JavaScript rendering for you, so you don't have to worry about the technical hurdles of modern web pages like React or Vue components. If you need to build a large knowledge base, you can perform recursive crawls to find and grab every subpage on a domain. It also lets you map out a website's structure or take full-page screenshots to keep a visual record of what's online. It's a reliable way to turn the messy web into structured data for your RAG pipelines. You'll find this is one of the most efficient ways to manage web data within the Vinkius catalog. You won't have to deal with broken tags or messy HTML, so your agent gets the clean text it needs to actually be useful. This means you spend less time debugging scrapers and more time building the actual features of your application. It links the web directly to your data storage without the usual headaches of manual extraction or complex coding. You can focus on the results while the tool handles the navigation and data cleaning. ## Tools ### get_api_usage Check your current Firecrawl credit balance. This helps you track your spending and remaining limits in real time. ### cancel_active_crawl Stop a crawl job that's currently running. Use this if a task is taking too long or you need to halt a process immediately. ### get_crawl_status Check the progress of a specific crawl job. This tells you if the job is still running, finished, or failed. ### start_crawl Start a recursive crawl of a whole website to find all subpages. It returns a job ID so you can track the progress as it builds your knowledge base. ### map_website_structure Identify every reachable link on a domain without downloading the content. This is great for understanding a site's hierarchy before you start scraping. ### scrape_url Turn a single URL into clean markdown. It automatically removes boilerplate like headers and footers so the data is ready for your agent to use. ## Prompt Examples **Prompt:** ``` Turn 'https://stripe.com/docs/api' into clean Markdown. ``` **Response:** ``` Scraping in progress... I've converted the Stripe API documentation into clean markdown, removing the navigation and footers. Would you like me to summarize the main endpoints for you? ``` **Prompt:** ``` Crawl 'https://docs.firecrawl.dev' recursively with a limit of 10 pages. ``` **Response:** ``` Crawl job started! I've initiated a recursive crawl of the documentation (ID: crl_123). I'll monitor the progress and notify you as soon as the 10 pages are indexed. ``` **Prompt:** ``` Map all internal links for 'https://github.com/vinkius'. ``` **Response:** ``` Mapping site structure... I've identified all reachable links for the requested domain. I found 25 internal URLs. Would you like the full list of mapped URLs? ``` ## Capabilities ### Turn URLs into clean markdown Convert any web page into high-fidelity markdown while automatically stripping out headers and footers. ### Crawl entire websites recursively Discover and scrape every subpage on a domain to build a complete knowledge base. ### Map out website structures Identify all reachable links on a domain to understand its hierarchy without downloading content. ### Capture full-page screenshots Take a visual record of any URL directly through your agent. ### Monitor credit usage and job status Track your remaining credits and check the progress of active crawl jobs in real time. ## Use Cases ### Building a RAG knowledge base A developer needs to feed 500 documentation pages into a vector store. They ask the agent to crawl the docs site recursively to get clean markdown. ### Automated market research An analyst needs to monitor a competitor's pricing across 50 pages. They ask the agent to scrape specific product URLs daily. ### Content curation A blogger wants to convert a long-form article into a markdown blog post. They ask the agent to convert the URL into clean text. ### SEO site mapping A marketer needs to see all internal links on a domain. They ask the agent to map the website structure to find broken paths. ## Benefits - Get clean markdown every time using scrape_url so your agent doesn't get lost in messy HTML tags or navigation menus. - Build massive knowledge bases automatically with start_crawl to index every subpage on a domain for your RAG pipeline. - Skip the manual work of finding links by using map_website_structure to see the whole site layout in seconds. - Keep a visual record of your data with visual capture so you can see exactly what your agent is seeing on the page. - Manage your costs and limits easily with get_api_usage to ensure your scraping jobs stay within your budget. - Stop runaway processes instantly with cancel_active_crawl to save your credits when a job isn't performing as expected. ## How It Works The bottom line is you get high-quality web data in a format your AI can actually read without any manual cleanup. 1. Get your Firecrawl API key from the dashboard. 2. Connect the Connector to your AI client like Claude or Cursor. 3. Ask your agent to scrape a URL or crawl a domain. ## Frequently Asked Questions **Can Firecrawl scrape websites that use JavaScript?** Yes, this Connector handles JavaScript rendering automatically, so it can scrape modern web apps that wouldn't work with basic scrapers. **How does Firecrawl handle messy HTML like headers and footers?** It automatically strips out boilerplate content like navigation menus and footers, giving you clean markdown that's ready for your AI to read. **Can I use Firecrawl to build a RAG pipeline?** That's a primary use case. You can crawl entire websites to build comprehensive knowledge bases for your RAG applications. **How do I see how many credits I have left?** You can ask your agent to check your Firecrawl credit usage at any time to see your remaining limits and active jobs. **Can I stop a crawl if it's taking too long?** Yes, you can tell your agent to cancel an active crawl job immediately to save your credits. **What is the difference between scraping a URL and crawling a site?** Scraping a URL gets data from a single page, while crawling a site follows every link on a domain to gather data from every subpage. **How do I find my Firecrawl API Key?** Log in to your [**Firecrawl dashboard**](https://www.firecrawl.dev/app/dashboard), and navigate to the **API Keys** section to copy your unique token. **Can I scrape content excluding headers and footers?** Yes! The `scrape_url` tool includes an `onlyMainContent` parameter. When set to true, Firecrawl uses AI to extract only the core article or page content. **How long does a recursive crawl take?** Crawl time depends on the site size and depth. Use the `get_crawl_status` tool to monitor progress and retrieve results once the job is complete.