• DATA
  • KNOWLEDGE
  • CONTROL
  • HUMAN

How to crawl entire websites with your AI

Connect Firecrawl to your agent to turn any website into a structured dataset. Once connected, your agent can navigate and scrape entire domains.

Ask AI about this page

Short answer

How can I crawl a whole site with my AI?

Your agent uses Firecrawl to follow every link on a target domain and pull the content into a readable format. This turns a website into a structured source for your agent. You'll get a complete map of the site's data.

The results.

Where the data goes.

The crawl process converts raw web pages into organized information.

DATA

Structured site maps

The AI maps out every URL and page structure. You get a clear hierarchy of the site's content.

KNOWLEDGE

Deep knowledge bases

The agent ingests the scraped text. This lets your AI answer questions based on the whole site, not just one page.

CONTROL

Active crawl management

You can monitor the progress or stop a run if it goes too deep. The AI keeps you updated on the status.

HUMAN

Manual review needed

The AI cannot decide which pages are irrelevant. You'll need to check the final data to ensure the scraper didn't hit junk pages.

The workflow

What your AI does when the URL arrives.

The AI handles the heavy lifting of navigating the web and gathering data.

  1. Start the crawl

    Your agent sends the starting URL to Firecrawl to begin the recursive search through the domain.

    start_crawl

  2. Check progress

    The AI checks in on the job to see how many pages have been processed and how many are left.

    get_crawl_status

  3. Stop the crawl

    If the crawl is taking too long or hitting the wrong areas, your agent can kill the task immediately.

    cancel_active_crawl

  4. Verify completion

    The AI uses the status tool to confirm the job is done and the data is ready for use.

    get_crawl_status

Try it

Copy these to start.

Use these prompts to trigger a crawl with your agent.

Starting points

These are opening lines. Swap out the website URLs for the ones you actually need to scrape.

Firecrawl Connector

You're all set. Choose your MCP client and follow the setup instructions.

Connector linkhttps://edge.vinkius.com/vk_preview_fc9lRjafM7OLZXZoNzbjhXgbJ3RhOnI5LpERnMdC/mcp

Claude Desktop

Follow the steps below to connect in seconds.

  1. 1In Claude Desktop, open Settings → Connectors.
  2. 2Click “Add custom connector” and paste the connector link above as the remote MCP server URL.
  3. 3Click Add and start a new chat — Firecrawl capabilities are ready to use.
Configuration · claude_desktop_config.jsonCopy
{
  "mcpServers": {
    "firecrawl-alternative-mcp": {
      "url": "https://edge.vinkius.com/vk_preview_fc9lRjafM7OLZXZoNzbjhXgbJ3RhOnI5LpERnMdC/mcp"
    }
  }
}
Copy into chat04

  • Start a crawl of https://docs.example.com so you can answer my technical questions.

  • Check the status of my current crawl to see if the data is ready.

  • Stop the active crawl on the site I just gave you; it's hitting too many subdomains.

  • Crawl the entire documentation site at https://help.software.com and summarize the main features.

  • Claude
  • ChatGPT
  • Cursor
  • VS Code
  • Windsurf
  • Claude Code
  • JetBrains
  • Cline

Start here

Connect Firecrawl once, then ask.

You only need to link the MCP once. Your credentials stay encrypted, and you can immediately start giving your agent web crawling tasks.

Connect Firecrawl to your AI

FAQ

How the crawler behaves

  • 01

    Can the AI crawl a site that requires a login?

    No. The current tools only support crawling publicly accessible URLs.

  • 02

    Will the AI delete any files on the website?

    No. The tools are strictly for reading and crawling data. They cannot modify or delete content on the target website.

  • 03

    How does the AI know when the crawl is finished?

    The AI uses the status tool to check the progress until it sees the job is complete.

  • 04

    Can I limit how many pages the AI visits?

    No. The tools perform recursive crawls based on the starting URL. You must manually stop the crawl if it goes too deep.

  • 05

    Does the AI store the website data permanently?

    No. The AI uses the scraped data to answer your immediate questions. It does not manage a permanent database for you.

  • More questions about Firecrawl? The Connector page answers them. See everything the Firecrawl Connector can do

Connect Firecrawl to Claude, Cursor, ChatGPT & more