• UPDATED
  • ACCURATE
  • SYNCED
  • HUMAN

How to reindex documentation websites with your AI

Connect Fern to your agent, then tell it to refresh its knowledge. It handles the crawling so your agent doesn't hallucinate on old docs.

Ask AI about this page

Short answer

How do I reindex my documentation site for my AI?

Use the index_website tool to trigger a fresh crawl of your site. This forces your agent to see every new page and update you've pushed since the last crawl. You'll get a clean, updated knowledge base for your agent to query.

The results of a crawl.

What your agent gains.

The AI handles the heavy lifting of visiting every URL and pulling in the new content.

UPDATED

Fresh knowledge base

The agent crawls the site and pulls in every new or modified page. Your agent's responses will now reflect your most recent documentation updates.

ACCURATE

Reduced hallucinations

By replacing old data with current text, the agent stops relying on outdated technical specs. This keeps your support or dev answers accurate.

SYNCED

Complete site coverage

The tool follows links to find new subdirectories or nested pages. Your agent gets the full picture of your site structure.

HUMAN

Manual review needed

The AI cannot verify if your documentation is actually correct or well-written. You still need to check the source content for accuracy.

The workflow

What your AI does when the crawl starts.

The agent uses the Fern connection to reach out to your web host and pull in the data.

  1. Initiate crawl

    Your agent sends a command to start a fresh pass of your specified domain.

    index_website

  2. Discover pages

    The tool traverses your site to find every live URL and updated file.

    index_website

  3. Ingest content

    The agent reads the text from the newly discovered pages to update its internal context.

    index_website

  4. Confirm readiness

    The agent lets you know when the new index is ready for questions.

    index_website

Try it

Copy these to start.

Use these prompts once you have connected Fern to your agent.

Starting points

Swap the bracketed text for your actual documentation URL or site name.

Fern Connector

You're all set. Choose your MCP client and follow the setup instructions.

Connector linkhttps://edge.vinkius.com/vk_preview_VqEKputPZIEstF5UI8C28igkFzBgP67EyQDUBcP9/mcp

Claude Desktop

Follow the steps below to connect in seconds.

  1. 1In Claude Desktop, open Settings → Connectors.
  2. 2Click “Add custom connector” and paste the connector link above as the remote MCP server URL.
  3. 3Click Add and start a new chat — Fern capabilities are ready to use.
Configuration · claude_desktop_config.jsonCopy
{
  "mcpServers": {
    "fern-mcp": {
      "url": "https://edge.vinkius.com/vk_preview_VqEKputPZIEstF5UI8C28igkFzBgP67EyQDUBcP9/mcp"
    }
  }
}
Copy into chat04

  • Reindex my documentation at https://docs.example.com so you have the latest version.

  • Run a fresh crawl of the example.com docs to pick up the new API changes.

  • Update your knowledge of the documentation site at https://docs.myapp.io.

  • Can you reindex the docs for the new release so your answers are current?

  • Claude
  • ChatGPT
  • Cursor
  • VS Code
  • Windsurf
  • Claude Code
  • JetBrains
  • Cline

Start here

Connect Fern once, then ask.

Click the button to link Fern to your client. Your credentials stay encrypted, and you can immediately start asking your agent to crawl your sites.

Connect Fern to your AI

FAQ

Common questions about reindexing

  • 01

    Does the AI change my actual website content?

    No. The tool only reads your website to update the agent's knowledge. It cannot edit, delete, or move files on your server.

  • 02

    How often should I reindex my docs?

    You should trigger a reindex whenever you push significant updates to your documentation to ensure your agent stays current.

  • 03

    Can the AI fix broken links on my site?

    No. The tool identifies the content on the pages, but it has no capability to modify your website's code or link structure.

  • 04

    Will this work for sites behind a login?

    No. The index_website tool crawls publicly accessible URLs. It cannot bypass authentication or access private, password-protected pages.

  • 05

    Does the agent remember the old docs after a reindex?

    The goal of a reindex is to replace the old context with the new. The agent will prioritize the most recent crawl data.

  • More questions about Fern? The Connector page answers them. See everything the Fern Connector can do

Connect Fern to Claude, Cursor, ChatGPT & more