• WEB
  • FILES
  • RAG
  • HUMAN

How to automate RAG knowledge ingestion with your AI

Connect Open WebUI once. After that, your agent handles the scraping, uploading, and organization of data into searchable collections.

Ask AI about this page

Short answer

How can I automate RAG knowledge ingestion?

You give your agent a URL or a file, and it handles the scraping and organization into RAG collections. This builds a structured knowledge base you can query immediately.

Data destinations.

Where your information lands.

The AI moves data from the web or your local files into organized, searchable collections.

WEB

Live web content

The AI pulls text from any URL you provide. This turns websites into searchable context for your agent.

FILES

Document uploads

Upload files directly. Your agent processes these documents and prepares them for retrieval.

RAG

Organized collections

Data is sorted into specific collections. This keeps your knowledge base clean and easy to navigate.

HUMAN

Manual oversight

You decide which files are ready and which collections need more data. The AI moves the files, but you manage the library.

The workflow

What your AI does when data arrives.

The AI acts as the bridge between raw information and your searchable knowledge base.

  1. Scrape websites

    When you provide a link, the AI visits the site and extracts the text content.

    process_web_url

  2. Upload files

    The AI takes the files you provide and prepares them for the system.

    upload_file

  3. Check status

    The AI tracks the progress of your uploads to ensure the data is ready for use.

    get_file_status

  4. Organize data

    The AI places the processed files into the specific RAG collections you've designated.

    add_file_to_collection

Try it

Copy these to start.

Use these prompts to trigger the ingestion process.

Starting points

Replace the URLs and file names with your actual data.

Open WebUI Connector

You're all set. Choose your MCP client and follow the setup instructions.

Connector linkhttps://edge.vinkius.com/vk_preview_YhwRL8adKsCUXpVOwybG0l7vG8Iky4FGyj9DhmOn/mcp

Claude Desktop

Follow the steps below to connect in seconds.

  1. 1In Claude Desktop, open Settings → Connectors.
  2. 2Click “Add custom connector” and paste the connector link above as the remote MCP server URL.
  3. 3Click Add and start a new chat — Open WebUI capabilities are ready to use.
Configuration · claude_desktop_config.jsonCopy
{
  "mcpServers": {
    "open-webui-mcp": {
      "url": "https://edge.vinkius.com/vk_preview_YhwRL8adKsCUXpVOwybG0l7vG8Iky4FGyj9DhmOn/mcp"
    }
  }
}
Copy into chat04

  • Scrape this website and add the content to my research collection: https://example.com/article

  • Upload this PDF and put it into the project documentation collection.

  • Check the status of my recent file uploads to see if they are ready for RAG.

  • Process the content from this URL and save it to the technical specs collection: https://docs.example.com

  • Claude
  • ChatGPT
  • Cursor
  • VS Code
  • Windsurf
  • Claude Code
  • JetBrains
  • Cline

Start here

Connect Open WebUI once, then ask.

Click the link to connect. Your credentials stay encrypted, and the full technical details live on the Connector page.

Connect Open WebUI to your AI

FAQ

How this task behaves.

  • 01

    Can the AI delete files from my collections?

    No. The AI can only upload and add files to collections; it cannot delete or remove existing data.

  • 02

    Can the AI create new collections?

    No. You must have existing collections for the AI to add files to.

  • 03

    Can the AI scrape websites behind a login?

    No. The AI can only process publicly accessible URLs.

  • 04

    How do I know when the data is ready to use?

    The AI uses the file status tool to check if the ingestion is finished before you query it.

  • 05

    Does the AI change the content it scrapes?

    No. The AI ingests the text exactly as it appears on the page.

  • More questions about Open WebUI? See everything the Open WebUI Connector can do

Connect Open WebUI to Claude, Cursor, ChatGPT & more