• VALIDATION
  • ERROR DETECTION
  • PIPELINE AUDIT
  • HUMAN REVIEW

How to test RAG pipeline accuracy with your AI

Connect Haystack to your agent to run specific pipelines and validate that the retrieved context is actually correct.

Ask AI about this page

Short answer

How can I test my RAG pipeline accuracy?

Your agent uses the Haystack connector to trigger specific pipelines and compare the output against your indexed documents. This lets you catch retrieval errors or hallucinations by checking if the agent's answers align with your actual data.

Testing outcomes

What you get from testing

The agent handles the heavy lifting of execution and comparison.

VALIDATION

Verified retrieval

The agent runs a pipeline and checks if the retrieved chunks contain the necessary facts. You get a clear view of whether your indexing is working.

ERROR DETECTION

Identify hallucinations

By comparing pipeline outputs to your source data, your agent flags when it makes things up. This helps you pinpoint exactly where the logic fails.

PIPELINE AUDIT

Pipeline inventory

Your agent can see every available pipeline in your Haystack setup. This ensures you are testing the right versions of your workflows.

HUMAN REVIEW

Manual correction

The agent identifies specific failures that require you to adjust your chunking strategy or embedding model. You decide how to fix the underlying data.

The workflow

What your AI does when the test starts.

The agent takes over the execution and comparison of your Haystack workflows.

  1. Inventory pipelines

    The agent looks at your Haystack instance to see which specific pipelines are ready for testing.

    list_pipelines

  2. Execute the test

    Your agent triggers the selected pipeline with specific inputs to generate a response or a set of results.

    run_pipeline

  3. Fetch source data

    To verify the answer, the agent pulls the original documents from your index to check for factual alignment.

    search_documents

  4. Compare and report

    The agent evaluates the pipeline output against the retrieved documents to find discrepancies.

    run_pipeline

Try it

Copy these to start.

Use these prompts to begin testing your pipelines.

Starting points

Swap out the pipeline names and document IDs with your actual Haystack identifiers.

deepset Cloud Connector

You're all set. Choose your MCP client and follow the setup instructions.

Connector linkhttps://edge.vinkius.com/vk_preview_Tf2g8zLZroyDlliFU39whBZWkEDykSKQVz38t3zr/mcp

Claude Desktop

Follow the steps below to connect in seconds.

  1. 1In Claude Desktop, open Settings → Connectors.
  2. 2Click “Add custom connector” and paste the connector link above as the remote MCP server URL.
  3. 3Click Add and start a new chat — deepset Cloud capabilities are ready to use.
Configuration · claude_desktop_config.jsonCopy
{
  "mcpServers": {
    "haystack-deepset-cloud-mcp": {
      "url": "https://edge.vinkius.com/vk_preview_Tf2g8zLZroyDlliFU39whBZWkEDykSKQVz38t3zr/mcp"
    }
  }
}
Copy into chat04

  • List all my Haystack pipelines so I can choose one to test.

  • Run the 'qa_pipeline' and tell me if the results match the documents in my index.

  • Check the 'retrieval_test' pipeline and search for documents related to the user query.

  • Run the 'summarization_pipeline' and verify the output against the source text.

  • Claude
  • ChatGPT
  • Cursor
  • VS Code
  • Windsurf
  • Claude Code
  • JetBrains
  • Cline

Start here

Connect Haystack once, then ask.

Just provide your Haystack credentials. Your connection is encrypted, and you can start running pipelines immediately.

Connect Haystack to your AI

FAQ

How the testing works

  • 01

    Can the AI change my Haystack pipeline configurations?

    No. The agent can only run existing pipelines or list them. It cannot modify your code or settings.

  • 02

    Can the agent delete my indexed documents?

    No. The tools only allow for listing pipelines, running them, and searching documents. Deletion is not possible.

  • 03

    How does the agent know if a result is a hallucination?

    The agent uses the search tool to pull the actual document text and compares it to the pipeline output to find mismatches.

  • 04

    Do I need to manually trigger every test?

    You can ask your agent to run them via chat, which automates the execution of the pipeline.

  • 05

    Can I test multiple pipelines at once?

    You can ask the agent to run them sequentially, but it executes each pipeline call individually.

  • More questions about Haystack? The Connector page answers them. See everything the Haystack Connector can do

Connect deepset Cloud to Claude, Cursor, ChatGPT & more