• ANALYSIS
  • IDENTIFICATION
  • REPORTS
  • HUMAN

How to compare RAG experiments with your AI

Connect Ragas to your AI client to analyze performance across different model versions or prompt configurations.

Ask AI about this page

Short answer

How can I compare RAG experiments with my AI?

You can pull specific experiment data and metric results into your chat to run side-by-side comparisons. Your AI handles the heavy lifting of analyzing which configuration produced the best scores. You'll get a clear view of how changes to your retrieval or generation affect performance.

Where your data lands.

The outcomes of your analysis.

The AI organizes your Ragas data into these specific outcomes.

ANALYSIS

Performance comparisons

The AI pulls results from your runs to show you exactly how different versions stack up against each other.

IDENTIFICATION

Top performing runs

Your agent identifies which specific configurations hit your target metrics most consistently.

REPORTS

Summary of results

Get a clean breakdown of metrics across all your recorded experiments.

HUMAN

Manual tuning

The AI identifies weak spots in your RAG pipeline, but you decide which prompt or model changes to implement next.

The workflow

What your AI does when the data arrives.

The AI acts as your data analyst for RAG evaluation.

  1. Find your runs

    The AI looks through your Ragas history to find the specific test suites you want to compare.

    list_experiments

  2. Select an experiment

    Once you pick a run, the AI pulls the specific configuration details for that test.

    get_experiment

  3. Pull the metrics

    The AI fetches the actual scores and evaluation data from the completed run.

    get_results

  4. Compare performance

    The AI uses the retrieved results to highlight differences in faithfulness, relevancy, or other metrics.

    get_results

Try it

Copy these to start.

Use these prompts to begin your analysis.

Starting points

Swap the experiment names or IDs with your actual Ragas run identifiers.

Ragas Connector

You're all set. Choose your MCP client and follow the setup instructions.

Connector linkhttps://edge.vinkius.com/vk_preview_mQUV7klN4rh09rRcp9ld3cwceCP25oL5Lk8res4B/mcp

Claude Desktop

Follow the steps below to connect in seconds.

  1. 1In Claude Desktop, open Settings → Connectors.
  2. 2Click “Add custom connector” and paste the connector link above as the remote MCP server URL.
  3. 3Click Add and start a new chat — Ragas capabilities are ready to use.
Configuration · claude_desktop_config.jsonCopy
{
  "mcpServers": {
    "ragas-mcp": {
      "url": "https://edge.vinkius.com/vk_preview_mQUV7klN4rh09rRcp9ld3cwceCP25oL5Lk8res4B/mcp"
    }
  }
}
Copy into chat04

  • List all my recent RAG experiments so I can see which ones to compare.

  • Compare the results of experiment 'rag_v1_test' and 'rag_v2_test'.

  • Show me the results for the experiment named 'production_baseline'.

  • Which experiment in my list has the highest faithfulness score?

  • Claude
  • ChatGPT
  • Cursor
  • VS Code
  • Windsurf
  • Claude Code
  • JetBrains
  • Cline

Start here

Connect Ragas once, then ask.

Just link your Ragas account. Your credentials stay encrypted, and you can immediately start asking your AI about your experiment data.

Connect Ragas to your AI

FAQ

How this task behaves.

  • 01

    Can the AI change my experiment settings?

    No. The AI can only read and analyze existing experiment data. You must change configurations in your Ragas environment manually.

  • 02

    Can the AI delete old experiment results?

    No. This MCP is read-only for your experiment data. It cannot delete or modify any of your Ragas records.

  • 03

    Can I compare two different RAG frameworks?

    Yes, as long as both sets of experiments are recorded within Ragas and accessible via the connector.

  • 04

    Does the AI calculate the metrics itself?

    No. The AI retrieves the metrics already calculated by Ragas and interprets them for you.

  • 05

    Can I see the raw data from a specific run?

    Yes. The AI can pull the specific results for any experiment you identify.

  • More questions about Ragas? See everything the Ragas Connector can do

Connect Ragas to Claude, Cursor, ChatGPT & more