• RUNS
  • METRICS
  • RESULTS
  • HUMAN

How to run RAG evaluations with your AI client

Connect Ragas once to let your agent run tests on your RAG pipelines and pull the results straight into your conversation.

Ask AI about this page

Short answer

How can I run RAG evaluations through my AI?

You tell your agent to trigger an evaluation on a specific dataset, and it uses Ragas to calculate scores like faithfulness or relevancy. You'll get the performance metrics back in your chat window immediately. This allows you to iterate on your prompts and context retrieval without leaving your workspace.

The outcomes.

Where your evaluation data lands.

The AI handles the heavy lifting of the math and data processing.

RUNS

New test cycles

The AI starts a fresh evaluation run on your provided datasets. You get a clean slate for every test iteration.

METRICS

Available score types

Your agent can check which specific metrics are available for your current setup. This helps you decide what to measure.

RESULTS

Performance scores

The AI pulls the final scores directly into your chat. You see exactly how your RAG system is performing.

HUMAN

Manual adjustments

You decide which datasets to test and which metrics matter most. The AI won't change your underlying data or test parameters without you asking.

The workflow

What your AI does when you request an evaluation.

The AI acts as the bridge between your chat interface and the Ragas engine.

  1. Identify metrics

    The AI checks which evaluation metrics are supported for your specific RAG task.

    list_metrics

  2. Start the test

    The AI kicks off the evaluation process using your specified dataset.

    run_evaluation

  3. Fetch scores

    Once the math is done, the AI grabs the resulting performance data.

    get_results

  4. Report findings

    The AI presents the final scores and metric breakdowns in your chat window.

    get_results

Try it

Copy these to start.

Use these prompts to kick off your first evaluation.

Starting points

Replace the bracketed text with your actual dataset names or specific metric requirements.

Ragas Connector

You're all set. Choose your MCP client and follow the setup instructions.

Connector linkhttps://edge.vinkius.com/vk_preview_mQUV7klN4rh09rRcp9ld3cwceCP25oL5Lk8res4B/mcp

Claude Desktop

Follow the steps below to connect in seconds.

  1. 1In Claude Desktop, open Settings → Connectors.
  2. 2Click “Add custom connector” and paste the connector link above as the remote MCP server URL.
  3. 3Click Add and start a new chat — Ragas capabilities are ready to use.
Configuration · claude_desktop_config.jsonCopy
{
  "mcpServers": {
    "ragas-mcp": {
      "url": "https://edge.vinkius.com/vk_preview_mQUV7klN4rh09rRcp9ld3cwceCP25oL5Lk8res4B/mcp"
    }
  }
}
Copy into chat04

  • Run an evaluation on my 'customer_support_v2' dataset.

  • What metrics can I use with Ragas for this dataset?

  • Get the results from the last evaluation run I triggered.

  • Run a new evaluation on 'product_docs_final' using faithfulness and relevancy metrics.

  • Claude
  • ChatGPT
  • Cursor
  • VS Code
  • Windsurf
  • Claude Code
  • JetBrains
  • Cline

Start here

Connect Ragas once, then ask.

Just follow the one-click setup to link Ragas to your client. Your credentials stay encrypted and secure on our side.

Connect Ragas to your AI

FAQ

How the Ragas connector behaves

  • 01

    Can the AI change my original dataset?

    No. The AI can only read datasets to run evaluations; it cannot modify or delete your source data.

  • 02

    Can I run evaluations on any random file?

    No. The AI can only run evaluations on datasets that are already accessible through the Ragas connection.

  • 03

    Does the AI pick the metrics for me?

    The AI can suggest metrics using the list tool, but you must tell it which ones to use for a specific run.

  • 04

    Can the AI fix my RAG pipeline automatically?

    No. The AI provides the scores and identifies issues, but you have to make the actual changes to your code or data.

  • 05

    How do I see the scores?

    You simply ask your AI to get the results from the most recent evaluation run.

  • More questions about Ragas? See everything the Ragas Connector can do

Connect Ragas to Claude, Cursor, ChatGPT & more