# Ragas MCP for AI Agents AI Agent Connect > Ragas lets you run professional RAG evaluations and track experiment metrics directly from your AI agent's chat. It handles dataset management, automated scoring for faithfulness and relevancy, and project organization so you can see exactly how your RAG pipeline is performing without leaving your workflow. ## Overview - **Category:** ai-frontier - **Price:** Free - **Endpoint:** https://edge.vinkius.com/vk_preview_mQUV7klN4rh09rRcp9ld3cwceCP25oL5Lk8res4B/ai-agent-connect - **Tags:** rag, llm-evaluation, metrics, dataset-management, model-performance, experiment-tracking ## Description Ragas lets you handle the heavy lifting of model evaluation right where you're already working. Instead of jumping back and forth between your RAG pipeline and a separate dashboard, this Connector lets you treat your evaluation data like a first-class citizen. You can have your agent look at your datasets, run a batch of tests to see how well your system is answering questions, and then immediately pull the metrics to see if your latest change actually helped or just made things worse. It takes the friction out of the iterative loop of improving a RAG system. You'll spend less time copy-pasting scores into a spreadsheet and more time actually tuning your retrieval logic. By connecting this to the Vinkius catalog, you get a centralized way to manage these complex ML workflows alongside your other tools. It's about turning a manual, multi-tab process into a conversation with your agent. You can quickly check if your model is drifting, verify that your chunking strategy is actually improving answer relevancy, and keep a clean history of every test you run. This means you can move from idea to production faster because you have a reliable way to measure success at every step of the development cycle. You get to stop guessing and start making data-driven decisions about your model's behavior. ## Tools ### list_datasets See every evaluation dataset you've uploaded. This helps you stay organized when managing multiple test sets for different projects. ### get_results See the final scores from a finished experiment. This gives you the hard numbers on your model's performance to help you make data-driven decisions. ### get_dataset Get the specific details for one dataset. Use this to check the contents of a particular evaluation group or see its metadata. ### list_experiments See all the tests associated with a specific dataset. This is the fastest way to find previous runs and compare results. ### get_experiment Pull up the full details for a single experiment. It's the best way to deep-dive into a specific test run and see the raw data. ### run_evaluation Start a new test run on a dataset. This triggers the scoring for your RAG pipeline so you can see how your model performs on real queries. ### list_metrics See every available scoring metric you can use. Use this to check what's available for your specific needs like faithfulness or relevancy. ## Prompt Examples **Prompt:** ``` Run an evaluation on the 'Legal_Q1_Test' dataset. ``` **Response:** ``` I've started the evaluation for 'Legal_Q1_Test'. You'll see the scores for faithfulness, relevancy, and other key metrics updated in your dashboard shortly. ``` **Prompt:** ``` What were the results for the 'Support Bot V3' experiment? ``` **Response:** ``` Here are the results for the 'Support Bot V3' test: - **Aggregate Score:** 0.89 - **Faithfulness:** 0.92 - **Answer Relevance:** 0.85 It looks like the model is very faithful to the source material, though the relevance could use a bit more work. ``` **Prompt:** ``` List all the datasets I have in my Ragas project. ``` **Response:** ``` I found 3 datasets in your project: - Legal_Q1_Test (ID: 01) - Medical_V2_Base (ID: 02) - General_FAQ_Validation (ID: 03) ``` ## Capabilities ### Trigger a new evaluation run Start a test on a specific dataset to get scores on faithfulness and relevancy. ### Fetch scores for specific datasets Pull the latest performance numbers for any dataset in your project. ### List all available experiments See a history of every test run associated with your evaluation data. ### View detailed metrics for a specific test Get a deep dive into the specific scores of a completed experiment. ### Organize evaluation data into projects Group your datasets and tests into specific projects for better organization. ### See all available scoring metrics Check the full range of metrics you can use to measure model performance. ### Retrieve results from a finished run Grab the final aggregate scores from a previous experiment instantly. ## Use Cases ### Checking for hallucinations A QA specialist notices a model is making things up. They ask the agent to run an evaluation on the 'Medical_V2' dataset to see the exact faithfulness score. ### Comparing chunking strategies A dev wants to see if smaller chunks help. They run evaluations on two different datasets and use get_results to compare the answer relevancy. ### Organizing test data A team has 50 different test sets. They use list_datasets and project organization to keep everything grouped by department. ### Quick metric lookups An engineer wants to know how the last test went. They ask the agent to fetch the results for 'Support Bot V3' to see the aggregate score. ## Benefits - Stop manual spreadsheet tracking. Use run_evaluation and get_results to see your scores instantly in your chat window. - Compare model versions faster. Use list_experiments to see how your changes affect your RAG pipeline over time. - Organize complex test data. Use list_datasets to keep your evaluation sets sorted by project or use case. - Standardize your QA process. Use list_metrics to ensure everyone on your team is measuring the same performance indicators. - Reduce context switching. You can initiate evaluations and query historical data without ever leaving your IDE or chat client. ## How It Works The bottom line is you get a way to audit your RAG performance in real-time without switching contexts. 1. Provide your Ragas Application URL and your generated Application Token. 2. Tell your agent which dataset or project you want to focus on. 3. Get back detailed scores on faithfulness, relevancy, and other metrics. ## Frequently Asked Questions **What is the Ragas MCP used for?** Ragas is used to evaluate RAG systems. It helps you measure how well your AI answers questions based on your data using standard metrics. **How does Ragas help with LLM hallucinations?** It provides specific metrics like faithfulness to help you identify and reduce hallucinations in your RAG pipeline. **Can I use Ragas to track my RAG experiments?** Yes, you can use it to list and view all your previous test runs to see how your model's performance changes over time. **Does Ragas support specific metrics like faithfulness?** Yes, it supports a variety of metrics including faithfulness and answer relevancy to give you a complete picture of performance. **How do I organize my RAG test data with Ragas?** You can organize your evaluation data into specific projects within the Connector to keep your different test sets neatly categorized. **Can my AI agent run evaluations automatically?** Yes, your AI agent can trigger new evaluation runs on your datasets directly from your chat or IDE. **How do I secure an App Token for Ragas?** Log into your provided Ragas dashboard. In your project's settings or dedicated security section, you will find the ability to generate a new Application Token. Copy it immediately, as it may only appear once. **What format is required to upload a dataset?** The tool uses common array formats through the Connector wrapper. When passing data, the AI maps arrays containing `question`, `ground_truth` and `contexts` natively matching Ragas base requirements. **Does the server evaluate prompts automatically during testing?** Yes. When triggering evaluations, Ragas uses its own sophisticated metrics (like Faithfulness, Answer Relevance) running internally. The Connector simply pipes these generated reports back to your chat.