# Braintrust MCP for AI Agents AI Agent Connect > Braintrust MCP helps you manage AI evaluations and prompt engineering directly through your agent. It lets you organize projects, run experiments, and query ground truth datasets without switching tabs. Use it to track model performance and manage prompt versions in one place. ## Overview - **Category:** brain-trust - **Price:** Free - **Endpoint:** https://edge.vinkius.com/vk_preview_cr8T7CpPGy155EtBgklGtlDgFsTWLRLYfzKoHIOF/ai-agent-connect - **Tags:** ai-evaluation, llm-benchmarking, prompt-engineering, model-testing, ai-observability, data-analytics ## Description This Connector connects your AI agent to Braintrust, giving it the ability to handle the heavy lifting of model evaluation and prompt management. Instead of manually digging through dashboards to see how your LLM is performing, you can ask your agent to pull specific metrics or run a new test suite on the fly. It handles the logistical side of AI development, like creating new projects for specific tasks or grabbing the exact text of a frozen prompt. When you're testing new iterations, the agent can insert new test cases into your datasets or list your existing experiments to see where things are breaking. It's a way to keep your evaluation logic consistent across your entire development cycle. Because Vinkius makes it easy to connect these tools, you can move from a high-level idea to a scored experiment without leaving your chat interface. It turns your agent into a capable member of your ML team that knows exactly where your data lives and how your prompts are versioned. ## Tools ### list_env_vars Check the Braintrust AI Gateway configurations for your model API keys. It's a quick way to probe your current setup. ### list_experiments Get a list of all evaluation experiments and their associated scores. Use this to monitor your testing progress. ### get_dataset Pull a specific dataset that contains the schemas for your LLM outputs. This lets you see exactly what data is being used. ### get_prompt Retrieve the literal text and variable contexts for a specific prompt. This is great for auditing your current instructions. ### insert_dataset_row Add new test cases into a dataset matrix for specific evaluations. It keeps your test data fresh without manual entry. ### list_projects See every AI evaluation project currently active in your Braintrust account. Use this to get a high level view of your work. ### list_prompts Find all version-controlled system prompts stored in Braintrust. This helps you manage multiple versions of your instructions. ### create_experiment Start a new historical trace to record your LLM pipeline tests. It helps you see how different iterations perform over time. ### create_project Set up a new project environment to track your AI evaluations and datasets. Use this to keep different tasks isolated. ### list_datasets See all isolated Ground Truth text banks used for automated scoring. This helps you find the right data quickly. ## Prompt Examples **Prompt:** ``` List all active test datasets configured under Braintrust. ``` **Response:** ``` I've fetched your Ground Truth repositories. There's 1 dataset active under ID 4a83b9c named 'Support-Responses-Testing'. Should I list the rows nested there? ``` **Prompt:** ``` Look up prompt template using specific ID XYZ. ``` **Response:** ``` Prompt XYZ returns successfully. It tracks specific {{user}} tags targeting strict instructions enforcing a professional tone. The JSON mapping version is 1.0.4. Do you need further metadata? ``` **Prompt:** ``` Analyze recent experiments across multiple models testing behavior. ``` **Response:** ``` Extracted the historical trace boundaries. Experiment run ID V3 generated a 94% alignment score compared to the previously logged V2 base structure matrix mapping differences on false positives. ``` ## Capabilities ### Create evaluation projects Set up new project environments to organize your AI testing and datasets. ### Run model experiments Execute and track historical traces to see how model changes affect performance. ### Query ground truth sets Pull specific datasets that contain the schemas and data for your LLM outputs. ### Retrieve versioned prompts Grab the exact text and variable contexts for your system prompts. ### Update test matrices Add new test cases directly into your existing datasets for automated scoring. ### Check gateway configs Probe your Braintrust AI Gateway settings to manage model API keys securely. ## Use Cases ### Testing a new model An engineer asks the agent to run a new experiment and compare it against the last version to see if accuracy dropped. ### Updating a prompt A product manager wants to see the current text of a prompt and asks the agent to pull it to check the tone. ### Expanding a dataset A data scientist tells the agent to add 50 new rows to a specific dataset for a new edge case. ### Project Audit A lead dev asks the agent to list all active projects to see which ones have finished evaluations. ## Benefits - Stop manual data entry by using insert_dataset_row to add test cases directly through your agent. - Keep your production prompts safe by using get_prompt to pull frozen versions instead of copying code. - Track every change with create_experiment to see how new model iterations affect your scores. - Organize your work better by using create_project to group related datasets and evaluations. - Access your data faster by using list_datasets to see all your ground truth banks at once. - Audit your security by using list_env_vars to check your AI Gateway configurations. ## How It Works The bottom line is that your agent becomes a direct interface for your Braintrust evaluation pipeline. 1. Add the Braintrust MCP to your AI client and provide your personal API ID. 2. Connect your Braintrust account to grant the agent access to your projects and datasets. 3. Ask your agent to run evaluations, pull prompt versions, or update your test matrices. ## Frequently Asked Questions **How does the Braintrust MCP help with my prompt engineering?** It lets you pull the exact, version-controlled text of your prompts directly into your chat. You can check instructions and variable contexts without digging through your code or documentation. **Can I use Braintrust MCP to update my test data?** Yes, you can ask your agent to insert new rows into your datasets. This makes it easy to add new test cases to your evaluation matrices on the fly. **What is the Braintrust MCP for model evaluation?** It connects your agent to your evaluation pipeline so you can run experiments, track scores, and see regressions across different models in one place. **Will Braintrust MCP help me manage my AI projects?** It allows your agent to list all active projects and create new ones. This keeps your different AI testing tasks organized and easy to find. **Can the Braintrust MCP see my model API keys?** The Connector can probe your AI Gateway configurations to check your settings. This helps you manage your model API keys and configurations securely. **Does Braintrust MCP support Ground Truth datasets?** Yes, it can query your specific Ground Truth sets to help your agent understand the schemas and data needed for accurate scoring. **Can I insert new test data dynamically tracking specific limits?** Yes. Utilizing the `insert_dataset_row` method, you can effortlessly inject exact JSON tracking payload mapping strings directly inside the text corpus evaluating the final results. **Does it pull out original Prompt definitions stored securely?** Certainly. The `get_prompt` command isolates and returns perfectly version-controlled bounding parameters slicing literal templates natively hosted under the Braintrust database. **How deeply can it inspect test regressions or scoring limits?** Using the robust `list_experiments` call, you can branch full arrays separating LLM version behaviors over massive iterations tracking the performance anomalies accurately.