• READ
  • WRITE
  • VERIFY
  • SYNC

How to expand evaluation datasets with your AI

Connect Braintrust to your AI client to grow your test sets through conversation. You just tell your agent what to add, and it handles the data entry.

Ask AI about this page

Short answer

How can I add new test cases to my Braintrust datasets?

You tell your AI client to add specific rows or new test cases to your existing Braintrust matrices. It uses your natural language instructions to pull current data and push new ground truth entries directly into your sets. You get a growing library of test data without manual CSV uploads.

Where your data lands.

The outcomes of your request.

The AI manages the movement of data between your chat and your Braintrust environment.

READ

Current dataset state

The AI pulls your existing evaluation rows so it knows the current context. This ensures new entries match your existing schema.

WRITE

New test cases

Your agent inserts new rows containing inputs and expected outputs. This expands your testing coverage instantly.

VERIFY

Human review needed

You decide if a generated test case is actually good. The AI waits for your signal before finalizing sensitive ground truth updates.

SYNC

Updated matrices

Your evaluation sets stay current with your latest model iterations. You'll find the new data ready for your next eval run.

The workflow

What your AI does when the request arrives.

The AI acts as a bridge between your conversation and the Braintrust API.

  1. Fetch existing data

    The agent looks up your current evaluation set to understand the existing inputs and expected outputs.

    get_dataset

  2. Prepare new rows

    The AI formats your natural language request into a structured row that fits your dataset's requirements.

    get_dataset

  3. Insert test cases

    The agent pushes the new ground truth data into your Braintrust matrix.

    insert_dataset_row

  4. Confirm updates

    The AI verifies the new row was successfully added to the dataset.

    insert_dataset_row

Try it

Copy these to start.

Use these prompts to begin interacting with your Braintrust data.

Starting points

Replace the bracketed text with your actual Braintrust dataset names or specific test case details.

Braintrust Connector

You're all set. Choose your MCP client and follow the setup instructions.

Connector linkhttps://edge.vinkius.com/vk_preview_cr8T7CpPGy155EtBgklGtlDgFsTWLRLYfzKoHIOF/mcp

Claude Desktop

Follow the steps below to connect in seconds.

  1. 1In Claude Desktop, open Settings → Connectors.
  2. 2Click “Add custom connector” and paste the connector link above as the remote MCP server URL.
  3. 3Click Add and start a new chat — Braintrust capabilities are ready to use.
Configuration · claude_desktop_config.jsonCopy
{
  "mcpServers": {
    "braintrust-mcp": {
      "url": "https://edge.vinkius.com/vk_preview_cr8T7CpPGy155EtBgklGtlDgFsTWLRLYfzKoHIOF/mcp"
    }
  }
}
Copy into chat04

  • Show me the current rows in my 'customer_support_eval' dataset.

  • Add a new test case to 'chatbot_v2' where the user asks about refund policies.

  • Look at my 'qa_test_set' and add a row with this input: 'How do I reset my password?'

  • Find the 'llm_benchmarks' dataset and add a new ground truth entry for the prompt about pricing.

  • Claude
  • ChatGPT
  • Cursor
  • VS Code
  • Windsurf
  • Claude Code
  • JetBrains
  • Cline

Start here

Connect Braintrust once, then ask.

Just link your account once. Your credentials stay encrypted, and you can immediately start managing datasets through your AI client.

Connect Braintrust to your AI

FAQ

How this task behaves.

  • 01

    Can the AI delete rows from my dataset?

    No. The AI only has the ability to retrieve data and insert new rows. It cannot delete or modify existing entries.

  • 02

    Does the AI automatically decide which test cases are correct?

    No. While the AI can suggest ground truth data, you remain in control of what is actually committed to your evaluation sets.

  • 03

    Can I use this to run an evaluation?

    No. This specific connector is for managing the data within your datasets, not for executing the evaluation runs themselves.

  • 04

    Do I need to format my data as a CSV first?

    No. You can simply describe the new test case in plain English, and the AI will handle the formatting for the dataset.

  • 05

    Can the AI update an existing row if I find an error?

    No. The current tools only allow for inserting new rows. To fix an error, you would need to add a corrected version or handle it in the Braintrust UI.

  • More questions about Braintrust? See everything the Braintrust Connector can do

Connect Braintrust to Claude, Cursor, ChatGPT & more