• ANALYSIS
  • DATA
  • HUMAN
  • LOGS

How to test prompt regressions with your AI

Connect Braintrust to your agent to run side-by-side comparisons. You'll catch subtle quality drops by checking new iterations against your established baselines.

Ask AI about this page

Short answer

How can I detect if a new prompt is worse than my old one?

Your agent pulls existing experiment traces and datasets to run a head-to-head comparison. This reveals exactly where a new prompt version fails compared to your baseline. You'll see the specific performance gaps immediately.

The outcomes.

Where your results land.

The AI handles the heavy lifting of data retrieval and comparison.

ANALYSIS

Regression detection

The AI identifies where new prompt outputs deviate from your baseline traces. You get a clear view of performance shifts.

DATA

Dataset alignment

The agent pulls the exact test sets you need to validate changes. It ensures you're testing against the right inputs.

HUMAN

Manual review

You decide which specific failures require a manual fix. The AI flags the issues, but you make the final call on prompt logic.

LOGS

Experiment history

The agent looks back through previous runs to find your gold standard. This keeps your testing grounded in real past performance.

The workflow

What your AI does when the request arrives.

The agent acts as a bridge between your chat interface and your Braintrust experiments.

  1. Identify the baseline

    The agent looks up your previous successful runs to establish what a good response looks like.

    list_experiments

  2. Fetch the prompt

    It pulls the specific version of the prompt you want to test or compare.

    get_prompt

  3. Load test data

    The agent grabs the specific input sets used for your evaluation.

    get_dataset

  4. Run comparison

    The agent compares the new outputs against the baseline traces to find regressions.

    get_prompt

Try it

Copy these to start.

Use these prompts to kick off your testing immediately.

Starting points

These are starting points. Swap out the experiment names and dataset IDs for your actual Braintrust identifiers.

Braintrust Connector

You're all set. Choose your MCP client and follow the setup instructions.

Connector linkhttps://edge.vinkius.com/vk_preview_cr8T7CpPGy155EtBgklGtlDgFsTWLRLYfzKoHIOF/mcp

Claude Desktop

Follow the steps below to connect in seconds.

  1. 1In Claude Desktop, open Settings → Connectors.
  2. 2Click “Add custom connector” and paste the connector link above as the remote MCP server URL.
  3. 3Click Add and start a new chat — Braintrust capabilities are ready to use.
Configuration · claude_desktop_config.jsonCopy
{
  "mcpServers": {
    "braintrust-mcp": {
      "url": "https://edge.vinkius.com/vk_preview_cr8T7CpPGy155EtBgklGtlDgFsTWLRLYfzKoHIOF/mcp"
    }
  }
}
Copy into chat04

  • Compare the prompt from experiment 'v2-final' against my baseline 'v1-stable'.

  • Run a regression test using the 'customer-support-test' dataset.

  • Check if the new prompt version in 'agent-refactor-04' has dropped in quality compared to the previous run.

  • Pull the data from the 'qa-eval-set' and compare it to my latest prompt version.

  • Claude
  • ChatGPT
  • Cursor
  • VS Code
  • Windsurf
  • Claude Code
  • JetBrains
  • Cline

Start here

Connect Braintrust once, then ask.

Just link your Braintrust account. Your credentials stay encrypted and secure, and you can start testing right away in your preferred AI client.

Connect Braintrust to your AI

FAQ

How this task behaves.

  • 01

    Can the AI change my prompts in Braintrust?

    No. The AI can only read prompts and experiment data. You must manually edit or save new versions in Braintrust yourself.

  • 02

    Will the AI delete any of my experiment traces?

    No. The tools only allow for reading and retrieving data. Your existing experiment history is safe.

  • 03

    Can I use this to run new experiments automatically?

    The AI can pull the data and compare results, but it doesn't trigger the actual execution of new large-scale experiment runs.

  • 04

    Do I need to provide the experiment IDs every time?

    It's easier if you do. While the AI can list experiments to find them, being specific helps avoid confusion.

  • 05

    Can the AI fix a regression for me?

    No. The AI identifies where the performance dropped, but you are responsible for rewriting the prompt to fix the issue.

  • More questions about Braintrust? See everything the Braintrust Connector can do

Connect Braintrust to Claude, Cursor, ChatGPT & more