• DATA
  • ANALYSIS
  • INSIGHT
  • HUMAN

How to compare experiment metrics with your AI

Connect Comet ML to your agent to pull data from specific training runs. Once connected, you can ask your AI to perform side-by-side comparisons of your model's performance.

Ask AI about this page

Short answer

How can I compare experiment metrics with my AI?

Your agent uses Comet ML tools to fetch numeric values from various training runs. You can then ask it to identify trends, calculate deltas, or find the best performing run. This breakdown shows how the data moves from Comet ML to your chat.

The outcomes

What you get from these comparisons

The AI handles the data retrieval so you can focus on the results.

DATA

Direct metric extraction

The AI pulls exact numbers for loss, accuracy, or custom metrics from your runs. You get clean data without manual exports.

ANALYSIS

Run-to-run comparisons

Ask your agent to compare two specific experiments. It calculates the differences in performance metrics automatically.

INSIGHT

Trend identification

The AI looks across multiple experiments to see if specific hyperparameter changes improved your results.

HUMAN

Decision making

The AI presents the numbers and the deltas. You decide which model version is actually ready for deployment.

The workflow

What your AI does when the data arrives.

The agent acts as a bridge between your Comet ML dashboard and your chat interface.

  1. Locate experiments

    The agent identifies the specific training runs you want to analyze by looking through your project history.

    list_experiments

  2. Fetch numeric data

    Once the runs are identified, the agent pulls the specific metric values you asked for.

    get_experiment_metrics

  3. Calculate differences

    The agent compares the values between the selected runs to find improvements or regressions.

    get_experiment_metrics

  4. Summarize findings

    The agent presents a clear comparison of the metrics in your chat window.

    get_experiment_metrics

Try it

Copy these to start.

Use these prompts to begin your analysis.

Starting points

Swap out the experiment names and metric keys with your actual Comet ML data.

Comet ML Connector

You're all set. Choose your MCP client and follow the setup instructions.

Connector linkhttps://edge.vinkius.com/vk_preview_C2PxqYqwffQi3V41WE8jlPynFSP048sA8SVR1PFT/mcp

Claude Desktop

Follow the steps below to connect in seconds.

  1. 1In Claude Desktop, open Settings → Connectors.
  2. 2Click “Add custom connector” and paste the connector link above as the remote MCP server URL.
  3. 3Click Add and start a new chat — Comet ML capabilities are ready to use.
Configuration · claude_desktop_config.jsonCopy
{
  "mcpServers": {
    "comet-ml-mcp": {
      "url": "https://edge.vinkius.com/vk_preview_C2PxqYqwffQi3V41WE8jlPynFSP048sA8SVR1PFT/mcp"
    }
  }
}
Copy into chat04

  • Compare the accuracy metric between experiment 'run_v1' and 'run_v2'.

  • List all experiments in my current project and tell me which one had the lowest validation loss.

  • What was the difference in training time between my last three runs?

  • Find the experiment with the highest F1 score in my recent training history.

  • Claude
  • ChatGPT
  • Cursor
  • VS Code
  • Windsurf
  • Claude Code
  • JetBrains
  • Cline

Start here

Connect Comet ML once, then ask.

One connection links your agent to your Comet ML data. Your credentials stay encrypted and secure on our side.

Connect Comet ML to your AI

FAQ

How this task behaves

  • 01

    Can the AI change my experiment settings?

    No. The AI can only read metrics and list experiments. It cannot modify your training runs or change hyperparameters.

  • 02

    Can the AI delete old experiments?

    No. The tools only allow for reading data. Deleting experiments must be done manually in Comet ML.

  • 03

    Do I need to export CSVs first?

    No. The AI pulls the data directly from Comet ML using the provided tools.

  • 04

    Can it compare metrics from two different projects?

    Yes, as long as the agent has access to both projects through the list_experiments tool.

  • 05

    Will the AI suggest new hyperparameters?

    The AI can identify which hyperparameters led to better metrics, but it cannot execute new training runs for you.

  • More questions about Comet ML? The Connector page answers them. See everything the Comet ML Connector can do

Connect Comet ML to Claude, Cursor, ChatGPT & more