• SUMMARY
  • DRILLDOWN
  • FEEDBACK
  • HUMAN

How to monitor project health metrics with your AI

Connect LangSmith to your agent to turn raw trace data into actionable performance summaries.

Ask AI about this page

Short answer

How can I track my LangSmith project metrics using AI?

Your agent uses LangSmith to fetch run counts, latency, and feedback scores across your specific projects. You get a consolidated view of how your LLM applications are actually performing in production.

Where your data lands.

Performance outcomes.

The AI processes your LangSmith traces into these specific outcomes.

SUMMARY

Aggregated health reports

The agent pulls data from multiple projects to show you high-level trends in latency and error rates.

DRILLDOWN

Specific trace analysis

You can ask for details on specific runs or high-latency traces to find exactly where a chain is slowing down.

FEEDBACK

User sentiment tracking

The AI gathers feedback scores to show you how users are rating your model outputs over time.

HUMAN

Manual intervention needed

When the AI identifies a sudden spike in errors or a drop in scores, it flags the specific project for you to review manually.

The workflow

What your AI does when the data arrives.

The agent moves from broad project discovery to specific metric analysis.

  1. Identify projects

    The agent scans your LangSmith account to find all active projects you want to monitor.

    langsmith_list_projects

  2. Gather metrics

    The agent looks at the run data within those projects to calculate averages and totals.

    langsmith_list_projects

  3. Analyze performance

    The agent compares latency and run counts to find outliers or unusual patterns.

    langsmith_list_projects

  4. Report findings

    The agent presents the final health status and flags any projects that need your attention.

    langsmith_list_projects

Try it

Copy these to start.

Use these prompts to begin monitoring your projects.

Starting points

Swap out the project names for your actual LangSmith project titles.

LangSmith Connector

You're all set. Choose your MCP client and follow the setup instructions.

Connector linkhttps://edge.vinkius.com/vk_preview_JBueIzhraYvzf2S9aR0UP9Q6bjldfIBgmwxTvRHt/mcp

Claude Desktop

Follow the steps below to connect in seconds.

  1. 1In Claude Desktop, open Settings → Connectors.
  2. 2Click “Add custom connector” and paste the connector link above as the remote MCP server URL.
  3. 3Click Add and start a new chat — LangSmith capabilities are ready to use.
Configuration · claude_desktop_config.jsonCopy
{
  "mcpServers": {
    "langsmith-mcp": {
      "url": "https://edge.vinkius.com/vk_preview_JBueIzhraYvzf2S9aR0UP9Q6bjldfIBgmwxTvRHt/mcp"
    }
  }
}
Copy into chat04

  • List all my LangSmith projects and show me the total run count for each.

  • Which of my LangSmith projects has the highest average latency right now?

  • Summarize the feedback scores for the 'Customer-Bot' project from the last 24 hours.

  • Compare the error rates between my 'Production-App' and 'Staging-App' projects.

  • Claude
  • ChatGPT
  • Cursor
  • VS Code
  • Windsurf
  • Claude Code
  • JetBrains
  • Cline

Start here

Connect LangSmith once, then ask.

Just link your account once. Your credentials stay encrypted, and you can immediately start asking your agent about your project metrics.

Connect LangSmith to your AI

FAQ

How this task behaves.

  • 01

    Can the AI change my LangSmith project settings?

    No. The AI can only read project data and metrics; it cannot modify your LangSmith configurations or settings.

  • 02

    Can the AI delete traces or projects?

    No. This connector is read-only. Your traces and project structures remain untouched.

  • 03

    Does the AI see all my data or just specific projects?

    The AI can see all projects it has permission to list, but it only analyzes the specific ones you ask about.

  • 04

    How does the AI know which project to check?

    It uses the project names you provide in your prompt to find the right data in LangSmith.

  • 05

    Can I use this to trigger automated alerts?

    The AI can identify issues and tell you about them, but you must be the one to take action or set up external alerting systems.

  • More questions about LangSmith? The Connector page answers them. See everything the LangSmith Connector can do

Connect LangSmith to Claude, Cursor, ChatGPT & more