# LangSmith MCP for AI Agents AI Agent Connect > LangSmith MCP for LLM observability. Track traces, audit prompt templates, and manage evaluation datasets directly from your AI client. Stop jumping between tabs to see why your agent is hallucinating or how many tokens you're burning. ## Overview - **Category:** superpower - **Price:** Free - **Endpoint:** https://edge.vinkius.com/vk_preview_ixIJrwVWDWlC1bEOf4IyffAPgAEw302Ns6BE1oLo/ai-agent-connect - **Tags:** llm-observability, tracing, prompt-management, evaluation, ai-debugging, llm-ops ## Description Managing LLM infrastructure usually means constant tab switching between your production environment and a complex dashboard. This Connector changes that by bringing your LangSmith data directly into your chat interface. You can ask your agent to pull up specific trace logs, check the status of your annotation queues, or grab the latest prompt templates from the Hub without leaving your workspace. It's about having a clear view of your model's performance and reasoning paths while you're actually building. By using this Connector through the Vinkius catalog, you get a way to audit human feedback loops and inspect multi-turn agentic workflows in real time. You'll see exactly where a chain broke or why a specific tool call failed, making it way easier to iterate on your prompts and datasets. You can move from high-level project overviews to granular run telemetry in a single conversation, giving you a much faster way to debug your production pipelines. ## Tools ### list_annotation_queues Lists active human-in-the-loop annotation queues. Use this to monitor how many reviews are pending. ### list_runs Lists specific LLM invocation runs within a selected project. It helps you isolate the raw interactions you need to see. ### get_run Pulls precise telemetry for a single LLM invocation run. Use this to see exact token counts and error strings. ### list_datasets Lists all evaluation and fine-tuning datasets mapped in LangSmith. This helps you verify your 'golden' data. ### list_prompts Extracts prompt templates hosted in the LangChain Hub. Use this to pull the latest instructions into your workflow. ### list_projects Shows all active LangSmith tracing projects and sessions. Use this to get an overview of your monitored pipelines. ## Prompt Examples **Prompt:** ``` Show me the logs for the 'Production-Bot-V2' project. ``` **Response:** ``` I've located the **Production-Bot-V2** project. Here are the recent runs: | Run ID | Status | Latency | Tokens | | :--- | :--- | :--- | :--- | | run-8821 | Success | 1.2s | 450 | | run-8822 | Failed | 0.8s | 120 | | run-8823 | Success | 2.1s | 890 | Would you like me to pull the telemetry for the failed run? ``` **Prompt:** ``` What was the token count for run-98765? ``` **Response:** ``` Retrieving data for **run-98765**... **Execution Summary:** - **Status:** Success - **Total Tokens:** 1,420 - **Prompt Tokens:** 850 - **Completion Tokens:** 570 - **Latency:** 2.4s The agent successfully used the 'Web-Search' tool before generating the final answer. ``` **Prompt:** ``` List some prompts from the Hub for summarization. ``` **Response:** ``` I found 3 prompts in the LangChain Hub related to summarization: * **summarization-agent-v1**: High-accuracy summary for news articles. * **data-extraction-helper**: Extracts key entities from long-form text. * **customer-service-v3**: Summarizes support tickets for agent review. I can retrieve the full instruction text or version history for any of these. ``` ## Capabilities ### Pull trace logs from specific projects Retrieve execution logs for specific LLM runs to see exactly what happened. ### Check token usage and latency metrics Get precise telemetry on token consumption and prompt latency for any run. ### Retrieve prompt templates from the Hub Access and pull managed prompt templates and version histories from the LangChain Hub. ### View evaluation and fine-tuning datasets Enumerate the curated datasets you use for automated prompt evaluation. ### Inspect human-in-the-loop annotation queues Monitor active queues where human reviewers assess your model's output. ### Analyze nested tool calls in agentic workflows Deep-dive into multi-turn reasoning paths to understand how your agent uses tools. ## Use Cases ### Debugging a failed production bot An engineer asks the agent to list_runs for a specific project and then calls get_run on a failed ID to see the exact error string and prompt. ### Grabbing a high-performing prompt template A developer uses list_prompts to find a summarization template in the Hub and pulls the version history directly into their chat. ### Checking human feedback backlog An analyst uses list_annotation_queues to see how many prompts are waiting for human review and reports the count to the team. ### Verifying evaluation datasets A team uses list_datasets to ensure the correct 'golden' dataset is being used for the latest automated evaluation of their model. ## Benefits - Stop manual dashboard hunting by using list_projects to see all active pipelines instantly. - Get exact token costs and latency data for any run using get_run to keep your budget on track. - Pull fresh prompt templates from the Hub with list_prompts to stay updated on best practices. - Check your golden datasets with list_datasets to ensure your evaluation logic stays consistent. - Monitor human feedback loops with list_annotation_queues to see how reviewers are scoring your model. - Debug complex agent logic by inspecting nested tool calls and reasoning paths without clicking through deep menus. ## How It Works The bottom line is it puts your LLM observability data into a conversational interface so you can debug faster. 1. Subscribe to the Connector and enter your LangSmith API Key and Endpoint. 2. Connect the Connector to your preferred client like Claude, Cursor, or Windsurf. 3. Ask your agent to list active projects or pull telemetry for specific runs. ## Frequently Asked Questions **Can I see my LLM traces with the LangSmith MCP?** Yes, you can pull logs and see exactly what happened during an LLM run. This includes the prompts sent, responses received, and any intermediate steps the agent took. **How does the LangSmith MCP help with prompt management?** It lets you pull prompt templates directly from the LangChain Hub into your chat. You can check version histories and instructions without leaving your current workspace. **Can I use the LangSmith MCP to see human feedback?** Yes, you can view active human-in-the-loop annotation queues. This helps you see how many reviews are pending and how human reviewers are scoring your model's outputs. **Does the LangSmith MCP show token usage?** Yes, it pulls precise telemetry for specific runs, including total tokens, prompt tokens, and completion tokens, helping you monitor your costs. **Can I manage my datasets with the LangSmith MCP?** You can list all evaluation and fine-tuning datasets mapped in LangSmith. This makes it easy to verify which datasets are being used for your automated tests. **Can I use the LangSmith MCP to debug agent reasoning?** Yes, it allows you to deep-dive into multi-turn agentic workflows. You can see nested tool calls and the internal reasoning paths the agent used to reach its conclusion. **Can I see the token usage for a specific LLM run through my agent?** Yes. Use the `get_run_telemetry` tool with a specific Run ID. Your agent will retrieve the exact token count (prompt + completion) and latency metrics calculated by LangSmith for that interaction. **How do I fetch a prompt template from the LangChain Hub using natural language?** The `list_prompts` tool allows your agent to navigate your hosted Hub repository. You can ask your agent to find a specific prompt by name to inspect its instruction text, variables, and version history. **Can my agent check the status of human annotation queues?** Absolutely. Use the `list_annotation_queues` tool to retrieve all active queues where human feedback is being collected. Your agent can report on the number of pending traces and general alignment scores established by your reviewers.