• DATA
  • QUEUE
  • HUMAN
  • REPORT

How to audit evaluation datasets with your AI

Connect LangSmith to your AI client to monitor your testing pipeline. Your agent handles the heavy lifting of data inspection and queue management.

Ask AI about this page

Short answer

How can I audit my evaluation datasets using AI?

Your AI client pulls your existing datasets and annotation queues directly from LangSmith. It then scans for inconsistencies or low-quality labels, flagging specific entries that require manual review. You get a clear view of where your evaluation data is failing or where humans need to step in.

Audit outcomes.

Where your findings land.

The AI organizes its findings into these specific categories.

DATA

Dataset health checks

The AI uses your existing datasets to find outliers or repetitive patterns. You get a summary of the data quality.

QUEUE

Annotation status

Your agent checks the current state of your annotation queues. It identifies which tasks are stuck or pending.

HUMAN

Manual review flags

The AI identifies specific rows or entries that don't meet your quality bar. These are sent to you for human intervention.

REPORT

Audit summaries

You receive a breakdown of the dataset coverage and label distribution. This helps you decide if you need more data.

The workflow

What your AI does when the audit starts.

The AI moves through your LangSmith environment to find and flag issues.

  1. Inventory datasets

    The AI looks through your LangSmith account to find the specific datasets you want to inspect.

    list_datasets

  2. Check annotation queues

    It pulls the current status of your human-in-the-loop queues to see what's being worked on.

    list_annotation_queues

  3. Identify data gaps

    The agent compares your datasets against your evaluation goals to find missing edge cases.

    list_datasets

  4. Flag human tasks

    The AI marks specific items in your queues that need a person to verify the labels.

    list_annotation_queues

Try it

Copy these to start.

Use these prompts to kick off an audit.

Starting points

These are starting points. Swap out the names for your actual LangSmith dataset or queue names.

LangSmith Connector

You're all set. Choose your MCP client and follow the setup instructions.

Connector linkhttps://edge.vinkius.com/vk_preview_ixIJrwVWDWlC1bEOf4IyffAPgAEw302Ns6BE1oLo/mcp

Claude Desktop

Follow the steps below to connect in seconds.

  1. 1In Claude Desktop, open Settings → Connectors.
  2. 2Click “Add custom connector” and paste the connector link above as the remote MCP server URL.
  3. 3Click Add and start a new chat — LangSmith capabilities are ready to use.
Configuration · claude_desktop_config.jsonCopy
{
  "mcpServers": {
    "langsmith-llm-observability-hub-mcp": {
      "url": "https://edge.vinkius.com/vk_preview_ixIJrwVWDWlC1bEOf4IyffAPgAEw302Ns6BE1oLo/mcp"
    }
  }
}
Copy into chat04

  • List all my LangSmith datasets and tell me which one has the most recent updates.

  • Check my annotation queues and tell me which ones have the most pending items.

  • Look at my 'model_eval_v2' dataset and find any entries with conflicting labels.

  • Summarize the current progress of the 'human_review_queue' in LangSmith.

  • Claude
  • ChatGPT
  • Cursor
  • VS Code
  • Windsurf
  • Claude Code
  • JetBrains
  • Cline

Start here

Connect LangSmith once, then ask.

Just link your LangSmith account through Vinkius. Your credentials stay encrypted, and you can immediately start asking your AI about your data.

Connect LangSmith to your AI

FAQ

How this task behaves.

  • 01

    Can the AI delete datasets in LangSmith?

    No. The AI can only list and read your datasets and queues. It cannot delete or modify any data in LangSmith.

  • 02

    Can the AI assign new tasks to humans?

    No. The AI can only list existing annotation queues. It cannot create new tasks or change the assignment of human workers.

  • 03

    How does the AI know which dataset to audit?

    You tell it. You can name a specific dataset or ask it to list all of them so you can choose.

  • 04

    Does the AI see my API keys?

    No. Vinkius manages the connection securely. Your AI client only interacts with the tools provided by the MCP.

  • 05

    Can I use this to run automated evals?

    The AI can help you prepare and audit the data used for evals, but it does not trigger the actual model evaluation runs.

  • More questions about LangSmith? See everything the LangSmith Connector can do

Connect LangSmith to Claude, Cursor, ChatGPT & more