How to Use the LangSmith (LLM Observability & Hub) MCP in Pydantic AI

Q: How do I configure LangSmith (LLM Observability & Hub) with Pydantic AI?

Install pydantic-ai-slimmcp. Use the unified MCPToolset approach with your HTTP URL and pass it to your Agent's toolsets array. Do not use the deprecated MCPServerHTTP class.

Q: Does Pydantic AI validate the responses from LangSmith (LLM Observability & Hub)?

Yes. Every payload returned by tools like getrun or listdatasets is validated against Pydantic models at runtime. If LangSmith returns an unexpected schema, the agent fails loudly.

Connect LangSmith (LLM Observability & Hub) to Pydantic AI to validate raw LLM traces and prompt templates with strict runtime type checks.

See Vinkius in Action

Works with every AI agent you already use

…and any MCP-compatible client

MCP Servers - Free for Subscribers

Connect LangSmith (LLM Observability & Hub) MCP to Pydantic AI

Create your Vinkius account to connect LangSmith (LLM Observability & Hub) to Pydantic AI and route execution through our secure gateway. The platform manages server hosting, runtime updates, and security layers. Configuration requires no manual server provisioning.

GDPR Free for Subscribers

Setup LangSmith (LLM Observability & Hub) with Pydantic AI

Ask AI about this MCP

ChatGPT

Claude

Perplexity

Validate prompt templates

The `list_prompts` tool extracts prompt templates hosted in the LangChain Hub directly into your Pydantic AI agent. Because this framework enforces strict typing, your agent validates the template structure immediately, failing loudly if the expected variables are missing. The agent also relies on `list_projects` to map out the boundaries of distinct AI pipelines currently monitored by LangSmith. It verifies that the prompts belong to the correct project environment before executing any further logic.

Type-check LLM telemetry via MCP Server

The `get_run` tool retrieves the precise telemetry for a single LLM invocation run. Pydantic AI parses this telemetry against your custom data models, guaranteeing that the trace contains the exact latency and token metrics you require. To analyze broader trends, the agent calls `list_runs`. This tool isolates the raw interactions containing prompts sent to and responses received from the AI models. If the API returns unexpected data structures, the framework throws a validation error instantly.

Audit evaluation datasets

The `list_datasets` tool fetches all evaluation and fine-tuning datasets mapped in LangSmith. Your agent pulls these records to run automated, type-safe checks against your baseline data. For manual review workflows, the agent uses `list_annotation_queues`. It lists active human-in-the-loop annotation queues, allowing your system to safely route pending reviews to your development team without silent data corruption.

Setup guide

Set up LangSmith (LLM Observability & Hub) MCP in Pydantic AI

Prerequisites

Python 3.10+ installed
pydantic-ai-slim[fastmcp] package
Active Vinkius subscription with a valid endpoint token

1

Install Pydantic AI with FastMCP
Run pip install "pydantic-ai-slim[fastmcp]". The FastMCP toolset replaces the deprecated MCPServerHTTP class with full protocol support.
2

Configure the FastMCPToolset
Pass a JSON-style config dict to FastMCPToolset with your Vinkius URL. Replace [YOUR_TOKEN_HERE] with your token from cloud.vinkius.com. Supports Streamable HTTP, SSE, and Stdio transports.
3

Create and run your agent
Pass the toolset to Agent(toolsets=[toolset]) and call agent.run(). Swap openai:gpt-4o for any supported model — Anthropic, Google, Mistral, or Groq.

agent.py

from pydantic_ai import Agent
from pydantic_ai.toolsets.fastmcp import FastMCPToolset

toolset = FastMCPToolset({
    "mcpServers": {
        "langsmith-llm-observability-hub-mcp": {
            "url": "https://edge.vinkius.com/[YOUR_TOKEN_HERE]/mcp"
        }
    }
})

agent = Agent(
    "openai:gpt-4o",
    toolsets=[toolset],
    system_prompt="You have access to LangSmith (LLM Observability & Hub) tools.",
)

result = await agent.run("List recent LangSmith (LLM Observability & Hub) transactions")
print(result.output)

Get your connection token →

Independent Platform Disclaimer: Vinkius is an independent platform and is not affiliated with, endorsed by, sponsored by, verified by, or otherwise authorized by LangSmith. All third-party trademarks, logos, and brand names are the property of their respective owners. Their use on this website is strictly for informational purposes to identify service compatibility and interoperability.

Why Choose Vinkius

Vinkius connects your tools to AI with real-time monitoring and automatic cost savings — all from one dashboard.

Connect LangSmith (LLM Observability & Hub) now

Real-time monitoring

Live

visibility into every interaction

Connect your favorite tools to your AI and see exactly what's happening — every request, every response, in real time.

Built-in savings

60%

lower AI costs

Vinkius compresses data between your apps and your AI automatically. Lower bills every month — no configuration required.

Single dashboard

One

place for every integration

Every tool your AI connects to, managed from a single screen. One account, complete control.

Common questions about LangSmith (LLM Observability & Hub) MCP in Pydantic AI

Install pydantic-ai-slim[mcp]. Use the unified MCPToolset approach with your HTTP URL and pass it to your Agent's toolsets array. Do not use the deprecated MCPServerHTTP class.

Yes. Every payload returned by tools like get_run or list_datasets is validated against Pydantic models at runtime. If LangSmith returns an unexpected schema, the agent fails loudly.

You can. Pydantic AI is model-agnostic. You can run a local model that connects to this server to audit your LangSmith traces and prompt templates without relying on a cloud LLM provider.

Yes. Pydantic AI's MCPToolset supports both Streamable HTTP and SSE transports out of the box. Just provide the Vinkius MCP endpoint URL and the framework handles the connection.

This MCP server handles sensitive evaluation and fine-tuning datasets mapped in LangSmith. Vinkius routes the traffic through a zero-trust, ephemeral V8 Isolate Sandbox. Authentication is managed server-side, meaning your raw API keys never touch the Pydantic AI runtime environment.

Use it with your favorite AI tools

Connect this server to Cursor, Claude, VS Code, and more.

OpenAI Agents SDK sdk-python

Google ADK sdk-python

Pydantic AI sdk-python

Vercel AI SDK sdk-typescript