Bring Llm Monitoring
to LlamaIndex

Connect your New Relic AI account to any AI agent and take full control of your LLM observability, token cost tracking, and performance analytics through natural conversation.

What you can do

• LLM Telemetry Audit — Retrieve detailed LLM chat completion messages and prompt inputs directly from your agent to understand literal model behavior in real-time
• Token Cost Tracking — Execute structural extraction of model costs to calculate exact USD token consumption across your entire AI infrastructure securely
• Performance Monitoring — Extract p95 latency matrices and average response times to ensure your LLM text generation remains performant and sub-second
• User Feedback Loop — Retrieve chronological feedback messages and 1-5 rating scores dumped by human supervisors to identify quality regressions natively
• Custom NRQL Execution — Run sophisticated read-only queries using the New Relic Query Language (NRQL) to extract rich insights from multi-tenant AI datasets instantly
• Custom Event Injection — Post atomic generic telemetry rows to track internal agent states and custom behavioral markers across your observability pipeline
• Resource Discovery — Enumerate active APM apps, dashboards, and alert policies to audit your AI environment's structural health and PagerDuty configurations

How it works

1. Subscribe to this server
2. Enter your New Relic API Key and Account ID
3. Start monitoring your AI stack from Claude, Cursor, or any MCP-compatible client

Who is this for?

• AI Engineers — monitor LLM prompt performance and verify model accuracy through natural conversation without manual dashboard navigation
• Observability Leads — track global AI token costs and p95 latency benchmarks directly from your workspace to optimize infrastructure spend
• DevOps Teams — audit APM app health and verify alert policy triggers across multiple AI environments efficiently