• REGRESSION
  • USER VOICE
  • HUMAN
  • TRENDS

How to monitor model output quality with your AI

Connect New Relic AI to your agent to turn user feedback into performance insights. It watches for quality drops so you don't have to.

Ask AI about this page

Short answer

How can I track if my model's quality is dropping?

Your AI uses the query_llm_feedback tool to pull star ratings and text comments from New Relic. It identifies patterns where performance has regressed, giving you a clear view of where the model is failing users.

Where insights land.

The outcomes of your analysis.

The AI categorizes feedback into specific outcomes based on the data it finds.

REGRESSION

Detected performance drops

The AI identifies when star ratings fall below your baseline. You get a list of specific model versions or prompts that are underperforming.

USER VOICE

Direct feedback summaries

The AI pulls raw text comments from users. It summarizes the main complaints or praise so you can read the sentiment quickly.

HUMAN

Manual review needed

When feedback is ambiguous or highly complex, the AI flags it for you. You decide if the issue requires a code change or a prompt tweak.

TRENDS

Quality over time

The AI tracks how ratings change across different deployment windows. This helps you see if a new update actually improved things.

The workflow

What your AI does when the feedback arrives.

The AI handles the heavy lifting of scanning logs and ratings to find the signal in the noise.

  1. Fetch feedback data

    The AI pulls the latest user star ratings and text comments from your New Relic instance.

    query_llm_feedback

  2. Identify regressions

    It compares current ratings against previous benchmarks to find sudden drops in quality.

    query_llm_feedback

  3. Summarize complaints

    The AI groups similar user comments to show you the most common reasons for low star ratings.

    query_llm_feedback

  4. Flag critical issues

    It highlights specific feedback entries that look like major bugs or system failures.

    query_llm_feedback

Try it

Copy these to start.

Use these prompts to begin analyzing your model performance.

Starting points

These are basic starting lines. Swap out the specific project or timeframes to match your setup.

New Relic AI Connector

You're all set. Choose your MCP client and follow the setup instructions.

Connector linkhttps://edge.vinkius.com/vk_preview_oqdaAroeFoXBv9yPI4WsHZZZZZuzqhwVoSn1YCyq/mcp

Claude Desktop

Follow the steps below to connect in seconds.

  1. 1In Claude Desktop, open Settings → Connectors.
  2. 2Click “Add custom connector” and paste the connector link above as the remote MCP server URL.
  3. 3Click Add and start a new chat — New Relic AI capabilities are ready to use.
Configuration · claude_desktop_config.jsonCopy
{
  "mcpServers": {
    "new-relic-ai-llm-observability-mcp": {
      "url": "https://edge.vinkius.com/vk_preview_oqdaAroeFoXBv9yPI4WsHZZZZZuzqhwVoSn1YCyq/mcp"
    }
  }
}
Copy into chat04

  • Show me the recent star ratings for the production model in New Relic.

  • Summarize the main complaints from users who gave a 1-star rating today.

  • Did the model performance drop after the last deployment according to user feedback?

  • Find all feedback related to the 'customer support' prompt in the last 24 hours.

  • Claude
  • ChatGPT
  • Cursor
  • VS Code
  • Windsurf
  • Claude Code
  • JetBrains
  • Cline

Start here

Connect New Relic AI once, then ask.

Just enter your credentials once. Your connection is secure, and you can immediately start asking your agent about your model's quality.

Connect New Relic AI to your AI

FAQ

How this task behaves

  • 01

    Can the AI automatically fix my model prompts?

    No. The AI identifies the problems and summarizes the feedback, but you must manually update your prompts or code.

  • 02

    Does the AI delete any feedback from New Relic?

    No. The AI only reads data using the query_llm_feedback tool. It cannot modify or delete your existing records.

  • 03

    Can I use this to monitor real-time streaming data?

    The AI works by querying the feedback data available in New Relic. It analyzes what has already been recorded.

  • 04

    Will the AI notify me when a rating drops?

    The AI doesn't send proactive alerts. You need to ask your agent to check for regressions or summarize recent feedback.

  • 05

    Can the AI see my entire New Relic dashboard?

    No. The AI only accesses the specific LLM feedback data provided by the query_llm_feedback tool.

  • More questions about New Relic AI? The Connector page answers them. See everything the New Relic AI Connector can do

Connect New Relic AI to Claude, Cursor, ChatGPT & more