• FLAKY
  • STABLE
  • HUMAN
  • SUMMARY

How to identify flaky tests with your AI

Connect Xray to your AI client to scan your test history. Once linked, your agent can find the noise in your test suites.

Ask AI about this page

Short answer

How can I find flaky tests using AI?

Your AI looks through Xray run history to find tests that pass and fail inconsistently. It flags these unstable tests so you can fix the underlying issues. You'll get a clear list of which tests are causing noise.

The results of your scan

Where your findings land.

The AI categorizes your test data into these groups.

FLAKY

Unstable tests identified

The AI flags tests that show inconsistent pass/fail patterns. You get a list of specific test IDs that need attention.

STABLE

Reliable test suites

Tests with consistent history are grouped together. This confirms your core logic is actually working.

HUMAN

Manual review required

Tests with messy or incomplete data are sent to you. You decide if they need a rewrite or a deeper look.

SUMMARY

Run history overview

The AI provides a summary of total runs and failure rates. This gives you the context to judge how bad the flakiness is.

The workflow

What your AI does when the data arrives.

The AI analyzes trends rather than just looking at the most recent result.

  1. Fetch run history

    The AI pulls the specific execution details for the tests you want to check.

    get_individual_test_runs

  2. Spot inconsistencies

    It compares results across multiple runs to find tests that flip-flop between states.

    get_individual_test_runs

  3. Group test behavior

    The agent sorts tests into stable or flaky categories based on the historical data.

    get_individual_test_runs

  4. Deliver the list

    The AI presents the problematic test IDs so you can start fixing them.

    get_individual_test_runs

Try it

Copy these to start.

Use these prompts to begin your analysis.

Starting points

Swap out the names in quotes for your actual Xray test or project names.

Xray (Test Management) Connector

You're all set. Choose your MCP client and follow the setup instructions.

Connector linkhttps://edge.vinkius.com/vk_preview_wm1FHp5m3EdTrKq9bAVa09Uad6Q4L0EjAfSyNZ5Y/mcp

Claude Desktop

Follow the steps below to connect in seconds.

  1. 1In Claude Desktop, open Settings → Connectors.
  2. 2Click “Add custom connector” and paste the connector link above as the remote MCP server URL.
  3. 3Click Add and start a new chat — Xray (Test Management) capabilities are ready to use.
Configuration · claude_desktop_config.jsonCopy
{
  "mcpServers": {
    "xray-test-management-mcp": {
      "url": "https://edge.vinkius.com/vk_preview_wm1FHp5m3EdTrKq9bAVa09Uad6Q4L0EjAfSyNZ5Y/mcp"
    }
  }
}
Copy into chat04

  • Analyze the last 10 runs for the 'Auth Service' test in Xray and tell me if it's flaky.

  • Look through the test runs for 'Project Phoenix' and list any tests that have failed and passed inconsistently.

  • Check the execution history for the 'UI Component' suite and identify unstable behavior.

  • Summarize the stability of all tests in the 'API Integration' folder based on recent Xray data.

  • Claude
  • ChatGPT
  • Cursor
  • VS Code
  • Windsurf
  • Claude Code
  • JetBrains
  • Cline

Start here

Connect Xray once, then ask.

Link your Xray account once. Your credentials stay encrypted on our side, and you can immediately start asking your agent about your test data.

Connect Xray to your AI

FAQ

How this task behaves

  • 01

    Can the AI fix the flaky tests for me?

    No. The AI identifies the patterns and flags the tests, but you have to go into your code to actually fix the logic.

  • 02

    Does the AI change anything in Xray?

    No. The AI only reads data using the get_individual_test_runs tool. It cannot modify, delete, or create test runs in Xray.

  • 03

    How far back can the AI look?

    The AI can look back as far as your Xray history contains data for the specific tests you ask about.

  • 04

    Can I use this with Cursor or VS Code?

    Yes, as long as you are using an MCP-compatible client.

  • 05

    Will the AI suggest new tests to write?

    No. The AI focuses on analyzing existing run data to find instability in what you already have.

  • More questions about Xray? The Connector page answers them. See everything the Xray Connector can do

Connect Xray (Test Management) to Claude, Cursor, ChatGPT & more