• ENGINE
  • ASSERTION
  • ENV
  • HUMAN

How to distinguish test errors from code bugs with your AI

Connect once to give your agent access to your logs. It will then pinpoint whether you need to fix your logic or just update a test.

Ask AI about this page

Short answer

Is it a bug or a bad test?

Your agent compares the actual output in the logs against the expected state defined in your test suite. It identifies if the code logic is flawed or if the assertion itself is incorrect. You'll see exactly which side of the failure needs a fix.

Failure outcomes

Where every issue lands.

The AI categorizes failures so you know exactly where to spend your time.

ENGINE

Actual code bug

The AI finds a mismatch between the code's logic and the requirements. You get a clear path to the broken function.

ASSERTION

Faulty test logic

The agent sees that the code works but the test is checking for the wrong thing. You'll know which assertion to rewrite.

ENV

Infrastructure issue

The AI detects if a missing dependency or a broken environment caused the crash. This prevents you from hunting for bugs that aren't there.

HUMAN

Needs manual review

If the logs are too messy or the failure is non-deterministic, the AI flags it for you. You decide how to handle the ambiguity.

The workflow

What your AI does when a test fails.

The agent handles the heavy lifting of reading through stack traces and source files.

  1. Log ingestion

    Your agent pulls the raw failure logs and the relevant source files to understand the context.

    diagnose_test_failure

  2. Pattern matching

    The agent compares the error message against known patterns of faulty assertions and engine defects.

    diagnose_test_failure

  3. Logic verification

    It traces the execution path to see if the code actually deviated from the intended logic.

    diagnose_test_failure

  4. Verdict delivery

    The AI presents a final determination of whether the issue is a bug, a test error, or an environmental fluke.

    diagnose_test_failure

Try it

Copy these to start.

Use these prompts to get the AI moving on a failure.

Starting points

These are just ways to kick things off. Swap out the repo names and file paths for your own.

QA Arbiter Connector

You're all set. Choose your MCP client and follow the setup instructions.

Connector linkhttps://edge.vinkius.com/vk_preview_X69TXPHsdjAfNffcyFTnx1dWNUhv3RVHL0mSZM7p/mcp

Claude Desktop

Follow the steps below to connect in seconds.

  1. 1In Claude Desktop, open Settings → Connectors.
  2. 2Click “Add custom connector” and paste the connector link above as the remote MCP server URL.
  3. 3Click Add and start a new chat — QA Arbiter capabilities are ready to use.
Configuration · claude_desktop_config.jsonCopy
{
  "mcpServers": {
    "qa-arbiter-mcp": {
      "url": "https://edge.vinkius.com/vk_preview_X69TXPHsdjAfNffcyFTnx1dWNUhv3RVHL0mSZM7p/mcp"
    }
  }
}
Copy into chat04

  • Analyze the failure in the auth-service repo and tell me if the assertion is wrong.

  • Look at the logs for the latest build in the payment-gateway repo. Is this a real bug?

  • Check the test failures in the user-api repo and find out if the environment is causing this.

  • Diagnose why the integration tests in the checkout-flow repo are failing.

  • Claude
  • ChatGPT
  • Cursor
  • VS Code
  • Windsurf
  • Claude Code
  • JetBrains
  • Cline

Start here

Connect QA Arbiter once, then ask.

Connect the MCP to your client in one click. Your credentials stay encrypted, and you can start debugging immediately.

Connect QA Arbiter to your AI

FAQ

How this task behaves

  • 01

    Can the AI fix the code for me?

    No. The AI identifies whether the issue is a bug or a bad assertion, but you have to write and commit the actual fix.

  • 02

    Does it have access to my entire codebase?

    It only reads the files and logs necessary to diagnose the specific failure you point it toward.

  • 03

    Can it run the tests itself?

    No. This MCP is designed to analyze existing failure data, not to trigger new test runs.

  • 04

    Will it delete any of my test files?

    No. The AI only has read access to your logs and code to perform its diagnosis.

  • 05

    What if the test failure is intermittent?

    The AI will attempt to find patterns in the logs, but if it can't find a consistent cause, it will flag the issue for your manual review.

  • More questions about QA Arbiter? The Connector page answers them. See everything the QA Arbiter Connector can do

Connect QA Arbiter to Claude, Cursor, ChatGPT & more