# QA Arbiter MCP for AI Agents AI Agent Connect > QA Arbiter helps your AI agent figure out why a test failed without guessing. Instead of just retrying a broken test, it forces the agent to trace the engine's logic, compare the actual results against that trace, and verify the expected values. It provides a deterministic verdict: is the test wrong, is the engine broken, or are both at fault? It stops agents from making 'fixes' that actually break things. ## Overview - **Category:** productivity - **Price:** Free - **Endpoint:** https://edge.vinkius.com/vk_preview_X69TXPHsdjAfNffcyFTnx1dWNUhv3RVHL0mSZM7p/ai-agent-connect - **Tags:** qa-automation, test-diagnostics, fault-localization, multi-agent, structured-reasoning, decision-pivots, agentic-pipeline, test-verification ## Description QA Arbiter helps your agent figure out why a test failed without guessing. Instead of just retrying a broken test, it forces the agent to trace the engine's logic, compare the actual results against that trace, and verify the expected values. It provides a deterministic verdict: is the test wrong, is the engine broken, or are both at fault? It stops agents from making 'fixes' that actually break things. When a QA agent hits a failing test, it usually starts guessing. It might rewrite the code to match a bad test, or it might keep retrying a flaky test until the pipeline stalls. This tool stops this cycle by forcing the agent to act like a methodical debugger. It uses a decision pivot pattern where the agent must prove its reasoning before it's allowed to suggest a fix. It has to trace the actual function execution, see what the engine produced, and then check if that matches what the test expected. If the engine did what it was supposed to do but the test expected something else, the agent knows the test is the problem. If the engine produced garbage, the agent knows there's a bug. It prevents the common mistake of 'fixing' a bug by just changing the test to match the broken behavior. This is a huge deal for teams running multi-agent pipelines where one agent writes the code and another writes the tests. Without a systematic way to separate these two, the agents can get stuck in a loop where they just keep passing the blame back and forth. By forcing a trace, you get a clear audit trail of the logic. You can find this and other high-quality tools in the Vinkius catalog to keep your agentic workflows from getting stuck in logic loops. ## Tools ### diagnose_test_failure QA Arbiter forces your agent to provide a step-by-step trace of an engine function to determine if a test failure is a code bug or a bad assertion. ## Prompt Examples **Prompt:** ``` I have a test that expects '05:25' but got '04:45'. The math is wake_time minus (cycles * 90 + 15). For 2 cycles from 08:00, what's wrong? ``` **Response:** ``` **Verdict: TEST_ERROR** **Trace:** * 2 cycles × 90 + 15 = 195 minutes * 08:00 (480 min) - 195 min = 285 min * 285 min = **04:45** The engine correctly returns '04:45'. Your expected value '05:25' is wrong — fix it to '04:45'. ``` **Prompt:** ``` The test expects '22:45' for 6 cycles from 08:00, but the engine returns '-02:-15'. Is this a bug? ``` **Response:** ``` **Verdict: ENGINE_DEFECT** **Trace:** * 6 cycles × 90 + 15 = 555 minutes * 480 min - 555 min = -75 minutes * JS modulo gives -2h-15m = **'-02:-15'** (incorrect) * Domain-correct: (-75+1440)%1440 = 1365 min = **22:45** **Root cause:** JavaScript % operator preserves sign on negative numbers. Keep this test — it documents the bug. ``` **Prompt:** ``` I have 6 failing tests. Give me a summary of what I should fix first. ``` **Response:** ``` **Session Summary** * **Analyzed:** 6 tests * **TEST_ERROR:** 4 * **ENGINE_DEFECT:** 1 * **BOTH_WRONG:** 1 **Recommendation: MIXED** Fix your 4 test assertions first, then report the engine defect with traced proof to the developer agent. ``` ## Capabilities ### Trace engine logic step-by-step QA Arbiter forces the agent to show every intermediate calculation and branch taken during execution. ### Compare actual results against traces QA Arbiter checks if the engine's output matches the logic the agent just traced. ### Verify test expectations QA Arbiter compares the test's expected value against the trace to find assertion errors. ### Identify test errors vs. engine defects QA Arbiter provides a deterministic verdict on whether the code or the test is at fault. ### Detect environment pollution QA Arbiter helps identify when a test fails only due to shared state or global config. ### Flag flaky test behavior QA Arbiter detects timing dependencies and suggests quarantining unstable tests. ### Audit code coverage QA Arbiter identifies when tests are providing 'theater' coverage instead of verifying behavior. ## Use Cases ### The Midnight Crossover Bug A test fails at 12 AM because of a JS modulo bug. QA Arbiter traces the math and identifies it as an ENGINE_DEFECT rather than a test error. ### The Tax Calculation Error A test expects 108 but gets 8. QA Arbiter shows the engine calculated the tax correctly, identifying a TEST_ERROR in the assertion. ### The Flaky CI Pipeline A test fails 5% of the time. QA Arbiter identifies the timing dependency and recommends quarantining the test to save CI time. ### The Ghost in the Machine A test passes in isolation but fails in the suite. QA Arbiter helps identify global state pollution from a previous test. ## Benefits - Stop 'fix introduces regression' cycles by proving the test was wrong before any code changes happen using diagnose_test_failure. - Eliminate infinite retry loops in multi-agent pipelines by forcing a verdict on every failure. - Spot environment pollution where tests pass alone but fail in a suite due to shared state. - Detect 'coverage theater' where tests run the code but don't actually verify any meaningful behavior. - Quarantine flaky tests to save hours of wasted CI time and team frustration. - Ensure your agent provides a traceable audit log for every bug fix it proposes. ## How It Works The bottom line is that your agent stops guessing and starts diagnosing with a verifiable logic trace. 1. Provide the failing test details and the engine's inputs to your agent. 2. The agent calls the tool to perform a structured trace and compare values. 3. You get a deterministic verdict (TEST_ERROR, ENGINE_DEFECT, etc.) with a clear explanation of the logic gap. ## Frequently Asked Questions **What does QA Arbiter do for my test suite?** It diagnoses why your tests are failing by forcing your agent to perform a step-by-step logic trace. This helps you determine if the bug is in your code or just a mistake in the test assertion. **How does QA Arbiter help with flaky tests?** It identifies timing dependencies and environmental issues that cause tests to pass sometimes and fail others. It helps you move those tests to a quarantine list so they don't break your CI. **Can QA Arbiter tell if my test is wrong?** Yes, it compares the actual engine output against the trace and your expected value. If the engine did what it was supposed to do but the test expected something else, it flags it as a test error. **Will QA Arbiter find bugs in my code?** It identifies engine defects by showing exactly where the code's logic deviates from the expected outcome. It provides the proof you need to give to your developers. **Does QA Arbiter run my tests?** No, it doesn't run the tests for you. It is a diagnostic tool used after a test fails to provide a clear, deterministic reason for the failure. **How does QA Arbiter prevent regressions?** It ensures that you don't 'fix' a bug by simply changing the test to match the broken behavior. By proving the engine is actually broken, it forces a real code fix. **Does QA Arbiter run my tests or compute expected values?** No. QA Arbiter performs zero computation and zero side effects. It forces the AI agent to structure its own reasoning into verifiable steps, then validates that the reasoning is logically consistent. Think of it as a reasoning enforcer — like Sequential Thinking, but specialized for test failure diagnosis. **What are Decision Pivots?** Decision Pivots are minimal, verifiable checkpoints that all correct reasoning paths must pass through — a concept from the ROMA research framework. In QA Arbiter, the two pivots are boolean fields: `receivedMatchesTrace` (does the engine's output match the hand-traced computation?) and `expectedMatchesTrace` (does the test's expected value match?). The verdict is derived deterministically from these two booleans, making it impossible to reach a wrong conclusion without contradicting yourself. **How does it prevent pipeline deadlocks in multi-agent systems?** In a typical QA→Developer pipeline, when tests fail, the system routes back to the developer. But if the tests themselves are wrong (QA's fault), the developer can't fix them — creating an infinite retry loop. QA Arbiter forces the QA agent to determine fault attribution BEFORE the pipeline routes: if it's TEST_ERROR, the QA agent fixes its own tests; if it's ENGINE_DEFECT, it routes to the developer with traced proof. The aggregate summary tells the orchestrator exactly what to do. **What happens if the agent lies about the boolean pivots?** The consistency validation catches direct contradictions — e.g., if the agent says both values match the trace but chose TEST_ERROR instead of FALSE_ALARM, the tool rejects it. For subtler misrepresentations, the `engineTrace` field creates an auditable trail: post-hoc analysis can cross-reference the trace against the actual engine source code. The structured format makes deception mechanically harder than with free-form text.