QA Arbiter MCP, Ready to Go

It diagnoses why your tests are failing by forcing your agent to perform a step-by-step logic trace. This helps you determine if the bug is in your code or just a mistake in the test assertion.

It identifies timing dependencies and environmental issues that cause tests to pass sometimes and fail others. It helps you move those tests to a quarantine list so they don't break your CI.

Yes, it compares the actual engine output against the trace and your expected value. If the engine did what it was supposed to do but the test expected something else, it flags it as a test error.

It identifies engine defects by showing exactly where the code's logic deviates from the expected outcome. It provides the proof you need to give to your developers.

No, it doesn't run the tests for you. It is a diagnostic tool used after a test fails to provide a clear, deterministic reason for the failure.

It ensures that you don't 'fix' a bug by simply changing the test to match the broken behavior. By proving the engine is actually broken, it forces a real code fix.

No. QA Arbiter performs zero computation and zero side effects. It forces the AI agent to structure its own reasoning into verifiable steps, then validates that the reasoning is logically consistent. Think of it as a reasoning enforcer — like Sequential Thinking, but specialized for test failure diagnosis.

Decision Pivots are minimal, verifiable checkpoints that all correct reasoning paths must pass through — a concept from the ROMA research framework. In QA Arbiter, the two pivots are boolean fields: receivedMatchesTrace (does the engine's output match the hand-traced computation?) and expectedMatchesTrace (does the test's expected value match?). The verdict is derived deterministically from these two booleans, making it impossible to reach a wrong conclusion without contradicting yourself.

In a typical QA→Developer pipeline, when tests fail, the system routes back to the developer. But if the tests themselves are wrong (QA's fault), the developer can't fix them — creating an infinite retry loop. QA Arbiter forces the QA agent to determine fault attribution BEFORE the pipeline routes: if it's TEST_ERROR, the QA agent fixes its own tests; if it's ENGINE_DEFECT, it routes to the developer with traced proof. The aggregate summary tells the orchestrator exactly what to do.

The consistency validation catches direct contradictions — e.g., if the agent says both values match the trace but chose TEST_ERROR instead of FALSE_ALARM, the tool rejects it. For subtler misrepresentations, the engineTrace field creates an auditable trail: post-hoc analysis can cross-reference the trace against the actual engine source code. The structured format makes deception mechanically harder than with free-form text.