Agent A/B Test Calculator Connector for AI agents.

You can use it to run controlled experiments on different agent prompts or workflows. By comparing the success rates of two versions, you can mathematically prove which one actually drives more conversions.

Yes. You can tell your agent how many daily interactions you have and what kind of improvement you want to detect, and it will tell you exactly how many samples you need and how many days the test will take.

Absolutely. It includes the necessary statistical corrections to ensure that when you test multiple variants simultaneously, you don't accidentally identify a false winner.

It uses rigorous statistical methods like p-values and Bayesian probability to quantify uncertainty, helping you understand if a small sample size is enough to trust the results.

Yes, as long as your client is MCP-compatible, such as Claude, Cursor, or Windsurf, you can use these statistical capabilities directly in your workflow.

You can use the analyze_variant_performance capability. It calculates the p-value to determine if the observed difference in conversion rates is statistically significant or likely due to chance.

Yes, the estimate_test_requirements capability calculates the required sample size and the estimated number of days needed to complete a test based on your daily traffic and desired sensitivity.

The frequentist approach (via analyze_variant_performance) focuses on p-values and significance thresholds, while the Bayesian approach (via calculate_bayesian_probability) provides the direct probability that one variant is better than another.