• STATISTICS
  • DISTRIBUTION
  • EFFECT SIZE
  • HUMAN

How to verify AB test results with your AI

Connect your experiment data to your AI client to run statistical sanity checks. It catches false positives before you ship changes.

Ask AI about this page

Short answer

How can I be sure my A/B test results aren't just noise?

Your AI uses the Data Analysis Prover to run power analyses and check if your sample size was large enough to detect the observed effect. It also validates that your data follows the necessary distribution assumptions. You'll get a breakdown of the mathematical reliability of your findings.

The outcomes.

What your AI uncovers.

The AI looks past surface metrics to find the mathematical truth of your experiment.

STATISTICS

Valid statistical power

The AI calculates if your sample size was sufficient to detect the effect you're seeing. You'll know if you're chasing ghosts or if the data is solid.

DISTRIBUTION

Assumption checking

It checks if your data meets the requirements for the tests you're running. This prevents you from using a t-test when the data is heavily skewed.

EFFECT SIZE

Real-world impact

The AI determines the magnitude of the difference between your groups. This helps you decide if a change actually matters for your business.

HUMAN

Decision support

The AI flags mathematical red flags but leaves the final call to you. You decide whether to trust the result based on the technical breakdown.

The workflow

What your AI does when the data arrives.

The AI moves from raw data ingestion to rigorous statistical validation.

  1. Data ingestion

    Your AI reads the raw experiment logs or spreadsheets you provide to build a clean dataset for testing.

    validate_data_analysis

  2. Power analysis

    The agent calculates the probability that your test actually had enough users to detect the change you observed.

    validate_data_analysis

  3. Distribution testing

    It runs tests to see if your data follows normal or other required distributions for standard statistical methods.

    validate_data_analysis

  4. Significance reporting

    The AI summarizes the effect size and p-values, highlighting where the math might be shaky.

    validate_data_analysis

Try it

Copy these to start.

Use these prompts to kick off the analysis in your AI client.

Starting points

Replace the bracketed text with your specific file names or data sources.

Data Analysis Prover Connector

You're all set. Choose your MCP client and follow the setup instructions.

Connector linkhttps://edge.vinkius.com/vk_preview_z0UOTGMw5z9e0sSzQndQJ5ZX9DZMH1doH3kzey3a/mcp

Claude Desktop

Follow the steps below to connect in seconds.

  1. 1In Claude Desktop, open Settings → Connectors.
  2. 2Click “Add custom connector” and paste the connector link above as the remote MCP server URL.
  3. 3Click Add and start a new chat — Data Analysis Prover capabilities are ready to use.
Configuration · claude_desktop_config.jsonCopy
{
  "mcpServers": {
    "data-analysis-prover-mcp": {
      "url": "https://edge.vinkius.com/vk_preview_z0UOTGMw5z9e0sSzQndQJ5ZX9DZMH1doH3kzey3a/mcp"
    }
  }
}
Copy into chat04

  • Analyze the A/B test results in my 'experiment_results.csv' file and check the statistical power.

  • Check the distribution assumptions for the conversion data in my Google Sheet.

  • Calculate the effect size for the variant B group in my 'test_logs.json' file.

  • Verify if the p-values in my 'marketing_test.csv' are reliable given the sample size.

  • Claude
  • ChatGPT
  • Cursor
  • VS Code
  • Windsurf
  • Claude Code
  • JetBrains
  • Cline

Start here

Connect Data Analysis Prover once, then ask.

Just link the MCP to your client. Your credentials stay encrypted and you can start running math immediately.

Connect Data Analysis Prover to your AI

FAQ

Common questions

  • 01

    Can the AI change my original data files?

    No. The AI only reads your data to perform calculations. It cannot modify or overwrite your source files.

  • 02

    Does the AI decide if I should launch the feature?

    No. The AI provides the statistical evidence, but you make the final business decision.

  • 03

    What kind of files can I use?

    The AI can process standard formats like CSV, JSON, and Google Sheets that you provide access to.

  • 04

    Can it run a Bayesian analysis?

    No. The current tools focus on frequentist validation like power and distribution checks.

  • 05

    Will it tell me if my sample size was too small?

    Yes. The power analysis capability is specifically designed to flag insufficient sample sizes.

  • More questions about Data Analysis Prover? The Connector page answers them. See everything the Data Analysis Prover Connector can do

Connect Data Analysis Prover to Claude, Cursor, ChatGPT & more