• STATISTICS
  • DECISION
  • DATA
  • HUMAN

How to validate A/B test significance with your AI

Connect this MCP to your AI client to turn raw conversion numbers into statistical certainty. Your agent handles the math so you can focus on the results.

Ask AI about this page

Short answer

How do I check if my A/B test results are actually significant?

Give your AI your control and variant conversion data, and it will run an independent t-test to calculate the p-value. This tells you if the difference you're seeing is real or just noise. You'll get a clear breakdown of the statistical significance for each test.

The outcomes

What your analysis produces

Once you provide the data, your AI delivers these specific insights.

STATISTICS

P-value calculation

The AI calculates the exact p-value for your experiment. This number tells you the probability that your results happened by chance.

DECISION

Significance verdict

Your agent interprets the math to tell you if the variant is a winner, a loser, or if the test is inconclusive. You get a plain English answer based on standard thresholds.

DATA

Metric summary

The AI organizes your control and variant means, standard deviations, and sample sizes. It cleans up the numbers so the math stays accurate.

HUMAN

Strategic direction

The AI provides the math, but you decide if the lift is worth the implementation cost. You take the statistical truth and turn it into a business move.

The workflow

What your AI does when the data arrives.

The AI takes over the heavy lifting of statistical computation once you feed it your experiment numbers.

  1. Data ingestion

    Your agent reads the conversion counts and sample sizes from your provided data source.

    calculate_t_test

  2. Statistical computation

    The AI executes the independent t-test formula to find the difference between your groups.

    calculate_t_test

  3. Significance check

    The agent compares the resulting p-value against your chosen alpha level to check for significance.

    calculate_t_test

  4. Result reporting

    Your AI summarizes the findings into a readable format that highlights the winner.

    calculate_t_test

Try it

Copy these to start.

Use these prompts to kick off your analysis.

Starting points

Replace the bracketed info with your actual numbers or file names.

jstat Connector

You're all set. Choose your MCP client and follow the setup instructions.

Connector linkhttps://edge.vinkius.com/vk_preview_EmtZwk2XpPdUyFvH15enT6zX1o0mdbBKs430ALWM/mcp

Claude Desktop

Follow the steps below to connect in seconds.

  1. 1In Claude Desktop, open Settings → Connectors.
  2. 2Click “Add custom connector” and paste the connector link above as the remote MCP server URL.
  3. 3Click Add and start a new chat — jstat capabilities are ready to use.
Configuration · claude_desktop_config.jsonCopy
{
  "mcpServers": {
    "t-test-statistics-engine-mcp": {
      "url": "https://edge.vinkius.com/vk_preview_EmtZwk2XpPdUyFvH15enT6zX1o0mdbBKs430ALWM/mcp"
    }
  }
}
Copy into chat04

  • Run a t-test on this CSV file to see if the conversion lift in variant B is significant.

  • Compare these two groups: Control has 500 conversions out of 10,000 visitors, and Variant A has 550 conversions out of 10,000 visitors.

  • Analyze the A/B test data in my Google Sheet and tell me if the p-value is below 0.05.

  • Check the significance of this experiment: Control mean is 0.12 and Variant B mean is 0.15, with sample sizes of 1200 each.

  • Claude
  • ChatGPT
  • Cursor
  • VS Code
  • Windsurf
  • Claude Code
  • JetBrains
  • Cline

Start here

Connect T-Test Statistics Engine once, then ask.

One connection gives your AI the math tools it needs. Your credentials stay encrypted and secure on our side.

Connect T-Test Statistics Engine to your AI

FAQ

How this task behaves

  • 01

    Can the AI change my original conversion data?

    No. The AI only reads your data to perform calculations; it cannot modify your source files or databases.

  • 02

    Does the AI decide which variant to deploy?

    No. The AI provides the statistical significance and the p-value, but you make the final decision on whether to roll out the change.

  • 03

    What kind of data does it need?

    It needs the number of successes (conversions) and the total sample size for both your control and your variant groups.

  • 04

    Can it run a Z-test instead of a T-test?

    No. This specific MCP is built to perform independent t-tests.

  • 05

    Will it automatically run tests every time my data updates?

    No. You must prompt your AI to run the calculation when you are ready to analyze the results.

  • More questions about T-Test Statistics Engine? The Connector page answers them. See everything the T-Test Statistics Engine Connector can do

Connect jstat to Claude, Cursor, ChatGPT & more