Free tool

Statistical significance calculator for A/B tests

Enter visitors and conversions for each version to get the p-value, the lift with its range and the chance to beat control.

Control (A)

Conversion rate: 3.00%

Variant (B)

Conversion rate: 3.60%

Confidence levelHow sure you want to be before calling a winner. 95% is the most common choice.
Test typeTwo-sided checks for a difference in either direction. One-sided only checks whether the variant is better, so it reaches significance sooner but can’t detect a variant that is worse.
Advanced
Planned traffic splitUsed for the sample ratio mismatch check.

Variant B is significantly better than control

Conversion rateConversions divided by visitors.3.60%control 3.00%
Relative liftHow much higher or lower the variant’s rate is than control’s, as a percentage of control. The range under it is where the true lift probably falls, at the confidence level you chose. A wide range means the result is uncertain.+20.0%95% range +3.2% to +39.5%
p-valueThe chance of seeing a difference at least this large if the two versions really performed the same. Lower means stronger evidence. At 95%, below 0.05 counts as significant if the lift range doesn’t cross zero.0.018
Chance to beat controlThe Bayesian probability that the variant’s true conversion rate is higher than control’s.99.1%

Traffic split looks fine (p = 1.000)

Get started →How Cuttl decides a test

The calculation runs in your browser. Nothing you enter is sent or stored.

A statistical significance calculator tells you whether the difference between two versions in an A/B test is bigger than chance would usually produce. Enter each version’s visitors and conversions. This one runs a two-proportion z-test for the p-value and a Bayesian model for the chance to beat control, and checks that your traffic split is sound.

Reading it

How to read your result.

The p-value and the chance to beat control answer different questions, so it helps to read them together with the lift range.

p-value: the chance of a difference at least this large if both versions really performed the same. At 95% confidence, a p-value under 0.05 counts as significant.

Chance to beat control: the probability that the variant’s true rate is higher than control’s. It is not the same number as 1 minus the p-value.

Lift range: where the true lift probably falls. “+20%, between +3% and +40%” is significant but still uncertain about the size of the win.

Traffic split check: if a 50/50 test ends up 52/48 on a lot of traffic, something is filtering visitors unevenly. That’s called a sample ratio mismatch, and it can bias the whole result.

Read the full guide: What is statistical significance in A/B testing?
One test, worked through
10,000 visitors each3.00% → 3.60%p = 0.018 · 99.1% chance to beat controlSignificant at 95%
1,000 visitors each3.00% → 3.60%p = 0.45 · 77% chance to beat controlNot significant yet

Control converts 300 of 10,000 visitors (3.00%) and the variant converts 360 of 10,000 (3.60%). The lift is +20.0%, the p-value is 0.018, and the chance to beat control is 99.1%, so the result is significant at 95%. With the same rates on 1,000 visitors each, the p-value is 0.45 and the chance to beat control is 77%: the gap looks the same, but there’s too little data to tell it from noise.

How it works

How this calculator works.

The calculator uses standard formulas and runs entirely in your browser.

Step 1

Conversion rates

Each version’s rate is conversions divided by visitors.

Step 2

p-value

A two-proportion z-test compares the two rates using their pooled rate. Two-sided by default; one-sided if you choose it.

Step 3

Lift range

The range for the relative lift comes from the log of the ratio between the two rates, at the confidence level you picked.

Step 4

Chance to beat control

Each version’s rate gets a Beta distribution starting from a flat prior (every rate equally likely before any data). The calculator then works out the exact probability that the variant’s rate is higher.

Step 5

More than two versions

Each variant is compared with control, and the significance threshold is divided by the number of comparisons (Bonferroni correction).

Step 6

Traffic split

A chi-square test compares the visitors each version got with the split you planned. A p-value below 0.001 is flagged as a sample ratio mismatch.

Mistakes

Mistakes that make a significant result wrong.

A calculator can only check the numbers you give it. These are the common ways a test produces a “significant” result that doesn’t hold up:

Measuring a change without an A/B test

Checking every day and stopping at the first green result

This is called peeking, and with a fixed sample size it pushes the false positive rate well above 5%. Wait for the planned sample size, or use a sequential test that’s built to be checked often.

No sample size plan

Decide the smallest lift worth detecting, the minimum detectable effect (MDE), before launch. At a 3% conversion rate, detecting a 10% relative lift at 95% confidence and 80% power needs about 53,000 visitors per variant.

Work out your sample size

Too many metrics or variants

Each extra comparison is another chance of a false positive. Pick one primary metric up front.

Stopping mid-week

Run tests for whole weeks so weekday and weekend visitors are both counted.

Small changes on small sites

If a test would take months, a before-and-after comparison may be the better tool.

Web Experiments

Get these checks on every live test.

Cuttl’s Web Experiments runs these checks for you on live tests, so you don’t need to copy numbers into a calculator.

  • Bayesian statistics by default, with sequential and fixed-horizon tests available
  • A pre-test planner that estimates how many weeks a test needs, with the minimum detectable effect set before launch
  • Peeking warnings, false-positive risk and sample ratio mismatch checks on every result
  • A launch decision with expected lift and chance to beat control, plus each variant’s own heatmaps and recordings
See test statistics in Web Experiments
Questions

Significance calculator questions

How do I know if my A/B test is statistically significant?

Enter each version’s visitors and conversions above. If the p-value is below 1 minus your confidence level (0.05 at 95%), the difference is statistically significant.

What confidence level should I use?

95% is the most common choice. Teams sometimes use 90% for low-risk changes and 99% when a wrong decision would be costly to undo.

Is a p-value of 0.03 a 97% chance that B is better?

No. The p-value describes how surprising the data would be if there were no difference. The chance that B is better is a Bayesian figure, which this calculator shows separately as the chance to beat control.

Should I use a one-sided or two-sided test?

Use two-sided unless you decided before the test that you only care whether the variant is better. A one-sided test reaches significance sooner, but it can’t tell you that a variant is worse.

Can I compare more than one variant?

Yes, up to three variants against control. The calculator applies a Bonferroni correction, so each variant needs a lower p-value to count as significant.

What is a sample ratio mismatch?

It’s when visitors aren’t split between versions the way the test planned, such as 52/48 instead of 50/50 on a lot of traffic. It usually means something is dropping visitors from one version, and it can bias the result.

How many visitors do I need?

It depends on your conversion rate and the smallest lift you want to detect. At a 3% conversion rate, detecting a 10% relative lift takes about 53,000 visitors per variant at 95% confidence and 80% power. Work it out for your page with the A/B test sample size calculator.

Run A/B tests with the statistics built in.

Cuttl plans the sample size, checks the traffic split and shows the chance to beat control on every test, alongside heatmaps and recordings for each variant.

Get startedTry the live demo