What is statistical significance in A/B testing?

What statistical significance means

Statistical significance in A/B testing is a check on whether the difference between two versions is bigger than chance alone would usually produce. Most teams use a 95% threshold. If control converts at 3.0% and the variant at 3.6% over 10,000 visitors each, the p-value is about 0.018, so the result counts as significant at 95%.

With the same two conversion rates and only 1,000 visitors each, the p-value is about 0.45. The gap looks identical on a dashboard, but with so few visitors there is too little data to tell it apart from noise.

How the p-value and the 95% threshold work

The p-value is the chance of seeing a difference at least this large if the two versions really performed the same. A test starts from that “no difference” assumption, called the null hypothesis, and asks how surprising the data would be under it.

Before the test, you pick a significance level, written α. At the usual α of 0.05, a result is called significant when the p-value falls below 0.05. Tools often show this as “95% significance” or “95% confidence”. Some teams use 90% for low-risk changes and 99% for changes that are costly to undo.

A 95% threshold means that when a change truly does nothing, about 1 test in 20 will still show a significant difference, half of them in the variant’s favour. That 5% is the false positive rate the threshold accepts, and a stricter threshold needs more visitors to reach a decision.

What significance does not tell you

A significant result says the difference is unlikely to be pure chance. It doesn’t say several things people often read into it:

• It isn’t the chance that the variant is better. A p-value of 0.02 doesn’t mean a 98% chance that B wins. That question needs a Bayesian analysis, which reports a “chance to beat control”.
• It doesn’t say how big the effect is. Look at the confidence interval. A lift of “+20%, between +1% and +39%” is significant but very uncertain.
• It doesn’t say the change matters. With enough traffic, a 0.1% lift can be significant and still not be worth the work.
• It doesn’t mean the win is real. If only a few of your ideas work, a fair share of significant wins will be false. At 95% significance, 80% power and 1 real winner in 5 ideas, about 11% of “wins” are false positives.

That last figure is called the false-positive risk. It depends on how often your past tests found real effects, so it varies from team to team.

Common mistakes that break significance

Most misleading A/B test results trace back to how the test was run. These are the usual causes:

• Peeking. Checking a fixed-horizon test every day and stopping the first time it crosses 95% raises the false positive rate well above 5%. Either wait for the planned sample size or use a sequential test, which is built to be checked as often as you like.
• No sample size plan. Decide the smallest lift worth detecting, the minimum detectable effect (MDE), before launch. At a 3% baseline, detecting a 10% relative lift at 95% significance and 80% power needs about 53,000 visitors per variant. A 20% lift needs about 14,000.
• Too many comparisons. Each extra variant or metric is another chance of a false positive. Choose one primary metric up front, and correct for multiple variants (a Bonferroni correction, for instance).
• A broken traffic split. If a 50/50 test ends up 52/48 with a lot of traffic, something is filtering visitors unevenly. This is called a sample ratio mismatch, and it can bias the whole result.
• Stopping on a weekday pattern. Run tests for whole weeks so that weekend and weekday visitors are both counted.

Fixed-horizon, sequential and Bayesian tests

There are three common ways to decide an A/B test, and each one handles significance differently. A fixed-horizon test is the classic method: plan a sample size, wait until you reach it, then read the p-value once.

A sequential test keeps its error rate valid however often you look, so you can stop early when the evidence is strong. The cost is that it needs a little more data than a fixed-horizon test when the effect is small.

A Bayesian test reports the chance that each version beats control, and the expected lift with a range. Many people find “a 96% chance B is better” easier to act on than a p-value. Our guide to A/B testing covers how to plan a test, and the guide to measuring a change without an A/B test covers sites with too little traffic for any of the three.

Statistical significance in Cuttl

Cuttl’s Web Experiments uses Bayesian statistics by default, and Cuttl checks each test once a day. Before the test’s planned end, a version is called better or worse only at a 99.9% chance. At the planned end, 95% is enough, and a test with no call by then is inconclusive. Both bars get stricter when more versions are compared.

Sequential and fixed-horizon tests are optional, with a threshold of 90%, 95% or 99% (95% by default). Bayesian results show a chance to beat control; sequential and fixed-horizon results show significance.

Before launch, a pre-test planner estimates how many weeks a test needs at your traffic. The primary metric, the MDE and the model are locked in when the test starts, and later changes are flagged. Every result also shows its false-positive risk, worked out from the site’s past tests.

Cuttl checks the traffic split once 100 visitors have joined a test, and warns of a sample ratio mismatch when the split is off at p below 0.001. Tests with several variants can use a Bonferroni correction, and every variant gets its own heatmaps and recordings so you can see why the winner won.

To check a finished test by hand, use the A/B test significance calculator. It shows the p-value, the lift range, the chance to beat control and a traffic split check.

Common questions