Minimum detectable effect: what it is and how to set it

What the minimum detectable effect is

The minimum detectable effect (MDE) is the smallest real change in a metric that an A/B test is designed to detect, at a chosen confidence level and statistical power. On a page that converts at 5%, a 10% relative MDE means the test is sized to spot a move from 5.0% to 5.5%. At 95% confidence and 80% power, that takes about 31,200 visitors per version.

You choose the MDE before the test starts. Together with your baseline conversion rate, it decides how many visitors the test needs and how long it has to run.

How the MDE sets sample size and test length

The smaller the effect you want to detect, the more visitors you need, and the relationship is steep: halve the MDE and you need about four times the visitors. These figures use one page at 2,000 eligible visitors a day, a 5% baseline, two versions, 95% confidence and 80% power:

• 5% MDE: about 122,000 visitors per version, or 122 days.
• 10% MDE: about 31,200 visitors per version, or 32 days.
• 15% MDE: about 14,200 visitors per version, or 15 days.
• 20% MDE: about 8,200 visitors per version, or 9 days.

The baseline matters too. A page that converts at 1% needs roughly five times the visitors of a page at 5% for the same relative MDE, because each visitor carries less information about a rare event. The guide to statistical significance explains confidence and power in more depth.

To see the numbers for your own page, enter your baseline and MDE in the A/B test sample size calculator.

Relative or absolute MDE?

Most A/B testing tools state the MDE as a relative change, a percentage of the baseline. A 10% relative MDE on a 5% conversion rate is 0.5 percentage points in absolute terms.

Mixing the two is a common planning mistake. A 10% absolute MDE on a 5% baseline would mean going from 5% to 15%, a change almost no page edit produces. When a calculator or a colleague quotes an MDE, check which one they mean before you compare sample sizes.

How to choose a minimum detectable effect

Choose the smallest change that would be worth acting on, then check that your traffic can detect it within a run you can accept. These questions help:

• What is a lift worth? If a 5% rise in checkout conversions pays for the work many times over, a small MDE is worth the wait. If the change is cheap and easy to undo, a larger MDE is fine.
• How long can the test run? Most teams cap a test at four to eight weeks, because seasonality, campaigns and cookie loss build up over longer runs.
• How bold is the change? A new headline rarely moves conversion by 20%. A redesigned checkout might. Match the MDE to the size of change you expect.

If the MDE you need is smaller than your traffic can support, don’t shrink the test to fit. Test a bolder change, move the test to a busier page, or measure the change with a before and after comparison. The guide to before and after analysis covers that option.

What the MDE is not

The MDE is a planning number: the true effect that a test at 80% power has an 80% chance of detecting. It says nothing about the size of the lift you will see.

• A test can reach significance with an observed lift smaller than the MDE. Observed results are noisy, so this happens regularly when a real effect exists.
• A result that falls short of significance doesn’t show there is no effect, only that any effect is probably smaller than the test was built to find.
• Changing the MDE halfway through a test changes the plan after seeing data, which raises the chance of a false positive. Set it once, before launch.

Minimum detectable effect in Cuttl

In Cuttl’s Web Experiments, the MDE is part of a test’s Statistics step, and a test can’t launch until it is set. Saving the MDE and the primary metric pre-registers the analysis.

• The MDE is a relative change, from 1% to 200%. Next to it you set the longest acceptable run (6 weeks by default, and at most 8 for a Bayesian test), the statistical power (70, 80, 90 or 95%) and an expected baseline.
• The pre-test planner reads the goal’s baseline on the test pages over the last 28 days, and the pages’ recent traffic at the test’s traffic share. It estimates visitors per version and the weeks the test needs, and it updates as you change the MDE.
• Its verdict is either “A/B feasible in about N weeks” or “Not enough traffic”, with the suggestion to use a before and after analysis (Change Impact) or test a bolder change instead.

AI assistants connected through the Cuttl MCP server can set the MDE on a draft test, and the same pre-registration rule applies before launch.

Common questions