Sample size for an A/B test: how to calculate it correctly
Why you can’t stop a test ahead of time, how to calculate the required number of users before launching, and what is test power.
Why is sample size so important?
Most A/B tests are stopped too early - this is one of the main mistakes in CRO. A test stopped at the first “green number” gives false results in 30-50% of cases.
Three calculation parameters
1. Base Conversion (CR) - current conversion of the control group
2. Minimum Effect (MDE) - the minimum improvement considered important
3. Statistical power - probability of detecting a real effect (usually 80%)
Evan Miller Formula
n = (Z_α/2 + Z_β)² × [p1(1-p1) + p2(1-p2)] / (p1-p2)²
Where:
Z_α/2 = 1.96 (at 95% significance level)
Z_β = 0.84 (at 80% power)
p1 = base conversion
p2 = expected conversion Practical examples
| Basic CR | MDE | Sampling per group |
|---|---|---|
| --- | --- | --- |
| 2% | +20% (up to 2.4%) | ~40,000 |
| 5% | +10% (up to 5.5%) | ~28,000 |
| 10% | +10% (up to 11%) | ~13,000 |
| 10% | +20% (up to 12%) | ~3 500 |
Conclusion: The smaller the MDE and base CR, the larger the sample required.
Rules for a good test
- ✅ Calculate sampling before launch, not during
- ✅ Do not stop the test early
- ✅ Test full weekly cycles (7N days)
- ❌ Do not change MDE after launch (p-hacking)
Calculate the sample size for your test using our calculator using Evan Miller''s method.