Why Sample Size Matters Before You Start Testing
Running an A/B test without calculating required sample size first is one of the most common testing mistakes. It leads to peeking (ending tests too early when results look good) and underpowered tests (declaring a winner before enough data exists). The required sample size depends on three inputs: your current (baseline) conversion rate, the minimum detectable effect (MDE) you care about, and your desired statistical power (typically 80%) and confidence level (95%). If your baseline is 3% and you want to detect a +1pp lift, you need approximately 10,000–15,000 visitors per variant.