Lester Leong
How to Calculate Sample Size for an A/B Test (Without a Statistics Degree)
Most A/B Tests Are Doomed Before They Launch
The single most common experimentation failure I see is not a flawed hypothesis or a buggy implementation. It is a test that never had a chance of concluding because nobody calculated how many users it needed. The team picks a metric, splits traffic, waits a few weeks, sees a result that is "trending positive but not significant," and argues about what to do. The honest answer is almost always that the test was underpowered from the start. It could not have detected the effect they wanted even if that effect was real.
I have watched this pattern repeat across three very different environments. In my consulting work across 20+ SMBs and startups at Gradient Growth, I am routinely handed an "inconclusive" test that was never powered to conclude anything. At a financial social media startup before its acquisition, where our traffic was small enough that getting the sample-size math wrong meant burning a month on a test that returned noise. And now on a GenAI squad at a major finance technology company, where traffic is abundant but the number of competing experiments means sample-size math is what decides which tests we can afford to run. In all three, the teams that planned sample size before launch shipped more real wins than the teams that launched first and reasoned later.
Sample size is not a statistics-degree problem. It is four inputs and one function call. This guide gives you the four inputs, the tradeoffs, a Python snippet you can run today, and the three mistakes that quietly waste most experimentation budgets.
The Four Inputs That Determine Sample Size
Every sample-size calculation for a conversion-style A/B test comes down to four numbers. Get these right and the arithmetic takes care of itself.
Input 1: Baseline Conversion Rate
This is the current conversion rate of the metric you are testing, measured on your control experience. If 5% of users who hit a checkout page complete the purchase, your baseline is 0.05. Pull this from real historical data, not a guess. The baseline matters because the variance of a proportion depends on the rate itself: a metric near 50% has more variance than one near 5% or 95%, which changes how many samples you need to distinguish two arms.
A practical note from consulting: teams frequently estimate the baseline from the wrong population. If your test only fires for logged-in users on mobile, your baseline must be the conversion rate for logged-in mobile users, not the blended site-wide number. The wrong baseline can throw your sample size off by a factor of two.
Input 2: Minimum Detectable Effect (MDE)
The MDE is the smallest improvement you care about detecting, and the input people understand least and get wrong most often. If your baseline is 5% and the smallest lift worth shipping for is half a percentage point, your MDE is an absolute 0.5pp, taking you from 5.0% to 5.5%. That is a 10% relative improvement.
The MDE is a business decision, not a statistical one. You are answering: how small a lift would still be worth the engineering and product cost of shipping this change permanently. If a 0.2pp lift would not change your roadmap, do not power your test to detect one. Smaller MDEs require dramatically more samples, and there is no point paying for precision you will never act on.
Input 3: Significance Level (Alpha)
Alpha is your tolerance for a false positive: declaring a winner when there is no real difference. The convention is 0.05, meaning you accept a 5% chance of being fooled by noise on any given test. Lowering alpha (say to 0.01) makes you more conservative and requires more samples. For most product experiments, 0.05 is the right default. Reserve tighter alpha for decisions that are expensive to reverse.
Input 4: Statistical Power
Power is the probability that your test detects a real effect of the size you specified in your MDE. The convention is 0.80: if a true lift of your MDE exists, you have an 80% chance of catching it and a 20% chance of missing it (a false negative). Higher power (0.90 or 0.95) means you miss fewer real wins, but it costs more samples.
Power is the input teams ignore most and pay for most. A test run at 50% power is a coin flip on detecting a real effect. When that test comes back non-significant, the team concludes "no effect," when the honest conclusion is "we ran a test that was always likely to miss this." More on that mistake below.
How the Inputs Trade Off
Two relationships govern every sample-size decision you will make.
1. Smaller MDE means more samples, and the cost is nonlinear. Sample size scales roughly with 1 / MDE^2, the most important single fact in experiment planning. Halving the effect you want to detect does not double your sample requirement. It roughly quadruples it. Chasing a 0.25pp lift instead of a 0.5pp lift is not twice as expensive. It is about four times as expensive.
2. Higher power and lower alpha both cost samples. Moving from 80% to 90% power increases your sample requirement by roughly a third. Tightening alpha from 0.05 to 0.01 increases it further. These are real costs, paid in traffic and time.
The practical implication is that the MDE has the most leverage. Before you calculate anything, the most valuable conversation your team can have is about the smallest effect actually worth detecting. Loosen that target from 0.25pp to 0.5pp and you cut your sample requirement by roughly 75%, which can be the difference between a test that concludes in two weeks and one that takes two months.
The Calculation in Python
You do not need to memorize the formula. The statsmodels library does it correctly in a few lines. The function below computes the per-arm sample size for a two-proportion test.
```python from statsmodels.stats.power import NormalIndPower from statsmodels.stats.proportion import proportion_effectsize
# baseline 5.0% vs target 5.5% (an absolute 0.5pp MDE) effect = proportion_effectsize(0.05, 0.055)
n = NormalIndPower().solve_power( effect_size=effect, alpha=0.05, power=0.8, alternative='two-sided', )
print(f"per arm: {n:,.0f}") print(f"total: {2 * n:,.0f}") ```
A few things to understand. `proportion_effectsize` converts your baseline and target rates into a standardized effect size (Cohen's h), which is what the power calculation consumes. `NormalIndPower` is the power-analysis engine for a normal-approximation two-sample test, the right model for comparing two conversion rates. The `solve_power` call returns the sample size needed per arm. A standard A/B test has a control and a treatment, so you double it for the total.
Treat the output as approximate. It rests on a normal approximation and assumes a clean 50/50 split, no sample-ratio mismatch, and independent observations. In practice I add a 10-20% buffer for users who enter the experiment but never reach the metric event, and for the messiness of real traffic.
A Fully Worked Example
Run the snippet above with a baseline of 5.0% and a target of 5.5%, and you get approximately 31,000 users per arm, or roughly 62,000 total. That is the cost of reliably detecting a half-point lift on a 5% baseline at standard alpha and power.
Now watch what the 1 / MDE^2 relationship does. Suppose you decide a 0.5pp lift is too small to matter and you only care about a full percentage point: baseline 5.0%, target 6.0%.
```python effect = proportion_effectsize(0.05, 0.06) # 1pp absolute MDE n = NormalIndPower().solve_power( effect_size=effect, alpha=0.05, power=0.8, alternative='two-sided' ) print(f"per arm: {n:,.0f}") # approximately 8,100 ```
The per-arm requirement drops from roughly 31,000 to roughly 8,100, and the total from about 62,000 to about 16,300. You doubled the MDE and the sample requirement fell to roughly a quarter. That is the 1 / MDE^2 relationship in action, and why the MDE conversation is the highest-leverage planning decision you will make.
The complete picture for a 5% baseline at alpha 0.05 and 80% power:
1. 0.5pp MDE (5.0% to 5.5%): about 31,000 per arm, about 62,000 total. 2. 1.0pp MDE (5.0% to 6.0%): about 8,100 per arm, about 16,300 total. 3. 2.0pp MDE (5.0% to 7.0%): about 2,200 per arm, about 4,400 total.
Now overlay your traffic. If 10,000 users hit this surface per week, the 0.5pp test takes about six weeks to fill both arms, the 1.0pp test under two weeks, and the 2.0pp test concludes in days. Same metric, same statistics, wildly different feasibility, all driven by one decision about how small an effect you will chase. The choice of which metric to power matters too; powering the wrong [conversion funnel stage](/insights/conversion-rate-funnel-stages) wastes the whole calculation.
The Three Costly Mistakes
The sample-size calculation is the easy part. The expensive errors happen in how teams behave around it.
Mistake 1: Peeking and Early Stopping
The most damaging mistake is checking the test repeatedly and stopping the moment it crosses significance. This feels rigorous. It is the opposite. Every time you peek at a running test and apply a 0.05 threshold, you take another independent shot at a false positive. Peek ten times during a run and your real false-positive rate is no longer 5%, it is closer to 20-30%. You will declare winners that are pure noise, ship them, and then wonder why your aggregate conversion rate never moves despite a string of "successful" tests.
The fix is to commit to your calculated sample size before launch and evaluate the result once, at the end. If you genuinely need to monitor a test in flight (for guardrail metrics, or to abort on a disaster), use a method built for it: sequential testing or alpha-spending boundaries that account for the repeated looks. The wrong approach is to run a fixed-horizon test and treat every Monday's dashboard as a decision point. This is one half of a broader trap I have written about, [running A/B tests on the wrong metric](/insights/ab-testing-wrong-metric): peeking and metric choice are the two most common ways a technically valid test produces an invalid decision.
Mistake 2: Calling an Underpowered Null "No Effect"
The second mistake is interpreting a non-significant result from an underpowered test as evidence that there is no effect. Absence of evidence is not evidence of absence. If you ran a test with enough traffic to detect a 2pp lift and the true effect was a real 0.5pp lift, your test was statistically blind to it, and a non-significant result tells you nothing about whether a smaller effect exists.
This is why you calculate the MDE up front: it defines the boundary of what your test could see. When a properly powered test comes back null, you can make a meaningful statement: "there is no effect of at least the size we cared about." When an underpowered test comes back null, the only honest statement is "this test was incapable of answering the question." Teams that conflate the two retire good ideas on the basis of tests that never tested them.
Mistake 3: Ignoring the MDE and Chasing Undetectable Effects
The third mistake is never specifying an MDE at all, which means implicitly chasing effects too small to detect at your traffic. A small product with 2,000 weekly users on a checkout flow simply cannot detect a 0.5pp change on a 5% baseline in any reasonable timeframe; it would take more than half a year to accumulate 62,000 users into the test. For that team the correct move is not to run the test. It is to either pursue changes big enough to produce a detectable effect, or pick an earlier, higher-volume metric where the MDE math works. Ignoring the MDE means you discover this constraint after wasting a quarter, instead of in a five-minute calculation before launch.
Sample Size Is a Throughput Constraint, Not Just a Per-Test Question
The deepest reason to take sample-size math seriously is that traffic is a shared, finite resource. Every user you route into one experiment is unavailable to another. Your sample-size requirements, summed across your backlog, set a hard ceiling on how many experiments you can run well in a given period. This is the real link between sample size and [experiment velocity](/insights/experiment-velocity-metric): velocity is not how many tests you launch, it is how many tests you can power to a conclusion.
This reframes the planning conversation. A team with 50,000 weekly users powering every test to detect 0.3pp effects can run perhaps one or two experiments a quarter, most still inconclusive. The same team, disciplined about MDE and willing to detect only effects of 1pp or larger, can run several conclusive tests in the same window and compound real wins. Tighter MDEs are not more rigorous. They are more expensive, and the expense is paid in tests you do not get to run.
The way out for traffic-constrained teams is to shrink the unit of test, not lower statistical standards. Test earlier in the funnel where event volume is higher. Test changes bold enough to produce large effects. Concentrate traffic on a few high-conviction bets rather than a dozen marginal ones. This is the discipline behind the [minimum viable experiment](/insights/minimum-viable-experiment): the smallest test that can produce a decision-grade answer with the traffic you actually have.
The Five-Minute Pre-Launch Checklist
Before any A/B test goes live, answer five questions in order. It is the highest-return five minutes in your experimentation process.
1. What is the baseline? Pull it from the exact population the test will run on, not a blended number. 2. What is the MDE? Decide the smallest lift worth shipping for. This is a business call, made before you touch any statistics. 3. What alpha and power? Default to 0.05 and 0.80 unless the decision is expensive to reverse, in which case tighten them and accept the larger sample. 4. What is the required sample, and how long will it take? Run the calculation, double it for two arms, add a buffer, and divide by your weekly traffic into the test. 5. Is that timeline acceptable? If the test would take longer than you can wait, do not launch it as designed. Raise the MDE, move to a higher-volume metric, or kill the test before it wastes your traffic.
The teams that run this checklist do not launch dramatically more tests than the teams that skip it. They launch dramatically more tests that actually conclude. That is the entire game. An experiment that cannot reach a decision is not a cautious experiment. It is a waste of traffic dressed up as diligence, and the sample-size calculation is how you tell the difference before you have spent the month finding out.
---
I help teams design experiments that are powered to conclude, so their traffic produces decisions instead of inconclusive dashboards. [lester@gradientgrowth.com](mailto:lester@gradientgrowth.com)