How much traffic do you need for an A/B test?
3 min read · updated
Far more than most people expect. A store converting 2 percent of visitors needs about 42,000 visitors in total to detect a 20 percent relative lift with a 50/50 split, at 95 percent confidence and 80 percent power. Keep 20 percent as a holdout instead of half and it rises to about 67,000. Halve the lift you hope to see and the requirement roughly quadruples. Work out the number before you start, not after.
What decides the sample size?
Four things. Your baseline conversion rate: the rarer the conversion, the more visitors you need. The smallest lift worth detecting, called the minimum detectable effect: small lifts hide in noise. How sure you want to be, usually 95 percent confidence and 80 percent power. And how traffic is split between the groups: an even split is the most efficient, and anything lopsided needs more in total.
Of the four, the minimum detectable effect moves the number most. Detecting half the lift needs about four times the visitors, because sample size grows with the square of one over the effect.
How many visitors for common conversion rates?
The table below gives total visitors across both groups, two sided at 95 percent confidence and 80 percent power. The first figure in each row assumes a 50/50 split, the second keeps 20 percent as a control group and shows the change to 80 percent, which is how an always on personalisation system usually runs.
| 1% baseline, +20% lift | 85,400 then 136,200 |
| 1% baseline, +30% lift | 39,700 then 63,800 |
| 2% baseline, +10% lift | 161,400 then 254,900 |
| 2% baseline, +20% lift | 42,200 then 67,300 |
| 2% baseline, +30% lift | 19,600 then 31,500 |
| 3% baseline, +20% lift | 27,800 then 44,400 |
| 5% baseline, +20% lift | 16,300 then 26,000 |
Why does a holdout need more traffic than a 50/50 test?
Because the smaller group sets the pace. With 20 percent in control, that group fills up four times more slowly than the treated group, and the comparison can only be as precise as its smaller side. The trade is deliberate: fewer visitors miss out on a change that is probably helping, at the cost of a longer wait for proof.
If you need an answer quickly and the change is reversible, a 50/50 test is faster. If the system runs continuously and the change is likely to help, a smaller holdout loses fewer conversions while it learns.
How do you turn visitors into a test duration?
Divide the total by the daily visitors who actually reach the page being tested, not site wide traffic. A store with 3,000 visitors a day, of whom 1,000 see product pages, testing a product page change at 2 percent baseline and a 20 percent lift needs about 42 days at 50/50. Run whole weeks so weekday and weekend behaviour are both included.
- Use visitors to the tested page, not the whole site
- Round up to whole weeks
- Avoid sale periods, which change who visits
- Decide the duration before starting and do not stop early on a good day
What if you do not have that much traffic?
Test bigger changes, so the effect you look for is larger. Test closer to the conversion, where the baseline rate is higher, such as cart to order rather than visit to order. Combine pages that share a template into one test. Or accept a lower confidence level for low risk, reversible changes and say so honestly in the result.
IntellQ's free Website Intent Audit includes a calculator that turns your monthly visitors and conversion rate into the days one test takes, and the standalone calculator on this site does the same for any split.
What does this mean for a typical Indian D2C store?
Take a store with 60,000 monthly visitors, a 1.8 percent conversion rate and about 20,000 visitors a month reaching product pages. A product page test hoping for a 20 percent lift needs roughly 47,000 visitors at 50/50, which is over two months of product page traffic. A cart page test, where the conversion rate to order is far higher, can finish in a few weeks.
So the same store should test big, high rate questions on few pages, and use customer research for the rest. Running five small product page tests at once would leave each of them unfinished for most of a year.
Questions
What is a minimum detectable effect?
The smallest relative lift your test is designed to detect reliably. Choose it before starting: smaller effects need far more visitors, roughly four times as many for half the lift.
Is 1,000 visitors enough for an A/B test?
Almost never for a purchase conversion. At a 2 percent conversion rate, 1,000 visitors produce about 20 orders, far too few to separate a real lift from noise.
Should I use 95 percent confidence?
It is the common default. For low risk, easily reversed changes, some teams accept 90 percent to get answers faster, but the risk of a false winner rises.
Does an 80/20 split need more traffic than 50/50?
Yes. With a 20 percent control group the test needs roughly 60 percent more visitors in total for the same precision, because the smaller group limits the comparison.