Peeking at A/B test results: why stopping early lies
3 min read · updated
Peeking means checking a test repeatedly and stopping as soon as it looks significant. Because random swings cross the significance line many times during a test, stopping at the first crossing produces far more false winners than the 5 percent you think you accepted. Decide the sample size before you start and read the result at the end, or use a method designed for continuous monitoring, such as sequential testing.
Why does checking early cause false winners?
A significance test assumes you look once, at a sample size you chose in advance. If you look every day, each look is another chance for noise to cross the line. Two identical pages will drift apart and back together many times during a test, and if you stop at the first moment they look different, you will declare a winner that does not exist.
The effect is large. Checking daily over a few weeks and stopping at the first significant result can multiply the false positive rate several times over.
Is looking at the dashboard itself the problem?
No. Looking is fine. Acting on what you see before the planned end is the problem. Watch for bugs, broken pages and disasters, and stop a test that is clearly harming customers. Do not stop one because it looks like it is winning.
Which stopping rules are safe?
A fixed horizon: calculate the sample, run to it, read once. Sequential testing: methods built for repeated looks, which raise the bar for significance at each look so the overall error rate stays where you set it. Or a Bayesian rule with a minimum sample: no decision until each version has a set number of visitors, then a probability threshold.
- Fixed horizon, read once at the planned sample
- Sequential tests with adjusted thresholds per look
- A minimum sample per arm before any decision
- A maximum duration, after which the test ends as inconclusive
What about tests that never reach significance?
Extending a test indefinitely in the hope it turns significant is peeking in reverse. Set a maximum duration in advance. If the test ends without a clear result, record it as inconclusive: the change, if it does anything, does less than the effect you designed the test to find.
How do automated systems avoid peeking?
By building the rule into the system so nobody can stop a test on a good afternoon. IntellQ does not report a verdict until each arm has its minimum sample, 200 visitors by default, and the probability that the change beats control reaches the threshold, 95 percent by default. Its audit and the calculator on this site estimate the days that will take before a test starts.
How much does peeking actually inflate errors?
Simulations in the statistics literature show the size of the problem. Two identical versions tested at the 95 percent level and checked once at the end produce a false winner about one time in twenty, as intended. Checked after every batch of visitors and stopped at the first significant reading, the same identical versions produce false winners several times more often, and the more often you look, the worse it gets.
That is why disciplined teams write the stopping rule down before launch. It costs nothing, and it is the single most effective protection against shipping changes that never worked.
If you must make decisions mid test, use a method built for it, and make sure everyone reading the dashboard knows which method that is.
A useful habit is to hide the result entirely until the planned end date and show only health checks in the meantime: visitors per group, error rates and page speed. Teams that cannot see the running lift are not tempted by it, and lose nothing, because the decision was always going to be made at the end.
Writing the rule down also protects the team from pressure. When a stakeholder asks to ship early, the answer is the rule, not an argument.
Questions
How often can I check my A/B test?
As often as you like for bugs and harm. Just do not stop it for looking like a winner before the planned sample is reached, unless you use a sequential method designed for that.
What is sequential testing?
A family of methods that allow repeated analysis during a test by adjusting the significance threshold at each look, keeping the overall false positive rate at the chosen level.
Does Bayesian testing solve peeking?
It reduces the damage but does not remove it. A minimum sample and a decision rule set in advance still matter.