Statistical significance in A/B tests, without the jargon
3 min read · updated
A result is statistically significant when a difference that large would rarely happen by chance if the two versions were really the same. At the usual 95 percent level, that means a gap this big appears less than one time in twenty from luck alone. It does not mean the change is 95 percent likely to be better, and it says nothing about whether the lift is big enough to matter. Read significance together with the size of the lift and its range.
What does a p-value actually tell you?
It answers one narrow question: if there were truly no difference between the versions, how often would random variation produce a gap at least as large as the one you saw? A p-value of 0.03 means three times in a hundred. Below 0.05, the result is called significant at the 95 percent level.
It does not tell you the probability that your change works, how big the effect is, or whether it will hold next month. Treating it as any of those is the most common misreading in experimentation.
Why do confidence intervals matter more than the headline lift?
A test that reports a 12 percent lift is giving you a single best guess. The confidence interval gives the range of lifts consistent with the data, for example from 1 percent to 23 percent. If that range includes values too small to be worth the change, the result is weaker than the headline suggests, even if it is significant.
Always report both. A lift of 12 percent with a range of 10 to 14 is a solid finding. A lift of 12 percent with a range of 1 to 23 is a hint.
What is statistical power?
The chance that your test detects a real effect of the size you care about. At 80 percent power, a real 20 percent lift will be found four times out of five, and missed once. Power is decided by sample size: a test that is too small has low power and mostly produces inconclusive results, which teams then misread as proof the change does nothing.
Which mistakes create false winners?
Stopping as soon as the result looks significant, which inflates the false positive rate well above the one in twenty you think you are accepting. Testing many metrics and reporting whichever one moved. Running many variants against one control without adjusting for multiple comparisons. And reading segments after the fact until one looks impressive.
- Fix the sample size and primary metric before starting
- Do not stop early because the result looks good
- Adjust for many variants or many metrics
- Treat segment findings as new hypotheses, not results
How should a tool report results?
With the raw conversion rate of each group, the lift against control, a range or probability, and a clear statement of whether the minimum sample has been reached. A tool that shows a green arrow before the sample is complete is encouraging exactly the stopping early mistake.
IntellQ's console shows the raw rate after a change next to the lift against control, labelled separately, and reports a verdict only once each arm has the minimum sample, by default 200 visitors, and the probability of being better crosses 95 percent.
How do you explain a result to a founder in one minute?
Say three things in this order: how big the effect probably is, how sure you are, and what you recommend. For example: the new product page raised orders by about 12 percent, with the likely range from 4 to 20 percent; the chance this is noise is small; we should keep it and check again in a month. That beats any p-value read aloud.
If a result is inconclusive, say so plainly and say what it rules out: we cannot tell the versions apart, but if the new one helps, it helps by less than 15 percent.
Questions
Does 95 percent significance mean 95 percent chance of winning?
No. It means a difference that large would appear by chance less than 5 percent of the time if the versions were the same. The probability your change is better is a different, Bayesian question.
What if my test is not significant?
It means the test could not separate the versions at that sample size. It does not prove they are equal. Check whether the test had enough power for the effect you cared about.
Is 90 percent confidence acceptable?
For low risk, reversible changes some teams accept it. It doubles the false positive rate compared with 95 percent, so use it knowingly.