What is a holdout group, and why every personalisation needs one
3 min read · updated
A holdout group is a fixed share of visitors who see your site exactly as it was, with no personalisation, experiments or new features. Comparing everyone else with them shows what all your changes together are really worth. Without a holdout, every tool reports its own success and nobody can say whether the site converts better than if none of them were running.
How is a holdout different from an A/B test control?
An A/B test control lasts for one test and is compared with one change. A holdout lasts across many changes and is compared with all of them together. The first answers whether this headline beats that one. The second answers whether your whole optimisation programme, or a whole personalisation tool, is earning its cost.
Many teams run both: a control inside each experiment, and a global holdout that never sees any experiment at all.
How big should a holdout be?
Large enough to measure the lift you expect within a time you can wait, and no larger, because every held out visitor misses changes that are probably helping. Ten to twenty percent is common. Smaller holdouts are cheaper in lost conversions but take much longer to show a reliable difference, since the smaller group sets the pace of the comparison.
How do you keep a visitor in the holdout?
Assign them by a stable hash of an identifier, not by a coin flip on each page view. The same visitor must land in the same group every time, or they see a mix of old and new and the comparison is muddied. A hash of a first party visitor ID and an experiment salt does this without storing a separate list.
Visitors who clear their browser storage will be reassigned, which adds a little noise, but not bias, since it happens equally in both groups.
- Assign by a stable hash, not per page view
- Keep the holdout share fixed while measuring
- Measure conversions, not clicks on the changed element
- Credit each conversion once, within a set window after exposure
What does a holdout reveal that dashboards hide?
Cannibalisation, mainly. A popup tool, a recommendation widget and a personalisation tool can each report impressive numbers by taking credit for the same buyers. The holdout shows the total effect on people who saw none of them. It also catches changes that win in their own test and then fade, because the holdout keeps measuring after the test is over.
How does IntellQ use a holdout?
IntellQ holds back 20 percent of visitors as control by default, assigned by a SHA-256 hash of the visitor and experiment so each person stays in one group. Every change is measured against that group, conversions are credited within an outcome window, 72 hours by default, and the console shows lift against control separately from the raw rate after a change. The share can be set between 5 and 50 percent.
What mistakes undermine a holdout?
Changing its size halfway through a measurement, which mixes two different comparisons. Excluding the holdout from some changes but not others, so it stops being a clean baseline. Letting a sale banner or new feature reach the holdout through a different tool. And reading the holdout on clicks instead of orders, which tells you about the changed element rather than about the business.
Another quiet error is forgetting the holdout exists when a new team member adds a tool. Write it down where everyone who changes the site will see it.
Finally, refresh a long running holdout every few months by reassigning a new random slice. Visitors who have been held out for a year have a different experience from everyone else, and their behaviour drifts for reasons unrelated to any one change.
- Keep the share fixed during a measurement
- Exclude the holdout from every change, in every tool
- Read it on orders or revenue
- Document it for everyone who changes the site
Questions
Does a holdout cost conversions?
Yes, if your changes are helping, the held out visitors miss them. That cost is the price of knowing whether the changes help at all, and it shrinks with a smaller holdout.
Should the holdout be permanent?
A global holdout is often kept for months and refreshed periodically, so it reflects current traffic and does not leave the same people out forever.
Can I use a holdout on a small site?
Yes, but the result takes longer. Size it with the traffic you have, and expect weeks rather than days before the difference is reliable.