The novelty effect: why your winning test faded

Why does novelty inflate early results?

People notice what changed. A returning shopper who has seen the same product page ten times will look at a new banner they would ignore on a first visit. That extra attention shows up as extra clicks, and sometimes extra conversions, in the first days of a test.

The opposite also happens. Some regular users dislike change and perform worse at first with a new layout, an effect usually called change aversion or primacy. That can make a good change look bad in week one.

How do you spot it in your data?

Plot the lift by day rather than looking only at the total. A novelty effect shows as a lift that starts high and declines steadily. Split by new and returning visitors: if the lift is concentrated in returning visitors and absent in new ones, newness is the likely cause, since new visitors have nothing to compare against.

• Daily lift that starts high and trends down
• Lift concentrated in returning visitors
• Clicks on the changed element rising much faster than orders

How do you protect against it?

Run for at least two full weeks even when the sample is reached sooner, so the newness has time to wear off. Judge on purchases rather than clicks on the changed element. And keep a holdout after the change ships, so a fading effect is caught rather than assumed to last.

Continuous systems should also weight recent evidence more than old evidence. A strategy that won in its first week should have to keep winning to stay promoted.

Is the novelty effect the same as seasonality?

No, though both make results fade. Seasonality comes from who visits: a festive sale brings different shoppers with different behaviour. Novelty comes from the same shoppers reacting to something new. Both are reasons to avoid testing during unusual weeks and to keep measuring afterwards.

How does IntellQ handle novelty?

Its experiments credit conversions within an outcome window after exposure, keep a fixed holdout running after a change is promoted, and give recent behaviour more weight than old, with a 48 hour half life on visitor signals. A change that fades is rolled back on evidence rather than kept because it won once.

What does a novelty effect look like in practice?

A store changes its add to cart button from grey to its brand red. In the first three days, add to cart rises 15 percent against control, almost all of it from returning visitors. By the end of the second week the lift is 3 percent, and orders show no difference at all. The red button was noticed, not preferred.

Had the test been stopped on day three, the team would have credited the colour with a large win and moved on. Reading daily lift, splitting new and returning visitors and judging on orders would each have caught it.

The broader lesson is to distrust any result that is mostly made of returning visitors reacting to something new. New visitors are the cleaner test of whether a page works.

If your site has few returning visitors, novelty matters less, because most of your audience sees every version for the first time. Sites with loyal, frequent visitors, such as subscription brands and marketplaces, need to plan for it in every test.

A simple guard for any team: make two full weeks the shortest test you will ever read, whatever the calculator says. It costs a few days on busy sites and saves the most common kind of false win.