Incremental lift: measuring what a change really added

How do you calculate lift?

Take the conversion rate of the exposed group and of the control group over the same period. The absolute lift is the difference, in percentage points. The relative lift is that difference divided by the control rate. With 3.0 percent exposed and 2.5 percent control, the absolute lift is 0.5 points and the relative lift is 20 percent.

Report both. A 20 percent relative lift sounds large and may be half a point. A half point sounds small and may be thousands of orders a month on a busy store.

Why does attribution overstate the effect?

Attribution counts who was touched, not who was changed. A recommendation widget seen by everyone who bought will be credited with every sale, including the ones that would have happened without it. Last click and similar models are useful for budgeting between channels, but they cannot say what a change added.

Only a comparison with an unexposed group answers that, which is why incrementality and holdouts go together.

What makes a lift number trustworthy?

A control group that is truly comparable, assigned randomly and kept stable. A sample large enough for the lift to stand clear of noise, with a range or probability attached. A fixed window for counting conversions after exposure, so late buyers are counted once and in the right group. And a measurement period that is not a sale or a launch week.

• Random, stable assignment to control
• Enough visitors in the smaller group
• A stated conversion window after exposure
• A normal period, not a sale or launch

How should lift be reported?

With the raw rates of both groups, the absolute and relative lift, a range or probability, the sample in each group, and the dates. Anything less invites the reader to assume more certainty than exists.

IntellQ's console reports a change this way: the raw rate after the change and the lift over control sit side by side and are labelled separately, so one is never presented as the other, and no verdict appears until each arm has its minimum sample.

Can lift be negative?

Yes, and it is worth looking for. Changes that raise clicks on themselves can lower orders by distracting shoppers from the main path. A system that only reports positive results is hiding the ones that should have been rolled back. A good process treats a measured negative lift as a success of the measurement, and removes the change.

What does a worked example look like?

Over four weeks, 40,000 visitors see a new delivery date line on product pages and 10,000 are held back. The exposed group places 1,200 orders, a 3.0 percent rate; the control places 250, a 2.5 percent rate. The absolute lift is 0.5 points and the relative lift 20 percent.

Scaled to the exposed group, that is about 200 extra orders in four weeks, not the 1,200 an attribution report would credit to the change. At an average order of 900 rupees, the change added around 1.8 lakh rupees of revenue in the period, which is the figure to weigh against what the change cost to build and run.

The same arithmetic works for any change, including a discount or a chat widget. Always ask what the control group would have done, multiply only the difference by the exposed audience, and compare that figure, not the attributed total, with the cost of running the change.