Your Email Dashboard Counts Wins You Would Have Gotten Anyway: Why Incrementality Is the New Retention Metric

I want to tell you about a number I used to report with a straight face.

We would send a win-back email to 1,000 lapsed customers. About 200 would buy within the week. The dashboard would call it a 20% conversion rate, and I would present it as a win. Nobody asked the awkward follow-up, least of all me: how many of those 200 were going to buy anyway?

Say the honest answer is 150. Then the email did not create 200 sales. It created 50. The other 150 were going to happen with or without us, and the dashboard quietly took the credit.

There is a name for the real number: incrementality. And if the product news from the first half of October is anything to go by, it is about to become the number growth teams get judged on.

What just happened

Three things landed within about a week of each other, and together they tell one story: holdout testing is being built into the tools marketers actually use.

A holdout test is not complicated. You leave a small, random group of customers out of a campaign, then compare what they did with what the contacted group did. The gap is what the campaign actually caused. Everything else is decoration.

On October 7, Google opened self-serve Conversion Lift studies to Search and Performance Max campaigns. The studies hold out users who never see the campaign, and the gap estimates the conversions the campaign truly added. There is a catch, and it is a real one: you need at least 1,000 observed conversions and a $5,000 minimum campaign budget to play. Smaller teams are locked out.

On October 15, a customer engagement platform called Optimove shared the first customer results from its AI product. One operator ran 44 real-time triggered activations reaching 354,000 player touches, each one measured against a control group. A content-only reactivation test produced 13.8% more deposits per 1,000 targeted players. Honestly, the numbers are not the interesting part. The interesting part is that they were reported against control groups at all. That is what lets a buyer check a vendor's homework.

On October 14, a measurement company called Lifesight rebuilt its platform around AI agents. The company says its scientists approve each model before it goes live, and that marketers keep control of budget decisions. It also says it serves more than 500 brands, though that is the company's own figure, not an independent count.

None of these is a small tweak. Together, they signal a shift in what counts as proof.

Why your dashboard counts phantom wins

Most dashboards run on attribution. Attribution is just a rule for assigning credit: the sale goes to whichever ad or email the customer touched last. It is a bookkeeping rule. It was never a test of what caused the sale.

Incrementality asks a different question: what would have happened if we had done nothing? That is the only question that tells you whether the money was well spent.

And the gap between the two can be uncomfortably large. Holdout studies covering dozens of direct-to-consumer brands put median retargeting incrementality near 28%. In plain terms, roughly 7 in 10 conversions the ads claimed were not caused by the ads at all. A separate reported study from a large social platform, across 15 advertisers, found only 34% of attributed conversions were genuinely incremental.

I would treat those as reported figures, not settled science. They come from vendor and marketing sources, not independent research. Your number could be better. It could also be worse. The only way to know is to run the test yourself.

Where this bites hardest is discounts. A 20%-off win-back email converts beautifully. But if that customer was about to pay full price, you have just handed over margin and learned nothing. The question is not "did the order happen?" It is "was the order worth the margin I gave away?"

The test itself is simple

I always compare it to a medicine trial. Half the patients get the pill, half get a sugar pill, and you compare who got better. A holdout test is the same idea, applied to your email list.

Pick a segment. Set aside 5 to 10% at random. Send them nothing. Compare purchase rates. Two things keep it honest: the split has to be genuinely random, not "the customers we like least," and the holdout group cannot quietly get the same offer through another channel during the test, or the comparison falls apart.

One more thing I have learned to say out loud in rooms: attribution is how you run the day to day, but incrementality is the audit. You need attribution to operate. Without a holdout test now and then to check it, though, it drifts into fiction. And when a test gives you a number, resist the urge to multiply it across your whole attribution report. Finance will ask how you got there, and "we scaled the test result" is not an answer that survives the meeting.

This applies to the newest channels too, by the way. Before paying commission on orders an AI assistant supposedly sent you, run a holdout and see how many of those orders your existing channels would have closed anyway. Some of these assistants are collecting rent on demand you already earned.

What I would do now

1. Run one holdout on your biggest recurring send. Your monthly win-back, your cart-recovery flow, your biggest loyalty email — pick one. Hold out 5 to 10% for one cycle. An email holdout needs no budget, only randomization. That is its big advantage over paid media.

2. Read the gap, not the open rate. Opens and clicks tell you who noticed. The holdout gap tells you who actually moved.

3. Put your discounts through the margin question. For each recurring offer, ask: was the order incremental enough to justify the margin? Retire the ones that fail. Keep the ones that pass, and keep them with confidence.

4. Ask every vendor one question. If a platform reports a lift, ask: against what control group? The better vendors are already answering this. Make it a condition of buying.

And the timing is genuinely good. A holdout run in late October tells you how much of your retargeting is real before Black Friday, the most expensive week of the year to be wrong. A dashboard that counts phantom wins does not just mislead you. It talks you into spending more on the illusion.

Sources: WebProNews (Oct 7, 2026); AgileBrandGuide (Oct 15, 2026); AgileBrandGuide (Oct 16, 2026); OmniConvert; CloseRev; AdBeacon.