6 min read

Hold back 10% of your list: the cheapest honest test of whether your retention emails actually bring customers back

A few years ago, at a large membership retailer where I led CRM and membership analytics, I sat in a review where the team was celebrating a win-back email. The dashboard said the email had brought back hundreds of lapsed customers.

A few years ago, at a large membership retailer where I led CRM and membership analytics, I sat in a review where the team was celebrating a win-back email. The dashboard said the email had brought back hundreds of lapsed customers. Everyone was pleased. Then I asked one question, and the room went quiet: how many of those people would have come back if we had sent nothing at all?

Nobody knew. The dashboard had never been asked to answer that. It only counted who bought after the email went out, and it gave the email the credit for all of them.

A founder asked me the same thing more recently, in plainer words: what's a simple incrementality test a small brand can run? She did not want a statistics lecture. She wanted one test she could run with the tools she already pays for, that would tell her whether her retention emails actually bring customers back — or whether they mostly take credit for people who were coming back anyway.

What a holdout actually is

A holdout is a small group of customers you leave out of a campaign on purpose. You pick them at random, you send them nothing, and then you compare what they did with what the people you emailed did. The gap between the two groups is what the email actually caused.

That group has a second name you will see in guides: a control group. Same thing. A group that shows you what happens without the treatment, so you can see what the treatment added.

And the number you get from it has a plain name too. Incrementality is just a long word for the extra buying the email caused, above what would have happened anyway. Not opens. Not clicks. Extra purchases that would not have happened without the email.

Here is why it matters so much for retention. Retention emails go to people who already know you. Many of them were going to buy again anyway. They bought from you before, they like the product, and sooner or later they come back on their own. An email sent to that group will always look successful, because the group was already likely to buy. The holdout is the only thing that separates the email's work from the customers' own habits.

How teams run one now

Pick one recurring send. Your win-back email, your cart recovery flow, your monthly retention email — whichever one you run most often and trust most. One test, one send.

Split the audience at random, 90% to receive the email and 10% held out. Random means random. Not your least engaged customers, not the people you think will not buy. A random 10% looks exactly like the other 90%, except they do not get the email. Most email tools now have a random split built into the journey builder, so this is a setting, not a project.

The holdout group gets nothing from that campaign. And here is the part teams get wrong: they must not get the same offer through another channel while the test is running. If the held-out customers get the same discount by text two days later, the comparison falls apart. Hold them out everywhere for that one campaign, for one full cycle.

Then wait. Run it for one full buying cycle — two to four weeks for most products, longer if people buy rarely. Do not peek on day two and call it. The point is to see who bought over the whole window, not who clicked on the first morning.

When the window ends, compare one number: who bought. Not who opened. Opens tell you who noticed the email. Purchases tell you who moved. Count everyone assigned to each group, including people who never opened anything.

One worked example

Take a small brand with a win-back list of 10,000 lapsed customers. Hold back 1,000 at random. Mail the other 9,000.

A week later, 450 of the mailed group have bought. That is 5%. The dashboard calls it a 5% conversion rate, and the team is happy.

Now look at the holdout. 40 of those 1,000 people bought too, with no email at all. That is 4%.

That 4% is the key to the whole test. If the mailed group had behaved like the holdout, 4% of 9,000 — 360 people — would have bought anyway, email or no email. So the email did not cause 450 purchases. It caused 450 minus 360. Ninety.

Ninety is a very different number from 450. At an average order of $60, the email caused $5,400 of extra sales. The dashboard would have claimed $27,000. Four out of every five purchases it counted were going to happen anyway.

It gets worse when there is a discount involved. Say the email carried 15% off. All 450 buyers used it, but 360 of them would have bought at full price. Fifteen percent of a $60 order is $9, so the brand gave away $9 times 360 — $3,240 — for sales it already had. The email still did real work: 90 extra purchases. But the discount paid for 450 purchases when only 90 needed paying for.

These figures are illustrative, not a real brand's results. But the shape is what I have seen again and again: a campaign that looks like a clear win, and a holdout that shows most of the win was already on its way.

What changed

A few years ago, this test needed a data team. Someone had to pull the list, randomise it in a spreadsheet, build the holdout segment by hand, and stitch the results together afterwards. Most small teams never got round to it, so they kept reporting the dashboard number and hoping it was roughly true.

That part has changed. The split, the suppression, and the comparison now live inside the tools most brands already pay for. The test itself is the same idea it always was — leave some people out, compare, read the gap — but the work around it has shrunk to almost nothing.

There is a second change, quieter. More vendors now report their results against a control group, because buyers have started asking for it. If a platform tells you a campaign drove a number, asking "against what holdout?" is now a normal question, not a hostile one.

What I would do on Monday

Pick your biggest recurring retention send. Hold back 10% for one cycle. Compare who bought in each group, and work out the gap the way the example above does.

If the gap is healthy, you have just bought yourself confidence. Keep the send, and stop second-guessing it. If the gap is small, you have found something better: a campaign that was mostly taking credit, and a discount budget you can now spend where it actually changes behaviour.

This is the kind of work I now do with B2C teams: small, honest tests before bigger spending decisions, so the money follows what actually brings customers back.

The Repeat Purchase Check — a free 15-minute self-check for B2C and DTC leaders. Five checks that show where repeat buying is leaking, and whether your personalization is doing anything about it: asksaumya-consulting.kit.com/retentioncheck

Sources: Rejoiner, How to Measure The True Profitability of Your Email Campaign; Ecommerce Fastlane, How To Measure Email & SMS Incrementality With Klaviyo. The worked example figures are illustrative, drawn from the author’s practice, not reported results from a named brand.

More articles