Compare sales before a change to sales after it, and the figure looks like a measurement. Most of the time it isn't. It can't separate a real effect from a season, a competitor's bad month, a trend that was already climbing, or ordinary noise snapping back toward average.
That gap does not shrink when the bet gets bigger. It grows. A $10M feature, product launch, or campaign judged on a pre-vs-post number is a $10M call that was never measured, presented as if it was.
Four cases follow. Two where a company assumed an effect and found none. One where a whole industry almost never checks. And one where the arithmetic shows why a real effect can stay invisible to a comparison this crude, no matter how carefully someone reads the chart.
The story in four parts
Without a real comparison group, there's no way to know what would have happened anyway. The number describes a coincidence in time, not a cause.
A real field experiment, not a guess: cutting a $100M-a-year budget line moved sales by an amount indistinguishable from zero.
Real observational estimates and real experiments on the identical campaigns disagreed by factors of three, ten, or more, in both directions.
The honest cost of skipping measurement isn't recklessness. It's not finding out, later, whether the money did anything.
The evidence at a glance
eBay's own randomised experiment found brand-keyword paid search added no measurable sales. Its CEO cut the budget by roughly $100 million a year.
One Facebook case study found an observational model estimating 1,306% lift, against a real randomised result of just 2.4%.
Applied to eBay's own data, the same style of regression estimated +4,100% ROI. The real experiment measured -63%.
Across 25 real retail experiments, the median sample needed to tell “worked” from “didn't” was 3.3 million exposed customers.
“Sales were up 12% after we launched it” sounds like a finding. In practice it is four different things jammed into one number. Only a real experiment peels them apart.
Without a group that didn't get the change, there's no version of events to compare the real one against. The 12% could be the change, or it could be what would have happened anyway.
A season, a competitor's outage, a macro shift, an unrelated feature shipped the same week. Pre/post has no way to separate the change from whatever else changed at the same time.
A change launched right after an unusually bad stretch will often look like it worked, purely because the bad stretch was never going to stay that bad.
A team runs several pre/post reads on the same launch, across different metrics and windows, and the one that looks best becomes the story that survives the debrief.
For years eBay paid to sit at the top of Google or Bing results for “eBay,” one line above the free organic listing for the same query. Internal before/after tracking said those ads drove incremental sales. Economist Steve Tadelis, working inside eBay, ran a randomised field experiment instead: brand-keyword ads off in some markets, left on in others.
Sales in the markets with ads off were statistically indistinguishable from the markets with ads on. Nearly every click the ad had been buying would have arrived anyway, through the free organic listing sitting right underneath it.
Tadelis pushed the test further, pulling ads for generic product-search terms too. The average effect on sales was, again, indistinguishable from zero. eBay's CEO subsequently cut the company's paid-search budget by roughly $100 million a year.
eBay wasn't careless before the test. The pre/post numbers looked strong, because people who clicked a brand-keyword ad were usually already heading to eBay. Only a market with the ads switched off could show that the same customer would have shown up for free.
Widen the lens past eBay's paid-search story and the pattern is industry-scale. Businesses spend hundreds of billions a year on advertising. Most of that spend has never faced a real experiment: a market or audience where the ad is simply withheld. Campaign launches, a number moves, credit gets assigned. Same pre/post logic this report is about.
That is not a claim that advertising never works. eBay's later experiments, and the case below, found real effects in other places. It is a claim about the ratio. A huge share of spend rests on a method that cannot rule out the alternative explanations in the taxonomy above. Very little of it is ever checked with a design that can.
The first two cases hang on plain before/after, the blunt version of the mistake. A regression on historical data looks more careful: it controls for known confounders. A large study that put those regression estimates beside 15 real, randomised Facebook ad experiments found that polish still wasn't enough.
Comparing what a careful observational model estimated against what the matched randomised experiment actually found, one case showed a real measured lift of 2.4%, against an observational estimate of 1,306%.
Applied to eBay's own non-brand search data, the same regression approach estimated an ROI of over 4,100%. The randomised experiment above found an ROI of roughly −63%. The two methods didn't just disagree on magnitude, they disagreed on the sign.
Sit with the paper's own conclusion: the errors do not point one tidy way you can correct for. A regression can inflate a real effect by an order of magnitude, or invent an effect that isn't there. There is no fixed discount you can apply to an observational number to make it trustworthy. The only way to know which side of that gap an estimate sits on is to run the experiment.
Even if confounding vanished, a second problem remains, and it is purely statistical. Individual purchase behaviour is noisy, noisier than most ad budgets are sized to detect. Across 25 real field experiments with major US retailers and brokerages, covering millions of exposed customers, researchers measured how much data a genuine effect needs before it stops looking like noise.
That's the median sample size the 25 experiments needed to distinguish a campaign that broke even from one that did nothing at all, a distinction most companies believe they can eyeball from a pre/post chart.
The sample size needed to reliably detect a genuinely real but modest 5% improvement, an effect size plenty of internal launch reviews claim to have found with nothing like this much data.
In those large experiments, the median confidence interval on a campaign's real ROI spans more than 100 percentage points. A pre/post read on a fraction of that data, with no control group, is not a cheaper version of the same measurement. It reports a single point where the honest answer is a range wide enough to hold both “this worked well” and “this lost money.”
None of these four cases say every feature, campaign, or product decision needs a randomised trial before it ships. That bar is too high for most calls: too slow for a small fix, too expensive for a low-stakes change, sometimes impossible to run cleanly. The narrower claim is harder to dodge: know which of two claims you are making. “We didn't have time to test it” can be honest and sometimes correct. “We tested it and it worked” is a claim about the world, and a pre/post number with no comparison group has not earned that claim.
Bet size should set the bar, not how confident the team feels. A $500 subject-line test can run on gut. A $10M feature, product line, or campaign is where the gap costs the most. The distance between “a number moved” and “we know why it moved” is expensive to get wrong, and hard to unwind once the internal story has settled on “it worked.”