At some point almost every product team, marketer, and policy office reaches for the same verdict: “this worked.” Behind that verdict can sit anything from a randomised trial to two dashboard lines that moved in the same direction. This field guide sorts those claims into a ranked ladder of causal evidence aimed at product, commercial, and digital decisions. It borrows the same identification tools economists bring to credit risk and public policy. The last stretch covers the mistakes that still wreck careful-looking analysis, with cases where the rung someone stood on changed the decision.
The story in four parts
Teams still call both a randomised trial and a dashboard correlation “evidence,” as if the word meant one thing.
Start with raw correlation, then matched comparison, then a natural experiment, then a randomised trial.
Especially in the week before a budget call, when almost nobody asks which rung the slide is standing on.
Cases further down cover paid search, ad measurement, credit scoring, and survivorship bias.
Product decks, marketing decks, and policy briefs all reach for the same casual label, “evidence that it worked,” for designs that sit on very different rungs. Read from the top: each step down drops another safeguard against a confound doing the work someone is attributing to X.
People (or accounts, or stores) are randomly split into treatment and control, so only the intervention differs between the two groups, on average, including on things nobody thought to measure.
Feature A/B tests · marketing lift tests · a randomised pilot of a new loan-approval rule
A real-world rule, cutoff, or accident of timing splits people as if at random, without anyone designing an experiment. Regression discontinuity at an eligibility threshold, difference-in-differences around a policy change, an instrument that shifts exposure for reasons unrelated to the outcome.
A price change rolled out region by region · a credit-limit rule that only applies above a score cutoff · a platform outage that hits some users and not others
Treated and untreated groups are matched or statistically adjusted to look similar on the traits you thought to record. Nobody was randomly assigned to anything, so any trait you didn't think to record can still be doing the real work.
A credit scorecard trained on approved applicants, matched to similar-looking rejected ones · a marketing report comparing “similar” cohorts before and after a campaign
Two things moved together in the data. Which one, if either, caused the other, and whether a third thing caused both, is still completely unknown.
Power users click feature X more often · most dashboard “insights” · most growth-hack case studies posted online
Climbing a rung does not make the lower ones useless. Correlation is a fair place to start hunting for a pattern. It is a poor place to stop, especially the week a budget gets locked.
Each phrase below can be factually true on its own terms. What slips is usually the framing: the surrounding sentence implies a higher rung than the design actually occupies.
“Users who did X converted Y% more.”
Correlation without a comparison group, and no account of who chose to do X in the first place.
“We saw a lift after launching X.”
Before-and-after on a single group, with other changes over the same stretch left unruled-out.
“Our top performers all do X.”
Survivorship bias: people who tried X and failed are usually absent from the sample being described.
“The data suggests a strong link.”
Correlational wording that tends to arrive at a decision-maker already wearing causal confidence it has not earned.
The hierarchy product and marketing teams reach for is the same one economists have used for decades on labour markets, credit, and public policy. Inside the field the usual label is the “credibility revolution”: a long shift toward designs that can carry a causal claim, rather than stopping at correlation.
Why compare fast-food jobs on two sides of a state line, instead of just before and after?
New Jersey raised its minimum wage in 1992. Most economists expected fast-food employment to fall. Card and Krueger compared restaurants in New Jersey with restaurants just over the Pennsylvania line, where the wage floor stayed put. The neighbouring state absorbed whatever else was moving in the regional economy at the time, so the wage change could be isolated more cleanly.
What it teaches: A single-place before-and-after cannot separate a policy from everything else that shifted in the same window. A comparable place where the policy never landed turns that weak before/after into a usable natural experiment.
How do you judge a lending rule on the applicants it never approved?
Lenders only see repayment for people they approved. Rejected applicants leave no outcome to learn from, so a model fit only on the approved slice is learning from a filtered, non-random cut of the pool. Credit-risk researchers call the correction reject inference.
What it teaches: Selection covers more than who volunteers. Institutions filter too, and the missing group is often the one needed to know whether the current rule works.
Does a result from a controlled lab travel to the market it's meant to describe?
Lab-versus-field gaps, and whether a small pilot's effect holds once it scales, already have their own report on this site. This page points there instead of restating that argument.
What it teaches: Randomisation makes the comparison fair. Setting, sample, and scale are separate questions about whether the result is the one someone actually needs. Those sit in full at the links below.
Each case below had real data, competent analysts, and a story that sounded plausible. Each still returned a materially wrong answer until someone checked it against a design higher on the ladder.
What happens when the people who click your ad were already going to buy?
Standard marketing measurement gives the sale to whichever ad a customer clicked last. Under that rule, eBay's brand-keyword paid search looked highly profitable, because clickers converted at a high rate. The confound was who those clickers were: people already searching for “eBay” by name were disproportionately people who would have visited and bought with or without the ad.
What it teaches: A large randomised field experiment turned brand-keyword search ads off across many U.S. markets while leaving them on in others. Incremental lift from that spend sat close to zero. The correlational read described who clicked accurately. It misread what caused the sale.
How far off is the ad-effectiveness number you already trust?
Most digital ad platforms estimate lift by comparing people who saw an ad with people who did not, using whoever the delivery system already exposed or skipped through ordinary delivery. That sample is not a random split of the audience. Facebook researchers ran large randomised experiments that held back a genuine control group with “ghost ads,” then ran the standard observational methods on the same campaigns to measure the gap between the two answers.
What it teaches: Observational methods overstated true ad effects by a wide margin, with the direction of the bias depending on the method. A platform's delivery algorithm never samples the audience at random. The gap was large enough to change which campaigns looked worth funding.
What's missing from a case study of companies that succeeded?
Survivorship bias has its own principle article on this site: the pattern behind “successful founders all did X” claims that leave out every founder who did the same X and failed.
What it teaches: The fix is rarely a fancier statistical test. Ask who is missing from the dataset before trusting a pattern that only shows up among the survivors.