A citation on every principle here still leaves a separate job: deciding what that citation can carry. In the 2010s, psychology and behavioural economics ran a hard audit of their own classics. Other labs repeated famous, heavily cited effects with larger samples and locked designs, and many of those effects shrank or disappeared. The field stayed useful. Headline trust stopped being enough.
Two checks do most of the work. Internal validity asks whether a fair, adequately powered test reproduced the effect in the first place. External validity asks whether it still shows up once people, place, and context change. The cases below cover both: findings that aged poorly under replication, and interventions that aimed one way and moved the opposite way in the field. For how strong the causal claim was before anyone talks about replication, start with How Strong Is That Evidence? A Field Guide to Causal Claims.
Through the 2010s, labs set out to reproduce classic results on purpose. Samples grew. Designs were pre-registered so analysis could not be reshaped after the data arrived. A striking share of famous effects collapsed or shrank under that pressure. How careful readers weigh a claim changed with it. Four cases, and the specific lesson each one carries:
Can standing like a superhero for two minutes really change your hormones?
Carney, Cuddy, and Yap's 2010 paper used 42 people. Two minutes in an expansive “power pose” appeared to raise testosterone, cut cortisol, and lift how powerful people said they felt. A TED talk past 20 million views pushed it into mainstream behavioural science lore. Ranehill and colleagues re-ran it in 2015 with 200 participants, enough power to see a real hormonal shift if one existed. Self-reported power came back. Hormones and risk-taking did not.
What it teaches: Power and polish can outrun the sample that produced them. A tiny original N can print a tidy chart that is mostly noise. Dana Carney, one of the original co-authors, has said publicly that she no longer treats the hormonal and behavioural claims as real.
Why did 23 labs together find nothing, after decades of studies said willpower runs out?
Self-control research treated willpower like a muscle for nearly twenty years. Spend it on task A, and task B gets harder because a limited resource is temporarily spent. Dozens of papers and a friendly meta-analysis built a consensus around that story. In 2016, twenty-three labs ran one shared, pre-registered protocol on 2,141 people in total. Pooled across labs, the ego-depletion effect was effectively gone.
What it teaches: A thick literature can still be thin evidence. Years of small, underpowered studies, plus a file drawer that keeps nulls off the shelf, can manufacture the look of agreement around a weak or missing effect. Pre-registration is there to stop that drift before it hardens into “everyone knows.”
Why did the pen-in-the-teeth effect vanish the moment anyone was watching on video?
Strack, Martin, and Stepper had people rate cartoons while holding a pen in their teeth (forcing a smile shape) or in their lips (forcing a pout). Cartoons scored funnier in the smile condition. The claim was sharp: the expression itself could shape the emotion people usually treat as its cause. A 2016 multi-lab replication, seventeen labs, more than 2,000 participants, protocol checked with the original author, returned a null.
What it teaches: A detail nobody flagged can decide whether an effect travels. One leading account: replication labs filmed participants, and being watched appears to damp the very feedback loop under study. A null replication can mean the original was wrong. It can also mean the effect only appears inside a boundary condition that had not been named yet.
Why did people only “walk slower” after old-age words when the experimenter expected it?
Bargh, Chen, and Burrows primed people with old-age words (“Florida,” “wrinkle,” “bingo”) and reported that those people walked more slowly when leaving the lab. The paper shaped a generation of priming work. Doyen and colleagues tried again in 2012 with infrared timing instead of a stopwatch in a human hand. Experiment 1 showed no priming. Experiment 2 recovered a slowdown only when the stopwatch holders had been told, beforehand, to expect slower walking.
What it teaches: Something real moved the stopwatch reading, and the cause was the wrong one. When the person scoring the outcome knows the hypothesis and is free to judge subjectively (“how slowly did they seem to walk”), expectation can leak into the score. Blinded, objective measurement is the internal-validity guardrail against that leak.
Read the four cases as one rule of thumb. A clean-looking single study is still one observation. Before treating any finding on this site, or anywhere else, as settled, check four things: Was the sample large enough to detect a real effect reliably? Was the analysis locked before the results arrived? Has a different lab, with different people, seen the same thing? Does the tested population and setting resemble the place someone wants to apply it? Behavioural science stays worth reading. Methodology has to sit in view beside the headline.
Replication in a lab still leaves another gap: whether the same mechanism keeps its sign once it hits a live system full of incentives and substitutes. A well-replicated finding can reverse under those conditions. Six published cases and six separate mechanisms now sit in their own full report.
A late fee, bottled-water ban, scare programme, neighbour letter, rat bounty, and donor reminder each flipped the wrong way. Six separate mechanisms explain those flips: crowding out, substitution, a mistaken audience theory, a masked subgroup, a gamed proxy, and an unwatched second number.
A clean lab demonstration and a result that survives a messy market share a vocabulary word, “effect,” and often little else as evidence. John List has spent much of his career measuring that gap. The honest reading is that it is frequently wider than people budget for.
Why doesn't how you play a lab game predict how generous you actually are?
A large share of published generosity and fairness findings come from paid student volunteers who know they are inside a study. Dictator-game giving and punishment of unfair offers are classic examples. Levitt and List reviewed that literature and found lab measures often fail to forecast how the same people act in comparable real situations. Anonymity, social context, and real stakes diverge from the booth.
What it teaches: Control buys precision and can also rewrite the behaviour under study. Being watched, knowing the session is an experiment, and dealing with strangers who will never reappear are lab features that markets usually strip away.
Why do pilots that work beautifully shrink the moment you roll them out for real?
List's applied programme keeps returning to the same pattern. A strong effect in a small pilot often shrinks, sometimes to a fraction of its size, once the programme runs at real scale. He labels that pattern the “voltage drop.” Fair warning on sourcing: the label and synthesis sit in List's popular book rather than one peer-reviewed paper alone. Treat it as a practitioner pattern from a long field-experiment career, not as a single controlled finding.
What it teaches: A successful pilot is useful evidence about that pilot. It is not yet evidence about a national rollout. Who gets recruited, who staffs the trial, and the novelty of a small test can all inflate results that fade once delivery becomes ordinary.
Asking “would you pay for this?” or “would you do this?” remains a standard research tool. It measures a different thing from what people do when money or effort is actually on the line, and that gap has a name of its own.
Why do people say they'd pay more than they actually do, even when warned about it?
When an answer costs nothing, people reliably report a higher willingness to pay in hypothetical surveys than they hand over once real money is required. Researchers call that gap hypothetical bias. Scripts that warn respondents about the bias beforehand (“cheap talk”) have generally done little reliable work closing it.
What it teaches: An honest stated intention answers a low-stakes question. It does not forecast the same person's behaviour under a real cost. Wherever the design allows it, an actual purchase or signed commitment beats a stated preference, and this site tries to keep that distinction visible rather than blurred.
Twenty-three companion pieces on evidence quality, research honesty, and applied mechanism design, written to the same standard as the guide above.
A field guide to causal evidence: a ranked ladder from raw correlation to randomised trials, framed around product and commercial decisions, with the selection-bias traps that sink careful-looking analysis.
John List’s argument that governments should build studies around full-scale constraints first, so the evidence fits the policy they mean to run rather than a pilot that cannot travel.
Why organisations flinch at the word ‘experiment,’ rename it a pilot or trial, and often drop the control group along with the label, on John List’s real advice and its cost.
Cash, debit, and credit aren’t just three ways to move the same money. Two real studies and the mechanism connecting them: why the payment method itself changes how much gets spent.
Thirteen real cases of experiments that didn’t lie and still misled: samples too small to see the effect, results stretched into a universal law, and tests run on a guess with no theory behind the metric they moved.
Prize-linked savings accounts swap a guaranteed rate for a chance at a prize sized to the balance; the deposit itself stays intact. Real programmes have moved millions into savings.
Nobody owns a bias: behavioural economics, psychology, applied behavioural science, and UX all draw on the same small set of findings. Sometimes the name survives the trip between them. Sometimes it doesn’t.
Temporal reframing, chunking, anchoring, bundling, sunk cost, and zero price. Six real, cited mechanisms, and the pricing-page patterns that run on each one.
A before/after number usually is not a measurement. Real cases where pre/post said one thing and a randomised experiment said another, eBay’s paid search among them.
Opt-in, opt-out, forced choice, timed defaults, and personalised defaults each work differently. Five researched settings, with the real studies, the numbers, and the famous organ-donation gap that shrank under a cleaner test.
Almost nobody opens a standard-form contract before agreeing. When a conflict is disclosed and read, advice can grow more biased while listeners still under-correct. One payday-store cost format did cut later borrowing.
Pricing pages already make choice-architecture calls. Eight cited mechanisms, from anchoring to bundling, that show up whether or not anyone named them.
MIT randomised apartments in 1946. A 2022 machine-learning paper looked at the MLB draft. Both found the same pattern: how near someone lives to the decision-maker predicts who gets picked, trusted, or called a good fit, apart from real skill.
Digital is cheaper to send, every time. Three real, cited studies on touch, reading comprehension, and the tactile premium people place on physical objects explain why organisations, NRMA’s own quarterly member magazine among them, still mail the paper copy anyway.
Progress Flow (steps and a progress bar) and Progressive Load (one continuous page) both borrow real psychology, chunking and the goal-gradient effect, but six studies ranked by fidelity show the same pattern doing nothing on a desktop and backfiring on a phone.
People over-trust fluent AI until one visible miss, then under-trust it past its real accuracy. A human-in-the-loop label does not reliably fix either swing. Ten cited studies on coding assistants, clinical AI, and knowledge work.
A fair coin as a teaching case for three misreadings: the gambler’s fallacy, what a p-value actually says, and why the smallest sample in any ranking looks the most extreme.
Defaults, loss aversion, present bias, and choice overload already steer Australian super at national scale. Four cited cases of real cost, plus the 2021 stapling law that pointed the same default the other way.
A clean, significant result can still fail on internal, external, ecological, construct, or statistical conclusion validity. Five cited cases, one for each failure mode.
A worked map of the outcome chain behind a feature launch: awareness, discovery, trial, completion, and persistence, walked through on one illustrative savings-goal example, with the proxy trap at each link.
Program theory is the decades-old evaluation discipline underneath every logic model: why a program fails for one of two very different reasons, shown on a real program that made crime worse while working exactly as designed, and one that got the theory right from day one.
A high school savings demo split pay into thirds, named each third, and hid one from view. Three separate real studies explain why each rule works, then the exact tap in a real banking app that does it.
Netflix is reportedly weighing always-on channels as engagement slips. Separately tested failures at choosing, queuing, and finishing point at the browse screen as the bottleneck.