The test was real. The conclusion wasn't.

Statistical power is the chance a study detects a real effect, given its sample size and the effect's true size. Designs with 20 or 30 people per group, the scale of several cases below, can sit under a 50 percent chance of catching a real moderate effect even when that effect is there. Misses are one half of the problem. The other half is uglier: when a small study still reports something “significant,” the true effect is usually smaller than the published number. Researchers call that inflation the winner's curse.

What it teaches: Small samples do more than risk a miss. A hit from an underpowered study is usually oversized on paper, which is why striking early findings so often shrink once a larger replication runs the same test.

A p-value answers one narrow question: if the true effect were zero, how often would random noise alone produce data this striking? Below 0.05 means less than a 5 percent chance under that single assumption. The number does not rank effect size, does not give the probability the claim is true, and does not forecast replication. Run several outcomes and publish only the one that clears the bar, or drop a few “messy” points, and noise can manufacture significance on demand. That habit now goes by p-hacking.

What it teaches: “Statistically significant” sounds broader than it is. Under one assumption, a pattern is unlikely to be pure chance. Importance, magnitude, and whether the finding is real are separate questions.

A confidence interval reports a plausible range for the true effect, not a lone point. “Improved outcomes by 10 points (95% CI: 1 to 19)” can clear significance while spanning a barely-there effect and a large one; the study cannot separate those stories. Small samples widen the interval because the data cannot pin the true value down. Headline numbers still travel without the uncertainty attached.

What it teaches: Two significant results can differ wildly in precision: one a tight estimate, the other barely distinct from finding nothing. The interval is the honest report; the headline number alone is not.

Cornell psychologist Daryl Bem published nine experiments in a top psychology journal in 2011. Eight reported statistically significant evidence that people could sense random future events before they occurred. Bem later said he had skipped pre-registration of hypotheses and methods. He had used the flexible exploratory analysis that was routine then: several ways of splitting and scoring the data, with whichever version cleared the threshold going into the paper.

What it teaches: Leave the hypothesis, sample, and analysis unlocked until after the results are in, and a determined analyst can pull “significant” effects from almost any noisy dataset. Three labs later tried a pre-registered replication of one of the nine effects and found no evidence of it. The episode helped push psychology towards pre-registration as a default.

A 2014 paper took 92 U.S. hurricanes from 1950 to 2012 and claimed storms with more feminine names killed more people. The proposed reason was that a “Hurricane Sandy” draws less seriousness than a “Hurricane Charlie.” Hundreds of news outlets ran with it within days. Statisticians who re-ran the work found the result leaned hard on a few unusually deadly storms and on particular model choices. Change the model, or drop a few outlier years, and the effect vanished or flipped.

What it teaches: Ninety-two storms is a thin sample for this kind of modelling, and many defensible model builds exist. Researchers can land on the exact setup that yields a striking result without intending to. That flexibility is often called researcher degrees of freedom, close kin to p-hacking.

A landmark 1985 paper treated the basketball “hot hand”, the idea that a player who just made several shots is likelier to make the next, as a myth. Hit-and-miss sequences looked like coin flips. That verdict held for about thirty years. In 2015, economists Joshua Miller and Adam Sanjurjo spotted a bias in the original method. In a short finite sequence, the usual measure of “chance of a hit right after a streak” leans toward finding no streak effect even when the data are truly random. Fix the bias, re-run the original numbers, and real hot-hand evidence appears.

What it teaches: Sample size and analysis flexibility leave other failure modes untouched. Sometimes the statistical procedure itself is biased toward a false negative. Null results draw less scrutiny than striking ones, so the bug can sit unnoticed for decades.

The Dunning-Kruger effect, that the least skilled people overrate their own ability the most, is among psychology's most cited findings. The usual test sorts people into quartiles by actual score, then compares each quartile's average self-rating to its score. Statisticians later showed that exact procedure yields a Dunning-Kruger-shaped curve even when ability and self-rating are pure, unrelated random noise. Sorting on the same score used to measure error creates a spurious correlation and a regression-to-the-mean pattern by itself. Methods that avoid that artefact shrink the effect well below the original claim.

What it teaches: A pattern that looks real can be mostly a property of how the analysis looks for it. Significance is not enough; ask whether the technique could carve that same shape out of noise.

Before the 1936 U.S. presidential election, Literary Digest mailed 10 million straw-poll ballots and got 2.4 million back, an enormous haul by any standard. It predicted Alf Landon over Franklin Roosevelt, 57 to 43 percent. Roosevelt won 62 to 37. The mailing list, drawn from telephone numbers, car registrations, and club memberships, tilted toward wealthier households in the Depression. A modern reanalysis found an even larger driver: the roughly 76 percent who never returned a ballot leaned toward Roosevelt far more than the 24 percent who did.

What it teaches: Millions of responses cannot rescue a sample that fails to represent the population it claims to describe. Survey statistics' oldest lesson is still the first check on any “huge dataset” claim: huge relative to what, and who is absent?

From the late 1960s, fewer than 90 children were offered one marshmallow now or two later if they waited. All attended a preschool on Stanford's campus, mostly from financially comfortable, highly educated families. Longer waiters showed modestly better later outcomes, and the study became a famous parable about willpower. In 2018, a larger, more economically and racially diverse sample of roughly 900 children showed the same basic pattern at about half the size. Once family income and early cognitive ability entered the model, the link nearly vanished.

What it teaches: An effect can be genuine inside a narrow, unrepresentative group and still misstate the cause once a broader population is tested. “Delaying gratification builds a better life” overlapped heavily with “growing up with financial security lets a child feel safe enough to wait, and also predicts a better life for other reasons.”

In 1971, Philip Zimbardo randomly assigned 24 male college volunteers to play guards or prisoners in a mock prison under Stanford. The standard telling says guards grew cruel on their own, and the study became a textbook staple: situations, more than personalities, corrupt behaviour. There was no control group, no pre-registered hypothesis, and no real statistical analysis, only a dramatic narrative from one uncontrolled run. A 2018 investigation using newly available recordings and interviews found the research team had coached guards to be “tough,” which undercuts the claim that cruelty sprang from the situation alone.

What it teaches: A vivid single case can feel like overwhelming evidence while missing the safeguards that separate a finding from a story: a comparison group, blinding, a real sample. It ran once, on 24 self-selected volunteers, with essentially no data analysis, and still shaped how millions of people were taught to think about human nature.

From 1924 to 1932, researchers at Western Electric's Hawthorne Works plant varied factory lighting and reported that productivity rose whichever way the lights moved, even when they dimmed. The claimed lesson was that workers respond to being watched more than to the change itself. “The Hawthorne effect” entered management language as the label for people changing behaviour simply because they know they are observed. For decades the original data were assumed lost. When economists Steven Levitt and John List located and formally analysed the records in 2011, the dramatic pattern largely evaporated. Day-of-week and pay-period effects explained the swings better than the lighting did.

What it teaches: A field can embed a term in its vocabulary and teach it for generations before anyone analyses the data underneath. “Everyone knows this effect is real” and “the data show this effect” are different claims, and for the study the concept is named after, they pointed in different directions.

In 1993, 36 college students briefly scored higher on one spatial-reasoning task after ten minutes of a Mozart piano sonata than after silence or relaxation instructions. The bump, roughly 8 to 9 IQ-equivalent points on that task, faded within 10 to 15 minutes. The original authors never claimed a link to general intelligence, children, or lasting development. Media coverage and popular books stretched it into “classical music makes babies smarter” anyway, and in 1998 Georgia's governor budgeted a classical CD for every newborn in the state. Later meta-analysis across many replications found any true effect is small and tied to certain spatial tasks, nothing like the popular intelligence claim.

What it teaches: Claim size and scope can balloon between a study and the policy built on it without anyone along the path deliberately lying. A short-lived bump on one narrow adult task became a lasting boost to infant intelligence, and a government spent real money on the gap.

James Q. Wilson and George Kelling's 1982 Atlantic essay argued that visible disorder, a broken window left unrepaired, signals that nobody is watching and invites more serious crime. The piece offered no original data or study; it was theory plus anecdote. It still became the intellectual basis for “zero-tolerance” and “order-maintenance” policing at city scale, most famously in 1990s New York, and sits among the most influential ideas in criminology. Decades later, a National Research Council review and later meta-analyses found little support for the claim that disorder itself causes serious crime. The most careful work found a modest crime drop from disorder policing, concentrated in cooperative, community-based approaches rather than the aggressive enforcement the essay's name came to stand for.

What it teaches: An intuitive, confidently argued idea can become national-scale policy without data behind it. The distance between a compelling essay and a tested causal claim never closed before the theory was already reshaping how millions of people were policed.

Facebook's News Feed ranking once weighted how often people clicked a link. By 2013 and 2014, publishers had adapted with headlines engineered only to trigger a click, “click-baiting,” without saying what the story contained. Click rates on that content were genuinely high. Facebook's own data showed other signals moving the other way. People who clicked a click-bait headline spent less time reading afterwards. They were also far less likely to like, comment, or share once they returned, exactly the behaviours the click was supposed to forecast.

What it teaches: Facebook had to rebuild ranking around bounce-back rate and post-click engagement rather than the click alone. The click had stopped meaning what teams assumed it meant. A bigger sample or a cleaner p-value would not have fixed that. The needed move was admitting the optimised metric had never been validated against the outcome it stood in for.

Waiting months for every test's true long-term effect is costly, so Netflix built a “surrogate index”: a model that combines several short-term signals into one number meant to forecast a longer-term outcome. They checked it against 1,098 test arms from 200 real A/B tests. Even that purpose-built, heavily validated model matched the true 63-day result about 95 percent of the time, and correctly flagged which tests were worth launching only 65 to 79 percent of the time.

What it teaches: A dedicated model, checked against 200 live experiments, is what Netflix needed to reach 95 percent confidence that a short-term number forecasts a long-term one. Treating an engagement metric as an “obvious cascade” down the funnel, with no model and no validation, is a reckless default anywhere.