The test was real. The conclusion wasn't.

Statistical power is the probability a study will detect a real effect, given how many people it tested and how big that effect truly is. A study with 20 or 30 people per group, common among the cases below, can easily have less than a 50 percent chance of detecting a real, moderate effect, even when one genuinely exists. That cuts both ways: a real effect can be missed entirely, and when a small study does turn up something “significant,” the true effect is usually smaller than reported, an inflation researchers call the winner's curse.

What it teaches: a small sample doesn't just risk missing a real effect. When it finds something anyway, that something is usually exaggerated, which is exactly why so many small, striking early results shrink once a larger study repeats the test.

A p-value answers one narrow question: if there were truly no effect at all, how likely would data this striking be, purely from random noise? A p-value under 0.05 means a result that size would turn up by chance less than 5 percent of the time under that one assumption, nothing more. It says nothing about how large the effect is, how likely the underlying claim is to be true, or whether it will replicate. Test several outcomes and report only the one that clears the bar, or quietly drop a few “messy” data points, and a significant p-value becomes easy to manufacture from pure noise, a practice now called p-hacking.

What it teaches: “statistically significant” is a far narrower claim than it sounds. It means a pattern is unlikely to be pure chance under one specific assumption, not that the finding is important, large, or even real.

A confidence interval gives a plausible range for the true effect, not just a single number. A study reporting “improved outcomes by 10 points (95% CI: 1 to 19)” is a real, significant result, but that range spans a barely-there effect and a genuinely large one; the study can't tell those two stories apart. Small studies tend to produce wide intervals, precisely because there isn't enough data to pin the true value down, and a single headline number often gets repeated without anyone mentioning how uncertain it actually was.

What it teaches: two studies can both clear the significance bar while one is a precise, trustworthy estimate and the other is barely distinguishable from finding nothing at all. The interval, not just the number in the headline, is the honest version of a result.

In 2011, Cornell psychologist Daryl Bem published nine experiments in a leading psychology journal, eight of which reported statistically significant evidence that people could sense random future events before they happened. Bem later acknowledged he had not pre-registered his hypotheses or methods, and had used the flexible, exploratory analysis common at the time, trying several ways of splitting and scoring the data and reporting whichever cleared the significance threshold.

What it teaches: a determined analyst can find “significant” effects in almost any noisy dataset if the hypothesis, sample, and analysis aren't locked in before looking at the results. A pre-registered attempt to replicate one of the nine effects, run independently by three labs, found no evidence of it at all. The episode became one of the direct triggers for psychology's push towards pre-registration.

A 2014 study analysed 92 U.S. hurricanes from 1950 to 2012 and reported that storms with more feminine names caused significantly more deaths, its proposed explanation being that people take a “Hurricane Sandy” less seriously than a “Hurricane Charlie.” The finding was picked up by hundreds of news outlets within days. Other statisticians who re-ran the analysis found the result depended heavily on a handful of unusually deadly storms and on specific choices inside the statistical model; changing the model or excluding a few outlier years made the effect disappear or reverse.

What it teaches: with a small number of independent events, 92 storms is a small sample for this kind of modelling, and many defensible ways to build the statistical model, researchers can end up choosing, often without meaning to, the exact setup that happens to produce a striking result. This is sometimes called researcher degrees of freedom, a close cousin of p-hacking.

A landmark 1985 study concluded that the “hot hand” in basketball, the belief that a player who just made several shots is more likely to make the next one, was a myth: players' sequences of hits and misses looked statistically indistinguishable from a coin flip. That conclusion stood as settled fact for three decades. In 2015, economists Joshua Miller and Adam Sanjurjo identified a subtle bias in the original method: the natural way of measuring “chance of a hit right after a streak” in a short, finite sequence is itself biased towards finding no streak effect, even in truly random data. Correcting for the bias and re-running the original numbers turned up real evidence of a hot hand after all.

What it teaches: the mistake isn't always a small sample or a flexible choice. Sometimes the statistical method has a hidden bias that manufactures a false negative rather than a false positive, and it can take decades to notice, since a null result rarely gets the scrutiny a striking one does.

The Dunning-Kruger effect, that the least competent people rate their own ability the most inaccurately high, is one of the most widely cited findings in psychology. Its standard test method sorts people into quartiles by their actual score, then compares each quartile's average self-rating to its actual score. Statisticians later showed this exact method produces a Dunning-Kruger-shaped pattern even when the underlying ability and self-rating numbers are pure, unrelated random noise, because sorting people by the same score used to measure their error creates a spurious correlation, and a regression-to-the-mean effect, on its own. Re-analysed with methods that avoid this, the effect shrinks to far smaller than the original claim.

What it teaches: a real-looking pattern can be partly, or even mostly, a property of the method used to look for it rather than the thing being measured. It's worth asking not just whether a result is significant, but whether the analysis technique itself could produce that exact shape out of noise.

Ahead of the 1936 U.S. presidential election, the magazine Literary Digest mailed 10 million straw-poll ballots and received 2.4 million back, an enormous sample by any standard, and confidently predicted Alf Landon would beat Franklin Roosevelt 57 to 43 percent. Roosevelt won in a landslide, 62 to 37. The mailing list, built from telephone numbers, car registrations, and club memberships, skewed towards wealthier households in the middle of the Great Depression, and a modern statistical reanalysis found an even larger driver: the roughly 76 percent of people who never returned a ballot leaned towards Roosevelt far more than the 24 percent who did.

What it teaches: a sample size in the millions cannot fix a sample that isn't representative of the population it claims to describe. It's the oldest lesson in survey statistics, and still the first thing worth checking about any “huge dataset” claim: huge compared to what, and who's missing from it?

Starting in the late 1960s, fewer than 90 children, all enrolled at a preschool on Stanford's own campus and mostly from financially comfortable, highly educated families, were offered one marshmallow now or two later if they waited. Children who waited longer showed modestly better outcomes years afterwards, and the finding became one of psychology's most famous parables about willpower and self-control. A 2018 study using a much larger, more economically and racially diverse sample of roughly 900 children found the same basic pattern, but at about half the size, and it disappeared almost entirely once researchers accounted for family income and the child's early cognitive ability.

What it teaches: an effect measured in one narrow, unrepresentative group can be genuine within that group and still misrepresent the underlying cause once tested on a broader population. “Delaying gratification builds a better life” turned out to overlap heavily with “growing up with financial security lets a child feel safe enough to wait, and also predicts a better life for other reasons.”

In 1971, Philip Zimbardo randomly assigned 24 male college volunteers to play guards or prisoners in a mock prison in a Stanford basement. Guards, the story goes, spontaneously grew cruel, and the study became one of the most cited demonstrations in psychology that situations, not personalities, corrupt behaviour, taught for decades in introductory textbooks worldwide. There was no control group, no pre-registered hypothesis, and no real statistical analysis, only a dramatic narrative from one uncontrolled run. A 2018 investigation using newly available recordings and interviews found guards had been explicitly coached by the research team on how to be “tough,” directly undercutting the claim that cruelty emerged spontaneously from the situation alone.

What it teaches: a vivid, well-told single case can feel like overwhelming evidence while containing none of the safeguards, a comparison group, blinding, a real sample, that separate a genuine finding from a compelling story. It ran once, on 24 self-selected volunteers, with essentially no data analysis, and still shaped how millions of people were taught to think about human nature.

Between 1924 and 1932, researchers at Western Electric's Hawthorne Works plant varied factory lighting and reported that worker productivity rose no matter which way the lighting changed, even when it got dimmer, supposedly because workers respond to being observed rather than to the change itself. The story became so influential that “the Hawthorne effect” entered the language as a general term for people changing behaviour simply because they know they're being watched. For decades the original data were assumed lost. When economists Steven Levitt and John List finally located and formally analysed the records in 2011, the dramatic pattern largely evaporated: once day-of-week and pay-period effects were accounted for, the productivity swings tracked the calendar far better than they tracked the lighting.

What it teaches: an idea can become permanently embedded in a field's vocabulary and taught for generations before anyone analyses the data it was built on. “Everyone knows this effect is real” and “the data shows this effect” are different claims, and for the study the whole concept is named after, they turned out to say quite different things.

In 1993, 36 college students briefly scored higher on one specific spatial-reasoning task after listening to ten minutes of a Mozart piano sonata, compared with sitting in silence or hearing relaxation instructions. The boost, roughly 8 to 9 IQ-equivalent points on that one task, faded within 10 to 15 minutes, and the original researchers never claimed it applied to general intelligence, children, or long-term development. Media coverage and popular books extended it into “classical music makes babies smarter” anyway, and in 1998 the governor of Georgia budgeted funds to mail every newborn in the state a classical music CD. A later meta-analysis of many replication attempts found the true effect, if any, is small and specific to certain spatial tasks, nothing like the general intelligence boost the popular version claimed.

What it teaches: the size and scope of a claim can grow enormously between the study and the policy built on it without anyone along the way deliberately lying. A short-lived effect on one narrow task, in adults, quietly became a permanent effect on general intelligence, in infants, and a government spent real money on the gap.

In a 1982 Atlantic essay, James Q. Wilson and George Kelling argued that visible disorder, a broken window left unrepaired, signals nobody is watching and invites more serious crime. The essay contained no original data or study; it was a theoretical argument illustrated with anecdotes. It nonetheless became the intellectual basis for “zero-tolerance” and “order-maintenance” policing adopted at city scale, most famously in 1990s New York, and is cited in criminology as one of the most influential ideas in the field's history. Decades later, a National Research Council review and later meta-analyses found little support for the specific claim that disorder itself causes serious crime; the most rigorous analysis found a modest crime-reduction effect from disorder policing, but concentrated in cooperative, community-based approaches rather than the aggressive enforcement the essay's name became attached to.

What it teaches: an idea doesn't need data behind it to become policy at national scale if it's intuitive and confidently argued. The gap between a compelling essay and a tested causal claim was never closed before the theory was already reshaping how millions of people were policed.

Facebook's News Feed ranking historically weighted how often people clicked a link. By 2013 and 2014, publishers had adapted with headlines engineered purely to trigger a click, “click-baiting,” without describing what the story actually contained. Clicks on this content were genuinely high. Facebook's own data showed something else moving the opposite way: people who clicked a click-bait headline spent less time reading afterwards and were far less likely to like, comment, or share once they returned, exactly the signals the click was supposed to predict.

What it teaches: Facebook had to explicitly rebuild its ranking to measure bounce-back rate and post-click engagement, not just the click itself, because the click alone had stopped meaning what it was assumed to mean. The fix wasn't a bigger sample or a better p-value. It was admitting the metric being optimised was never validated against the outcome it stood in for.

Waiting months for every test's true long-term effect is expensive, so Netflix built a “surrogate index,” a statistical model combining several short-term signals into one number meant to predict a longer-term outcome, and checked it directly against 1,098 test arms from 200 real A/B tests. Even this purpose-built, heavily validated model only agreed with the true 63-day result about 95 percent of the time, and correctly identified which tests were actually worth launching only 65 to 79 percent of the time.

What it teaches: if it takes a dedicated model, tested against 200 real experiments, to get to 95 percent confidence that a short-term number predicts a long-term one, assuming an engagement metric will “obviously cascade” down the funnel, with no model and no validation at all, is not a safe default anywhere.