An experiment can clear its own design checks, hit significance, and still leave almost nothing a decision-maker can use. Clearing one validity test does not imply the other four cleared with it. When someone says “the study found X,” the sentence almost never names which of the five tests that finding actually survived.
Reading the Research already leans hard on two of those words: internal validity and external validity. Three more sit beside them: construct validity, statistical conclusion validity, and ecological validity. Treating the first pair as if they covered the rest is how an honest study gets aimed at the wrong question.
Each of the five gets one cited case already on this site. The thread through all of them is practical: what belief does this particular result actually support, and where does that licence stop?
The story in four parts
When the person collecting the outcome knows the hypothesis, their expectations can create a real-looking result that the treatment never produced.
A finding can be airtight in the exact conditions of the original test and disappear after a small change to the setting.
A metric can shift in a clean, statistically solid way while standing in for a construct it never measured.
Many small, individually plausible studies can harden into agreement that one adequately powered test then dissolves.
The evidence at a glance
A 2012 replication of a famous priming study found nothing with infrared timing. The slower-walk result returned only when the person with the stopwatch expected it in advance.
A facial-feedback finding that had replicated across 17 independent labs disappeared as soon as participants knew they were on video.
Bank data showed daily-login engagement moving the opposite way from financial wellbeing, against the usual assumption that the metric tracks loyalty.
Decades of papers treated willpower as a depleting muscle. One adequately powered, pre-registered replication across 23 labs found no meaningful effect.
Internal validity is the narrowest of the five questions. Inside this study alone, did the treatment produce the measured effect, or could a confound, selection pattern, or the measurement process itself explain it just as well? That last route is easy to miss, because collecting the score feels like logistics rather than part of the design.
Why did people only walk slower after old-age words when the experimenter expected it?
One highly cited study reported that people primed with words linked to old age walked more slowly as they left the lab. A 2012 replication timed exits with infrared sensors instead of a human stopwatch and found no priming effect. A second experiment brought the effect back, but only when the person holding the stopwatch already believed participants would walk slower.
What it teaches: if the person scoring the outcome can guess the hypothesis and is not blinded to condition, their expectations can seep into the number. Think “how slowly did that person seem to walk.” Blinded, objective measurement is what internal validity is built to protect.
External validity asks whether the result travels. Change the population, the setting, or the moment in time, and does the same effect still appear? A study can be internally perfect (the treatment really caused the outcome in that room) and still describe a thin slice of the world that does not match the decision being made.
Why did the pen-in-the-teeth effect disappear the moment anyone was watching on video?
The original facial-feedback study found people rated cartoons as funnier when a pen held in their teeth forced a smile shape. In 2016, 17 labs and more than 2,000 participants ran a protocol the original author helped design and found no effect. One leading account: unlike the original, the replication labs filmed participants, and being watched appears to suppress the effect.
What it teaches: a failed replication is not always proof the original effect was invented. Sometimes the effect has a real boundary condition (here, being observed) that the first study never tested and that the finding depended on the whole time.
Ecological validity gets mixed up with external validity often enough that the split is worth keeping sharp. External validity asks whether a result generalises to other people or settings. Ecological validity asks whether the task inside the study resembles the real behaviour it claims to stand for. A finding can be internally sound and travel well across populations, and still fail this test if the task was never a realistic stand-in for what people actually do.
Why do people say they'd pay more than they actually do, even when warned about it?
Surveys that ask “would you pay for this?” measure something real. They rarely measure the thing people treat them as measuring. When answering costs nothing, stated willingness to pay runs higher than the amount paid once real money is at stake. Warning respondents about that bias in advance has generally done little to close the gap.
What it teaches: a stated intention answers a real question. It is usually the wrong question for predicting paid behaviour: a lower-stakes task standing in for a higher-stakes one, and the two do not reliably move together.
Construct validity asks whether the number on the page represents the idea people assume it stands for. “Engagement,” “loyalty,” and “satisfaction” are constructs, not directly observable objects. The operational metric behind any of them can be completely real, statistically clean, and still be tracking something else.
Why were a bank's most engaged customers its most financially stressed ones?
One study combined a nationally representative US survey with administrative data from 194,678 accounts at a large Australian bank. Daily app engagement correlated negatively with financial wellbeing (r = -0.34). The most digitally engaged customers were the least financially well, the opposite direction the common “engagement equals loyalty” reading predicts.
What it teaches: the correlation itself is solid. The failure sits one level up, in what “engagement” was taken to mean. A customer who checks a balance daily from anxiety produces the same metric as one who checks from satisfaction, and the metric alone cannot separate them.
This is the question the other four leave alone. Even with a clean design, a fitting population, and a metric that really captures the intended construct, an underpowered test can invent an effect that is not there, or miss a real one entirely. This site's coin-flip report walks through the mechanics with exact numbers. The case below shows the same failure landing on a famous published finding rather than a hypothetical coin.
Why did a finding from 42 people become one of the most famous results in behavioural science?
A 2010 study of 42 participants reported that holding an expansive “power pose” for two minutes raised testosterone, lowered cortisol, and increased self-reported feelings of power. A TED talk viewed more than 20 million times helped make it one of the field's best-known results. A 2015 replication with a properly powered sample of 200 recovered the self-reported feeling of power and found no hormonal or behavioural effect.
What it teaches: a small sample can produce a result that looks clean on the page and still be noise. One of the original co-authors has since said publicly she no longer believes the hormonal and behavioural effects are real.
None of these five failures require dishonesty. Most of the studies above came from careful researchers. They are ordinary routes by which a result can mislead, which is why each one is worth checking on purpose, one at a time.