John List has partnered on field experiments with hundreds of organisations. The tip he keeps handing researchers who want a first collaboration is blunt: never call it an experiment. Say pilot. Say trial. Plenty of partners will run the same design under those labels and still bristle when someone says “experiment” out loud, as if the word itself were slightly improper. That flinch is useful as a signal. What it usually marks is something more expensive than branding: dodging the discipline the word implies, a real control group, a fair comparison, room to be shown wrong.
The story in four parts
List tells researchers to lead with pilot or trial. Plenty of partners treat “experiment” as a word that should not be said in the room.
Rename the work a “pilot,” launch it to everyone, skip the comparison group, and the design slides down the evidence ladder even if the slides still look polished.
Budgets for running a test get debated in detail. The bill for never learning whether the untested version works almost never shows up on the same spreadsheet.
Uber's apology field test and List's early-childhood trial in Chicago Heights both ran because someone treated the design as a real experiment and said so.
The evidence at a glance
In his practitioner guide for researchers pitching organisations, List's opening move is simple: cut the word “experiment” from the pitch and talk about a pilot or a trial.
One field experiment on customer apologies found that saying the same apology again after a second failure made things worse. That is the opposite of the fix most teams reach for first.
List and Heffetz put governments in a category of their own: more resistant to randomised trials than companies that already A/B test as routine practice.
“Experiment” spooks people partly because it is overloaded. Put a researcher, a product manager, and a UX designer in one meeting, and the same noun can point at three methods with three very different standards of rigor. Naming which design is actually on the table does more work than arguing over the label.
Glenn Harrison and John List (the List this report cites throughout) published a framework that sorts field work by how real the subjects, stakes, and setting are. At one end sits a conventional lab: researcher-built room, paid volunteer pool. At the other sits a natural field experiment, people going about ordinary decisions with no idea a study is underway. Between those poles sit usability sessions, live A/B tests, hypothetical survey items, and the rest of applied work, whether the person running it would call any of that an experiment or not.
Realism and control usually trade off. Randomised field experiments (the RCTs and A/B tests in the upper right) are the rare design that gets both at once. Most other points on the chart buy realism by giving up certainty, or buy certainty by stepping out of the real world.
Ask what someone would do, or pay, inside a scenario they only picture. They never actually live through it.
Strength: cheap and fast, and often the only workable option when the situation is too risky or too far off to build for real, including a product that does not exist yet.
Weakness: stated intent and real behaviour diverge on a reliable schedule. Across 29 studies, a meta-analysis found hypothetical willingness-to-pay and willingness-to-accept overstated incentivised values by roughly a factor of three, on average (List & Gallet, 2001).
Real participants get randomly assigned to conditions inside a setting the researcher builds, so one variable can be isolated at a time.
Strength: confounds get the tightest possible leash. When the lab shows an effect, coincidence is an unlikely explanation.
Weakness: the room, the incentives, and often the subject pool are artificial. Holding in the lab is no guarantee of surviving real stakes and real distractions.
A small set of real users tries real tasks on the real product while someone watches, usually with think-aloud commentary.
Strength: fast, dense feedback. Jakob Nielsen's model puts five users at catching roughly 85% of usability problems (Nielsen, 2000), for a fraction of a larger study's cost.
Weakness: no control group and no counterfactual. You learn where people got stuck. You do not learn whether version A converts more than version B, or by how much.
Real people get randomly split across two or more live conditions, and a real outcome gets measured. Product teams call the same design an “A/B test” when it runs inside an app or website.
Strength: random assignment plus real stakes is why people reach for the phrase “gold standard”: a measured gap is much likelier to be caused by the change itself.
Weakness: detecting an effect at all needs enough traffic or participants (see this site's sample-size guidance). Trying one change at a time also crawls when the backlog of candidates is long.
Same randomised skeleton as an A/B test, applied to several factors together: headline, image, and price display in one go, instead of one change per round.
Strength: surfaces interaction effects a queue of single-factor A/B tests would miss, such as a headline that only works beside one particular image.
Weakness: each added factor multiplies the combination count. The same traffic then spreads thinner across more cells, and every individual comparison carries less statistical certainty.
Nobody designs the comparison ahead of time. A policy shift, a price change, or an eligibility cutoff creates something close to random variation on its own, and the researcher spots it afterward.
Strength: realism at its peak: ordinary people making ordinary decisions, with almost no chance the study itself alters what they do.
Weakness: treating the comparison as “as good as random” is an assumption that has to earn its keep. The whole result hangs on how well that assumption holds. See Natural Experiments for how those checks get run.
List lays this out in his practitioner guide for field experiments. Many organizational partners treat the word “experiment” as close to repugnant, so a researcher trying to open a collaboration is better off dropping it and talking about a “pilot” or “trial” instead. He has said the same thing more casually in public. A 2022 post branded it “Rule #1” for anyone approaching organisations, and pointed readers at a companion piece he co-wrote on why governments in particular resist evidence-based policymaking.
Side by side, the peer-reviewed tip and the later public restatement cover the same ground at different distances. The academic paper records professional advice forged across List's long run of standing up organizational field experiments. Eleven years on, the post repeats that advice in plain language for a wider audience. Cite the paper as the load-bearing source; treat the post as its accessible restatement. One claim, told twice.
Swapping labels would be cheap if the test underneath stayed intact. The usual failure mode is that ditching “experiment” travels with ditching the pieces that make a test worth running: a genuine control group, and room for an outcome that might disappoint. A “pilot” rolled out to everyone at once, with no comparison arm and no pre-registered way to fail, sits lower on the evidence ladder this site maps elsewhere, whatever name appears on the slide deck.
List's other practical point about organizational resistance shifts where the cost actually sits. Inside most organisations the debate centres on the price of running a test (time, headcount, a visible miss) and barely touches the opportunity cost of skipping one. That opportunity cost is keeping money and scale behind a programme that has never been measured against a fair alternative. Once that bill is written down, talking yourself out of testing gets much harder.
This site's “Not Testing Is Still a Bet” report covers the product and commercial twin of the same avoidance. Teams ship an untested change to everyone because a proper test on real users feels riskier and less ethical. Usually the risk ranking runs the other way, and the pre/post figure later used to declare a win was never measuring the claim either.
Which is actually riskier: testing a change on some users, or shipping it to everyone untested?
The full report argues that shipping untested gets treated as the “safe” default far more often than the evidence supports. It then walks through cases where the number a team trusted as proof never measured the claim at all.
What it teaches: “We didn't want to experiment on our users” almost always means skipping measurement. The risk did not leave the room; it just stopped being measured. The full report has the reasoning and the cases.
What did a company find once it stopped guessing and ran the test anyway?
This site's experiment blueprint on costly, specific apologies builds on List's Uber field experiment: an organisation that ran the test and got a counterintuitive result back. A repeated identical apology can backfire. That finding only shows up when someone is willing to be shown something unflattering.
What it teaches: Teams that will say “experiment” and mean a real comparison learn things a pilot already dressed as a done deal never surfaces.