Departments and large organisations already say they practise “evidence-based policy.” In a 2024 Nature paper, economist John List argues that the slogan usually puts the study first and the eventual rollout second.
The cited numbers often come from a small pilot that was never asked to work under national conditions. Legislation then inherits a result the study was never built to underwrite.
List's remedy reverses that sequence. Build the evidence-generating study around the constraints of the full-scale programme from day one: ordinary staff, the whole population, and the size the policy will actually reach. He calls the product of that design policy-based evidence.
The story in four parts
Most of the evidence on the slide came from a small trial that never had to survive national delivery.
The same programme, once expanded to a whole population, often keeps only a fraction of its pilot effect. List calls that drop the voltage effect.
A country-wide rollout cannot hire enough of the pilot's strongest staff to keep the same quality everywhere.
Use realistic staffing, the full population, and true operating scale in the earliest evidence-generating trial.
The evidence at a glance
List's 2024 Nature paper labels it the voltage effect: a real pilot result that reliably shrinks once the same programme runs across a full population.
List and Heffetz describe governments as unusually resistant to randomised policy trials, while companies treat ad A/B tests as routine.
A national programme cannot recruit enough of the pilot's own best performers to deliver the same quality in every site.
Scared Straight-style programmes kept winning budgets for decades on a compelling narrative, without a controlled comparison.
A typical policy pilot sits with an enthusiastic delivery team, a small and often self-selected participant group, close oversight from the researchers who designed it, and a short enough window that the novelty of being studied still holds. None of that is dishonest. None of it matches the conditions of a permanent programme delivered by an ordinary public agency with ordinary staff and ordinary funding.
This site already treats the general version of that gap as the voltage effect: measured impact that shrinks, sometimes to a fraction of the pilot size, once rollout is real. See Reading the Research: the voltage drop at scale and the field session Why your best pilot is the one most likely to lose its voltage for the full mechanism. Government policy adds a second step. A pilot with thin external validity still gets written into permanent legislation or a national programme. By the time anyone asks whether it will hold at this size, with this staff, for this whole population, the funding and political commitment are already spent.
The site's cautionary case for the same pattern is “Scared Straight” programmes. A plausible intervention scaled nationally and kept being funded for decades on the strength of a story rather than a control group. List's argument here is the broader version of that failure. Badly designed pilots are only part of it. A properly randomised pilot can still produce evidence that never answered “what happens at scale,” because the study was never built to answer that question.
List wants the usual sequence reversed. Drop the convenient small study that later hopes, or lobbies, for scale. Put the scaling constraints into the evidence-generating study from day one. Delivery quality in the trial should match what the full rollout can actually hire, not a specially recruited pilot team. The sample should span the full range of people the policy will eventually serve, including the hard-to-reach and less motivated. The design also needs power to detect general-equilibrium effects (prices, capacity, spillovers onto people not directly treated) that only appear once a programme runs at real scale.
“Policy-based evidence” names that product: evidence whose design already answers to the shape of the eventual policy. “Evidence-based policy” works the other way: the policy is assembled afterwards on a study that only ever answered a narrower, more convenient question. List frames the point as a call for social scientists to change how they design studies meant to inform public policy. He is not asking readers to distrust every existing finding.
A study built this way costs more, takes longer, and is politically harder to approve than a small, fast, encouraging pilot. That cost structure explains why the default stays small.
Why is a government far more reluctant to run a fair test than a company is?
List and co-author Ori Heffetz argue that governments resist randomised policy trials even when a contested question would be settled by one. Holding a policy back from some citizens at random reads as unfair in a way a company's A/B test on a marketing email never does, even though the statistical logic matches.
What it teaches: The barrier to scaled, well-designed evidence in government is often political rather than technical. The randomisation step itself needs its own political and ethical case before the study can start.
What happens when a pilot's best people aren't the people running it nationally?
One recurring driver of the voltage effect, already covered on this site as the general pattern, is structural. Small pilots often draw unusually motivated or skilled teachers, caseworkers, or delivery staff, because a handful is enough. Expanding the same programme to a full population requires far more people than the pool of similarly exceptional workers can supply. Average delivery quality falls as the programme grows.
What it teaches: Part of a pilot result is a result about the specific people who ran it. Evidence meant to survive scaling has to test with realistically staffed delivery. Otherwise it measures a programme that cannot actually be run at the size it is meant for.