Natural Experiments

When you can't flip a coin

To show A caused B, the textbook move is a randomised controlled trial: flip a coin, treat one half of the sample, compare what happens. Plenty of questions worth asking refuse that design. You cannot randomly place an MLB scouting director in a city, force a takeover of one firm rather than another, or rewrite a country's statute for the sake of a study.

Researchers still need a causal answer they can defend. Each entry here walks through one published study that found a substitute for the coin flip, and says plainly what that substitute can and cannot carry.


How do you prove distance biased 30,000 draft picks when nobody can randomly assign where a scout lives?

Major League Baseball's amateur draft already behaves like a natural experiment nobody designed. About thirty scouting directors grade thousands of prospects each year. Hiring and hometowns were never randomised. In 2022 a research team treated that setup as data. Holding measured skill fixed, does living nearer a team's scouting director raise the odds of being drafted by that club? And do those nearer picks cost the club on the field?

A randomised trial cannot settle either question. Directors' addresses and prospects' hometowns sit outside anyone's experimental control. What follows is the identification strategy they used in place of a control group, and the checks that make the claim worth taking seriously.

Observational Study Machine Learning Controls
Propinquity Effect Selection on Observables Robustness Checks
The obstacle

The causal fork is sharp. Either distance itself tilts how a director drafts, or nearer players look better because directors cover their own region more carefully and surface genuine talent there. Random assignment would close the fork: place directors in markets by lottery, or move prospects' hometowns at random, then watch whether draft picks shift.

Those designs stay off the table. Clubs hire directors for operational reasons, and a prospect's hometown is locked in long before any study exists. The only usable path is the draft record the league already produced.

The workaround

The paper studies MLB drafts from 2000 through 2019: about 30,000 drafted players drawn from a pool larger than a million. Machine learning built a flexible forecast of each prospect's draft-worthy skill from the file a real department would already hold: game statistics, scouting grades, and physical measurables.

Once that skill score is held fixed, the test shrinks. Take two prospects the model rates as equal. Does the one living closer to the scouting director still get selected more often? The model is doing statistically what random assignment would have done up front: stripping out the skill confound that could otherwise explain the whole geographic pattern.

Why it's still convincing

One controlled correlation would be thin evidence. The paper leans on more structure than that. The proximity bias concentrates in later rounds, where a director's personal call matters most and outside ranking pressure is lightest. Early rounds, where national consensus rankings bind tighter, show a weaker version of the same pattern.

A leftover measurement glitch would tend to appear wherever skill is noisy. It would not line up so cleanly with the organisational moment when one person's judgment carries the most weight. Effects that intensify where the mechanism says they should, and fade where it says they should not, do more work than a single average percentage ever could.

The finding

With the model's skill estimate held constant, a player is about 7.1% more likely to be drafted by a given team for every 1,000km closer he lives to that team's scouting director. The same estimate puts him about 4.9% more likely for every 1,000km closer to the team's home city.

The cost shows up in performance. Prospects drafted with that proximity edge were 38% less likely to appear in even one MLB game than similarly drafted players without it, and played roughly 25 fewer games on average. Geography did more than reorder pick slots. It pulled measured roster quality down.

What this method can and can't prove
What it proves
  • A stable link between geographic closeness and draft selection that remains after controlling for every skill signal the dataset records.
  • A shape that fits bias rather than noise: the effect peaks where scouts have the freest hand, and softens where national rankings constrain them.
  • A performance penalty on proximity-favoured picks, so the pattern carries a roster cost on top of any reordering of names.
What it can't rule out
  • Skill dimensions that exist in the world but never entered the model's inputs could still explain part of the gap. Observational controls never guarantee the full set of confounds a true random assignment would have erased.
  • At time of writing the piece remains an NBER working paper rather than a peer-reviewed journal article, so treat the exact percentages as provisional until that status changes.
  • The sample is one league across two decades. The same magnitude in another sport, country, or era is outside what this paper alone tests.
Source

Ahmadi, M., Durst, N., Lachman, J., List, J. A., List, M., List, N., & Vayalinkal, A. (2022). “Nothing Propinks Like Propinquity: Using Machine Learning to Estimate the Effects of Spatial Proximity in the Major League Baseball Draft.” NBER Working Paper No. 30786

Figures above are paraphrased from the paper's public summary and independent academic write-ups. The full text sat behind this site's network restrictions, so nothing here is a verbatim quotation from the PDF.

See also: Propinquity Effect, the psychological mechanism this draft result scales up, first measured in a 1950s housing complex. How Strong Is That Evidence? places designs like this on the site's ladder from raw correlation to a randomised trial. Proximity Gets Mistaken for Merit reads the housing study and this draft paper as a single argument.


Who's behind this

I'm Grant Burrow. I've spent years running behavioural experiments inside real consumer products, the kind written up as Harvard Business School case studies, not just textbooks. I care about the applied part: not reciting theory, but showing you the sign, the screen, the menu, and exactly why it works.


One field note, whenever there's something worth sending

Real examples, plainly explained. No spam, no sleazy tricks.

Unsubscribe in one tap, anytime.