When you can't randomise

Sometimes you can't randomise: it would be unethical, impossible or too expensive. Researchers then look for natural experiments: situations where something close to random assignment happened on its own.

John Snow and the water companies

In 1850s London, cholera was thought to spread through bad air. The physician John Snow suspected water. Two companies supplied water to the same neighbourhoods, often to houses side by side. One, Lambeth, had moved its intake upstream of the city's sewage; the other, Southwark and Vauxhall, still drew water downstream. Households hadn't chosen their company based on health. Snow found cholera deaths were many times higher among Southwark and Vauxhall customers. It was close to a randomised experiment that nobody designed.

The minimum wage study

In 1992, New Jersey raised its minimum wage; neighbouring Pennsylvania didn't. David Card and Alan Krueger compared fast-food restaurants on both sides of the border, before and after. This difference-in-differences design helped Card win a share of the 2021 Nobel Prize in economics.

Common natural-experiment designs

• Difference-in-differences: compare the change over time in a group that was affected with one that wasn't.
• Regression discontinuity: compare people just above and just below a cut-off, like a scholarship score.
• Lotteries: when places are given out by lottery, winners and losers are randomised by nature.

The catch

Natural experiments rest on an assumption: that the groups were comparable. Always ask what else differed.

RememberCan't randomise? Look for a situation where chance, a border or a cut-off did it for you, and question whether the groups truly match.