p-hacking and forking paths

Every analysis involves choices: which outliers to drop, which variables to control for, when to stop collecting data, which outcome to report. If those choices are made after seeing the data, false results become very easy to find.

p-hacking

p-hacking means trying different analyses until one gives p < 0.05, then reporting only that one. It is often done without bad intent: each choice seems reasonable at the time.

In 2011, Simmons, Nelson and Simonsohn showed how strong this is. By combining just four common flexible choices (two outcome measures, adding a control variable, collecting more data if p was close, and dropping a condition), the chance of a false "significant" result rose from 5% to about 61%. They even "showed", with real data, that listening to a particular Beatles song made people younger.

The garden of forking paths

Andrew Gelman and Eric Loken pointed out that you don't need to try many analyses for this to happen. If your choices would have been different had the data looked different, the p-value is already misleading. Every dataset offers many forking paths.

The fix

• Decide the analysis before you see the data. Write it down (pre-registration).
• Report everything you tried, not just what worked.
• Separate exploring from confirming. Explore freely on one part of the data, then confirm on new data.

RememberIf the analysis was chosen after seeing the data, the p-value is too optimistic. Decide first, then look.