What a p-value is

A p-value answers a narrow question: if there were really no effect, how surprising would data like ours be?

Step by step

1. Assume the null hypothesis: no difference, no effect.
2. Imagine running the study many times in that no-effect world.
3. The p-value is the share of those imaginary results that would be at least as extreme as what you actually got.

A small p-value (say 0.01) means: in a no-effect world, results like yours would be rare. That makes the no-effect explanation harder to believe.

ExampleYou flip a coin 10 times and get 9 heads. If the coin were fair, getting 9 or more heads (or 9 or more tails) happens about 2% of the time. p ≈ 0.02. Surprising for a fair coin, but not impossible.

The 0.05 line

By convention, p < 0.05 is often called "statistically significant". This line, popularised by R. A. Fisher, is arbitrary. p = 0.049 and p = 0.051 are practically the same evidence.

The ASA statement

In 2016 the American Statistical Association issued its first-ever statement on a single statistical idea. Its six principles include:

• p-values can indicate how incompatible the data are with a specified model.
• They do not measure the probability that the hypothesis is true.
• Scientific conclusions should not be based only on whether p passes a threshold.
• A p-value does not measure the size or importance of an effect.

RememberA p-value tells you how surprising your data would be if nothing were going on. Nothing more.