Why small studies mislead

Statistical power is the chance that a study will detect an effect if it is really there. A common target is 80%.

Low power is common

In 2013, Button and colleagues estimated that the median power of studies in neuroscience was about 21%. Most studies had less than a one-in-four chance of detecting the effects they were looking for.

Low power does more than miss effects

You might think a weak study just finds nothing. Worse things happen:

1. Inflated effects. With a small sample, only results that happen to be unusually large cross the significance line. So the effects that get published are exaggerated. This is sometimes called the winner's curse.
2. More false positives among the "hits". When power is low, a larger share of significant results are flukes.
3. Failure to replicate. A bigger follow-up study finds a much smaller effect, or none.

ExampleA small study of 15 students finds a new study method raises scores by 20 points, p = 0.04. A follow-up with 500 students finds 3 points. The first study wasn't necessarily dishonest; small studies that "win" tend to overestimate.

What to do

• Be sceptical of surprisingly large effects from small samples.
• Look for replications and larger studies.
• In your own work, plan the sample size before collecting data.

RememberSmall studies don't just miss effects; when they find one, it's often exaggerated. Big claims need big samples.