How noisy review is

Peer review is valuable, but it is done by a few busy humans. How consistent is it?

The NeurIPS 2014 experiment

In 2014, the organisers of NeurIPS, a top AI conference, ran an experiment. About 10% of submissions were reviewed by two separate committees, each making its own accept or reject decision.

The committees disagreed on about a quarter of these papers. Looking at the papers one committee accepted, the other committee rejected roughly half of them. Corinna Cortes and Neil Lawrence later analysed the results and found that reviewers' scores did a reasonable job of spotting weak papers, but were poor at picking which strong papers would be most cited later.

A repeat experiment at NeurIPS 2021 found a similar level of disagreement.

What this means for you

• Rejection is partly luck. A rejected paper is not necessarily a bad paper. Revise and resubmit elsewhere.
• Acceptance is partly luck too. Don't treat "accepted at a top venue" as proof a result is right.
• Clear writing reduces noise. Reviewers skim; a paper whose message and evidence are obvious is less likely to be misjudged.

Why have review at all?

Review still catches errors, improves papers and filters out much weak work. Most authors say review made their paper better. It's a useful filter, just not a perfect one.

RememberReview is useful but noisy: about half of accepted papers at one top conference would have been rejected by a second committee. Don't let one decision define your work.