Blinding

People's expectations change results, even when they try to be fair. Blinding hides who got what, so expectations can't leak in.

Clever Hans

In the early 1900s in Berlin, a horse called Clever Hans seemed to answer arithmetic questions by tapping his hoof. In 1907, the psychologist Oskar Pfungst showed that Hans was reading tiny, unconscious cues from the questioner, who relaxed when the right number of taps was reached. When the questioner didn't know the answer, Hans failed. The questioner's knowledge was the hidden variable.

Levels of blinding

• Single-blind: participants don't know which group they're in. This controls the placebo effect.
• Double-blind: neither participants nor the people measuring outcomes know. This also controls observer bias.
• Blind analysis: the analyst decides the method before seeing which group is which.

In ML and evaluation

• Human ratings of model outputs should hide which model produced which output, and shuffle the order.
• Peer review is often double-blind for the same reason.
• If you label test data yourself while knowing your model's predictions, your labels may drift toward them.

RememberIf someone's expectations could change a measurement, hide the information from them.