Sanity checks

A sanity check asks a simple question: does my method's output actually depend on the thing it claims to measure?

A famous example

Saliency maps are pictures that highlight which pixels a neural network "looked at". In 2018, Adebayo and colleagues ran simple sanity checks on several popular methods. They randomised the model's weights, destroying everything it had learned, and also trained models on random labels. Some methods still produced nearly the same maps. Those maps were mostly showing edges in the image, not the model's reasoning.

The methods looked convincing for years. Two simple checks showed some of them were not measuring what people thought.

Sanity checks you can run

• Randomise the model: does the explanation or output change?
• Randomise the data or labels: does performance drop to chance?
• Overfit a tiny batch: a correct training setup should drive the loss on a handful of examples close to zero. If it can't, there's a bug.
• Check by hand: look at 20 random inputs and outputs yourself.

Why they matter

Sanity checks are cheap and catch bugs that no amount of statistics can. Many published errors would have been caught by an afternoon of them.

RememberBreak your method on purpose. If the output doesn't change when it should, the method isn't measuring what you think.