Benchmark
Definition
A fixed test set used to compare models — which measures one thing well and everything else badly.
Benchmarks are indispensable and routinely over-read. A score is evidence about performance on that distribution, and once a benchmark is popular enough to optimise against, it stops measuring generalisation and starts measuring how hard people tried on it.