The Review Lottery: Calibrating an Observational Estimator of Peer-Review Noise (ICLR 2017-2025)

Excerpt

arXiv:2610.06591v1 Announce Type: new Abstract: How much of a conference accept/reject decision would change if the same paper were reviewed by a different set of reviewers? Running a second independent program committee is the gold standard for answering this, but it is prohibitively expensive: done only twice (NeurIPS 2014 and 2021). We build an observational estimator of this quantity from public review data alone, calibrate it twice, and apply it to nine years of ICLR (2017-2025; 36,113 pape