Excess Separability: Nuisance-Controlled Residual-Stream Probing for Benchmark Contamination Detection

Excerpt

arXiv:2608.12652v2 Announce Type: replace-cross Abstract: Benchmark contamination is diagnosed with n-gram overlap, likelihood-based membership inference, or canary strings, and each needs something usually unavailable: the training corpus, a well-chosen test statistic, or foresight at release. A recent alternative reads it off a linear probe on internal activations. We show the natural way to do this does not work, specify one that survives measurement, then find that the correction making it w