Will the Judge Flip? Predicting Position-Sensitive LLM Judgments from Residual Stream Activations

Excerpt

arXiv:2610.07115v1 Announce Type: new Abstract: The order in which candidate responses are presented can change an LLM judge's verdict. Detecting such a position flip ordinarily requires judging each pair in both orders, which doubles the number of judgments. We investigate whether residual stream activations recorded immediately before the initial verdict can predict a flip. We use nested grouped cross-validation to evaluate regularized linear probes on 534 JudgeBench pairs for three Qwen3 judg