When In-Distribution Gains Fail: Evaluating Weak-to-Strong Reward Models under Preference Shift

Excerpt

arXiv:2605.25629v3 Announce Type: replace-cross Abstract: Weak-to-strong (W2S) generalization is a promising framework for scalable oversight, yet existing evaluations often test students under matched train-test distributions. Therefore, we study W2S preference learning under zero-shot distribution shift and find that strong students trained on weak preference labels can appear successful in-distribution while failing to transfer across preference datasets. We provide evidence for a representat