Safety is Contextual, LLM-Judges Are Not: Navigating the Rigid Priors of Evaluators

Excerpt

arXiv:2606.07874v2 Announce Type: replace Abstract: LLMs-as-judges are the primary way to evaluate safety at scale. Despite their importance, LLM-judges themselves are rarely evaluated beyond human agreement in simple, static benchmarks. We therefore investigate two under-explored but crucial properties of LLMs-as-judges: their sensitivity to in-context information, and their steerability to differing safety definitions, which may not align with their internal safety priors. We evaluate the safe