Risk-Aware Adaptive Evaluation: Finding High-Impact Failures Under Limited Budgets

Excerpt

arXiv:2609.38914v1 Announce Type: cross Abstract: Evaluating interactive agents is expensive. Agent behavior is stochastic, so reliability must be measured over repeated trials, but failures are rare and differ widely in how much they matter. Standard benchmarks spend this budget uniformly: a read-only lookup is sampled as often as an irreversible payment action. We instead formulate evaluation as a sequential allocation problem. Given a fixed trial budget and a set of scenarios whose failure be