What Does a Harness Repair? A Preregistered Study of Visibility, Baseline Adequacy and Evaluation Defects

Excerpt

arXiv:2610.05533v1 Announce Type: cross Abstract: Harness search keeps a change to the prompts, reasoning switches, token budgets or parsers around a frozen model if the change raises a score. Such a gain can come from answers the parser could not read before, a weak comparison, or a defect in the evaluation. We preregistered a study of where these gains come from, with three small models, three benchmarks, replication and test partitions, a GEPA search arm and six evaluation defects injected on