Rethinking the Evaluation of Harness Evolution for Agents

Excerpt

arXiv:2607.12227v4 Announce Type: replace Abstract: Harness evolution is an iterative search procedure that repeatedly evaluates and revises candidate harnesses used for LLM agents using task feedback. We revisit the evaluation of such automatic harness evolution procedures and identify two fundamental issues in the protocol. First, prior work does not compare these approaches with simple task-level search baselines under matched feedback and inference budgets. Second, prior work searches for ha