Risk-Controlled KV-Cache Eviction: From Memory Budgets to Risk Targets

Excerpt

arXiv:2609.27981v1 Announce Type: new Abstract: KV-cache eviction is typically evaluated through average quality-memory trade-offs, yet a small average loss can hide requests whose utility degrades materially. We reformulate eviction as a deployment risk-control problem: a material degradation occurs when eviction lowers task utility by more than a deployment-specified tolerance relative to full-KV inference on the same request, and deployment risk is the population frequency of such events. Giv