InferOpt: Constrained Multi-Objective Search for LLM Inference Configurations

Excerpt

arXiv:2610.04473v1 Announce Type: new Abstract: Serving an LLM means setting dozens of inference-time knobs, from per-layer KV retention to per-layer expert counts. Practice sets them with mechanism-specific heuristics that return a single operating point and do not scale to layer-wise search spaces. We recast inference configuration as constrained multi-objective black-box optimization and build InferOpt, a reusable search framework that requires only variable bounds, a deterministic resource c