Which and When to Admit: Gradient Admission for Data-Centric Small Language Model Finetuning

Excerpt

arXiv:2610.07553v1 Announce Type: new Abstract: LoRA fine-tuning adapts small language models (SLMs) to heterogeneous instruction data within a low-rank update subspace, making it vulnerable to three structural problems: conflicting gradients that cancel, static data selection that cannot track evolving learning dynamics, and subspace saturation that causes later updates to overwrite useful directions. We argue that effective adaptation therefore requires controlling which data-induced gradients