Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning

Excerpt

arXiv:2610.02190v1 Announce Type: new Abstract: Step-size selection remains a central challenge in large-scale neural network optimization; conservative steps slow convergence, while aggressive steps can destabilize it. We combine \textbf{Z}ero-and-\textbf{F}irst-\textbf{O}rder optimization~(ZFO) and propose a lightweight framework that decouples direction selection from step-size. ZFO uses a trusted first-order optimizer to determine the direction and performs zeroth-order evaluations only alon