AF-Muon: An AdamW-Free Muon Optimizer for Tied-Embedding Models

Excerpt

arXiv:2610.01395v1 Announce Type: new Abstract: Muon improves large-scale training by applying a spectral-norm steepest-descent update to matrix parameters, but practical models also contain parameter blocks that do not fit dense-matrix geometry. One important case is the tied vocabulary table, which appears in language models and other token generators and can receive multiple structurally different gradient sources, from sparse input lookups to dense output-classifier updates. In the reference