Muon Is Theoretically Wrong For Convolutions, But Empirically Effective

Excerpt

arXiv:2610.07103v1 Announce Type: new Abstract: Muon, an optimizer known for its efficiency, has a clear interpretation for matrix-valued updates, but convolutional kernels are stored as four-dimensional tensors. Standard implementations reshape these tensors into matrices, a shortcut which breaks the theoretical understanding behind Muon. To investigate this, we formalize the corresponding optimization objective directly in convolutional operator geometry and introduce Convolutional Newton-Schu