Sven: Singular Value Descent as a Computationally Efficient Natural Gradient Method

Excerpt

arXiv:2604.01279v2 Announce Type: replace-cross Abstract: We introduce Sven (Singular Value dEsceNt), a new optimization algorithm for neural networks that exploits the natural decomposition of loss functions into a sum over individual data points, rather than reducing the full loss to a single scalar before computing a parameter update. Sven treats each data point's residual as a separate condition to be satisfied simultaneously, using the Moore-Penrose pseudoinverse of the loss Jacobian to fin