Learning the identity: a case study of how SGD selects among functional decompositions

Excerpt

arXiv:2610.00615v1 Announce Type: new Abstract: One might think that learning the identity function with a deep linear residual network is trivial - the path along residual connections already implements the identity, and so the network need only drive its weights to zero. However, this zero-weight solution is just one point on an entire manifold of population-loss minimizers, each corresponding to a different decomposition of the identity across the network's layers. Although the population los