Retrieval-Centric Deep Learning in Growing Nonparametric Neural Networks

Excerpt

arXiv:2610.03858v1 Announce Type: cross Abstract: We investigate a general-purpose layer for deep learning that, instead of compressing arbitrary-size training data into fixed-size weight matrices, stores a new pair of key-value representations for every data point during training, and retrieves and recombines these representations through an attention mechanism at inference time - resulting in a growing neural net (NN). While Irie et al. (arXiv:2202.05798) have put forward this perspective from