The Conflict Between Logic and Memory: Learning Higher-Order Interactions in Shallow MLPs

Excerpt

arXiv:2610.00403v1 Announce Type: new Abstract: A network can fit its training examples while failing to recover the rule that generated their labels. We examine this separation in single-hidden-layer multilayer perceptrons (MLPs), using synthetic tasks that control interaction order and the presence of nuisance inputs. We establish elementary benchmark properties: pure parity contains no predictive lower-order marginals, admits an exact Bayes posterior, and can be represented on clean latent in