How much of the weight-space perception gap is actually symmetry? Evidence from ~1.8M fitted SIRENs [R]

Excerpt

I’ve been looking at a fairly basic question in weight-space learning that I don’t think gets separated cleanly enough: Why does reading semantics directly from neural network weights work pretty well when the networks share an initialization, but collapse when the networks are fitted independently? The usual explanation is parameter symmetry. Permute hidden units, flip equivalent signs, etc., and two parameter vectors can represent the same function while looking completely different to a downs