Geometric and Behavioral Stratification in Transformer Residual Streams

Excerpt

arXiv:2608.12447v2 Announce Type: replace-cross Abstract: Trained transformer models develop privileged bases: coordinate axes whose statistics differ from the rest of the residual stream. But what kind of direction does such a basis select? We investigate the prediction direction, the unembedding direction of the token a model currently predicts, and find that it functions as a content-defined privileged anchor. Measured with respect to this anchor, residual-stream variation is geometrically an