Local and Global Regimes of Geometric Complexity in Language Model Representations

Excerpt

arXiv:2608.14361v1 Announce Type: new Abstract: Intrinsic dimensionality (ID) is widely used to probe the representational complexity of language models, but it remains unclear whether ID differences reflect properties of language itself or artefacts of how the underlying dataset was constructed. In this paper, we focus specifically on how lexical diversity, the number of unique last-token items present in a dataset, affects ID estimates of that dataset. We find a scale-dependent transition betw