Left-Branching Transformers Excel at Right-Branching Languages: Data Shapes Word Order Preferences in Language Models

Excerpt

arXiv:2608.15129v1 Announce Type: cross Abstract: We systematically compare word order preferences in decoder-only language models across 192 artificial languages and typologically diverse natural languages. On artificial languages, models exhibit a left-branching preference that aligns with neither natural language universals nor human word order learning biases. On natural languages, monolingual models show no clear base word order bias at small scales, but as data grows, a preference for righ