Shifting Mechanisms: How Positional Encoding Choice Shapes In-Context Retrieval

Excerpt

arXiv:2609.38530v1 Announce Type: cross Abstract: Language models increasingly use architectures that vary attention span and positional encoding across layers, such as applying RoPE with sliding-window attention and NoPE with global attention (SWA NoPE). However, how these choices shape in-context retrieval remains unclear. To study this question, we take a mechanistic view, tracing how positional encoding (PE) choice shapes the internal mechanisms models use for in-context retrieval. Across 22