Targeted Retrieval, Compact Representations: How CoT Reasoning Improves Long-Context Counting

Excerpt

arXiv:2609.38958v1 Announce Type: cross Abstract: Large language models (LLMs) have been rapidly improving in long-context tasks, powered by Chain-of-Thought (CoT) reasoning. However, the internal mechanisms underlying this improvement remain unclear. We investigate these mechanisms through a needle-in-a-haystack (NIAH) counting task, where an LLM is asked to count the number of records dispersed in a long text. Across twelve model comparison groups, Thinking (or reasoning) improves counting acc