One QK Channel, Many Sources: Tracing Low-Precision Attention Collapse

Excerpt

arXiv:2608.02091v2 Announce Type: replace Abstract: A bfloat16 transformer can train normally, then collapse abruptly. Prior work links collapse to structured attention errors and shows QK normalization disrupts their compounding. Distinct low-precision errors trigger the same collapse, leaving unclear whether each needs a fix at its source or one shared route can be blocked instead. We isolated the fault behind a reproduced GPT-2-class collapse to the streaming-softmax accumulator, where an fp3