Beyond Accuracy: Prefix-Invariant Realizations of Low-Precision Fast Matrix Multiplication

Excerpt

arXiv:2609.39816v1 Announce Type: new Abstract: Fast matrix multiplication saves multiplications through exact cancellation, but rounding sums that mix token rows can leave contributions from later tokens in earlier language model outputs. This threatens prefix invariance, which multiple-choice likelihood scoring relies on: a scored likelihood must depend only on its allowed prefix. On Qwen2.5-14B-Instruct, two fast FP8 realizations repaired to ordinary-looking accuracy still change the answers