Listening to the Wise Few: Query-Key Alignment Unlocks Latent Correct Answers in Large Language Models

Excerpt

arXiv:2410.02343v2 Announce Type: replace-cross Abstract: Large language models (LLMs) routinely fail to output the correct option in multiple-choice question answering (MCQA) while encoding the answer internally. We expose this latent knowledge via the Query--Key (QK) score, defined for an attention head as the inner product between the last-token query and the key at the end-of-line token following option $i$, evaluated before rotary positional embedding is applied. Its argmax identifies a uni