Scaling Attention Head Analysis via Gradient-Based Attribution in Context-Aware Machine Translation

Excerpt

arXiv:2609.28117v1 Announce Type: new Abstract: In this paper, we introduce a gradient-based head attribution strategy where the Token-level Max-Margin loss is backpropagated to the attention maps. This framework enables a large-scale causal analysis of attention heads, making it suitable for LLMs. We evaluate our method on the task of disambiguation in Context-aware Machine Translation, where we analyze 50 phenomena across 4 models and 4 language directions. We empirically show the alignment of