LightMTP: Lightweight Latent Multi-Token Prediction

Excerpt

arXiv:2610.06031v1 Announce Type: cross Abstract: Next-token prediction (NTP) is the standard pretraining objective for large language models, yet it provides an explicit training signal only for the immediate next token, which can lead models to exploit local patterns instead of capturing longer-range structure and ideas. Multi-token prediction (MTP) addresses this by training models to predict several future tokens. However, existing MTP methods often introduce a large number of new parameters