Compressing History into Memory: Distilling Transformers into Recurrent Transformers

Excerpt

arXiv:2606.21562v2 Announce Type: replace-cross Abstract: Transformers are AI's workhorse but their computational cost becomes prohibitive when processing long sequences. We target long-horizon streaming vision and robotics applications, where it is particularly impractical to store and maintain a history of observations. Recurrent Transformers address this limitation by maintaining fixed-size memory but their performance lags behind that of transformers operating over the full observation histo