Contrastive Attention Mitigates Spectral Bias in Spiking Transformers

Excerpt

arXiv:2610.01403v1 Announce Type: new Abstract: Spiking Transformers merge the energy-efficiency of spiking neural networks (SNNs) with the representational power of self-attention, creating a promising architecture for high-performance, energy-efficient computation. However, a performance gap persists versus its counterparts in artificial neural networks (ANNs). Unlike prior works attributing this to binary activations, we reveal that both spiking neurons and spiking self-attention (SSA) act as