Revisiting the Efficient Channel Attention paper (2019, 12k citations) - the central hypothesis isn't quite right [D]

Excerpt

ECA was positioned as a successor to SE . The idea behind ECA is quite simple. Unlike SE which reduces the channel means into a smaller hidden layer, it directly uses a 1d convolution kernel on the channel means themselves, avoiding the need for dimensionality reduction. The results are undeniable: ECA is a clear improvement over SE. The authors claim that cross-channel interaction is a key ingredient. But on a conceptual level, the design of ECA doesn't make much sense. Let's take a step back.