The CEA architecture is a bigger deal than I initially thought

Excerpt

I initially saw CED as just an efficiency improvement, but the more I read about it, the more it feels like an inference architecture leap. The encoder/decoder split has some pretty interesting implications for GPU pooling. Instead of treating every GPU the same, you could have prefill-specialized GPUs for the encoder and decode-specialized GPUs for the decoder, each optimized for a different part of inference. Or just using more modern GPUs for the prefill phase and old HBM cards for decode in