EDGC: Entropy-driven Dynamic Gradient Compression for Efficient LLM Training

Excerpt

arXiv:2511.10333v2 Announce Type: replace Abstract: Training large language models (LLMs) at scale incurs substantial communication overhead, while static gradient compression cannot adapt to gradient evolution and may degrade model quality. We propose EDGC, an entropy-driven dynamic gradient compression framework that adapts compression ranks to gradient entropy during training. EDGC combines efficient entropy estimation through gradient sampling, a theoretical model relating entropy to compres