Exact Quantile Balancing and Load-Error Injection for Mixture-of-Experts

Excerpt

arXiv:2609.28053v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) training requires global load balance to prevent expert under-utilization and local balance for efficient expert-parallel execution. Existing distributed Quantile Balancing (QB) uses shard-dependent or approximate global quantiles, while token-independent expert biases cannot ensure microbatch-level balance. We introduce Exact Quantile Balancing (EQB), which computes exact global-batch BF16 quantiles with negligible commu