LLM Digest

Story

arxiv_cs_cl · Nov 2, 2026 · paper

Source brief

Sonar-MoE: In-Backward Expert Updates for Memory-Efficient Mixture-of-Experts Training

arxiv.orgNov 2, 2026
original source linked

In brief

Most parameters of a Mixture-of-Experts (MoE) model belong to its experts, and standard training keeps every expert gradient in GPU memory until the optimizer step. This memory grows with the number and size of the experts, so it rises as MoE models scale. We introduce Sonar-MoE,

Read the original at arxiv.org →Open in live feed Read that day’s brief

Sonar-MoE: In-Backward Expert Updates for Memory-Efficient Mixture-of-Experts Training

Earlier in this thread 4 items

MatMoE: Matryoshka Mixture-of-Experts with Dynamic Mixed-Precision Quantization for Efficient Inference

Multi-View MoE: Letting Every Router Shape Expert Computation

Rimuru: Unified Memory Management for Modern LLM Serving via Elastic Expert Offloading

Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data