Optimizing 124B MoE inference on a single Nvidia DGX Spark
Vendor guide on fitting a 124B MoE onto a single DGX Spark.
4 items · 3 sources · 4 days
Operational story trace
Follow in this browser to see new updates on your Live feed.
Latest change
Hugging Face (Nov 18) describes expert streaming in Transformers that runs a 180B-parameter MoE in 32GB of RAM, the largest model-to-memory ratio in this thread.
This is a pattern across independent work, not one event: early November brought an NVIDIA post on fitting a 124B MoE onto one DGX Spark and a Show HN project, MoEShift, claiming faster MoE inference with half the RAM. A Nov 13 arXiv paper then took the same data-movement problem to TPUs with a fused kernel that also balances uneven expert loads.
Arc
Vendor guide on fitting a 124B MoE onto a single DGX Spark.
Hobbyist project claims the same memory win on Mac, at half the RAM.
Moves the problem to datacenter TPUs and adds uneven expert load as a second bottleneck.
Brings expert streaming into mainstream Transformers at a 180B-to-32GB ratio.
Vendor guide on fitting a 124B MoE onto a single DGX Spark.
Hobbyist project claims the same memory win on Mac, at half the RAM.
Moves the problem to datacenter TPUs and adds uneven expert load as a second bottleneck.
Brings expert streaming into mainstream Transformers at a 180B-to-32GB ratio.
What to watch — open questions
Storylines are threaded mechanically from the feed: stories that share a distinctive anchor across multiple days and sources. Each item links to its original source. The evidence trace, current state, and open questions are written by the editor routine and refreshed whenever a new beat lands.