LLM Digest
Subscribe

AI Storyline

4 items · 3 sources · 4 days

View as JSON

Operational story trace

MoE Inference

Latest change

Hugging Face (Nov 18) describes expert streaming in Transformers that runs a 180B-parameter MoE in 32GB of RAM, the largest model-to-memory ratio in this thread.

Earlier contextThe story so far

This is a pattern across independent work, not one event: early November brought an NVIDIA post on fitting a 124B MoE onto one DGX Spark and a Show HN project, MoEShift, claiming faster MoE inference with half the RAM. A Nov 13 arXiv paper then took the same data-movement problem to TPUs with a fused kernel that also balances uneven expert loads.

editor-curated · source-linked

Arc

Nov 2Nov 18 · now
SMALL HARDWARE · Nov 2–3
Large MoEs squeezed onto a DGX Spark and a Mac
2 sources · show sources ▾
KERNELS · Nov 13
Fuse-All kernel targets data movement and expert imbalance on TPUs
1 source · show source ▾
NOW · Nov 18
Transformers streams experts: 180B parameters in 32GB of RAM
1 source · show source ▾

What to watch — open questions

  • What tokens-per-second does Transformers expert streaming reach at 32GB, and on which storage tier?
  • Do the DGX Spark and MoEShift results publish RAM and throughput figures others can reproduce?
  • Does the Fuse-All kernel's load balancing ship in a public TPU serving stack?
How this thread was built
editor wrote the arc · 3 beats

Storylines are threaded mechanically from the feed: stories that share a distinctive anchor across multiple days and sources. Each item links to its original source. The evidence trace, current state, and open questions are written by the editor routine and refreshed whenever a new beat lands.