Show HN: Shard – Distributed AI Inference Across Multiple Computers
Hobbyist end of the theme: pooling commodity machines for one model.
3 items · 3 sources · 3 days
Operational story trace
Follow in this browser to see new updates on your Live feed.
Latest change
AWS launched the SageMaker HyperPod Global Inference Gateway, one managed endpoint that routes requests across HyperPod clusters in one Region or several and fails over between them.
This is a loose grouping, not one event: three unrelated items from a hobby project, an arXiv paper and AWS share one theme, routing LLM inference across more than one pool of compute. It began with Shard, a Show HN project that splits inference across several computers, and continued with Cap-and-Track, a paper on dispatching requests when fleet capacity shifts.
Arc
Hobbyist end of the theme: pooling commodity machines for one model.
Research view of the same problem: dispatch that tracks shrinking or growing capacity.
Managed-cloud version: a single AWS endpoint with cross-cluster and cross-Region failover.
Hobbyist end of the theme: pooling commodity machines for one model.
Research view of the same problem: dispatch that tracks shrinking or growing capacity.
Managed-cloud version: a single AWS endpoint with cross-cluster and cross-Region failover.
What to watch — open questions
Storylines are threaded mechanically from the feed: stories that share a distinctive anchor across multiple days and sources. Each item links to its original source. The evidence trace, current state, and open questions are written by the editor routine and refreshed whenever a new beat lands.