Story

arxiv_cs_lg ยท Nov 18, 2026 ยท paper

Source brief

Cap-and-Track: Optimizing LLM Inference Request Dispatch Across Shifting Capacity

arxiv.orgNov 18, 2026
original source linked

In brief

A fixed dispatch rate for LLM inference becomes stale as serving fleets resize and request sizes change. Generative output length is unknown at admission, and shared provider pools can lose capacity. We present Cap-an...

Continue reading

Read the original at arxiv.org โ†’Open in live feedRead that dayโ€™s brief

Earlier in this thread 4 items