The SIMPLE Law for Energy Cost of LLM Inference
Sets the premise: inference energy rises non-linearly along a sequence, which flat per-token billing hides.
3 items · 2 sources · 3 days
Operational story trace
Follow in this browser to see new updates on your Live feed.
Latest change
A Nov 18 arXiv paper proposes timeout cascades to cut the GPU cost of coding agents, two days after an open-source benchmark compared OSS inference vendors on cost and latency.
This is a loose research thread on inference cost, not one event. It opened with an arXiv paper arguing that per-token billing hides cost: each token added to the KV-cache makes later tokens more expensive, so energy grows non-linearly with sequence length.
Arc
Sets the premise: inference energy rises non-linearly along a sequence, which flat per-token billing hides.
Adds a reproducible harness for comparing OSS inference vendors on cost and latency.
Moves the thread to agents, proposing timeout cascades to reduce coding-agent GPU spend.
Sets the premise: inference energy rises non-linearly along a sequence, which flat per-token billing hides.
Adds a reproducible harness for comparing OSS inference vendors on cost and latency.
Moves the thread to agents, proposing timeout cascades to reduce coding-agent GPU spend.
What to watch — open questions
Storylines are threaded mechanically from the feed: stories that share a distinctive anchor across multiple days and sources. Each item links to its original source. The evidence trace, current state, and open questions are written by the editor routine and refreshed whenever a new beat lands.