LLM Digest
Subscribe

AI Storyline

3 items · 2 sources · 3 days

View as JSON

Operational story trace

Cost Inference

Latest change

A Nov 18 arXiv paper proposes timeout cascades to cut the GPU cost of coding agents, two days after an open-source benchmark compared OSS inference vendors on cost and latency.

Earlier contextThe story so far

This is a loose research thread on inference cost, not one event. It opened with an arXiv paper arguing that per-token billing hides cost: each token added to the KV-cache makes later tokens more expensive, so energy grows non-linearly with sequence length.

editor-curated · source-linked

Arc

Nov 2Nov 18 · now
THE MODEL · Nov 2
Paper argues tokens are not equally expensive
1 source · show source ▾
MEASUREMENT · Nov 16
Open-source benchmark compares OSS inference vendors on cost and latency
1 source · show source ▾
NOW · Nov 18
Timeout cascades proposed to cut coding-agent compute
1 source · show source ▾

What to watch — open questions

  • Do any inference providers price long-context tokens to reflect KV-cache cost?
  • How much do timeout cascades save on a public coding-agent benchmark, and at what pass-rate loss?
  • Do the inference-bench vendor rankings hold across model sizes and context lengths?
How this thread was built
editor wrote the arc · 3 beats

Storylines are threaded mechanically from the feed: stories that share a distinctive anchor across multiple days and sources. Each item links to its original source. The evidence trace, current state, and open questions are written by the editor routine and refreshed whenever a new beat lands.