LLM Digest

Story

arxiv_cs_lg · May 13, 2026 · paper

Source brief

Attention Once Is All You Need: Efficient Streaming Inference with Stateful Transformers

arxiv.orgMay 13, 2026
original source linked

In brief

Conventional transformer inference engines are request-driven, paying an O(n) prefill cost on every query. In streaming workloads, where data arrives continuously and queries probe an ever-growing context, this cost i...

Read the original at arxiv.org →Open in live feed

Attention Once Is All You Need: Efficient Streaming Inference with Stateful Transformers

Earlier in this thread 4 items

Semiparametric Efficient Test for Interpretable Distributional Treatment Effects

Transformers with Selective Access to Early Representations

Article: From Batch to Micro-Batch Streaming: Lessons Learned the Hard Way in a Delta Index Pipeline

Carbon-Taxed Transformers: A Green Compression Pipeline for Overgrown Language Models