Multi-layer Indexer: Overlapping Selection and Attention in DeepSeek Sparse Attention
Anchors the thread in DeepSeek Sparse Attention, overlapping the indexer's selection with attention compute.
4 items · 4 sources · 4 days
Operational story trace
Follow in this browser to see new updates on your Live feed.
Latest change
DTS (Nov 19) scores attention tiles by deltas to decide which blocks to skip, targeting the prefill bottleneck of long documents and codebases.
This is a loose cluster of four independent arXiv papers, not one event: each attacks the cost of long-context attention by letting a query attend only to a selected subset of keys. It opened with work on DeepSeek Sparse Attention's learned indexer, then two training-free methods that choose keys by semantic blocks or per-head structure.
Arc
Anchors the thread in DeepSeek Sparse Attention, overlapping the indexer's selection with attention compute.
Adds a training-free option: a prompt-specific key index built from coarse semantic blocks for prefill and decode.
Argues sparsity should vary by head, chosen from each head's query-key interaction structure.
Shifts focus to prefill, scoring tiles by deltas to decide which blocks block-sparse attention skips.
Anchors the thread in DeepSeek Sparse Attention, overlapping the indexer's selection with attention compute.
Adds a training-free option: a prompt-specific key index built from coarse semantic blocks for prefill and decode.
Argues sparsity should vary by head, chosen from each head's query-key interaction structure.
Shifts focus to prefill, scoring tiles by deltas to decide which blocks block-sparse attention skips.
What to watch — open questions
Storylines are threaded mechanically from the feed: stories that share a distinctive anchor across multiple days and sources. Each item links to its original source. The evidence trace, current state, and open questions are written by the editor routine and refreshed whenever a new beat lands.