Causal Chain of Thought: Mitigating Attention Leakage via Context Isolation
Argues chain-of-thought steps may not causally drive answers and proposes context isolation.
AI Storyline
8 items · 4 sources · 6 days
Operational story trace
Attention Context
Follow in this browser to see new updates on your Live feed.
Current stateDeveloping
Latest change
Muir (Nov 5) reworks gated delta-rule linear attention with sensitivity-allocated gating, while QRAE compresses DeepSearch agent context using query-relevant attention entropy.
Earlier contextThe story so far
Late-October papers attacked long-context cost from several sides: isolating context in chain of thought, initializing linear-attention memory, and learned sparse attention. By early November the work moved to production sparse attention, TPU kernels and agent-side context compression.
editor-curated · source-linked
Arc
Oct 22Nov 5 · now
FRAMING · Oct 22Attention leakage in chain of thought1 source · show source ▾
CHEAPER MEMORY · Oct 30Linear and sparse attention target ultra-long context2 sources · show sources ▾
DAMI: Dual-Axis Memory Initialization for Enhanced Long-Context Retrieval in Gated Linear Attention
Initializes gated linear attention memory to improve long-context retrieval.
Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention
Latent sparse attention aimed at contexts above 1M tokens for long-horizon agents.
PRODUCTION SPARSITY · Nov 2–3DeepSeek Sparse Attention and training-free block indexes2 sources · show sources ▾
Multi-layer Indexer: Overlapping Selection and Attention in DeepSeek Sparse Attention
Overlaps selection and attention in DeepSeek Sparse Attention.
Coarse Semantic Blocks for Efficient Training-Free Sparse Attention
Training-free sparse attention using prompt-specific semantic block indexes.
HARDWARE · Nov 4Gated delta attention co-designed for TPUs1 source · show source ▾
Systolic Linear Attention: Co-Designing Tile-Grouped Gated Delta Attention for TPUs
Adapts gated delta attention to TPU hardware constraints.
NOW · Nov 5New gating design and agent context compression2 sources · show sources ▾
QRAE: Query-Relevant Attention Entropy for Context Compression in DeepSearch Agents
Compresses retrieved context for DeepSearch agents to cut tokens and latency.
Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout
New gating and readout design for gated delta-rule linear attention.
Day 1 Thursday, Oct 22, 2026
Causal Chain of Thought: Mitigating Attention Leakage via Context Isolation
Argues chain-of-thought steps may not causally drive answers and proposes context isolation.
Day 2 Friday, Oct 30, 2026
DAMI: Dual-Axis Memory Initialization for Enhanced Long-Context Retrieval in Gated Linear Attention
Initializes gated linear attention memory to improve long-context retrieval.
Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention
Latent sparse attention aimed at contexts above 1M tokens for long-horizon agents.
Day 3 Monday, Nov 2, 2026
Multi-layer Indexer: Overlapping Selection and Attention in DeepSeek Sparse Attention
Overlaps selection and attention in DeepSeek Sparse Attention.
Day 4 Tuesday, Nov 3, 2026
Coarse Semantic Blocks for Efficient Training-Free Sparse Attention
Training-free sparse attention using prompt-specific semantic block indexes.
Day 5 Wednesday, Nov 4, 2026
Systolic Linear Attention: Co-Designing Tile-Grouped Gated Delta Attention for TPUs
Adapts gated delta attention to TPU hardware constraints.
Day 6 Thursday, Nov 5, 2026
QRAE: Query-Relevant Attention Entropy for Context Compression in DeepSearch Agents
Compresses retrieved context for DeepSearch agents to cut tokens and latency.
Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout
New gating and readout design for gated delta-rule linear attention.
What to watch — open questions
- Will any of these linear or sparse attention methods appear in a released open-weight model?
- Do the long-context retrieval gains hold on independent benchmarks?
How this thread was built
editor wrote the arc · 5 beatswatcher 1 status change
Storylines are threaded mechanically from the feed: stories that share a distinctive anchor across multiple days and sources. Each item links to its original source. The evidence trace, current state, and open questions are written by the editor routine and refreshed whenever a new beat lands.