{"slug":"attention-context","label":"Attention Context","item_count":8,"day_count":6,"source_count":4,"first_seen":"2026-10-22T17:52:56+00:00","last_updated":"2026-11-05T17:53:49+00:00","generated_at":"2026-11-07T00:12:50.771141+00:00","sources":["arxiv_cn_open_weight_reports","arxiv_cs_ai","arxiv_cs_cl","arxiv_cs_lg"],"days":[{"date":"2026-10-22","items":[{"title":"Causal Chain of Thought: Mitigating Attention Leakage via Context Isolation","url":"http://arxiv.org/abs/2610.31639v1","source":"arxiv_cs_lg","type":"paper","summary_1line":"Chain-of-thought (CoT) prompting elicits step-by-step rationales that appear to explain model predictions, but there is no guarantee these steps causally drive the final answer. In standard autoregressive generation,...","why_it_matters":"Matches feed focus: eval.","sid":"97282ff9b80798e7","published":"2026-10-22T17:52:56+00:00","editor_note":"Argues chain-of-thought steps may not causally drive answers and proposes context isolation."}]},{"date":"2026-10-30","items":[{"title":"DAMI: Dual-Axis Memory Initialization for Enhanced Long-Context Retrieval in Gated Linear Attention","url":"http://arxiv.org/abs/2610.41994v1","source":"arxiv_cs_cl","type":"paper","summary_1line":"Linear attention scales linearly with sequence length, but its finite-state memory can limit long-context retrieval. In gated linear recurrences, memory retention depends on state decay, which is shaped jointly by gate parameters A_log and dt_bias and by input activations. In con","why_it_matters":"Matches feed focus: eval.","sid":"35220abd16206ae7","published":"2026-10-30T16:39:37+00:00","editor_note":"Initializes gated linear attention memory to improve long-context retrieval."},{"title":"Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention","url":"http://arxiv.org/abs/2610.42147v1","source":"arxiv_cs_ai","type":"paper","summary_1line":"Agentic workflows are increasingly shifting towards long-horizon tasks, such as open-ended scientific work, where models benefit from long context windows. However, at ultra-long context (above 1M tokens), full attention is prohibitively expensive. A promising approach is sparse","why_it_matters":"Matches feed focus: agentic.","sid":"36261eb7a1c4b20b","published":"2026-10-30T17:59:34+00:00","editor_note":"Latent sparse attention aimed at contexts above 1M tokens for long-horizon agents."}]},{"date":"2026-11-02","items":[{"title":"Multi-layer Indexer: Overlapping Selection and Attention in DeepSeek Sparse Attention","url":"http://arxiv.org/abs/2611.03768v1","source":"arxiv_cn_open_weight_reports","type":"paper","summary_1line":"Sparse attention cuts the cost of long-context inference by making the attention matrix sparse: each query attends only a subset of the sequence, chosen by a fixed pattern or, in recent production LLMs, learned per query. In DeepSeek Sparse Attention (DSA), the attention of DeepS","sid":"2ef0f5e9028f8dae","published":"2026-11-02T15:46:01+00:00","editor_note":"Overlaps selection and attention in DeepSeek Sparse Attention."}]},{"date":"2026-11-03","items":[{"title":"Coarse Semantic Blocks for Efficient Training-Free Sparse Attention","url":"http://arxiv.org/abs/2611.04313v1","source":"arxiv_cs_cl","type":"paper","summary_1line":"We introduce a training-free sparse attention method that builds a prompt-specific key index to accelerate both prefill and decoding. The method clusters the keys into large semantic blocks that group tokens from nonc...","why_it_matters":"Matches feed focus: eval.","sid":"2d77f36eefa1549d","published":"2026-11-03T04:46:30+00:00","editor_note":"Training-free sparse attention using prompt-specific semantic block indexes."}]},{"date":"2026-11-04","items":[{"title":"Systolic Linear Attention: Co-Designing Tile-Grouped Gated Delta Attention for TPUs","url":"http://arxiv.org/abs/2611.06828v1","source":"arxiv_cs_lg","type":"paper","summary_1line":"Gated-delta linear attention layers such as Gated DeltaNet (GDN), Kimi Delta Attention (KDA), and GDN-2 are tuned for GPUs, where small tensor-core tiles and fast vector registers make per-channel decay rescaling and sequential in-block solves cheap. On Google TPUs, which concent","sid":"392a1228ae5178b1","published":"2026-11-04T17:42:58+00:00","editor_note":"Adapts gated delta attention to TPU hardware constraints."}]},{"date":"2026-11-05","items":[{"title":"QRAE: Query-Relevant Attention Entropy for Context Compression in DeepSearch Agents","url":"http://arxiv.org/abs/2611.07976v1","source":"arxiv_cs_lg","type":"paper","summary_1line":"DeepSearch LLM agents repeatedly search and carry retrieved sources into subsequent reasoning steps. This expanding context increases token consumption and latency, although many results do not contribute to later rea...","why_it_matters":"Matches feed focus: agent, evaluation.","sid":"7726cf50890f3ade","published":"2026-11-05T13:08:43+00:00","editor_note":"Compresses retrieved context for DeepSearch agents to cut tokens and latency."},{"title":"Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout","url":"http://arxiv.org/abs/2611.08376v1","source":"arxiv_cs_lg","type":"paper","summary_1line":"Linear attention compresses context into a fixed-size key-value state for linear-time training and constant-memory inference. Gated delta-rule models decay this state and erase its prediction for an incoming key befor...","why_it_matters":"Matches feed focus: eval.","sid":"50bc93990145bd05","published":"2026-11-05T17:53:49+00:00","editor_note":"New gating and readout design for gated delta-rule linear attention."}]}],"editorial":{"tldr":"Late-October papers attacked long-context cost from several sides: isolating context in chain of thought, initializing linear-attention memory, and learned sparse attention. By early November the work moved to production sparse attention, TPU kernels and agent-side context compression.","stale":false,"whats_new":"Muir (Nov 5) reworks gated delta-rule linear attention with sensitivity-allocated gating, while QRAE compresses DeepSearch agent context using query-relevant attention entropy.","why_it_matters":"Long-context cost drives inference spend and latency for agents. Sparse and linear attention variants, plus agent-level context compression, are the levers that decide how much context a deployment can afford.","take_for_builders":"These are preprints with no shared benchmark, so do not plan around them yet. Track which ones ship in open-weight models or serving stacks, and rerun your long-context retrieval evals before adopting any.","status":{"state":"Developing","tone":"rising","changed":"2026-11-05"},"beats":[{"kicker":"FRAMING","tone":"launch","headline":"Attention leakage in chain of thought","sids":["97282ff9b80798e7"]},{"kicker":"CHEAPER MEMORY","tone":"rising","headline":"Linear and sparse attention target ultra-long context","summary":"Memory initialization for gated linear attention and latent sparse attention above 1M tokens.","sids":["35220abd16206ae7","36261eb7a1c4b20b"]},{"kicker":"PRODUCTION SPARSITY","tone":"turn","headline":"DeepSeek Sparse Attention and training-free block indexes","sids":["2ef0f5e9028f8dae","2d77f36eefa1549d"]},{"kicker":"HARDWARE","tone":"rising","headline":"Gated delta attention co-designed for TPUs","sids":["392a1228ae5178b1"]},{"kicker":"NOW","tone":"now","headline":"New gating design and agent context compression","sids":["7726cf50890f3ade","50bc93990145bd05"]}],"open_questions":["Will any of these linear or sparse attention methods appear in a released open-weight model?","Do the long-context retrieval gains hold on independent benchmarks?"],"generated_at":"2026-11-07T00:30:00+00:00"}}