Continual Learning Exam: Benchmarking Long-Horizon RSI and Experiential Learning in LLM Agents
Opens the thread with a benchmark for continual learning from experience in LLM agents.
AI Storyline
10 items · 4 sources · 8 days
Operational story trace
Evaluating Tasks
Follow in this browser to see new updates on your Live feed.
Current stateDeveloping
Latest change
A Nov 6 paper argues one run is not enough and proposes a Monte Carlo protocol for measuring LLM reliability on data science tasks, following new consumer-task (LifeVal) and multilingual tool-use (τ-bench-LILT) benchmarks.
Earlier contextThe story so far
Late-October arXiv papers framed long-horizon agent work as an evaluation problem, with benchmarks for continual learning and web search alongside new critic, memory and multi-agent methods. On Nov 1 a paper questioned whether long-horizon terminal benchmarks actually measure depth.
editor-curated · source-linked
Arc
Oct 25Nov 6 · now
BENCHMARKS · Oct 25–26Long-horizon agents get exam-style benchmarks2 sources · show sources ▾
LongSearchEval: Evaluating Long-Horizon Web Search Capabilities of Large Language Models via a Trajectory-Aware Agent Framework with Failure-Node Localization
Adds a long-horizon web-search evaluation with failure-node localization.
METHODS · Oct 26–29Critics, memory governance and goal-aware collaboration target long-horizon tasks3 sources · show sources ▾
FOCUS: Active Critic Learning for Long-Horizon Language Model Agents
Proposes active critic learning to address credit assignment under sparse outcome rewards.
CALM: Conflict-Aware Lifecycle Memory Governance for Long-Horizon Language Agents
Addresses long-horizon agent memory by governing outdated versus still-needed memories.
Goal-Aware Dynamic Multi-Agent Collaboration for Long-Horizon Tasks
Proposes goal-aware dynamic multi-agent collaboration for long-horizon tasks.
THE TURN · Nov 1–3Is a long-horizon benchmark really deep?A terminal-task paper questions turn count as a proxy for depth, while hierarchical multi-agent RL for web tasks continues.2 sources · show sources ▾
Is Your Long-Horizon Benchmark Really Deep? Measuring and Synthesizing Depth in Terminal Tasks
Challenges the assumption that task length in turns reflects depth in terminal benchmarks.
JHiMA-WebRL: A Joint Hierarchical Multi-Agent Reinforcement Learning Framework for Long-Horizon Web Tasks
Applies hierarchical multi-agent reinforcement learning to long-horizon web tasks.
NOW · Nov 4–6Evals widen to consumer tasks, other languages and repeat runs3 sources · show sources ▾
LifeVal: Evaluating AI Model Performance on Personalized Agentic Consumer Tasks
Introduces LifeVal, a benchmark for personalized agentic consumer tasks.
τ-bench-LILT: Evaluating Conversational Agents on Realistic Multilingual Tool-Use Tasks
Extends tool-use conversational agent evaluation beyond English.
One Run Is Not Enough: A Monte Carlo Protocol for Evaluating LLM Reliability on Data Science Tasks
Argues a single run is insufficient and proposes a Monte Carlo reliability protocol for data science tasks.
Day 1 Sunday, Oct 25, 2026
Continual Learning Exam: Benchmarking Long-Horizon RSI and Experiential Learning in LLM Agents
Opens the thread with a benchmark for continual learning from experience in LLM agents.
Day 2 Monday, Oct 26, 2026
LongSearchEval: Evaluating Long-Horizon Web Search Capabilities of Large Language Models via a Trajectory-Aware Agent Framework with Failure-Node Localization
Adds a long-horizon web-search evaluation with failure-node localization.
FOCUS: Active Critic Learning for Long-Horizon Language Model Agents
Proposes active critic learning to address credit assignment under sparse outcome rewards.
Day 3 Wednesday, Oct 28, 2026
CALM: Conflict-Aware Lifecycle Memory Governance for Long-Horizon Language Agents
Addresses long-horizon agent memory by governing outdated versus still-needed memories.
Day 4 Thursday, Oct 29, 2026
Goal-Aware Dynamic Multi-Agent Collaboration for Long-Horizon Tasks
Proposes goal-aware dynamic multi-agent collaboration for long-horizon tasks.
Day 5 Sunday, Nov 1, 2026
Is Your Long-Horizon Benchmark Really Deep? Measuring and Synthesizing Depth in Terminal Tasks
Challenges the assumption that task length in turns reflects depth in terminal benchmarks.
Day 6 Tuesday, Nov 3, 2026
JHiMA-WebRL: A Joint Hierarchical Multi-Agent Reinforcement Learning Framework for Long-Horizon Web Tasks
Applies hierarchical multi-agent reinforcement learning to long-horizon web tasks.
Day 7 Wednesday, Nov 4, 2026
LifeVal: Evaluating AI Model Performance on Personalized Agentic Consumer Tasks
Introduces LifeVal, a benchmark for personalized agentic consumer tasks.
τ-bench-LILT: Evaluating Conversational Agents on Realistic Multilingual Tool-Use Tasks
Extends tool-use conversational agent evaluation beyond English.
Day 8 Friday, Nov 6, 2026
One Run Is Not Enough: A Monte Carlo Protocol for Evaluating LLM Reliability on Data Science Tasks
Argues a single run is insufficient and proposes a Monte Carlo reliability protocol for data science tasks.
What to watch — open questions
- Will other benchmark authors adopt a depth measure beyond turns needed to pass a task?
- Will multi-run Monte Carlo reporting become standard in agent benchmark releases?
- Do results on τ-bench-LILT show a measurable gap between English and other languages for tool-using agents?
How this thread was built
editor wrote the arc · 4 beatswatcher 1 status change
Storylines are threaded mechanically from the feed: stories that share a distinctive anchor across multiple days and sources. Each item links to its original source. The evidence trace, current state, and open questions are written by the editor routine and refreshed whenever a new beat lands.