{"slug":"evaluating-tasks","label":"Evaluating Tasks","item_count":10,"day_count":8,"source_count":4,"first_seen":"2026-10-25T00:00:25+00:00","last_updated":"2026-11-06T17:06:21+00:00","generated_at":"2026-11-10T00:13:24.787725+00:00","sources":["arxiv_agent_systems_research","arxiv_cs_ai","arxiv_cs_cl","arxiv_cs_lg"],"days":[{"date":"2026-10-25","items":[{"title":"Continual Learning Exam: Benchmarking Long-Horizon RSI and Experiential Learning in LLM Agents","url":"http://arxiv.org/abs/2610.33956v1","source":"arxiv_cs_cl","type":"paper","summary_1line":"Continual learning is the ability of LLM agents to improve over time by acquiring, refining, and applying knowledge from experience while retaining useful prior knowledge. It is essential for agents learning over long...","why_it_matters":"Matches feed focus: agent, harness, evaluation.","sid":"55edc2615764e1b5","published":"2026-10-25T00:00:25+00:00","editor_note":"Opens the thread with a benchmark for continual learning from experience in LLM agents."}]},{"date":"2026-10-26","items":[{"title":"LongSearchEval: Evaluating Long-Horizon Web Search Capabilities of Large Language Models via a Trajectory-Aware Agent Framework with Failure-Node Localization","url":"http://arxiv.org/abs/2610.35150v1","source":"arxiv_agent_systems_research","type":"paper","summary_1line":"Long-horizon web search requires LLM-based agents to integrate information across web pages through extended interactions under task-specific constraints. Evaluating this capability is important not only for measuring...","why_it_matters":"Matches feed focus: agent, evaluation.","sid":"5f97c21b490b1ce9","published":"2026-10-26T10:15:38+00:00","editor_note":"Adds a long-horizon web-search evaluation with failure-node localization."},{"title":"FOCUS: Active Critic Learning for Long-Horizon Language Model Agents","url":"http://arxiv.org/abs/2610.35801v1","source":"arxiv_cs_lg","type":"paper","summary_1line":"Sparse outcome rewards make it difficult for long-horizon language-model agents to assign credit to earlier actions. Learned critics provide value estimates for intermediate states, but these estimates can be inaccura...","why_it_matters":"Matches feed focus: agentic.","sid":"3ca73f33ea96bd6a","published":"2026-10-26T17:34:47+00:00","editor_note":"Proposes active critic learning to address credit assignment under sparse outcome rewards."}]},{"date":"2026-10-28","items":[{"title":"CALM: Conflict-Aware Lifecycle Memory Governance for Long-Horizon Language Agents","url":"http://arxiv.org/abs/2610.38573v1","source":"arxiv_cs_ai","type":"paper","summary_1line":"A memory can be outdated for present-day reasoning yet indispensable for reconstructing the past. Relevance-based retrieval overlooks this tension: semantic relevance, evidence-grounded validity, and query-specific el...","why_it_matters":"Matches feed focus: agent, eval.","sid":"423c688c0621c177","published":"2026-10-28T10:06:57+00:00","editor_note":"Addresses long-horizon agent memory by governing outdated versus still-needed memories."}]},{"date":"2026-10-29","items":[{"title":"Goal-Aware Dynamic Multi-Agent Collaboration for Long-Horizon Tasks","url":"http://arxiv.org/abs/2610.39193v1","source":"arxiv_agent_systems_research","type":"paper","summary_1line":"LLM-based agents are increasingly used to complete complex, long-horizon tasks without human intervention. To make progress toward the task goal, agents need to perform different duties, such as planning and verificat...","why_it_matters":"Matches feed focus: agent.","sid":"0b7565240a3894fa","published":"2026-10-29T02:15:35+00:00","editor_note":"Proposes goal-aware dynamic multi-agent collaboration for long-horizon tasks."}]},{"date":"2026-11-01","items":[{"title":"Is Your Long-Horizon Benchmark Really Deep? Measuring and Synthesizing Depth in Terminal Tasks","url":"http://arxiv.org/abs/2611.01927v1","source":"arxiv_cs_ai","type":"paper","summary_1line":"Long-horizon terminal tasks are used to evaluate and train agents. Human-curated datasets are scarce, so many synthesis methods have emerged. They usually measure the length of a task by the turns a model needs to pas...","why_it_matters":"Matches feed focus: agent, eval.","sid":"52a3c89c275c2670","published":"2026-11-01T14:05:27+00:00","editor_note":"Challenges the assumption that task length in turns reflects depth in terminal benchmarks."}]},{"date":"2026-11-03","items":[{"title":"JHiMA-WebRL: A Joint Hierarchical Multi-Agent Reinforcement Learning Framework for Long-Horizon Web Tasks","url":"http://arxiv.org/abs/2611.05263v1","source":"arxiv_agent_systems_research","type":"paper","summary_1line":"Multimodal web agents have shown promising capabilities for interacting with websites through graphical user interfaces (GUIs), yet realistic web tasks often require long-horizon, multi-step planning that remains chal...","why_it_matters":"Matches feed focus: agent, eval.","sid":"08c314bb237b4b82","published":"2026-11-03T17:49:14+00:00","editor_note":"Applies hierarchical multi-agent reinforcement learning to long-horizon web tasks."}]},{"date":"2026-11-04","items":[{"title":"LifeVal: Evaluating AI Model Performance on Personalized Agentic Consumer Tasks","url":"http://arxiv.org/abs/2611.06687v1","source":"arxiv_cs_ai","type":"paper","summary_1line":"We introduce LifeVal, a benchmark evaluating AI model performance on personalized agentic consumer tasks. LifeVal covers the most valuable life tasks that a personalized digital superassistant could automate, represen...","why_it_matters":"Matches feed focus: agentic, evaluation.","sid":"3b93b07bf2154e4f","published":"2026-11-04T15:55:20+00:00","editor_note":"Introduces LifeVal, a benchmark for personalized agentic consumer tasks."},{"title":"τ-bench-LILT: Evaluating Conversational Agents on Realistic Multilingual Tool-Use Tasks","url":"http://arxiv.org/abs/2611.07203v1","source":"arxiv_agent_systems_research","type":"paper","summary_1line":"Frontier evaluations of tool-using conversational agents are conducted almost exclusively in English, leaving a critical gap for organizations deploying agents in other languages and regions. We present $τ$-bench-LILT...","why_it_matters":"Matches feed focus: agent, evaluation.","sid":"51abf5f84d7f5698","published":"2026-11-04T22:47:38+00:00","editor_note":"Extends tool-use conversational agent evaluation beyond English."}]},{"date":"2026-11-06","items":[{"title":"One Run Is Not Enough: A Monte Carlo Protocol for Evaluating LLM Reliability on Data Science Tasks","url":"http://arxiv.org/abs/2611.09729v1","source":"arxiv_cs_lg","type":"paper","summary_1line":"Large language models (LLMs) have shown promise for automating data science workflows. Benchmarks such as DataSciBench (2026) and StatQA (2024) have advanced the evaluation of their capabilities across data science an...","why_it_matters":"Matches feed focus: evaluation.","sid":"0d4497db5df885d5","published":"2026-11-06T17:06:21+00:00","editor_note":"Argues a single run is insufficient and proposes a Monte Carlo reliability protocol for data science tasks."}]}],"editorial":{"tldr":"Late-October arXiv papers framed long-horizon agent work as an evaluation problem, with benchmarks for continual learning and web search alongside new critic, memory and multi-agent methods. On Nov 1 a paper questioned whether long-horizon terminal benchmarks actually measure depth.","stale":false,"whats_new":"A Nov 6 paper argues one run is not enough and proposes a Monte Carlo protocol for measuring LLM reliability on data science tasks, following new consumer-task (LifeVal) and multilingual tool-use (τ-bench-LILT) benchmarks.","why_it_matters":"Agent benchmark scores are only as trustworthy as the task design and run count behind them: depth, language coverage and run-to-run variance all change what a number means for your deployment.","take_for_builders":"Rerun your internal agent evals several times and report the spread rather than a single score, and add non-English tool-use cases before trusting an English-only benchmark result.","status":{"state":"Developing","tone":"rising","changed":"2026-11-06","detail":"Research is shifting from adding long-horizon benchmarks to questioning their depth, language coverage and single-run reliability."},"beats":[{"kicker":"BENCHMARKS","tone":"launch","headline":"Long-horizon agents get exam-style benchmarks","summary":"Continual-learning and long-horizon web-search evaluations arrive in the same two days.","sids":["55edc2615764e1b5","5f97c21b490b1ce9"]},{"kicker":"METHODS","tone":"rising","headline":"Critics, memory governance and goal-aware collaboration target long-horizon tasks","sids":["3ca73f33ea96bd6a","423c688c0621c177","0b7565240a3894fa"]},{"kicker":"THE TURN","tone":"turn","headline":"Is a long-horizon benchmark really deep?","summary":"A terminal-task paper questions turn count as a proxy for depth, while hierarchical multi-agent RL for web tasks continues.","sids":["52a3c89c275c2670","08c314bb237b4b82"]},{"kicker":"NOW","tone":"now","headline":"Evals widen to consumer tasks, other languages and repeat runs","sids":["3b93b07bf2154e4f","51abf5f84d7f5698","0d4497db5df885d5"]}],"open_questions":["Will other benchmark authors adopt a depth measure beyond turns needed to pass a task?","Will multi-run Monte Carlo reporting become standard in agent benchmark releases?","Do results on τ-bench-LILT show a measurable gap between English and other languages for tool-using agents?"],"generated_at":"2026-11-10T00:30:00+00:00"}}