{"slug":"benchmarking-real-world","label":"Benchmarking Real-World","item_count":7,"day_count":7,"source_count":4,"first_seen":"2026-11-03T13:23:37+00:00","last_updated":"2026-11-16T18:32:05+00:00","generated_at":"2026-11-18T00:14:17.482304+00:00","sources":["arxiv_cs_ai","arxiv_cs_cl","arxiv_cs_lg","arxiv_llm_reliability"],"days":[{"date":"2026-11-03","items":[{"title":"One Assistant, Many Memories: Benchmarking Real-World Group Assistance When Recall Is Not Enough","url":"http://arxiv.org/abs/2611.04867v1","source":"arxiv_cs_cl","type":"paper","summary_1line":"The value of memory lies not only in recall but in using it to provide appropriate assistance. Recent benchmarks test memory use in personalisation and agentic tasks, but seldom evaluate assistance without fixed answe...","why_it_matters":"Matches feed focus: agentic, evaluation.","sid":"ef4c20cc8b60cd08","published":"2026-11-03T13:23:37+00:00","editor_note":"Opens the cluster: memory evaluated by the quality of assistance it enables across many users."}]},{"date":"2026-11-05","items":[{"title":"MCPHallu: Benchmarking Reasoning, Execution, and Memory Hallucinations in MCP Agents","url":"http://arxiv.org/abs/2611.07689v1","source":"arxiv_llm_reliability","type":"paper","summary_1line":"Large language model agents are increasingly deployed through the Model Context Protocol (MCP), which allows them to discover and invoke tools across multiple servers at runtime. Existing MCP benchmarks primarily meas...","why_it_matters":"Matches feed focus: agent, eval.","sid":"45b80d77d473d30f","published":"2026-11-05T09:14:26+00:00","editor_note":"Shifts to failure modes: hallucinations in reasoning, execution and memory for agents using MCP tools across servers."}]},{"date":"2026-11-06","items":[{"title":"MemoryDojo: Benchmarking Poisoning Through LLM Agent Memory","url":"http://arxiv.org/abs/2611.09721v1","source":"arxiv_cs_cl","type":"paper","summary_1line":"LLM agents benefit from using memory systems to improve performance on long-horizon, multi-session tasks. However, memory introduces a critical attack surface. Malicious instructions encountered in one session may be...","why_it_matters":"Matches feed focus: agentic, evaluation.","sid":"138e084674254809","published":"2026-11-06T17:00:09+00:00","editor_note":"Adds a security angle: malicious instructions from one session persisting through agent memory into later ones."}]},{"date":"2026-11-09","items":[{"title":"Infrathon: Benchmarking Language Agents for Real-World, Long-Horizon AI Infra Optimization","url":"http://arxiv.org/abs/2611.11995v1","source":"arxiv_cs_cl","type":"paper","summary_1line":"In this work, we study whether frontier agent can continually improve real-world AI infrastructure over long horizons. We introduce Infrathon, a benchmark of diverse, long-horizon AI infrastructure optimization tasks...","why_it_matters":"Matches feed focus: agent, evaluation.","sid":"23c0753cd928bfa7","published":"2026-11-09T07:24:54+00:00","editor_note":"Moves the real-world framing to infrastructure: long-horizon optimization tasks for agents."}]},{"date":"2026-11-10","items":[{"title":"ARE-bench: Benchmarking Agents' Research Ability in Real-World Researcher Sessions and Workspaces","url":"http://arxiv.org/abs/2611.14364v1","source":"arxiv_cs_cl","type":"paper","summary_1line":"Research agents achieve increasingly strong benchmark scores, yet these scores do not necessarily translate into reliable delivery in ongoing research projects. Many research-agent benchmarks emphasize standalone task...","why_it_matters":"Matches feed focus: agent, eval.","sid":"d56fa1c78b5794b5","published":"2026-11-10T17:21:31+00:00","editor_note":"Applies it to research agents, scoring delivery inside real researcher sessions rather than standalone tasks."}]},{"date":"2026-11-13","items":[{"title":"PrivacyPR: A Dataset of Real-World Privacy Fixes for Benchmarking Privacy-Aware Code Review","url":"http://arxiv.org/abs/2611.18293v1","source":"arxiv_cs_ai","type":"paper","summary_1line":"Coding agents increasingly review code, yet whether they catch the privacy defects that laws such as the GDPR require software to avoid remains largely untested, partly because few datasets connect real privacy defect...","why_it_matters":"Matches feed focus: agent, harness, evaluation.","sid":"004673f14a75854b","published":"2026-11-13T15:18:33+00:00","editor_note":"Extends it to code review with a dataset of real privacy fixes tied to their defects."}]},{"date":"2026-11-16","items":[{"title":"Benchmarking memory and nonlinear processing capacity of quantum recurrent neural networks for time series prediction","url":"http://arxiv.org/abs/2611.21552v1","source":"arxiv_cs_lg","type":"paper","summary_1line":"Quantum machine learning has emerged as a promising approach to time series prediction. While quantum reservoir computing has been widely explored for this setting, variational quantum models tailored to sequential da...","why_it_matters":"Matches feed focus: eval.","sid":"46d71c45a0641225","published":"2026-11-16T18:32:05+00:00","editor_note":"Keyword match only: a quantum ML time-series benchmark that adds nothing to the agent-evaluation thread."}]}],"editorial":{"tldr":"This is a research trend, not one event: six unrelated arXiv papers in ten days each build an agent benchmark from real sessions, real tools or real fixes. It began with group-assistant memory (Nov 3), added MCP hallucinations and memory poisoning, then moved to long-horizon infra work, researcher workspaces and privacy-aware code review (Nov 13).","stale":false,"whats_new":"The newest member (Nov 16) is a quantum recurrent-network benchmark for time-series prediction; it shares only the word \"benchmarking\", so PrivacyPR (Nov 13) remains the last agent-relevant entry.","why_it_matters":"Each agent paper argues that standalone task scores overstate how agents behave on stateful, tool-heavy, long-horizon or compliance-sensitive work, which is the work platform teams actually deploy them on.","take_for_builders":"If your agents persist memory or call MCP tools, add an eval that replays a poisoned instruction across sessions and logs tool-call hallucinations; keep gating rollouts on one eval built from your own logged sessions, not public scores alone.","beats":[{"kicker":"MEMORY","tone":"launch","headline":"Group-assistant benchmark tests memory use, not just recall","summary":"Evaluates whether an assistant shared by many users turns remembered context into appropriate help when there is no fixed answer.","sids":["ef4c20cc8b60cd08"]},{"kicker":"FAILURE MODES","tone":"turn","headline":"MCPHallu and MemoryDojo benchmark how agents break","summary":"One measures reasoning, execution and memory hallucinations in MCP tool use; the other tests whether malicious instructions persist through agent memory across sessions.","sids":["45b80d77d473d30f","138e084674254809"]},{"kicker":"LONG HORIZON","tone":"rising","headline":"Infrathon and ARE-bench test agents on sustained real work","summary":"One asks whether agents keep improving real AI infrastructure over long horizons; the other scores research agents inside real researcher sessions and workspaces.","sids":["23c0753cd928bfa7","d56fa1c78b5794b5"]},{"kicker":"CODE REVIEW","tone":"rising","headline":"PrivacyPR targets privacy defects in agent code review","summary":"A dataset linking real privacy fixes to the defects they repaired, aimed at coding agents that review pull requests.","sids":["004673f14a75854b"]},{"kicker":"OFF-THREAD","tone":"neutral","headline":"Quantum RNN time-series benchmark joins on a shared keyword","summary":"Quantum machine-learning research with no agent or platform angle; clustered here only by the word \"benchmarking\".","sids":["46d71c45a0641225"]}],"open_questions":["Will any of these benchmarks publish a public leaderboard with frontier-model scores?","Which MCP hallucination type in MCPHallu do current agents fail most often: reasoning, execution or memory?","Does MemoryDojo show poisoned memory surviving into later sessions for the memory systems teams actually ship?","Do agents that score well on standalone suites rank lower on Infrathon or ARE-bench, as the authors suggest?","What share of PrivacyPR's real privacy defects do current coding-agent reviewers catch?","Will the datasets and harnesses be released openly enough to rerun as internal evals?"],"generated_at":"2026-11-18T00:13:29Z"}}