LLM Digest
Subscribe

AI Storyline

7 items · 4 sources · 7 days

View as JSON

Operational story trace

Benchmarking Real-World

Latest change

The newest member (Nov 16) is a quantum recurrent-network benchmark for time-series prediction; it shares only the word "benchmarking", so PrivacyPR (Nov 13) remains the last agent-relevant entry.

Earlier contextThe story so far

This is a research trend, not one event: six unrelated arXiv papers in ten days each build an agent benchmark from real sessions, real tools or real fixes. It began with group-assistant memory (Nov 3), added MCP hallucinations and memory poisoning, then moved to long-horizon infra work, researcher workspaces and privacy-aware code review (Nov 13).

editor-curated · source-linked

Arc

Nov 3Nov 16 · now
MEMORY · Nov 3
Group-assistant benchmark tests memory use, not just recall
1 source · show source ▾
FAILURE MODES · Nov 5–6
MCPHallu and MemoryDojo benchmark how agents break
One measures reasoning, execution and memory hallucinations in MCP tool use; the other tests whether malicious instructions persist through agent memory across sessions.
2 sources · show sources ▾
LONG HORIZON · Nov 9–10
Infrathon and ARE-bench test agents on sustained real work
2 sources · show sources ▾
CODE REVIEW · Nov 13
PrivacyPR targets privacy defects in agent code review
1 source · show source ▾
OFF-THREAD · Nov 16
Quantum RNN time-series benchmark joins on a shared keyword
1 source · show source ▾

What to watch — open questions

  • Will any of these benchmarks publish a public leaderboard with frontier-model scores?
  • Which MCP hallucination type in MCPHallu do current agents fail most often: reasoning, execution or memory?
  • Does MemoryDojo show poisoned memory surviving into later sessions for the memory systems teams actually ship?
  • Do agents that score well on standalone suites rank lower on Infrathon or ARE-bench, as the authors suggest?
  • What share of PrivacyPR's real privacy defects do current coding-agent reviewers catch?
  • Will the datasets and harnesses be released openly enough to rerun as internal evals?
How this thread was built
editor wrote the arc · 5 beats

Storylines are threaded mechanically from the feed: stories that share a distinctive anchor across multiple days and sources. Each item links to its original source. The evidence trace, current state, and open questions are written by the editor routine and refreshed whenever a new beat lands.