Silent Instrument Errors Reorder Agent Leaderboards
Opens the theme on the eval side: harness errors that raise no alarm can change which agent looks better.
4 items · 3 sources · 4 days
Operational story trace
Follow in this browser to see new updates on your Live feed.
Latest change
A Nov 13 companion paper moves from measuring silent errors to catching them: a small model in a browser tab flags clean, well-formed orders that the customer never placed.
This is a research trend, not one developing event: four independent arXiv papers in two weeks study outputs that pass every check and are still wrong. The first two found them in agent leaderboard harnesses and in LLM code repairs after API migrations.
Arc
Opens the theme on the eval side: harness errors that raise no alarm can change which agent looks better.
Shows the same failure mode in LLM code repair, backed by a preregistered 73-task study across three runtimes.
Turns output format into a design lever for silent error rates in small fine-tuned parsers.
The companion paper moves from measurement to detection, running the checker in a browser tab.
Opens the theme on the eval side: harness errors that raise no alarm can change which agent looks better.
Shows the same failure mode in LLM code repair, backed by a preregistered 73-task study across three runtimes.
Turns output format into a design lever for silent error rates in small fine-tuned parsers.
The companion paper moves from measurement to detection, running the checker in a browser tab.
What to watch — open questions
Storylines are threaded mechanically from the feed: stories that share a distinctive anchor across multiple days and sources. Each item links to its original source. The evidence trace, current state, and open questions are written by the editor routine and refreshed whenever a new beat lands.