LLM Digest
Subscribe

AI Storyline

4 items · 3 sources · 4 days

View as JSON

Operational story trace

Silent Errors

Latest change

A Nov 13 companion paper moves from measuring silent errors to catching them: a small model in a browser tab flags clean, well-formed orders that the customer never placed.

Earlier contextThe story so far

This is a research trend, not one developing event: four independent arXiv papers in two weeks study outputs that pass every check and are still wrong. The first two found them in agent leaderboard harnesses and in LLM code repairs after API migrations.

editor-curated · source-linked

Arc

Nov 3Nov 13 · now
EVALS · Nov 3
Silent instrument errors reorder a public agent leaderboard
1 source · show source ▾
CODE REPAIR · Nov 5
LLM repairs after API migrations run cleanly but return wrong results
Preregistered study of 73 tasks over 60 documented changes in CPython, Node.js and SQLite; typical-input release checks ship the bad repairs.
1 source · show source ▾
PARSING · Nov 12
Output format choice controls how often small parsers fail silently
1 source · show source ▾
NOW · Nov 13
Catching well-formed wrong orders with an in-browser language model
1 source · show source ▾

What to watch — open questions

  • Do public agent leaderboards re-score entries after correcting the harness errors the Nov 3 paper describes?
  • Does the format effect seen on PIZZA hold for larger models or for tool-call JSON in agent loops?
  • How many wrong-but-valid outputs does the browser-tab checker miss, and how often does it flag correct orders?
How this thread was built
editor wrote the arc · 4 beats

Storylines are threaded mechanically from the feed: stories that share a distinctive anchor across multiple days and sources. Each item links to its original source. The evidence trace, current state, and open questions are written by the editor routine and refreshed whenever a new beat lands.