What the Trace Shows and What the Label Says: Auditing Behavioral Detectors in Agent Evaluation
Frames trace labels (answer correctness, tool verification, fault propagation) as measurements that need auditing.
AI Storyline
3 items · 3 sources · 3 days
Operational story trace
Auditing Evaluation
Follow in this browser to see new updates on your Live feed.
Current stateDeveloping
Latest change
A new paper tests model self-confidence and LLM-as-a-judge as cheaper ways to audit human-in-the-loop evaluation, aiming for statistically sound accuracy estimates.
Earlier contextThe story so far
Three papers in six days audit the measurements behind agent and model evaluation rather than the models. First came trace-derived labels, then a headline memory gain in XLand-MiniGrid.
editor-curated · source-linked
Arc
Oct 21Oct 26 · now
LABELS · Oct 21Trace-derived labels are measurements, not ground truth1 source · show source ▾
NOISE · Oct 24A 25-point memory gain gets audited against seeds, maps and noise1 source · show source ▾
Memory or a Clock? Auditing a Memory Advantage in XLand-MiniGrid with Seeds, Maps and Evaluation Noise Apart
Applies the audit mindset to a reported LSTM-over-MLP gain of 25 points, separating seed, map and evaluation noise.
NOW · Oct 26Self-confidence and LLM judges proposed to scale human-in-the-loop audits1 source · show source ▾
From Self-Confidence to Statistical Accuracy: Scaling Human-in-the-Loop Evaluation with Direct and LLM-Assisted Auditing
Moves from diagnosing evals to a method: self-confidence and LLM judges to cut human audit cost.
Day 1 Wednesday, Oct 21, 2026
What the Trace Shows and What the Label Says: Auditing Behavioral Detectors in Agent Evaluation
Frames trace labels (answer correctness, tool verification, fault propagation) as measurements that need auditing.
Day 2 Saturday, Oct 24, 2026
Memory or a Clock? Auditing a Memory Advantage in XLand-MiniGrid with Seeds, Maps and Evaluation Noise Apart
Applies the audit mindset to a reported LSTM-over-MLP gain of 25 points, separating seed, map and evaluation noise.
Day 3 Monday, Oct 26, 2026
From Self-Confidence to Statistical Accuracy: Scaling Human-in-the-Loop Evaluation with Direct and LLM-Assisted Auditing
Moves from diagnosing evals to a method: self-confidence and LLM judges to cut human audit cost.
What to watch — open questions
- Do agent benchmarks begin publishing detector and label audits alongside scores?
- Does LLM-assisted auditing hold its accuracy estimates on agentic traces, not just single-turn outputs?
How this thread was built
editor wrote the arc · 3 beatswatcher 1 status change
Storylines are threaded mechanically from the feed: stories that share a distinctive anchor across multiple days and sources. Each item links to its original source. The evidence trace, current state, and open questions are written by the editor routine and refreshed whenever a new beat lands.