LLM Digest
Subscribe

Agent Know-How

Agent engineering · know-how

LLM-as-judge: model-graded evaluation of traces and outputs

🛠️ Solution·active·18 entries·18 sources·updated 2026-10-28

Use a model to grade a model: give an LLM the agent's output (or its full trace) plus a rubric, and have it return a structured verdict. It scales the judgment human raters can't keep up with — every production trace, every CI run — and is the practical backbone of agent evaluation when answers are open-ended.

Latest updates
  1. Novelty judges swing by 50+ points on small prompt changes
  2. Simulated control evaluations test agent monitors with an LLM that fakes tool outputs
  3. Decision models replace LLM judges only after tuning the 0.50 threshold on your own labels
  4. Audit an LLM judge for stability, chance-corrected human agreement, and ensemble divergence
  5. Mechanical checks catch errors LLM judges accept
State of the art

The pattern has moved past "ask a model to rate 1–5". Current judges read the whole trace and return categorized failures with causes, and they now run in three places: offline over production traces, before release in simulated deployment, and inside the agent loop as a grader that sends per-criterion feedback back to the agent.

Cost is the first lever teams pull. A frontier judge over every trace is too expensive, so teams fine-tune small open judges on their own failure clusters (about 1/100th the cost) or swap the generative judge for encoder classifiers that score many signals in one pass.

Trust is the open problem. Judges show position, verbosity, and language bias that raw accuracy hides, and an LLM-written rubric can be wrong even when the judge is calibrated. The working practice: validate the judge against held-out human labels, ensemble cheap judges to cut false positives, and use a deterministic check instead of a judge whenever the task allows one.

Trade-offs

The judge is itself a non-deterministic model: it has biases (verbosity, position, self-preference) and can be gamed. It needs its own validation against human labels, or it just launders noise.

Cheap fine-tuned judges narrow the cost gap, but they can overfit to the trace distribution they were trained on and miss novel failure modes. Ensembling judge personas cuts false positives but multiplies judge calls per artifact.

LLM-as-judge works best with a rubric, a held-out human-labeled set, and a need for explanations (which step failed) rather than one opaque score.

Why it matters for platform engineers

This is what makes continuous agent eval affordable: a judge you can run in CI and on live traffic to catch regressions a model upgrade or prompt change introduces.

The cost knob — frontier judge, fine-tuned local judge, or encoder classifier — is a real budget decision, and the judge itself becomes a dependency you must monitor and re-validate like any other piece of infra. Pairs with agent benchmarks for the fixed-task side of evaluation.

Developments 18

From a 1-5 score to structured, trace-level verdicts

3 entries · latest Aug 27, 2026

Judges now read whole trajectories and return categorized failures, run before release, or sit inside the agent loop as a grader.

·1 source

Deep Agents RubricMiddleware puts the grader inside the agent loop

LangChain's Deep Agents RubricMiddleware takes a newline-delimited checklist at invocation time and hands it to a separate grader sub-agent. The grader can call tools, such as a test runner, to gather evidence before it rules.

When a criterion fails, the grader's per-criterion feedback is injected back into the conversation and the agent re-runs, up to a configured cap. The rubric becomes a runtime input the agent iterates against, not an offline scoring pass.

Making the judge cheap enough to run on every trace

5 entries · latest Oct 20, 2026

Small fine-tuned judges, encoder classifiers, and shared-backbone heads cut judge cost and latency by one to two orders of magnitude.

·1 source

Decision models replace LLM judges only after tuning the 0.50 threshold on your own labels

Hamel Husain compared six decision models (Jev, Perplexity Decider, Liquid d1, Cloudflare Clef Flash, OpenAI Decisions) on 100 labeled traces from a leasing agent. The default 0.50 threshold is often wrong: tuning with cross-validation lifted Liquid d1 and Clef Flash, which otherwise missed his 80% precision and recall floor.

On cost, Perplexity cost 0.82 cents per 100 eval runs at F1 0.79. OpenAI Decisions reached F1 0.87 at 3.72 cents, about 4.5x more. Scores were not calibrated probabilities: one Clef Flash bucket predicted 8% failure but failed 60% of the time. Platt scaling fixed several models, not all. This is one small dataset.

For builders: tune and calibrate the threshold on saved scores before alerting on them.

·1 source

Morph Reflexes scores many failure modes in one pass over a trace

Morph Reflexes reads an agent trace once through a shared backbone and scores behavioral signals — looping, reasoning leakage, user frustration — with separate classifier heads off the same forward pass.

Reusing the KV cache gives sub-30ms inference and under 2ms of marginal latency per added signal, so "judge every failure mode" costs one read of the trace instead of N model calls.

Show 2 earlier entries

·1 source

Encoder classifiers can match generative judges for guardrail verdicts

"Do Encoders Suffice?" compares encoder classifiers with decoder (generative) judges for safety evaluation. For guardrail-style verdicts, a cheaper, lower-latency encoder often matches the generative judge.

Use an encoder when you need a fast inline check; keep a generative judge when you need a free-text explanation.

Auditing the judge, the rubric, and whether you need one

10 entries · latest Oct 28, 2026

Judges carry position, verbosity, and language bias, and rubrics can be wrong; calibrate against human labels or replace the judge with a deterministic check.

·1 source

Novelty judges swing by 50+ points on small prompt changes

Ai2's "Old Ideas, Novel Problems" benchmarks LLM novelty judges on 154 high-novelty ideas drawn from ICLR 2026 peer reviews. Across six judges (GPT-5.1/5.2/5.4, Claude Sonnet 4.5, Opus 4.5/4.6), changing a few words in the prompt shifted pairwise accuracy by more than 50 percentage points (paper-reported).

Retrieval and extra reasoning did not help consistently, and two purpose-built novelty systems scored below a plain prompted Opus 4.6 baseline. For builders: a judge validated on human-written papers may not transfer to generated ideas, so re-check any judge after every prompt or setup change.

·1 source

Simulated control evaluations test agent monitors with an LLM that fakes tool outputs

Redwood Research's paper on simulated control evaluations (SCEs) tests whether a monitor, the model that flags or blocks an agent's risky actions, would catch an agent deliberately trying to cause harm, such as quietly inserting a security flaw. An LLM simulates the tool outputs, so tests run under conditions closer to real deployment than static transcripts allow.

For builders: a monitor's catch rate on canned attacks overstates its real one. Measure it against an adaptive adversary in a simulated environment before trusting it as a gate.

Also relevant to Measuring whether an agent actually worked is hard

·1 source

Audit an LLM judge for stability, chance-corrected human agreement, and ensemble divergence

A Google Cloud consulting post lists the judge biases to test for: verbosity, position, halo, agreeableness, and self-enhancement. It cites swapped option order flipping verdicts over 75% of the time and judges missing up to 75% of bad outputs (figures as cited by the vendor, no methodology given).

Its audit has three steps:

  • Stability: score the same output several times and measure variance; high variance means the rubric is ambiguous.
  • Human alignment: use chance-corrected agreement, because a lenient judge looks good on mostly valid data.
  • Ensembles: one judge per criterion, and at least two models on the same output to flag divergence.

For builders: report chance-corrected agreement, not raw accuracy, before trusting a judge in CI.

Show 7 earlier entries

·1 source

Mechanical checks catch errors LLM judges accept

A paper on verifying Lean autoformalizations pairs an LLM semantic review with mechanical numerical and vacuity checks. Statements are checked by computation, not only by a model's reading of faithfulness.

On three labeled datasets the combined method had the lowest accepted-error rate (fraction of incorrect statements it accepted) among the methods the authors evaluated. Expert review also found translation and source defects in existing datasets. Results are paper-reported, in a formal-math domain.

For builders: where a claim can be checked by execution, arithmetic, or a vacuity test, run that check alongside the judge and return its output as checkable feedback. Keep the LLM judge for what only a model can assess.

Also relevant to Agents give fluent, confident-looking output even when it's wrong

·1 source

Zero-temperature judges still flip verdicts; the most consistent one was right only 51% of the time

A reliability audit stress-tested six frontier models as judges across four benchmarks, five prompt formats, two presentation orders, three temperatures, and ten repetitions. Verdicts changed across identical runs at temperature zero, and swapping answer order flipped the majority of verdicts on hard tasks. The most deterministic judge was perfectly consistent only by repeating wrong verdicts, agreeing with ground truth 51% of the time.

The authors propose a trustworthy verdict rate: the joint probability a verdict is reproducible, order-invariant, and accurate. Holistic rubric scoring beat pairwise win-rate and any single prompt tweak.

For builders: measure consistency and order-invariance per item, not just agreement, and prefer rubric scores over pairwise comparisons.

Also relevant to Measuring whether an agent actually worked is hard

·1 source

Ensembling judge personas cuts false positives more than a bigger model

Dan Luu's notes on agentic coding report that running independent judges under different personas — including one briefed to argue the opposite verdict — over the same artifact cuts false positives more reliably than upgrading to a stronger single model.

The cost is more judge calls per artifact, so it pays off where false positives are expensive to triage.

From related topics
  1. Measuring whether an agent actually worked is hardJev ships in LangSmith Evals as a faster, cheaper alternative to LLM judges
  2. Measuring whether an agent actually worked is hardMulti-perspective adjudication catches subtle factual errors single annotators miss
  3. Measuring whether an agent actually worked is hardGitHub evaluates an LLM for secret scanning with a recall guardrail it cannot trade away
  4. Measuring whether an agent actually worked is hardEvaluators can keep the right verdict while their reasons stop tracking the evidence
  5. Measuring whether an agent actually worked is hardLangSmith Tuned Evaluators attach a Perceived Error signal to production traces
  6. Measuring whether an agent actually worked is hardSimilarweb's first report rubric rewarded source quantity until it was recalibrated
  7. Measuring whether an agent actually worked is hardChecking 100 human-annotated traces shows automated evals diverge from human raters
Evidence · 18 sources
All sources behind this page