If you grade agent traces with an LLM judge, the score is not a fact about the agent. It is the agent filtered through a second model with its own failure modes. Before wiring a judge into CI gating or a dashboard, find out whether it favors a slot, a length, or a language, and whether its accuracy on your eval set predicts catching the production failure you care about.
"},{"heading":"Short answer","html":"LLM-as-judge accuracy on a held-out set is necessary but not sufficient. Judges carry systematic biases toward position, length, and language that raw accuracy hides, and a judge that passes a benchmark can still miss the real failure the eval exists to catch.
\nA deeper gap sits underneath: a judge grades the answer's plausibility, not whether the trajectory earned it. In GroundEval's case study, two frontier judges scored an answer 0.85 or higher although the agent never retrieved the evidence it depended on. Treat the judge as a component under test, and where trajectories have checkable evidence, replace it with a deterministic scorer.
"},{"heading":"Builder model","html":"A judge is a classifier with a prompt instead of a training loop, and classifiers fail in ways one aggregate accuracy number doesn't show. Judge designs sit on a spectrum:
\nBoth need what an agent needs: a held-out test set built from real failures, not the cases the judge was tuned on.
\nEither design grades what the agent said, not what its trajectory can prove. When the task has a checkable evidence path — a document that should have been retrieved, a time window to reason within, a causal mechanism to use — a deterministic scorer over the trajectory closes a gap no judge tuning can.
"},{"heading":"Mechanism","html":"An LLM judge produces a verdict conditioned on the response, a rubric, and often a second response to compare. Because the verdict is a model output, it inherits model-level artifacts:
\nSwapping the order of compared responses probes position bias directly: if the verdict flips with the slot, the judge is responding to position, not content. BabelJudge shows how much raw accuracy hides here; order consistency fell to 0.480 in Swahili, near random.
\nAgent trajectories multiply the places bias can hide. The judge scores a sequence of tool calls and decisions, and can be well calibrated on final-answer correctness while unreliable on whether the path was sound. Cheaper judges (fine-tuned encoders, distilled small LLMs) trade rubric flexibility for cost and speed, and face the same requirement: measure them against the failures you care about, not the cases used to tune them.
\nDeterministic trajectory scoring removes the judge where evidence is checkable. GroundEval generates questions from a domain configuration, lets the agent answer freely, then scores the answer and recorded trajectory on three tracks:
\nEach track grades against ground truth the system already knows — what was retrievable, when, and through what path — so it catches a fluent, ungrounded answer that a judge reading only the final text cannot detect by construction.
"},{"heading":"How to apply","html":"Before trusting a judge's verdicts, run these checks:
\nSee agent evaluation for grading agent trajectories, LLM-as-judge for the model-graded pattern this page interrogates, and does a high benchmark score predict production reliability? for whether the benchmark itself predicts real-world behavior.
"}],"evidence":[{"id":"google-cloud-audit-llm-evaluators-2026","kind":"primary-doc","tier":"primary-doc-backed","title":"Who judges the judge? Three helpful ways to audit your LLM evaluators","note":"Google Cloud's audit recipe: measure score variance across repeated passes on identical inputs, compare against human experts with chance-corrected metrics, and use at least two judge models on decomposed criteria, flagging divergence. Cites prior work that swapping option order can flip a verdict over 75% of the time. Gives no thresholds or pass counts.","added":"2026-10-21","url":"https://cloud.google.com/blog/topics/consulting/three-ways-to-audit-your-llm-evaluators","sid":"4e6e796fa0c5bfc0"},{"id":"hamel-decision-model-evals-2026","kind":"production-field-report","tier":"production field-report-backed","title":"How To Use Decision Models for Evals","note":"Tests six small decision models on 98 apartment-leasing agent traces (61 failures) with five-fold cross-validation. Tuning the default 0.50 threshold recovers performance; the cheapest model scored F1 0.79 at 0.82 cents per 100 runs versus 0.87 at 3.72 cents. Calibration must be measured: one group predicted 8% failure but 60% failed. Small, single dataset.","added":"2026-10-21","url":"https://hamel.dev/blog/posts/decision-model-evals","sid":"58b479d20c57cce8"},{"id":"groundeval-2026-judge-free-agent-evaluation","kind":"benchmark-result","tier":"benchmark/result-backed","title":"GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation","note":"In a case study, two frontier LLM judges scored a plausible agent response 0.85 or higher while the recorded trajectory showed the agent never retrieved the artifact its answer depended on; GroundEval scored it 0.000. Replaces the judge with a deterministic scorer over grounded, time-bounded, access-controlled evidence on three tracks: Silence, Perspective, and Counterfactual. The authors' case studies suggest the failure is common, not exceptional.","added":"2026-07-10","url":"https://arxiv.org/abs/2606.22737"},{"id":"babeljudge-2026-judge-bias","kind":"benchmark-result","tier":"benchmark/result-backed","title":"BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories","note":"Builds gold-labeled pairs by perturbing known-good answers (no human annotation) and measures position bias, verbosity bias, order inconsistency, and cross-lingual degradation in a Qwen2.5-7B-Instruct judge. Raw accuracy (0.835 Hindi vs. 0.660 Swahili) understates the gap: bias-penalized reliability falls from 0.714 to 0.550, and order consistency collapses to 0.480 under slot swaps, near random. One judge model tested.","added":"2026-06-28","url":"http://arxiv.org/abs/2606.22329v1"},{"id":"encoder-decoder-safety-judges-2026","kind":"benchmark-result","tier":"benchmark/result-backed","title":"Do Encoders Suffice? A Systematic Comparison of Encoder and Decoder Safety Judges for LLM Adversarial Evaluation","note":"Benchmarks fine-tuned encoder classifiers against LLM-based judges for detecting harmful outputs across several attack techniques, testing whether a much cheaper, lower-latency judge architecture can substitute for an LLM judge without a major accuracy loss. Safety-classification setting only; no headline figure is recorded here.","added":"2026-06-28","url":"http://arxiv.org/abs/2606.25782v1"},{"id":"langchain-fireworks-trace-judge-2026","kind":"production-field-report","tier":"production field-report-backed","title":"Building a 100x Cheaper Trace Judge with Fireworks","note":"LangChain and Fireworks fine-tuned a small open model on production trace labels and matched frontier-judge performance at roughly 1/100th the cost. Vendor-reported, on their own narrow trace-grading task. Shows a judge's behavior can be distilled and re-validated rather than locked to whichever frontier model wrote the first version.","added":"2026-06-28","url":"https://www.langchain.com/blog/building-a-100x-cheaper-trace-judge-with-fireworks"},{"id":"linear-sales-email-eval-miss-2026","kind":"production-field-report","tier":"production field-report-backed","title":"Why most AI evals would miss the Linear sales email failure","note":"Practitioner postmortem on a real production failure that a typical eval suite and judge would have scored as a pass. Single incident, no rates; shows judge accuracy on a benchmark does not guarantee the judge catches the failure that actually matters.","added":"2026-06-28","url":"https://tenureai.dev/writing/why-most-ai-evals-would-miss-the-linear-sales-email-failure"},{"id":"llm-judge-reliability-editorial-synthesis","kind":"editorial-inference","tier":"editorial inference","title":"LLM Digest synthesis","note":"For agent builders, an LLM-as-judge score is an output of a measurement instrument with its own bias profile, not a ground-truth label, so the judge needs the same validation discipline as the agent it grades.","added":"2026-06-28"}],"related_topics":[{"slug":"agent-evaluation","title":"Measuring whether an agent actually worked is hard"},{"slug":"llm-as-judge","title":"LLM-as-judge: model-graded evaluation of traces and outputs"}],"related_playbook_cards":["pb-audit-llm-judges-for-position-and-language-bias"],"related_storylines":[]}