If you evaluate a per-step model router — swap a cheaper model in for one step of a logged agent trajectory and check whether the rest of the recorded trajectory still looks right — you are measuring a world that stops existing the moment you make the swap. A controlled study that actually forked live SWE-bench trajectories at the swap point found 61-94% of the agent's later actions changed once a different model took over, and a standard replay evaluator scored every swap-dependent outcome wrong as a result. If you shipped a router using replay numbers, you don't actually know whether it works.
"},{"heading":"Short answer","html":"No. Replaying a logged trajectory and substituting one model's output for another's assumes the rest of the trajectory would have happened the same way regardless of which model produced that step — an assumption the underlying paper tests directly and rejects. Forking live agent runs at the swap point and continuing them for real, instead of replaying the original log, shows the agent takes a substantially different path after a model swap: 61-94% of post-fork actions differ from what was logged, and static replay evaluators built on the old log mispredict the resulting outcome almost every time it actually mattered.
"},{"heading":"Builder model","html":"Two different things can happen after you swap a model mid-trajectory, and only one of them is what replay evaluation checks for:
\nThe paper's branching-rollout method makes this fork explicit instead of assuming it away: fork the real trajectory at the swap point, rebuild the actual sandbox, and let the new model run for real. Comparing that against a same-model control fork (same swap mechanics, same model on both sides) isolates how much of the divergence is the model swap itself versus ordinary sampling noise any two runs would show.
"},{"heading":"Mechanism","html":"An agentic trajectory is a sequence of (model output → environment response → next model input) steps, and each step's input depends on everything the environment returned from the step before it. Swap the model at step N, and step N+1's input is now built from a different model's tool call, file edit, or command — not the one the log recorded. The next model reasons over a different context than the original run ever produced, so its own output diverges, which changes the environment response again, compounding at every subsequent step. A replay evaluator that keeps consuming the original log's later steps is checking the new model's swapped-in step against a continuation the swap itself invalidated.
\nThe study's numbers show how fast this compounds: 74-77% of early swaps diverge at the very first action after the fork (versus 6-35% for same-model controls, which isolates how much of that is just normal run-to-run variance rather than the swap), and by the time the trajectory reaches its end only 3% of the originally logged states are still states the forked run actually passes through. Divergence shrinks the closer the swap happens to the end of the trajectory simply because there are fewer remaining steps left to diverge in — not because late swaps are safer to replay-evaluate.
\nThe outcome-level consequence is asymmetric and rare but real: all five task-outcome flips (an unsolved instance becoming solved, or the reverse) happened in swap forks, never in same-model controls, meaning a router evaluated only by looking at final-state accuracy on replayed logs can miss exactly the outcome changes a real deployment would produce.
"},{"heading":"Evidence","html":"Benchmark-result-backed: a controlled empirical study (COLM 2026) using branching rollouts on live SWE-bench agent trajectories, with a same-model control arm specifically designed to separate the model-swap effect from ordinary sampling and replay noise, and multiplicity-corrected confidence intervals on the reported divergence gap. The 3% valid-state figure, the 61-94% action-rewrite range, and the 0.00-0.11 patch-similarity result under a log-stitching replay evaluator are all measured outcomes of that experiment, not modeled estimates. Editorial synthesis: framing this as a caution specifically for agentic model-routing evaluation, distinct from the routing-policy question itself, is LLM Digest's own read of what the result implies for builders.
"},{"heading":"How to apply","html":"See when should an agent route a call to a cheaper model? for the routing-policy side of this decision — the shape production routers use to decide when to escalate — and what should an agent eval actually measure? for the broader discipline of auditing an eval's own correctness before trusting the score it reports, which this concept applies specifically to the case of a per-step model router.
"}],"evidence":[{"id":"gonuguntla-2026-replay-gap","kind":"benchmark-result","tier":"benchmark/result-backed","title":"The Replay Gap: Static Evaluation of Model Switching in LLM Agents Scores the Wrong World","note":"Ashritha Gonuguntla, accepted at COLM 2026. Tests the standard practice of evaluating a per-step agent model router by replaying a logged trajectory and substituting another model's recorded output, which assumes the rest of the trajectory is unaffected. The study forks live SWE-bench agent trajectories at controlled points, rebuilds the sandbox environment, continues each fork with a different model, and compares against same-model control forks that isolate ordinary sampling and replay noise. Across six paired runs (~900 rollouts), model-swap forks exceed their matched same-model control floors by +0.25 to +0.66 normalized edit distance (multiplicity-corrected confidence intervals exclude zero), rewriting 61-94% of the actions taken after the fork point. 74-77% of early swaps diverge at the very first post-fork action, versus 6-35% of same-model controls, leaving only 3% of the originally logged post-fork states still valid to replay against. Divergence decreases the deeper into the trajectory the fork happens, in both swap and control arms. All five task-outcome flips observed in the study occur in swap arms (upgrades rescuing an otherwise-unsolved instance, one downgrade losing the sole solve) and zero occur across 359 control forks. When the same swaps are scored with a log-stitching replay evaluator instead of a live rollout, the replay evaluator mispredicts every outcome call that actually depended on the swap and predicts patches with only 0.00-0.11 similarity to what the live rollout actually produced.","url":"https://arxiv.org/abs/2608.08239"},{"id":"story-4e6b8920803e5949-replay-gap","kind":"story","tier":"source story","title":"Static Evaluation of Model Switching in LLM Agents Scores the Wrong World","note":"","sid":"4e6b8920803e5949"},{"id":"model-switching-replay-gap-editorial-synthesis","kind":"editorial-inference","tier":"editorial inference","title":"LLM Digest synthesis","note":"This is a methodology result about evaluating agentic model routers, not a routing-strategy result — it says nothing about which routing policy to use (see the model-routing concept for that), only that the common way teams check whether a routing policy is safe to ship is measuring something other than what will actually happen in production."}],"related_topics":[{"slug":"agent-evaluation","title":"Measuring whether an agent actually worked is hard"},{"slug":"agent-benchmarks","title":"Agent benchmarks: fixed tasks that exercise real tool use"},{"slug":"agent-cost","title":"Agent token costs are unpredictable and easily run away"}],"related_playbook_cards":[],"related_storylines":[],"covers_evidence":["gonuguntla-2026-replay-gap","story-4e6b8920803e5949-replay-gap","model-switching-replay-gap-editorial-synthesis"]}