Agent Builder Foundations

No. Replay substitution assumes the rest of the trajectory would unfold the same way whichever model produced the swapped step. Forking live agent runs at the swap point and continuing them for real rejects that assumption: the agent takes a different path, only 3% of logged states remain reachable, and replay evaluators get the outcome wrong exactly when it matters.

A model swap is a fork, not an edit.

• Replay treats the swap as a local patch. Substitute one step, keep every later logged step, diff the ending. Cheap: no execution, no environment.
• Reality branches. From the swap on, the new model reads tool outputs and errors produced by its own actions, so every later step differs. The two runs share a prefix and nothing after it.

The fix is a branching rollout: fork the real trajectory at the swap, rebuild the sandbox, let the new model run, and compare against a same-model control fork. The control isolates how much divergence is the swap versus ordinary sampling noise.

An agent trajectory is a chain of model output, environment response, and next model input. Each input depends on what the environment returned for the previous action. Swap the model at step N and step N+1's input is built from a different tool call, edit, or command. The next output diverges, which changes the next environment response, and the difference compounds every step. A replay evaluator that keeps consuming the original log is grading the swapped step against a continuation the swap invalidated.

Three features of the measured divergence matter for builders:

• It starts immediately. Most early swaps diverge on the very first post-fork action, far above the same-model control rate.
• It shrinks late in a run only because fewer steps remain, not because late swaps are safe to replay.
• Outcome flips are rare but real. Every solved-to-unsolved or unsolved-to-solved flip came from a swap arm, never a control, so final-state accuracy on replayed logs misses exactly the changes a deployment would produce.

• Discount replay-measured router accuracy. Treat it as unvalidated until confirmed with live rollouts.
• Evaluate with branching rollouts. Fork at the decision point, run the swapped-in model against a real or faithfully rebuilt environment, and grade the outcome, not a diff against the log.
• Add a same-model control arm. Without it, sampling noise and swap effects are indistinguishable.
• Sample early swap points heavily if your router switches models early, where first-action divergence is highest.
• Decide what history crosses the swap. A recipient imitates the habits in raw history it inherits; test summarized handoffs against raw ones, especially when a weaker model's history goes to a stronger one.
• Keep cost and correctness as separate claims. Fewer frontier calls is measured correctly from logs; unchanged success rate needs live evidence. See model routing for the cost side.

• Reporting replay-substitution accuracy as if it were a live production measurement.
• Assuming a swap affects only its own step.
• Favoring late swap points in evals because divergence looks smaller there.
• Running swap experiments without a same-model control, misattributing sampling noise to the swap or the reverse.
• Treating a router's cost savings as proof it is safe to ship.

See model routing for the routing policy itself and agent eval design for auditing an eval's correctness before trusting its score.