Agent Builder Foundations

Agent memory's payoff is conditional, not automatic, on three axes these evaluations independently expose: how good the underlying retrieval and reasoning still is (the leaderboard's top score is 58.02 out of 100 — state of the art is still far from solved), whether the memory content itself distorts reasoning on the current task (MemTrapBench found every tested memory framework underperforms no memory at all, by more than 10% at best), and which model it's attached to (ALTK-Evolve measured gains from +16.1 percentage points down to +0.0 across eight models on the identical benchmark). Treat memory as an intervention you A/B test per model and task, not a component you install once.

Three distinct ways a memory system can land, and each needs a different check before you trust it in production:

• It helps, and the size of the help depends on the model. ALTK-Evolve's guideline-extraction memory gained gpt-oss-120b +16.1pp and DeepSeek-V3.2 +9.5–16.1pp, but Claude Opus 4.6 only +4.1pp and GLM-5 nothing — the same mechanism, eight different outcomes. A model that's already strong on a task has less headroom for memory to fill.
• It does nothing measurable. GLM-5's 0.0pp result is the "saturated" case: the model already had the capability the memory guidelines were meant to supply, so the memory added token overhead and complexity for no return.
• It actively hurts. MemTrapBench's finding is the sharpest: memory that is stored and retrieved with perfect accuracy can still distort the model's reasoning on the current task through reasoning fixation or belief distortion, and every framework the authors tested landed here — below the no-memory baseline.

A recall-accuracy check ("did the memory system retrieve the right fact?") only catches the mechanics. It cannot catch the third failure mode, because the retrieved memory can be exactly correct and still make the model perform worse.

The Agent Memory Leaderboard fixes a system boundary that makes memory systems comparable at all: the memory implementation only owns Add and Search, while the platform owns Answer and Eval, so a submitted system can't tune its score by controlling how answers are graded. Under that boundary, evaluated across fact recall, multi-hop integration, temporal understanding, governance, personalization, rule execution, safety, and privacy, the best of 69 completed submissions in the first cycle scored 58.02 out of 100 on the Commercial Products track — a concrete signal that current memory systems, even purpose-built commercial ones, are still a coin flip's width from a passing grade on their own designed benchmark.

MemTrapBench probes a mechanism most memory benchmarks don't test: does the *content* of a retrieved memory bias the model's reasoning on the current, unrelated-in-substance task? Its two named traps — reasoning fixation (the model over-anchors on a retrieved prior approach) and belief distortion (a retrieved fact shifts the model's belief state in a way that leaks into unrelated reasoning) — are constructed so that a memory system can pass a standard "did it retrieve the right fact" check and still fail here. Across two model families and five memory frameworks, that gap wasn't rare: every framework tested underperformed a no-memory control, with the best still losing more than 10%. The authors' fix, AdaptiveMem, works at inference time by instructing the model to recognize when a retrieved memory looks like it's about to bias current reasoning and discount it — a mitigation layered on top of retrieval, not a change to what gets stored.

ALTK-Evolve's mechanism is a self-distillation loop: an agent's own trajectories, both successful and failed, get mined for behavioral guidelines, which are consolidated into a reusable set and reinjected into future runs. The reason its effect varies by model isn't a bug in the method — it's that a guideline only helps a model that doesn't already reliably produce the behavior the guideline describes. Measuring across eight models on AppWorld's 585 multi-step tasks is what surfaced the dosing pattern: strong models with real capacity gap benefited from the full guideline set, weaker models did best with a compact core plus task-specific retrieval (minimizing token overhead), and a model already at ceiling on the task gained nothing regardless of how the guidelines were dosed.

• Benchmark-result-backed (Agent Memory Leaderboard): a standardized, fixed-boundary public evaluation with 136 registered teams and 69 completed submissions, reporting exact top-3 scores for the first cycle.
• Benchmark-result-backed (MemTrapBench): a controlled comparison across two model families and five memory frameworks against a no-memory baseline, with an explicit quantitative drop (>10% for the best method) and a named, reproducible failure mechanism.
• Benchmark-result-backed (ALTK-Evolve / IBM Research): a controlled eight-model study on a fixed 585-task benchmark (AppWorld), reporting per-model percentage-point deltas rather than an aggregate claim.
• Editorial inference: that these three, run independently and not citing each other, converge on "memory's effect must be measured, not assumed" is LLM Digest's synthesis across three differently designed evaluations.

• Measure your memory system's effect on your own model and task before trusting it, using a no-memory control. ALTK-Evolve's per-model spread (+16.1pp to +0.0pp) on the identical mechanism means a result from someone else's model tells you little about yours.
• Don't stop at recall accuracy. A memory system can retrieve the exactly correct fact and still make your agent worse, per MemTrapBench — add a check for whether retrieved memory content changes the model's behavior on tasks it's otherwise unrelated to.
• Size the guideline or memory payload to the model, not to the theoretical maximum. ALTK-Evolve's own finding — weaker models did best with a compact core plus targeted retrieval, not the full set — means "more memory" is not a safe default even when memory helps at all.
• Treat a memory product's leaderboard rank as a starting point, not a verdict. The current best public score (58.02/100) is well short of solved, and track-to-track comparisons on the leaderboard are explicitly not valid, so a top rank in one track doesn't transfer to your production task or track.
• Re-test after a model swap. Because ALTK-Evolve shows the same memory mechanism's payoff is model-specific, upgrading or switching the underlying model invalidates a prior memory A/B result — re-run it rather than assuming the win carries over.

• Shipping a memory layer on the strength of a vendor's leaderboard rank or a paper's aggregate claim, without measuring its effect against a no-memory control on your own model and task.
• Validating a memory system only on recall accuracy (did it retrieve the right fact?) and missing that correctly retrieved content can still distort reasoning and lower task performance, per MemTrapBench.
• Assuming a memory mechanism that helped a smaller or weaker model will help equally after a model upgrade, when the ALTK-Evolve results show a saturated, already-capable model can gain nothing from the same mechanism.
• Dosing every model with the same full guideline or memory payload regardless of size or capability, adding token overhead for models that get no measurable benefit from it.

See agent memory for the architecture problem of what to persist and how to recall it, and agent evaluation for the broader difficulty of measuring whether an agent's trajectory — not just its final answer — actually worked.