Agent Builder Foundations
Evidence-tiered mechanisms for agent builders
Agent foundations
Can you trust what your agent remembers?
Concept·Memory and context·4 sources·updated 2026-10-06
If your agent writes facts, decisions, or reasoning traces to a store it reads in later sessions, that store is writable by an attacker or by the agent's own drift. Untrusted content processed today (an email, a tool result, a retrieved document) can plant an entry that survives into every future session and is treated as trusted state with no further scrutiny.
No, not by default. 2026 benchmarks show three ways memory goes wrong:
- Forged reasoning: FARMA reaches up to 100% attack success under baseline conditions.
- Poisoned facts via routine content: MemGhost reaches 87.5% end-to-end success against a real agent stack and transfers across architectures and backends.
- Sycophancy: agents over-trust stored memories over fresh, correct evidence, with no attacker involved.
Keyword filters and consensus checks do not close the gap; only purpose-built defenses tested so far do.
Memory-trust failures come in three classes:
- Fact or preference poisoning. Untrusted content the agent ingests is written to memory and reused as ground truth. The attacker never touches the agent, only what it reads.
- Reasoning poisoning. The attacker forges the record of *why* the agent decided something, so later self-consistency or majority checks see corroboration instead of an outlier.
- Sycophancy (no attacker). The agent weights a stored memory over better, current evidence because retrieval surfaced it and nothing forces re-verification.
The shared root cause: once something is in memory, most pipelines read it back as trusted without asking how it got there or whether it still holds.
Reasoning forgery beats surface defenses. A forged reasoning entry phrased evasively slips past keyword filters. If it cites or echoes itself across turns, consensus checks count it as agreement. FARMA uses both. The SENTINEL defense instead scores each candidate entry against structural forgery signals before it is trusted.
Poisoning rides the normal ingestion path. MemGhost needs no jailbreak prompt. It crafts content, such as an email, that the agent processes, writes to memory without the user noticing, and acts on in a later, unrelated session. On WhisperBench the payloads transfer across agent architectures and memory backends and survive the defenses tested, including 71.4% success against the Claude Code SDK.
Both attacks exploit one gap. The write path stores agent-generated or external content as safe, and the read path treats anything already stored as more trustworthy than it should be.
It mirrors compaction failure. In does compacting an agent's context put its safety rules at risk?, a legitimate constraint is silently dropped. Here, an illegitimate one is silently added and kept.
- Treat every memory write path as untrusted input. Tool outputs, retrieved documents, parsed emails, and the agent's own reasoning summaries all need prompt-injection-level scrutiny before a future session trusts them.
- Do not rely on keyword filters or consensus checks alone. FARMA is built to beat both. For reasoning-forgery defense, score entries structurally, as SENTINEL's Reasoning Guard does.
- Run an adversarial replay test on your real ingestion path. Feed attacker-controlled content through your actual email, tool, and document ingestion and check whether it lands unnoticed in the persistent store. Vendor agent tooling does not protect memory writes by default.
- Test sycophancy separately. Give the agent a stored memory that conflicts with correct, fresh evidence and check whether it defers to the memory. An adversarial-only suite will not catch this.
- Trusting a memory entry because the agent wrote it, without checking how it got there.
- Defending memory with keyword filters or consensus checks only.
- Assuming single-turn prompt-injection defenses cover memory, which is written once and replayed every session.
- Never testing whether attacker content can reach persistent storage through real ingestion.
- Treating sycophancy as out of scope because it is not an attack.
See agent memory for tiered-memory design, prompt injection for the single-turn threat model, and does compacting an agent's context put its safety rules at risk? for the mirror-image failure.
newest first
- BenchmarkYour Agent's Memories Are Not Its Own: Forged Reasoning Attacks on LLM Agent Memory and Defensesbenchmark/result-backed
FARMA poisons an agent's remembered reasoning traces rather than its facts. Evasive language bypasses keyword defenses; self-referential reinforcement defeats consensus defenses. Attack success reaches up to 100% under baseline conditions across 50 trials. The paper's SENTINEL defense, a Reasoning Guard scoring five weighted forgery signals, cuts success to as low as 0% with no false positives across 326 benign traces. Attack and defense come from the same authors.
- BenchmarkWhen Claws Remember but Do Not Tell: Stealthy Memory Injection in Persistent Personal Agentsbenchmark/result-backed
MemGhost generates one-shot payloads delivered through untrusted content such as email, which get silently written to persistent memory and reused later as trusted state. On WhisperBench (108 cases, five risk categories, fact and preference poisoning), end-to-end success reaches 87.5% against OpenClaw with GPT-5.4 and 71.4% against the Claude Code SDK with Sonnet 4.6. Attacks transfer across agents (NanoClaw, Hermes Agent) and backends (filesystem, Mem0) and survive the input-, model-, and system-level defenses tested.
- BenchmarkMemSyco-Bench: Benchmarking Sycophancy in Agent Memorybenchmark/result-backed
Argues most memory benchmarks only check storage, retrieval, and updates. Proposes five tasks testing whether an agent rejects memory as insufficient evidence, respects its scope, resolves memory-vs-evidence conflicts, tracks updates, and uses valid memory for personalization. Surfaces sycophancy, over-trusting a stored memory over fresh correct evidence, as a non-adversarial failure mode.
- EditorialLLM Digest synthesiseditorial inference
Persistent agent memory deserves the same threat model as prompt injection: anything written into long-term memory from an untrusted source, or generated by the agent itself, can later be replayed as trusted context, and a reasoning trace is exactly as forgeable as a fact.