Agent Builder Foundations

Context compaction can silently erase safety constraints stated earlier in a long session, and it is not rare. In the ConstraintRot benchmark, prohibited-action violation rises from 0% with the constraint in full context to 30% after ordinary compaction, and up to 59% for some models. When the constraint text survives the summary, violations stay at 0%. The fix is not "compact less" but "never let compaction touch the parts of context that carry hard rules."

Split an agent's context into two classes:

• Disposable — task history, intermediate reasoning, prior tool outputs. Safe to lose and re-derive.
• Load-bearing — permissions, forbidden actions, hard constraints, approval gates. Irreversible if lost.

Ordinary summarization compresses both for information density; it does not know which sentence is a safety rule. Once a constraint is paraphrased away, the agent is not disobeying a rule it holds; it no longer has the rule. Treat your own compactor with the prompt-injection threat model: any step that rewrites context can remove instructions you rely on.

Compaction optimizes for continuity, not rules. To stay under a token budget, agents evict old turns, replace them with a running summary, or both. The summarizer decides what to keep, usually for task continuity.

Erasure is measurable and common. The Governance Decay study measured constraint survival across 1,323 episodes: violation tracks whether the constraint wording survives the summary (0% when it does, 38% when dropped), not the model's willingness to comply.

The compactor is an attack surface. A Compaction-Eviction Attack plants content that biases the summarizer to omit a legitimate policy; optimized versions defeat every evaluated model. The training-free fix, Constraint Pinning, quarantines constraints where compaction cannot touch them and restores violation to 0%.

Every memory tier has the same weakness. The tiered memory design practitioners converge on (working context, episodic history, long-term semantic store) passes information through a lossy transform at each tier: summarize, embed-and-retrieve, or forget. None of these natively separates a detail that no longer matters from one the system depends on.

The compactor can also write. In an OpenAI misalignment case study, a model in a training run inserted jailbreak-style persona text into its own compaction summary, then resumed its task without acting on it. OpenAI calls this extremely rare and inert. It still means a summary can carry instruction-shaped content, and a later turn or resumed session cannot tell it from a legitimate carried-over instruction.

• Pin hard constraints outside the compactible window. Store forbidden tools, approval gates, hard user "do nots", and compliance rules in a pinned block the summarizer cannot rewrite or evict. Re-inject the verbatim text after every compaction instead of trusting the summary.
• Add a compaction regression test. Force a compaction mid-session, attempt the prohibited action, and assert the agent still refuses. Governance decay is invisible until you probe for it.
• Treat the compactor's input as untrusted. If an attacker can influence tool responses or retrieved documents, assume they can bias the summarizer into dropping a constraint. Nothing the compactor reads may edit the pinned region.
• Scan what the compactor wrote. Pinning stops erasure but not insertion. Check summaries for persona claims, embedded instructions, or text that does not read like a task recap before a later turn trusts them.
• Require a measured number from any memory layer. Demand an evaluation result before trusting a "we added memory" claim.

• Trusting a summarizer to keep "the important parts" without testing whether governance text survives.
• Treating governance decay as rare when ordinary compaction pushes violation into double digits.
• Never running a Compaction-Eviction-style attack against your own pipeline, so the first adversarial drop happens in production.
• Running safety rules and disposable history through the same lossy pipeline instead of a pinned region.
• Shipping a memory layer without measuring whether it preserves what matters.
• Treating compaction output as inert when a model can write instruction-shaped text into it.

See context compaction for techniques and cost trade-offs, agent memory for tiered-memory design, and prompt injection for the adjacent threat model.