Agent Engineering Wiki

The root cause is now usefully framed as role confusion: an LLM has no reliable channel that separates "instructions from my operator" from "data I was asked to process," so text arriving as a tool result or a fetched page can assume the operator's role and be obeyed. Naming it this way clarifies why prompt hygiene can't fix it — the model is doing exactly what it was built to do, treating in-context text as authoritative — and why the durable controls live in *authorization* rather than in detecting "malicious" strings. There is no clean fix, only layered mitigation, and each layer has known holes.

Guardrail models that screen inputs/outputs are the common defense, but recent work shows the very reasoning that makes them effective also makes them a target — "From Shield to Target" demonstrates denial-of-service attacks that weaponize a guardrail against the agent it protects.

Sandboxing is necessary but not sufficient: a coding-agent sandbox contains code execution yet does nothing about credential authorization — the agent inside the sandbox still holds tokens that injected instructions can abuse.

The threat compounds in multi-agent systems, where one compromised agent's output is another's trusted input; new benchmarks (Deep-XPIA) are emerging specifically to measure cross-agent (indirect) prompt-injection exposure.

The durable lesson is least privilege: scope what the agent can touch so a hijack has a small blast radius. The operational framing is consolidating around agent-as-identity: an autonomous agent holds credentials and takes actions, so it is a non-human identity that needs the same lifecycle, scoping, and audit as a service account. Security teams warn that most organizations don't yet treat agents that way, leaving an ungoverned class of actors with standing privileges that injection can borrow.

Red-teaming practitioners (Gray Swan, with OpenAI's Zico Kolter) push the same point from the offensive side: agent security is *not* "cybersecurity with AI sprinkled on" — the attack surface is the model's behavior under adversarial input, so it needs dedicated red-teaming of the agent's decisions and tool use, not just the perimeter around it.

A subtler erosion comes from the agent's own plumbing: "Governance Decay" shows that the context compaction used to keep long sessions affordable can silently evict the safety and governance constraints stated up front, so a guardrail that held at turn one is simply gone by turn fifty — meaning the defenses against injection have to be pinned outside the compactible window, not trusted to survive summarization.

Industry framings are converging on where the ReAct loop actually breaks: practitioner guidance now locates the vulnerabilities separately in context (what gets read in), reasoning (what the model decides), and tool execution (what it's allowed to do), naming memory poisoning and rogue tool execution as the concrete failure modes and recommending defense-in-depth — layered controls plus an LLM-as-judge critic reviewing the agent's own decisions — structured against a named threat model (MAESTRO) rather than ad hoc rules.

Model providers are also treating jailbreak resistance as an ongoing, versioned release concern, not a one-time hardening pass: Anthropic's redeployment of Claude Fable 5 ships updated cybersecurity safeguards alongside a new industry jailbreak framework, evidence that the red-teaming push (Gray Swan, Kolter) is feeding back into shipped model updates.

That framework is getting concrete follow-through, not just an announcement: Anthropic has since published what its cyber classifiers do and don't block alongside a first draft of a jailbreak *severity* framework — grading how bad a successful jailbreak is, not just detecting one, which lets a provider triage and prioritize fixes instead of treating every bypass as equally urgent.

The harness default is also moving toward stricter authorization: Claude Code changed its default permission mode to "Manual" across the CLI, VS Code, and JetBrains (and stopped AskUserQuestion dialogs from auto-continuing) — shipping least-privilege as the out-of-the-box behavior rather than an opt-in setting, which matters because most successful injections exploit exactly the gap between what a default configuration permits and what a user actually intended to authorize.

Jailbreak defense is getting more granular: Anthropic followed the Fable 5 redeploy with a published account of what its cyber classifiers do and don't block plus a first-draft jailbreak *severity* framework — grading how bad a bypass is, not just whether one occurred. In parallel, a harness default shifted toward least privilege out of the box: Claude Code now defaults to "Manual" permission mode across its CLI and IDE integrations, closing the gap between what a default configuration permits and what a user actually authorized — the same gap most successful injections exploit.

That builds on the standing shift from "filter the prompt" to "govern the actor" (agent-as-identity, per-parameter permissions, approval-gated writes), the role confusion framing of injection's root cause, the network-egress perimeter as a managed control (Google Cloud VPC Service Controls), and a named threat model for the ReAct loop's three attack points (MAESTRO).

This is the security boundary of the whole agent stack, and it maps to ordinary ops controls done right: scoped credentials, per-tool authorization, network egress limits, and human approval on high-impact actions. The mistake is treating a sandbox or a guardrail model as the answer; both are layers, and both have published bypasses. Every tool you connect (see tool use) widens the attack surface, so authorization and blast-radius limits — not prompt hygiene alone — are the real control.