Agent Know-How

OpenAI's Vinoth Govindarajan argues, using incidents such as OpenClaw, that production agents fail beyond model hallucination. He places the fix in the harness:

• explicit state ownership
• serialized concurrent state mutations, so two in-flight actions cannot corrupt shared state
• execution authority scoped per step
• actions validated at the user-visible edge, not by the model's own account of what it did

Each item is an invariant a team can audit its harness against.

Irakli Betchvaia's Typed Domain Grounding embeds a domain-specific language inside a mainstream typed language, so the host compiler and type checker, not a second LLM pass, reject invalid generated output. He evaluates it on kUML benchmarks and an infrastructure-as-code example with generate-compile-repair loops.

For code generation, the cheapest verifier is often a compiler you already have.

In an InfoQ talk on LLM-powered selection systems, Jendrik Jördening restricts model output to a constrained schema, separates semantic extraction (where the model is needed) from the deterministic code that acts on it, and validates the model's choices with a discriminator model before they reach the database.

He structures the pipeline as an MVC-style split so non-determinism stays in one layer instead of leaking into storage and downstream logic.

Brex routes production onboarding-agent workflows through Temporal Cloud instead of a bespoke retry loop. Long-running completion rose from roughly 96% to 99.9%, and the workflow code stayed unchanged because the durable runtime is swapped in underneath it.

Checkpoint recovery and exactly-once execution become infrastructure you buy, not logic you re-implement per agent. The same article covers running that code in-process for evals; see agent evaluation.

A three-person agency got a $14,000 AWS bill in one day after attackers extracted static access keys with unrestricted Bedrock access and burned them on Claude calls. In the DN42 incident, an autonomous agent with open-ended AWS access provisioned $6,531 of oversized infrastructure in 24 hours. A credit-card charge caught both, because Cost Explorer and Budgets work off data that lags roughly a day.

The fix is scoped credentials and action-time alerts applied to spend: IAM roles instead of static keys, service-control policies blocking expensive instance families, and CloudTrail alerts on RunInstances and InvokeModel.

An essay splits agent reliability into three legs:

• Identity has no purpose-built primitive. Platforms retrofit workload identity: SPIFFE-based identities (Gemini Enterprise), service principals plus token brokers (Microsoft Entra). The fit is poor because two runs of the same agent can behave differently.
• Execution borrows the distributed-systems playbook: checkpoint recovery, exactly-once guarantees, cgroup quotas, per-session microVM or gVisor isolation.
• Intent is least solved. Models drift from the task or report false completion. Fixes split between LLM-graded trajectory checks and cheaper encoder classifiers that score task completion, an auditability and cost trade-off.

The identity leg overlaps with sandboxing and scoped credentials.

DialFork targets a failure factual checks miss: each utterance is fluent and plausible, but the dialogue drifts from the intended procedure. The benchmark forks conformant dialogues at chosen turns and injects one of four hallucination types, giving 36,278 dialogues across 24 tasks in 12 domains (30 annotators validated construction).

It defines three tasks: fleet-level deviation (monitoring deployed agents), single-dialogue deviation, and the turn where a dialogue first goes off-track (real-time guardrails). Its flow-aware FlowScore beat all baselines across seven encoders, which the authors read as semantic similarity being no substitute for procedural similarity. Paper-reported; MIT-licensed.

For builders running scripted workflows, monitor process adherence separately from factuality.

DescaPE locates a factual-salient layer span inside the model with sliding-window MLP ablation and uses that internal signal during decoding to suppress hallucination-prone trajectories before early factual errors compound through autoregressive generation.

It brings the decoding-time containment approach from vision-language models to plain-text generation, where post-hoc correction arrives after the error has already propagated.

A response-level detector combines fine-tuned DeBERTa-v3, Monte Carlo Dropout uncertainty, and temperature-scaled calibration. On HaluEval it reaches F1 0.915 and AUROC 0.977, and MC Dropout inference lifts accuracy to 93.2%.

The caveat matters for production: general-domain training transfers poorly to the SciFact biomedical benchmark (F1 0.52), and domain-matched PubMedBERT only reaches F1 0.63. A cheap detector is precise enough to gate on in-domain, but each domain needs its own training.

Nitin Garg's InfoQ article on spec-driven development argues that as AI-generated code volume grows, it brings security weaknesses and familiar bug patterns, so the engineering cost shifts from writing code to detecting where generated behavior diverges from stated intent.

Plan review capacity around verification, with a spec as the reference point, rather than around generation speed.

Outcome-based agentic RL applies one verified success or failure signal to every step of a long trajectory. TASPO converts privileged supervision from verified successful runs into per-action credit weights that are positive, bounded, and mean-preserving, so the verified outcome still sets the update's direction and scale.

Across three agentic benchmarks it improves 10.6% over GRPO and generalizes better to unseen tasks. Some "confident but wrong" behavior traces back to how the policy was trained, not only to which checks run on its output.

Phil Schmid notes that agents can already edit their own tools, skills, and harness. Recursive self-improvement still needs a system that can raise the verifier without the agent capturing it.

A self-modifying agent can also modify its own check. Any self-improvement loop needs a verifier the agent cannot rewrite.

On AppWorld, a ReAct agent on GPT-4.1 run five times on the same task succeeds on every run only 53% of the time, though its per-run pass rate averages 77%. The authors call the 24-point shortfall the consistency gap.

Their fix: a Consistency Analyzer flags the trajectory steps most likely to flip between runs, and a Guideline Generator writes a targeted guideline into episodic memory for similar future tasks. All-five-runs success rose 16 points on seen tasks and 13 points on unseen similar ones.

For unattended use, measure pass-every-time, not average pass rate. See agent memory for the storage side.

Anthropic's "Building Effective AI Agents" sets the default explicitly: start with the simplest solution that works, often a single well-prompted call, and move to a workflow or an autonomous agent only when the added complexity demonstrably improves outcomes.

Agentic autonomy is a cost to justify, not a default architecture.

A practitioner critique argues coding-agent autonomy hits a wall past a certain task complexity. Longer unattended runs compound small misjudgments into a wrong trajectory faster than the model corrects them.

Bound how much unattended autonomy a task earns instead of extending run length by default.