Agent engineering · know-how
🧱 Obstacle·planning·active·40 entries·41 sources·updated 2026-10-14
Give an agent a goal that takes ten steps and it will often take the wrong ones: charge ahead on an ambiguous request instead of asking, follow a plan that drifts, get stuck in a retry loop, or skip a step it needed. Planning, turning a goal into the right ordered actions and knowing when to stop or ask, is a distinct failure mode from tool use or memory.
Robust planning comes from structure around the loop, not a better prompt. The ReAct loop (reason, act, observe, repeat) is the shared base, and the field now treats it as engineered infrastructure: framework APIs with persisted state and detached turns, portable loop libraries, and hooks that change an agent's tools mid-run. LangGraph, GitHub, and OpenAI describe the same practice under different names: graph, loop, or harness engineering.
Three refinements sit on top of the loop:
How the harness carries state between steps matters as much as the model. Retaining reasoning and enabling compaction roughly tripled one model's ARC-AGI-3 score.
The open problem is measurement. Self-improving loops are only as good as their evaluator, and an end-to-end pass/fail cannot tell whether the agent or the loop failed. Benchmarks that score the loop controller separately and test procedures hundreds of pages long are first steps; production teams still lean on humans for most planning decisions.
Bad planning turns a capable model into an unreliable agent. It causes runaway loops (a cost problem), confidently wrong work on ambiguous tickets, and the long-horizon failures that erode trust. The job is to wrap the model in a controllable harness: bounded loops, explicit decomposition, clarification checkpoints, verification, and re-planning on failure. Prove it with trajectory-level eval rather than hoping a bigger model plans better. Planning sits upstream of orchestration: once you can decompose reliably, the question becomes who executes each step.
The reason-act-observe loop is now packaged as frameworks, portable libraries, and production harnesses; vendors agree that structure around the loop, not a cleverer prompt, keeps multi-step runs on track.
Keel (release 0.10) bills itself as "a conductor, not an agent loop" for AI coding agents that prove their work.
It is an explicit alternative to the loop-as-primitive framing most harnesses assume: an orchestrator that directs agents and checks evidence, rather than one open loop. The Show HN post gives no measured results.
The GitHub Podcast devotes an episode to the new vocabulary in developer conversations: loop engineering, harnesses, squads, hill climbing, and open weights.
The loop-engineering vocabulary has spread from research and conference circles into mainstream developer media.
Per InfoQ, Cloudflare cut GitHub issue-triage work on the Astro project 85% with AI agents built into the workflow via GitHub Actions, Cloudflare Workers, and Flue, with humans kept in the loop.
It is a measured production case for wrapping a structured workflow around the model rather than routing raw model calls at each issue.
Astro creator Fred Schott's Flue 2 borrows React's hooks: useSkill(), useTool(), useSubagent(), and custom hooks among 16 built-ins, so tools, resources, and state can attach or change during a run instead of being fixed in config up front.
His framing names the planning gap: a support or triage bot "can't be fully configured in advance... it has to adapt in real-time." The harness is the foundation, not an afterthought, which gives a concrete mechanism for re-planning mid-run.
A look at how ChatGPT optimizes its own agent loop treats the harness, the API, and inference as one system to tune together.
It is the loop-as-infrastructure view seen from inside a shipping consumer product rather than a framework or a paper.
GitHub's "The harness is all you need (mostly)" lays out a repeatable Copilot workflow (prototype, plan, implement, review) and argues for building discipline around it instead of re-architecting every time a new AI tool ships.
A second major coding-agent vendor backs the loop-as-infrastructure view: harness stability is worth more than chasing novelty.
LangChain's three-year LangGraph retrospective argues graph engineering, loop engineering, and harness engineering are the same practice: placing model reasoning inside an explicit, inspectable control structure, with the right context at each step.
The convergence is industry-wide, not one vendor's pattern. Pick the control structure deliberately rather than trusting a single prompt to plan.
"Agents in the Wild" surveys agentic systems moving from research prototypes to production in software engineering, scientific discovery, and finance. It contrasts academic benchmark focus with deployment needs.
The deployment evidence backs the structure-around-the-loop thesis: decomposition, checkpoints, and guardrails carry production systems, not a stronger model alone.
Google released the Genkit Agents API in preview for TypeScript and Go. It packages message history, the tool-call loop, streaming, and state persistence behind one chat() interface.
Two primitives matter for planning. Detached turns let an agent keep working after the client disconnects. Interruptible tools add human-in-the-loop approval mid-plan, with anti-forgery validation on resume. "Ask vs. proceed" gets a framework-level mechanism instead of a bespoke one.
An engineer at Featherless kept rebuilding the same loop (call model, run tools, feed results back, stop), so he extracted it and put every piece behind an interface. The MIT-licensed result works with any OpenAI-compatible endpoint and does not own the UI or control flow.
The loop itself becomes portable infrastructure, separable from any one framework or provider.
Latent Space's dispatch from the AI Engineer World's Fair reports that "loops" and "software factories", production setups that wrap a planning loop in enough infrastructure to run it repeatedly, were a dominant theme alongside forward-deployed engineering and open models.
Planning as a harness problem has moved from research framing into mainstream practitioner conversation.
An AWS write-up of Stripe's financial-compliance agent describes a ReAct framework (reason, act, observe, repeat) running on a dedicated agent service, with guardrails, human oversight for accountability, and lessons on task decomposition.
The ReAct loop alone does not keep multi-step runs on track at production scale. Planning reliability is an architecture problem, solved with infrastructure around the loop, not a prompt problem.
Clarification benchmarks, uncertainty signals, live state models, scope estimation, and reasoning-effort dials give a harness explicit moves before it commits to a plan.
Matt Pocock's /wayfinder skill is for greenfield projects or work where the way forward is unclear. It treats planning as navigating a "fog of war": explore first, then commit to a plan.
It adds a third option beside asking for clarification and proceeding: scout the territory when decomposition is not obvious up front.
Sebastian Raschka explains how LLMs learn low, medium, and high reasoning-effort modes, which providers now expose to trade latency and cost for deliberation depth.
A harness can set "how hard should the model think before acting here" per step, instead of applying one fixed reasoning budget to every step of a plan.
Agents often follow a maximum-context-first strategy, turning a one-line edit into a codebase audit. E3 (Estimate, Execute, Expand) estimates a minimal operating point, executes a minimum-sufficient path, and expands scope only when verification fails.
On a 121-edit benchmark it matches the strongest baseline's 100% success while cutting cost 85%, tokens 91%, and files inspected 92%. A live gpt-4o harness showed milder but real over-reading. Deciding how much work a task needs before executing is the cheapest fix for over-scoped plans.
Candidly built an IO-HMM over per-turn signals such as message length and semantic alignment. It infers whether a conversation is Engaged, Detailed, Guided, or Disengaging and steers the agent's next turn accordingly.
In production, disengaging turns fell from 23% to 11% and the Engaged state rose from 53% to 64%. Inferring mid-episode whether the plan is working, not just scoring the end result, was worth the extra model.
CoMet targets uncertainty estimation in multimodal models by decomposing what kind of uncertainty a model has, from context and from multiplicity of valid answers.
"Knowing what you don't know" is the input a planning loop needs to choose between asking a clarifying question and proceeding. CoMet works on producing that signal; benchmarks like DiscoBench only score the decision.
Most search-agent benchmarks assume the user's query is complete. DiscoBench measures clarification-aware deep search: whether an agent recognizes an under-specified goal and asks instead of confidently planning down the wrong path.
It treats "ask a question" as a first-class planning action that a harness should test for, not a sign of weakness.
Bounded worker/critic pairs, rubric graders with iteration caps, test-anchored coding, and multi-hypothesis failure diagnosis replace blind retry with an explicit check-then-revise step.
With the model fixed (gpt-5.2-codex), LangChain reports (vendor-reported) a 13.7-point gain on Terminal Bench 2.0 from harness edits alone.
Also relevant to Agents give fluent, confident-looking output even when it's wrong, Tracing and trace analysis for agent runs
LangChain's RubricMiddleware for Deep Agents turns a newline-delimited checklist of success criteria into a grade-then-revise cycle. A grader sub-agent with its own model, prompt, and optional tools checks the output, injects per-criterion feedback on failure, and the agent revises until it passes or hits a max-iteration cap.
Verification as a planning step ships as reusable middleware rather than a bespoke harness. The grader side is covered on LLM-as-judge.
Existing approaches use generated tests as static post-hoc validators, which limits their effect and can mislead when the tests are wrong. TDD-Agent uses test-driven reasoning so tests guide implementation during repository-level code generation.
The plan stays anchored to passing tests rather than a plausible-looking diff.
Netflix open-sourced an agentic workflow for observational causal inference. Given observational data and the analyst's own plan, an actor-critic loop estimates causality, writes a report, and suggests next steps, reducing analysis toil.
The worker/critic structure from coding agents carries over to a data-science task, with the human supplying the plan the agent executes against.
The Krystal Loop Protocol structures a coding agent's loop as a bounded worker/critic pair: one role proposes work, a second checks it, within an explicit bound rather than an open-ended retry.
It packages the propose-then-check structure as a named, reusable protocol instead of a bespoke harness detail.
Anthropic's guide shows how to turn a developer's own manual checks (does it compile, does it match the spec, did the test pass) into reusable skills, so Claude runs the verification step and closes its own feedback loop.
Verification becomes a packaged, repeatable step of the plan instead of a human re-checking every output.
Autonomous research agents usually recover from a failed experiment with a single free-form reflection that compresses metrics, logs, and design choices into one critique. This work generates several candidate explanations for the failure, weighs them, and re-plans around the most likely cause.
Failure diagnosis becomes a planning step, not a trigger for blind retry.
Morph Reflexes serves semantic signals from agent traces over an API, with classifier heads for looping, reasoning leakage, and user frustration. Its pitch: judging every turn with a frontier model is too slow and expensive at scale.
A planning loop that stalls or repeats needs a detector cheap enough to run on every turn. Classifier signals make "is the agent stuck" a runtime check instead of a post-mortem finding. Cost details are on LLM-as-judge.
Agents improve plans with hindsight experience, RL on real tool interactions, evolvable harnesses, and plans written as reusable code; the evaluation signal remains the limiting factor.
Claude Code (Opus 5) and Codex (GPT-5.6 Sol, GPT-6 Astra) were each given a task description, simulator access, and a fixed budget to synthesize a reusable planning program, which was then frozen and tested on unseen instances. Across 28 environments (980 programs, 98,000 episodes), all three beat hand-engineered planners: 56-95% mean success versus 47% on the 16 environments with a planner baseline.
As object counts grew, the agents' programs kept their edge with an order of magnitude less computation per instance. Letting an agent write the plan as code generalizes better and cheaper than re-planning each episode.
OpenAI reports that two API settings, retaining reasoning state across steps and enabling context compaction, together roughly tripled GPT-5.6's ARC-AGI-3 score and improved efficiency.
How the harness carries state between planning steps is itself a planning lever, not just which model runs them. The eval framing is on agent evaluation.
PoTRE (Poly-Topological Reasoning Ensembles) runs four heterogeneous agents (adversarial refinement, hierarchical strategic planning, spectrum search, direct chain) and reconciles them through a task-adaptive aggregation layer using candidate selection, semantic synthesis, or neuro-symbolic verification.
It reaches 49.92% on Humanity's Last Exam, above the previous best official score, with similar or fewer inference tokens than scaled single-stream baselines. Decomposing long-horizon reasoning across specialized roles can beat scaling one reasoner at comparable cost; see multi-agent.
Lilian Weng's survey names three recurring harness patterns: goal-oriented plan, execute, observe, improve loops; the file system as persistent memory; and parent agents spawning inspectable sub-agents. It goes further by treating the harness code itself as an artifact an LLM mutation operator can evolve (AlphaEvolve, Darwin Gödel Machine).
Her caveat matters as much as the taxonomy: self-improvement loops are only as good as their evaluation signal, and weak evaluators remain the bottleneck. Pair any harness-evolution experiment with trajectory-level eval.
AWS's best practices for multi-turn RL in SageMaker AI list the operational steps for training agents on long interactions:
It is the checklist underneath "just fine-tune on tool interactions".
Paul Bakaus, discussing his tool Impeccable, argues against one-shot AI design: iterative, human-curated task specs ("skills") get better plans than hoping a single prompt gets them right, even in a "loopmaxxing" era.
Treat task specs as artifacts you refine across runs, with human judgment in the loop.
Introspection co-founder Roland Gavrilescu describes autoresearch: agents iterate on their own task recipes across runs, closing a feedback loop over the plan itself rather than over individual steps.
He is explicit that humans remain central to steering the loop. Self-improving plans still need someone choosing what "better" means.
Agent RFT fine-tunes reasoning models with reinforcement learning against real-time tool interactions and custom reward signals. The InfoQ talk frames it as solving credit assignment: which of many steps in a long trajectory caused success or failure.
Speakers report enterprise cases where it removed long-tail token loops. Training now targets planning directly, not only the harness around it.
Small open multimodal web agents are cheap and private but plan poorly and generalize weakly across websites. This work has them autonomously explore interfaces and reuse hindsight experience from earlier attempts, which improves task decomposition on repetitive GUI tasks over zero-shot planning.
A plan learned from prior runs beats one re-derived cold each time, which matters most when you want a small model instead of a frontier one.
DAIN (Dynamic Agent-based Interaction Network) reframes multimodal fusion as a dynamic network of interacting agents whose collaboration and reasoning structure adapts per task, instead of a static mixture-of-experts routing.
The planning lesson: the structure of who reasons about what can itself be chosen per task rather than fixed up front.
New benchmarks score the loop controller separately from the agent, test reasoning about the harness's own lifecycle, and stretch planning horizons to hundreds of pages of procedures.
Most multi-hop benchmarks are short-horizon, needing only a few retrieval or inference steps. Tasks over Application Manuals (TAM) tests whether a model can follow real procedures spanning hundreds of pages of interdependent guidelines, and finds gaps that short benchmarks miss.
It tests whether a plan stays consistent with a long manual, not a small task graph; related long-horizon gaps are tracked on agent evaluation.
In loop engineering, a controller monitors progress, assigns work, runs checks, and decides what the agent does next. LoopArena benchmarks models in that controller role: whether the loop trusts a stale progress note, skips verification, or stops before the task is safe to submit.
A single end-to-end pass/fail cannot tell whether the agent or the loop caused the outcome. LoopArena separates them.
Dynamic harnesses let a model change the software that shapes its own execution, so a local plugin change can propagate through dependencies and cleanup. CordisBench is a 1,200-question benchmark of that lifecycle reasoning, combining a controlled formal setting with real programs.
The harness's own state becomes something the model has to reason about, not just run inside.
QUALITY.md is an open format, agent skill, and CLI for holistic quality evaluation of a project. Its author found it suited loop engineering, and wants to move teams from a reactive review-and-repair mindset to defining quality up front.
It names harness quality as something to measure directly instead of leaving it implicit in end results.