ali's garden

harness engineering

Properties1
tagslearning, ai, software-engineering, harness-engineering

5 min read

Learning in progress

Part of the harness. The spine of the cluster: every philosophy plugs into one of its components. Bullet structure intentional.

Engineer the environment, not the prompt. A harness is the controlled environment an agent runs in: guides (feed-forward), sensors (feedback), and a loop (orchestration) that ties them together. Everything else in this cluster is a bet on one of those three.

why we need it

  • Models are capable but unreliable: they hallucinate, drift, forget across sessions, and produce plausible-but-wrong code with full confidence.
  • Prompting alone has diminishing returns. In professional work the bottleneck is not phrasing; it’s preventing and catching errors.
  • The environment is the real lever: good context up front plus fast automatic checks raise the quality floor far more than a cleverer prompt.
  • It converts “hope it works” into “the system catches it when it doesn’t”: determinism where you can, human judgment where you must.
  • It is what lets you delegate more without losing control: you supervise the system, not every keystroke.
  • Evidence: OpenAI built ~1M lines leaning on the harness; Fowler & Böckeler argue harness quality beats prompting skill for professional work (Fowler, OpenAI).

the three components

Vibe coding bets on none of these (just eyeball the output). Formal spec bets on two at once (its spec is a guide, its verifier is a sensor). Most philosophies pick one.

Guides · feed-forward

Purpose: prevention. Give the agent intent, conventions, architecture, examples, and memory so it does not have to guess. Lowers the error rate before it writes a line.

  • Plugs in here: pseudocode, light spec, SPDD, spec-driven, rules files, and the spec half of formal spec
  • Deep dive: context & memory architecture is the persistent-state slice of guides (CLAUDE.md, specs, ADRs, worklogs)
  • The discipline of choosing what to feed and when is context engineering; the failure mode is a bloated guide that rots and crowds out the task

Sensors · feedback

Purpose: detection and self-correction. Catch errors after the agent acts, automatically and fast, and feed failures back so it repairs itself.

  • Plugs in here: tests and contracts, lint and types, evals, and the proof half of formal spec
  • Two kinds (sensors for coding agents):
    • Computational - tests, lint, types: cheap, deterministic, run on every change
    • Inferential - LLM-as-judge: costly and flaky, run selectively
  • Self-repair in practice: the agent reads sensor output and fixes itself in a loop (Aider auto lint/test)

Loop · orchestration

Purpose: control and division of labor. Decide who does what, when work passes between human and agent(s), and how state survives across context windows.

  • Plugs in here: human-on-the-loop, multi-agent orchestration
  • Patterns: self-repair loops; an initializer + coding-agent split that bridges many context windows for long-running work (Anthropic: long-running agents)

supporting components

Less philosophical, but a real harness needs them:

  • Agent-computer interface (ACI) - purpose: make the agent’s actions legible and guard-railed. Purpose-built view/search/edit commands beat raw shell (SWE-agent)
  • Execution sandbox - purpose: run and test safely, with no blast radius (OpenAI sandbox)
  • Observability / logging - purpose: let the agent reproduce and verify its own work (logs, metrics, traces, DevTools)
  • Evals / benchmarks - purpose: grade whole-task performance, not just single checks (SWE-bench)

how the loop runs

  1. Intent plus guides go in (context, rules, spec, examples, memory)
  2. The agent acts (writes or edits code)
  3. Sensors check automatically (tests, lint, types, review gates)
  4. Failures feed back; the agent revises
  5. Repeat until sensors pass; the human on the loop sets direction and reviews the result

The whole point is that steps 3 and 4 happen without you watching every keystroke.

other approaches & variants

  • Fowler / Böckeler - guides + sensors - the cybernetic framing; computational vs inferential; “keep quality left” (harness engineering)
  • OpenAI Codex - engineer the environment - architecture enforced by custom linters, docs/ as system-of-record, “garbage-collection” agents (harness engineering)
  • SWE-agent / SWE-bench - ACI + eval harness - interface as the lever, paired with a standard benchmark (paper, benchmark)
  • Aider - deterministic test/lint loop - auto-lint and auto-test after every edit, retry with structured error feedback (docs)
  • Anthropic - long-running-agent harness - initializer + incremental coding agent bridging many context windows (write-up)

open problems

  • Reward / eval hacking - agents pass tests by cheating (special-casing, editing or deleting tests). CoT monitoring helps but is fragile (OpenAI, CoT monitoring)
  • Benchmark overfitting & contamination - public benchmarks leak into training and get scaffold-tuned, inflating scores (SWE-bench Pro)
  • Verification gap - tests pass but code is wrong; correctness is outside any sensor’s remit if intent was under-specified. Fowler calls the “behaviour harness” the elephant in the room
  • Flaky / slow / costly feedback - inferential sensors can’t run on every commit; forces quality-left tradeoffs and re-runs instead of hard blocks
  • Context limits vs. memory - “context rot”; each session starts amnesiac; a monolithic rules file rots and crowds out the task
  • Security of autonomous tool use - sandbox escape and unsafe actions motivate dedicated sandboxing (OpenAI sandbox)
  • Non-determinism & reproducibility - model stochasticity plus non-deterministic sensors make outcomes hard to reproduce
  • Harness coherence at scale - keeping guides and sensors in sync, non-contradictory, and measurable (“code coverage, but for harnesses”)
  • Where human judgment stays - taste, accountability, organizational memory, which conventions are load-bearing vs. habitual; the goal is to direct human input, not remove it

sources

Graph View

Backlinks