A coding agent says it fixed the bug. The diff looks reasonable. Then you discover it changed the test, skipped the boundary case, or ran the checks before its final edit. I want the engineering environment to catch those mistakes before I review the patch.
That environment is the harness: the code and operating rules that supply context, expose tools, run checks, retain evidence, and decide when work can stop. A better prompt helps the model choose an action. The harness determines what that action can do and what counts as completion.
In this issue, I show you how I build a small Python harness around an ordinary shipping-price ticket. You will run a local Ollama model, inspect its actual actions, and replay the resulting repair. The ticket is an authored, executable example. I tested it with llama3.2:3b and mistral:7b; the results below include both successful repairs and bounded failures.
What You Are Building
The ticket is specific: a cart worth exactly £50 is charged £4.99 delivery, although free delivery starts at £50 inclusive. Amounts are integer pence. Below the threshold, delivery stays 499 pence. Negative subtotals must still raise an error. Here is the supplied implementation:
def shipping_pence(subtotal_pence: int) -> int:
if subtotal_pence < 0:
raise ValueError("subtotal must be nonnegative")
return 0 if subtotal_pence > 5000 else 499The repair is small enough to understand immediately: change the comparison to >= 5000. The engineering work is making the agent read the right context, change only the permitted file, demonstrate the behavior, and leave evidence that another engineer can inspect.
This extends the repository instructions in Issue #28 and the model comparison in Issue #29. Here, the unit under test is the execution environment around a coding task: prompts, action schemas, file policy, verification, budgets, and completion.
What Harness Engineering Actually Means
Harness engineering means giving a model the context, tools, feedback, and controls it needs to do useful work. As LangChain explains, this includes how an agent runs tools, manages its workspace, and keeps track of context. An execution harness supports the work; an evaluation harness checks the results.
Your coding agent may already provide much of this machinery. Your role is to configure repository instructions, commands, permissions, and checks, the outer harness described by Birgitta Böckeler. In this example, AGENTS.md guides the repair, test failures provide feedback, and executable controls enforce which files can change and when the task is complete.
Longer tasks may also need saved progress and handoffs between sessions, as shown in Anthropic’s example. Our eight-step runner keeps conversation history in memory and saves evidence. It does not resume sessions or compact context.
How I Engineer the Harness
I use the following sequence to turn a recurring coding problem into a better working environment. It is a practical synthesis of these approaches:
- Define the task and acceptance evidence. Reproduce the £50 shipping error, specify neighboring amounts, and identify who reviews the patch.
- Prepare the workspace and context. Start from a known source revision, provide the relevant rules, and make the source and check results easy to inspect.
- Set action authority and budgets. Decide which files and tools are available, where execution is isolated, and when the run must stop.
- Connect actions to feedback. Return useful test failures to the agent, invalidate stale test success, and retain the candidate, observations, and final outcome.
- Inspect failures and improve one boundary. Add the observed failure as a regression case, change the responsible rule or tool, replay it, then run fresh model trials to check whether task performance improves.
OpenAI’s engineering account makes repository knowledge, observable application behavior, and mechanical checks part of this work. The practical question I ask is: what information, capability, or feedback was missing when the agent went wrong? Sometimes the fix is a clearer rule; sometimes it is a runnable test or a better tool result.
System Structure
The model proposes a typed action. Python owns validation, file access, execution, and acceptance. Recorded actions can replace the model at the same boundary, so I can test the host independently of model decisions.
The temporary directory keeps each fixture run clean. It is not a security sandbox. The supplied runner executes candidate Python with your OS permissions. Use the benign fixture locally; move untrusted generated code into an isolated container or VM before adopting this execution pattern in production.
Runtime Configuration First
The settings below live in harness.toml, with validated integer HARNESS_ environment overrides. Eight steps allow inspection, reproduction, an edit, another test, and completion. They also stop an agent that repeatedly reads the same file. These are chosen limits for this example, not universal defaults.
max_steps = 8
max_seconds = 90
test_timeout = 5
model_timeout = 30
max_response_bytes = 32768The host checks remaining wall time before operations and completion, and passes bounded timeouts to the model request and verifier. This is a cooperative run deadline, not a hard process supervisor. Python’s subprocess documentation explains child-process timeout behavior; production also needs process-tree termination and resource quotas.
Give the Agent a Small, Useful Context
I put the ticket and action contract in prompt.txt, then supply fixture/AGENTS.md as repository context. It says amounts are pence, names the inclusive threshold, preserves negative-input rejection, and limits edits to shipping.py. The model can read the source rather than guessing its contents.
The agent does not receive the expected patch or scripted solution. It receives the behavior to achieve. The five acceptance checks remain harness-owned. Their test output can guide a repair, but their source is outside the tool allowlist. In a larger repository, I would add only the relevant module map and documented check commands, then retrieve extra context when needed.
Make Tool Authority Executable
For local generation, the Ollama adapter supplies a simple object schema with an operation enum; the server rejected the original generated union grammar. Python still validates every response against the stricter action union, which accepts only read, replace, test, and finish. Pydantic rejects extra keys. A model cannot turn a test request into an arbitrary command by adding a shell argument.
class Strict(BaseModel):
model_config = ConfigDict(extra="forbid", strict=True)
class Replace(Strict):
op: Literal["replace"]
path: str
old: str = Field(min_length=1, max_length=16000)
new: str = Field(max_length=16000)The host permits replacement only when the path equals shipping.py, and the old text occurs exactly once. Through these tools, it rejects ../prompt.txt, absolute paths, and attempts to edit the verifier. An invalid request terminates this small run with an explicit reason. I prefer that visible stop to silently expanding permissions to help the model continue.
The test action invokes a fixed interpreter and verifier with an argument list, a fixed working directory, and no shell. That controls the tool interface. It does not make imported Python trustworthy: candidate code could access the host or forge a successful process exit. Production isolation and independent result collection must enforce that stronger boundary.
Bind Completion to the Code You Tested
The harness-owned verifier checks five inputs: 0 and 4999 must cost 499 pence; 5000 and 5001 must cost zero; -1 must raise ValueError. A fix that gives free shipping at 4999 therefore fails even though it repairs the reported cart.
The subtle failure is editing after a successful test. I bind test success to the source hash and clear it on every replacement. The acceptance condition is small enough to inspect:
# After a successful verifier process:
tested_hash = digest((work / "shipping.py").read_bytes())
# On every replacement:
tested_hash = None
# On finish:
passed = tested_hash == digest((work / "shipping.py").read_bytes())
Anthropic’s agent-evaluation guide distinguishes the transcript from the outcome. Both matter here: the trace explains the path, while running the checks against the resulting file provides the task evidence. SWE-bench uses repository tests to evaluate generated patches at a much larger scale; this fixture borrows that observable-outcome approach without claiming benchmark equivalence.
Walking a Real Local Model Run
Start Ollama with your installed llama3.2:3b model, then run these commands from the companion directory. Live execution is the default; --model overrides HARNESS_MODEL or the default llama3.2:3b. Use a new output directory for each attempt.
python -m venv .venv
.venv/Scripts/python -m pip install -e ".[test]"
.venv/Scripts/python -m pytest -q
.venv/Scripts/bugfix-harness live --model llama3.2:3b --out runs/live-llama
.venv/Scripts/bugfix-harness live --model mistral:7b --out runs/live-mistralIn the first final Llama trial, the model chose four actions: read the module, replace the return expression with an inclusive comparison, run the five checks, and finish. Its one-line patch fixed the ticket. It skipped the requested pre-edit test: that is guidance in the prompt, while testing the final source is enforced by the host. The scripted regression case separately demonstrates failing checks before the repair. Here, accepted means the fixture’s checks passed for the final source; merging or deploying still requires your normal review and CI checks.
{
"passed": true,
"reason": "accepted",
"steps": 4
}This is a subset of the actual live report. With the final adapter and prompt, Llama completed 3/3 attempts in 1.53–2.09 seconds; Mistral completed 0/3, repeatedly reading the unchanged file until the eight-step limit stopped it. Trials ran sequentially on Ollama 0.31.1 with temperature zero. These are local observations on one ticket after prompt development, not a held-out benchmark or a general model ranking. Loading and hardware affect timing.
The initial live runs were useful too: both hit the schema error; after the adapter fix, Llama invented nonexistent source text and Mistral looped on reads. I clarified the inspect→edit→test workflow without supplying the patch. All attempts are retained. The final harness tests passed 16/16; the seven scripted scenarios still independently check valid repairs and expected stops.
Replay the Actions, Reexecute the Checks
Replay repeats saved actions in a fresh workspace and runs the tools and tests again, without calling the model. Use it after changing a harness rule to check whether a valid repair still passes and an invalid action still stops. Because the actions are fixed, replay cannot show how a model would respond to different test output. Compare the saved manifests yourself; the CLI records input hashes but does not check that they match the original run. Use the command below to replay the saved Llama actions.
.venv/Scripts/bugfix-harness replay --actions runs/live-llama/actions.json --out runs/replayedTo check whether the model chooses useful actions and completes the ticket, make a new model run. To repeat that check with both installed models, run python scripts/run_local_trials.py --out runs/local-trials. It makes three attempts per model, saves model digests and the Ollama version, and preserves each outcome. Repeating this development ticket does not replace a held-out task set.
Keep Evidence That Explains a Failure
Each run saves manifest.json, trace.jsonl, actions.json, test logs, candidate.py, patch.diff, and report.json. The manifest hashes the fixture, instructions, prompt, verifier, and harness implementation, and records configuration, Python, OS, and provider identity. Live runs also save model-exchanges.json: actual request context, returned actions, and Ollama timing and token-count fields. HTTP errors retain the server’s bounded error response. Invalid action text is retained with its error. The trace stores observable actions and results, not private model reasoning.
The suite summary hashes the case file and compares case IDs before checking baseline regressions. Every expected stop must match, so denying every action cannot produce a green gate. Keep the reviewed baseline outside the candidate change in CI. If the task set changes, review a new baseline; do not silently compare unlike experiments.
These local records are unsigned. Logs have no disk quota, socket timeouts are not a hard total deadline, and generated code can defeat checks without execution isolation. I would add read-only verifier mounts, no host secrets, restricted egress, CPU and memory limits, bounded logs, and externally collected test results before trusting arbitrary patches.
Why This Architecture Works
There are two feedback loops. The coding loop turns a ticket and observations into a proposed repair. The engineering loop turns a failed run into a better harness and a permanent regression case. When I see “tests passed before the last edit,” I add the source-hash completion rule and keep that exact failing sequence in the suite.
Keep the harness as small as the task allows. In Anthropic’s later harness-design account, the team tested simplifications one component at a time as model capabilities changed. I would use the same discipline here: measure whether a control helps before adding more orchestration. Keep correctness and authority requirements explicit when testing those changes.
Potential Enhancements
My next step would be a small collection of daily maintenance tickets: a pagination off-by-one error, an inclusive date filter, and a retry that duplicates an operation. Give each an immutable starting revision, a readable ticket, protected behavior checks, and independent attempts. Compare completion, policy violations, elapsed time, and model usage by ticket; a single aggregate pass rate can hide an expensive regression.
Use deterministic checks for exact amounts, file scope, and completion evidence. Reserve a calibrated human or model judge for questions such as whether the patch explanation is useful. A persuasive explanation must never compensate for failing shipping tests. Keep a held-out set separate from the failures used to improve the harness.
Final Notes
I build a harness around work I can reproduce and results I can check. The model proposes the repair, while the application controls what can change and verifies the code before accepting it.
You can run this example with Ollama, inspect the model?s actions, and replay them after changing a harness rule. When a run fails, use the trace to find what needs improving and keep that failure as a regression case.
Explore the source code in the GitHub repository.
See you in the next issue.
Stay curious.