This is the second post in the Paper → Live demo → Honest post series: take a recent paper, build a demo anyone can try, measure it carefully, and report what came out. The first one was about memory for agents. This one is about a question I find more uncomfortable: can an agent improve itself when nobody tells it whether it’s doing well?
The paper is Harness², presented in Harness²: Recursive Agent Harnessing for an Open World by Ruiyao Xu, Yanfei Chen, Zhongying CuiZhu, Bhavana Dalvi Mishra, Yifei Ming, Han Yu, Rujun Han, Chen-Yu Lee and Tomas Pfister, from Google (arXiv:2610.14224). What hooked me was how practical the idea is: most of us will never retrain a model, but we all write prompts, rules, tool guidance and memories for our agents. If that can be improved systematically, it’s a lever within anyone’s reach.
I also had a more personal reason to try it. Several of my demos are agents whose prompts I tuned by hand, run after run: in the dreaming agent I changed the prompt examples again and again until the model stopped copying them. If Harness² can do that work systematically, it’s a tool I want in my next projects. So this demo is also a test bench: I wanted to know whether I’ll use it, and under what conditions.
The usual disclaimer, out of respect for the authors: this is not a reproduction of the paper, and it doesn’t evaluate it. My demo takes the shape of the loop and simplifies it a lot, with very small models on CPU. My numbers say something about my version, not about Harness².
And as in the previous posts, I built it together with a coding assistant, in long sessions of decisions and measurements, with a devlog that records every problem and how it was solved.
What the paper proposes
The starting point is that an agent’s performance doesn’t depend only on the model, but on its execution harness: everything around the model while it works. In the paper that’s eight components, each one an editable file: system prompt, guardrails, memory, skills, tool guidance, scripts, subagents and a middleware plugin.
The problem they tackle is “open-world” tasks: tasks nobody saw when the harness was designed and that, when they have to be solved, have no verifier with the right answer. They propose two levels of recursion:
- At the task level, a Propose → Probe → Compose step. Propose: the current harness runs as a reference, and two edited candidates are proposed, one with general improvements (planning, tool use) and one with domain-specific knowledge. Probe: each candidate runs on the same task, and its trajectory is contrasted with the reference to see whether the edit fixed the weakness it targeted, changed nothing, or caused a regression. Compose: an editor reads that evidence and decides which changes to keep, which to revise, or whether to stay with the current harness, and every decision has to rest on a concrete observation.
- At the harnessing level, an improvement library that accumulates experience across tasks: what worked, in what situation and with what evidence, so later tasks start from a better place.
They report gains of up to 11.5 points on WorkBuddy-Bench, 4.5 on JobBench and 11.8 percentage points on LAB, with GLM-5 and Gemini 3.5 Flash, without touching the models’ weights.
One detail matters for everything that follows: the paper’s Probe is not a score. It’s a contrast between executions, interpreted by a capable editor model that justifies each decision with evidence. The authors themselves say that evidence is imperfect, because of the randomness of executions and the biases models share.
The first version was a mock-up
I’m telling this because it’s part of the story. The first version of the demo had the whole interface: you picked a scenario, watched the Propose, Probe and Compose phases advance, the metrics go up, and the optimized output next to the original. It looked great. But the traces were precomputed: the demo replayed examples written in advance and didn’t run any model.
The demo’s panel said so, but a demo showing improvements that never happened is exactly what this series wants to avoid. I hid it from the site and the goal changed: make the loop run for real, with a real model, even if that meant a slower demo and less flashy results. The mock-up survives as Cached mode, labeled as an illustrative example.
The constraints shaped the design (again)
The demo lives on the site’s ephemeral infrastructure, with its usual rules: everything local with Ollama and open models, zero external APIs, CPU only, and no state. With that, the gap with the paper is large:
| Paper | My version | Why |
|---|---|---|
| GLM-5 and Gemini 3.5 Flash | Qwen2.5 1.5B Instruct (default) and Gemma 2 2B, on CPU | Zero-API rule and a 2 vCPU / 4 GiB pod |
| Eight harness components as files | Only lines appended to the system prompt | Simplicity; the “memory” or “tool guidance” labels are just labels |
| Two candidates per step (general and domain) | One candidate per iteration | Time: every model call costs seconds on CPU |
| Probe: trajectory contrast interpreted by an editor | A judge scoring 0 to 100, the same model that proposes | There’s no other model available inside the pod |
| Compose: selective decision justified with evidence | Accepted if the candidate’s score is higher | It’s the simplest rule that can be shown |
| Improvement library across tasks | Nothing persists between runs | Ephemeral, stateless demo |
| WorkBuddy-Bench, JobBench, LAB | Short scenarios inspired by those names | They are not runs of the official benchmarks |
The table already hints at the problem. Of the whole loop, the piece I simplified the most is exactly the one the paper treats most carefully: how to know whether a change helped.
How the demo works
You pick a model, a domain, and write a task. The engine builds two test cases: your task as is, and a variation with a domain-specific instruction (in WorkBuddy, for example, “separate known facts from assumptions and order actions by dependency”).
- Base. The agent answers both cases with a minimal instruction: “You are an AI assistant. Complete the task directly and concisely in no more than 70 words.”
- Propose. The same model receives the task and the current instructions, and returns a JSON with a new instruction, the component it belongs to, and its justification.
- Probe. The agent answers both cases again, now with the candidate instruction added to its system prompt. The same model, as judge, receives the four answers and scores each one from 0 to 100.
- Compose. If the candidate’s average beats the current version’s, the instruction is kept. Otherwise it’s discarded.
Everything streams over SSE: you see each token as it’s generated, how many tokens were spent, and which phase it’s in. A run allows one or two iterations and a budget of up to 8,192 tokens. At the end, the demo shows the base and optimized answers side by side, with the instructions that were added.
Small models and JSON
The first live runs failed before reaching any interesting question, and almost always for the same reason: format.
Gemma, when asked for JSON matching a schema, sometimes returned the schema itself: {"type": "object", "properties": {...}} instead of the values. The judge could do it too, and strict validation rejected the whole judgment. The fix had three parts: a concrete example answer in every prompt (“return values, not the schema”), a function that unwraps that envelope when it shows up, and a retry with explicit feedback when the output doesn’t validate.
Even so, of the 18 runs I recorded in the WorkBuddy scenario, 7 ended without a result because the proposal was invalid JSON: in all of them, an unterminated string after roughly 390 to 490 characters, which is consistent with the output hitting the proposal’s 120-token limit: the models write longer instructions and justifications than fit. Those runs stayed in the record as failures; they weren’t repeated until something pretty came out.
Shipping it: the demo that got destroyed mid-run
On CPU, two iterations take about 4.5 minutes with Qwen and about 7 with Gemma. That forced the pod up to 2 vCPU and 4 GiB, about 5 minutes of startup time (the image weighs 2.8 GB with both models inside), and an SSE keepalive every 10 seconds so the connection wouldn’t look dead.
What I didn’t expect was this: in production, a long run ended with a generic model error. The model was fine. The problem was the infrastructure. The site’s gateway recorded activity only once, when the request arrived, and the process that cleans up idle environments destroys them after 8 minutes without activity. A run streaming tokens for several minutes was, from the gateway’s point of view, an abandoned environment. It deleted it while the model was still writing.
The fix was for the gateway to renew activity while it streams a response the client is still receiving. If the client leaves, data stops flowing and the renewal stops, so the session limit still works. That case is now in the gateway’s local test too. No earlier demo had needed responses that long, and this one exposed the assumption.
The judge that always said yes
With the loop running for real, the part that matters arrived: measuring. The repository has three batches of local runs in the WorkBuddy scenario, with two iterations, temperature 0.2 and both models, keeping everything: every generation, every proposal, every judge decision and every error.
The first result: all 26 completed comparisons accepted the candidate. No proposal was rejected. The only rejection in the project is a unit test with a mocked model.
The second one I found by counting the scores. In 18 of the 26 comparisons, the judge gave exactly the same numbers: 85 and 80 for the current version, 90 and 85 for the candidate. An average of 82.5 versus 87.5, five points in favor of the change, again and again, with different instructions, different models and different tasks. That doesn’t look like an evaluation. It looks like a small model that learned the shape of the answer expected from it.
The third is the clearest, because it can be checked without opinion. I added a task with arithmetic: two engineers with 3 hours each and 12 checks of 45 minutes. Each engineer can do 180 / 45 = 4, so together they complete 8, and 4 remain. In one of the Qwen runs:
- The base answer said: 4 completed and 8 remaining.
- The optimized answer said: 4 completed and 8 remaining.
- The final harness had two nearly identical instructions, accepted in consecutive iterations.
- The judge preferred the candidate both times, 87.5 versus 82.5, and in its rationale wrote that the base version “accurately calculates the remaining checks”.
None of the six final answers in that batch gave both numbers right. In a later study, with two arithmetic controls and three repetitions, the agent with the “optimized” instructions got 2 of 6 right, and the base agent 4 of 6. These are very small samples and say nothing general about the models, but they do say something concrete about this optimizer: its acceptances don’t track the quality of the answers.
Why this happens
When the same model proposes the change and judges it, the loop has no source of information it didn’t have from the start. The judge shares the proposer’s blind spots: if the model doesn’t know that 180 / 45 is 4 per engineer, it won’t notice the answer got it wrong either. What it can notice is that the candidate answer is longer, more specific, or mentions more things from the task, and that’s what it rewards. The judge’s rationales say it almost literally: “more detailed”, “more specific”.
This doesn’t contradict the paper; it’s the extreme version of a limitation the paper acknowledges. Harness² doesn’t ask a small model for a score: it contrasts full executions, with a capable editor that has to justify each decision with concrete evidence, and even so its authors warn about shared biases. My demo removed exactly those safeguards. The result shows what happens when they’re missing.
What I deliberately didn’t do
- No improvement library. The paper’s second level of recursion, the one that accumulates experience across tasks, doesn’t exist in the demo. Every run starts from scratch.
- Not the eight components. Everything accepted ends up as text in the system prompt.
- No independent judge. There’s no other model in the pod, and using an external API would break the zero-API rule.
- No human review, yet. The repo has a more careful study on questions about the future of artificial general intelligence: instructions frozen before seeing the evaluation questions, randomized A/B pairs, and blind packets ready for two reviewers. But nobody has reviewed them yet, so I’m not claiming anything about the quality of those answers.
And the honest limitation of the measurement: synthetic tasks, few runs, and the rubric checks were done by the coding assistant, not by a blinded person. These are exploratory observations, not a benchmark.
What I’d do differently
If I started over, I’d put something in the Probe that the model doesn’t control. For the arithmetic task, that’s trivial: a function that compares the numbers with the right answer and rejects any candidate that makes it worse, no matter what the judge thinks. Open-ended tasks have no such function, but they do have partial checks: that the answer follows the requested structure, that it doesn’t invent data that isn’t in the task, that it doesn’t exceed the requested length. They’re the same checks I did by hand with the rubric, turned into code.
I’d also let the judge say “I don’t know” and count a tie or a tiny difference as a rejection. An optimizer that never rejects isn’t optimizing; it’s just piling up text. It’s the same lesson as the dreaming agent and the trip planner: the model proposes, the code decides.
Would I use it in my projects?
This was the underlying question, and the answer has two parts.
This demo’s version, no. A loop that accepts everything it proposes doesn’t improve an agent: it adds text to it. In the arithmetic run it ended with two nearly identical instructions and the same wrong answer. If I set it to tune the prompts of another project, it would give me the feeling of progress without the progress.
The pattern, yes, with conditions. What the demo taught me is what I’d need for it to be worth it:
- A signal that doesn’t come from the model that proposes. Checks in code where they exist, or a more capable editor that contrasts executions, as in the paper. A judge the same size as the proposer isn’t enough.
- A set of tasks with known answers that works as a regression check: if a change makes any of them worse, it’s rejected, whatever the judge says.
- Use it at design time, not in production. Every step multiplies the model calls. On a CPU pod that’s minutes per iteration; as a tool for designing a harness before shipping a demo, with a larger model, the cost does make sense.
The first place I’d try it is the dreaming agent. There I already have what this demo lacked: a benchmark of 28 questions with known answers, run three times per condition. That’s the independent signal an optimizer needs to know whether it improved.
What I’m taking away
Three ideas stay with me.
The first: an optimizer is only as good as its signal. Propose and Compose are the flashy part of the loop, but the Probe decides everything. If the signal is noise, the loop piles up noise with great confidence.
The second: a judge that never rejects is an alarm, not a result. 26 out of 26 acceptances looked like a success until I counted the scores. From now on, in any evaluation with a model as judge, the first thing I’ll look at is the distribution of its scores and how many times it said no.
The third: verifiable checks are worth more than their size. An arithmetic task, which has nothing to do with the purpose of the demo, is what showed that the acceptances didn’t track quality. Without it, all I’d have is scores going up.
None of this takes anything away from Harness². It leaves me wanting to try what the demo doesn’t have: a capable editor that contrasts executions and justifies every change, and a library that learns across tasks, which is where the paper places its bet. Thanks to its authors for framing so clearly a question that anyone writing prompts for agents should be asking.
You can request the demo directly at alexisalulema.com/projects?demo=harness2-playground: it’s provisioned when you request it, runs on CPU with Qwen or Gemma, and destroys itself. Write a task, watch every proposal and every score, and notice how many times the judge says no. The code, with the full devlog and the records of every run, failed ones included, is at github.com/alulema/harness-playground.
If the judge gives you 87.5 versus 82.5, now you know why.