Fix the ruler before ranking the models: turning a Code Wiki failure into an Agent benchmark
Open Nua Engineering
The most dangerous way to decide whether an Agent model can use tools is to give it a small task, watch it succeed once, and call the question settled.
Real engineering work is less tidy. A model must understand an existing workspace, explore a controlled file system, distinguish source from generated output, revise a connected set of documents, preserve evidence, and pass quality gates within token, call, and time budgets. A failure in any layer—the model, provider, prompt adapter, tool mount, accounting, or grader—can look like the same thing: “the model failed.”
Open Nua's first Agent benchmark came from exactly this kind of real failure.
Ten million tokens later, we still did not know what had failed
The task was to revise an existing Plugins Code Wiki. Many pages were missing sources, and the structural review still had cross-page requirements. The Agent needed to read a controlled source snapshot, repair the skeleton, and restore evidence page by page.
Several failures looked similar from the outside:
- the model repeatedly listed a directory that did not exist and received empty results;
- after the virtual path was corrected, it could repeatedly write files without increasing evidence coverage;
- long runs consumed millions of tokens before stopping;
- remote and local models produced very different traces, but the workflow could not reliably separate model failures from system failures;
- non-streaming requests made TTFT and output TPS unavailable.
The first important finding was that the directory loop was not simply a weak-model problem. The runtime summary had failed to expose the real virtual root, so the model was exploring the wrong map. The second was that a successful tool response is not necessarily progress. Rewriting the same content is still a loop.
The question changed. Instead of asking which model was stronger, we first asked:
Can this failure become a task with a fixed starting point, repeatable execution, deterministic grading, and layered attribution?
Freeze the world, not the current checkout
The first rule of a useful benchmark is that every run faces the same world.
We derived a content-addressed fixture from the failed checkpoint. It fixed:
- 107 source-only files totaling 888,238 bytes;
- 62 planned Wiki pages and their existing drafts;
- inherited evidence, missing evidence, and unresolved structural requirements;
- content hashes, a fixture digest, a reproducible build process, and a redaction audit.
The fixture builder rejects bundles, caches, binaries, secrets, and host paths. The runtime cannot read the current checkout to discover a newer answer, because source changes between runs would invalidate the comparison.
The fixture locks facts and quality invariants, not final prose. Models may take different routes and write differently as long as their sources are real, the structure is complete, and the quality gates pass.
Exercise the real product, not a benchmark toy
The benchmark runner exposes a narrow interface:
runBenchmark({
caseId,
modelProfile,
runType: "qualification" | "formal",
trials,
budget,
})
The runner does not know about the Desktop database, provider secrets, or the observability service. It delegates each trial to a narrow Adapter, which invokes the same prompt, file tools, and Code Wiki generation flow used by the product. The benchmark therefore measures a model under Open Nua's real tool protocol instead of a simplified evaluation-only environment.
Three boundaries matter:
- The source snapshot, Wiki workspace, and task skills are readable; normal skill loading is not an authorization violation.
- Only the Wiki output area is writable. The model cannot alter the source fixture or skills.
- Traces and the operations console are observation surfaces, not grading facts. Local artifacts must remain sufficient even when observability is offline.
Every run leaves five immutable records:
- a manifest for the case, profile, budget, and runtime identity;
- sanitized model, tool, validation, and lifecycle events;
- deterministic grader findings and scores;
- a machine-readable summary;
- a human-readable report.
Replace one expensive long run with three qualification gates
The original qualification was effectively another full Wiki revision. It was realistic but wasteful: a model that could not read source correctly might still spend millions of tokens before stopping.
We split qualification into three ordered gates:
- Protocol: discover and read source successfully in an isolated, read-only probe;
- Evidence sample: repair exactly five Host-selected pages while keeping the page structure locked;
- Focused QA: let the Host audit declared paths, fixture bytes, content hashes, and the evidence ledger.
A failed stage stops the run. Later stages and the three formal trials never start. The qualification now answers whether the model understands the tool contract and whether it can converge on a small but real repair batch.
The measuring instrument could lie too
Smaller tasks did not automatically make the results trustworthy. The protocol itself needed several corrections:
| Stage | What was wrong | Correction |
|---|---|---|
| Long-task baseline | Qualification mixed in the complete task and made failure expensive | Split source protocol, five-page evidence repair, and focused QA |
| Path calibration | Valid skill reads looked unauthorized; nested calls were undercounted | Separate readable and writable boundaries; include nested calls in stage budgets |
| Budget calibration | High cache hits still consumed raw-total budget; call limits stopped runs before token budgets | Use provider-aware token budgets; keep call counts as loose safety ceilings |
| Grading calibration | Setup leaked into graded stages; the candidate participated in its own QA; tool failures could be double-counted | Exclude setup, move QA to deterministic Host checks, record tool errors separately |
Token fairness was particularly easy to miss.
The OAuth reference profile uses raw total tokens for convergence. Other providers default to:
budget tokens = uncached input tokens + output tokens
Raw input, cached input, output, reasoning, and total usage are still recorded. Cost, model calls, tool calls, wall time, repeated discovery, repeated mutation, and no-change writes remain independent safeguards. The purpose is not to relax the benchmark; it is to avoid treating a very high cache hit rate as equivalent new work.
We also removed candidate self-grading. Focused QA no longer calls the candidate model. The Host verifies:
- whether declared source and test files belong to the authorized fixture;
- whether the files exist and their hashes match;
- whether the evidence ledger proves an actual read or hash-verified inheritance;
- whether the page structure stayed locked;
- whether any unauthorized, repeated, or no-change tool operation occurred.
Only then does PASS mean more than generated text.
Make execution status a product fact
Long-running work has a simple operational requirement: people must know what is actually running.
Desktop publishes explicit execution lifecycle events. Gateway records execution-level usage. The management service projects those facts into operations, where the console shows status, model and tool calls, tokens, launch parameters, and drill-down details.
This corrected two early problems:
- an execution could remain “running” after the process had ended;
- a random execution ID did not reveal the provider, model, reasoning setting, or budget used to launch it.
The operations console is still not the final judge. Its projection can be repaired or rebuilt; local run artifacts are the immutable source of truth.
The first valid results: qualification is not a default-model decision
The first valid qualification runs started from the same fixture:
| Profile | Qualification | Model calls | Tool calls | Budget tokens | Raw total | Cached input | Wall |
|---|---|---|---|---|---|---|---|
| OpenAI OAuth reference | PASS | 17 | 51 | 596,631 | 596,631 | 47.68% | 131.7 s |
| DeepSeek 4 Flash | PASS | 47 | 89 | 85,641 | 2,609,801 | 97.51% | 170.2 s |
Both runs had zero grader violations, repeated discovery or mutation events, unauthorized tools, and tool-runtime errors. DeepSeek's raw total was high, but most of it was cached input; under its profile, the convergence budget consumed 85,641 tokens.
This establishes that both remote profiles have the basic tool capability to enter formal trials. It does not establish which model should be the Desktop default. That decision still requires repeated trials and comparison of success rate, variance, cost, and performance.
The local Gemma candidate does not yet have a valid score that can be compared with these runs. A historical protocol probe never entered source reading, while a run that ends only in an Adapter error and lacks execution and phase data is not a model result. “No valid score” is more honest than turning a broken measurement into a model failure.
Five lessons
1. The best Agent benchmarks often begin as real failures.
Synthetic tasks may measure whether a model knows an answer. A failed checkpoint measures whether it can explore, revise, recover, and converge.
2. A fixture freezes the world, not the answer.
It fixes source, prior state, digests, and quality invariants while allowing different paths and prose.
3. Tool success is not progress.
Progress needs input hashes, target paths, before-and-after content, evidence deltas, and stage state.
4. Validate the evaluation protocol before comparing models.
Adapter failures, self-graded QA, and incorrect path policies do not belong in a leaderboard.
5. Qualification grants entry; it does not choose the winner.
Its job is to stop unsuitable profiles early and reserve expensive formal trials for credible candidates.
This is only the first case in the benchmark suite. Future cases should follow the same principles: exercise real product seams, freeze factual inputs, use deterministic Host audits, preserve immutable artifacts, and separate model, provider, tool-runtime, and orchestration failures.
Fix the ruler first. Rank the models second.