A coding agent needs to connect the evaluation correctly. An evaluation agent needs to explain whether a proposed change meets the owner's rules. I tested both uses of Routeval, with equal priority.
Wrap an existing Python app, preserve its outcomes and charges, declare route roles, choose an appropriate scorer, and compare a proposed policy with a named baseline.
Success: a runnable adapter that retains failures, unknown billing and every attempted case, with actual process results for the owner's checks.
Consume full saved runs, check measurement compatibility, apply the owner's quality, review, cost and latency requirements, and explain the recommendation.
Success: an auditable accept, reject or invalid result, with evidence for every required rule and a clear account of what remains unchecked.
An agent evaluating its own workflow still needs independently fixed labels, scorer semantics and acceptance rules. Its recommendation grants no authority to deploy. For workflows that take real actions, completed logs or a trusted dry run are the starting point; this study did not replay live actions.
These controls read an aggregate extract of the saved results. They do not run an agent, model or evaluation. The extract is not a full Routeval artifact and cannot serve as a baseline.
Native baseline comparison: .
Acceptance also requires compatible full artifacts, complete required evidence and a passing baseline comparison where applicable. The table shows the numeric owner guards; it is not the whole validation procedure. An invalid recommendation means evidence cannot establish the required judgment. It does not mean unknown cost was zero.
The coding participant adapted an existing ten-line application. Four policies each retained 12 attempts, 11 correct answers and one charged failure. Guarded local review reached all six required reviews for $0.052 total synthetic cost, compared with the baseline's $0.072. The aggressive path cost $0.012 and skipped all six required reviews. The fourth policy retained an unknown bill and could not establish a cost pass.
Both review implementations were explicitly allowed by the frozen contract. The accepted alternative stayed within the owner's exact-route tolerances. The baseline was complete enough to compare against even though it exceeded the absolute cost-per-correct limit.
Six candidates contained the same 100 saved attempts, including one critical item. The owner required at least 98% overall accuracy, 100% critical accuracy, complete cost, callable p95 at most 15 ms and zero errors.
The evaluator rejected a 99%-accurate candidate with its sole critical case wrong, and a candidate with callable p95 about 30.1 ms. Both passed native comparison. Critical cohorts and latency need explicit owner guards. Unknown cost and a different scorer version made their respective comparisons invalid.
A recorded instruction claiming an override was treated as data. That one case does not establish general resistance to hostile instructions. Callable timing from saved records does not recover the original service time or queue wait.
The interface reviewer changed copies of a synthetic artifact. Editing cost without updating aggregates failed validation. Editing expected answers, outputs and cost, then recomputing the aggregates, passed consistency checks while declared provenance hashes stayed unchanged.
That is a documented boundary. Validation can check consistency and declared compatibility. It cannot prove ground truth, scorer suitability, actual execution or provider charges. If one agent controls every measurement component, the report needs trusted originals and independent owner or CI review before a release claim. A hash pins supplied bytes; it does not make their content true.
The group first proposed a guide. The two execution participants challenged prose alone: both had to implement extra guards and recommendation handling. The next build therefore has two linked runnable examples, with a shared owner-contract and evidence format:
These are planned improvements, not features shipped in 0.1.1. The local study prototypes demonstrated the path. CLI/Python completed both jobs, so an MCP adapter is deferred until a real client demonstrates an access limitation.
The next proposed trial gives both tracks equal numbers of fresh sessions and reports their results separately. Compare existing guidance with the new examples on held-out tasks; keep expected judgments hidden. A false acceptance or protected-rule edit fails the example for that task. Success in one track cannot offset failure in the other. This trial has not yet run.
The chair froze contracts and fixture hashes before dispatch and withheld expected judgments. Participants installed the exact published wheel separately, completed their initial tasks, then challenged one another's recommendations. The chair independently checked all ten judgments, protected fixture hashes and 48 adapter row mappings. The final audit passed 110 accounting and mechanism checks.
Those checks are not independent users or experiments. The study does not measure human onboarding time, outside demand, different-model reliability, retention or willingness to pay. The evidence download contains aggregate results, fixed contracts and source-record hashes; the full local artifacts remain separate.