Local / hosted cascade
A local fast path, a hosted escalation, and a free local review. Check that price never determines a route's role.
Replay examples through your AI pipeline. Compare answer quality, routing decisions, and cost in one report — for a model cascade, a human handoff, or a rules-and-LLM flow.
Python 3.10+ · no required package dependencies · seven offline workflow recipes · no API key needed
python -m pip install routeval==0.1.1. Both public release files match the verified hashes. A fresh public installation, the offline demo, all seven workflow recipes and the tolerance-boundary fix passed verification. The trial kit includes the same verified wheel and source archive.These are five synthetic cases, evaluated by the installed Routeval package. The second policy skips a required local review. Its final answer stays correct and its API cost stays the same — the review is free. Its escalation recall still falls.
| Answer correct | Answer wrong / failed | |
|---|---|---|
| Route matches label | Both checks passed | Chosen path failed |
| Route differs | Correct answer, policy mismatch | Both checks failed |
These route labels express a reference policy. A mismatch does not prove an answer was accidental or that another path would have produced a better answer. The costs are illustrative; zero API spend excludes local compute and energy.
In a fresh folder, run:
python -m pip install routeval==0.1.1
routeval init
routeval run
The bundled demo needs no API key. Expect 20 attempts, 16 correct answers, one deliberately allowed failure, and $0.0613 synthetic total cost. Failed attempts remain in the accuracy denominator.
Start with a policy change you need to accept or reject. Define correctness, mandatory guardrails, route roles and the whole decision's cost before comparing. A lower cost per correct outcome can still hide lost answer coverage or wrong actions.
Routeval 0.1.1 includes seven runnable workflow recipes. Scaffold one, then follow its README. Every recipe uses synthetic data and prices; no API key is required.
| Workflow | Start here | What to check |
|---|---|---|
| Async cascade | routeval init --recipe cascade cascade-demo |
Earlier charges, billed failures and missed review |
| Human handoff | routeval init --recipe handoff handoff-demo |
Completed resolution versus a queue acknowledgment; packet coverage |
| Structured extraction | routeval init --recipe extraction extraction-demo |
Required fields and versioned whole-record scoring |
| RAG with abstention | routeval init --recipe rag rag-demo |
Supported answers and useful coverage on answerable questions |
| Response cache | routeval init --recipe cache cache-demo |
Valid reuse, initial state and cold-versus-warm comparisons |
| Tool actions | routeval init --recipe tools tools-demo |
Approval, idempotency, committed effects and safely handled failures |
| Completed logs | routeval init --recipe logs logs-demo |
Reliable joins and complete bills without repeating model or tool calls |
The RAG, cache, action and log examples come from a second test round: 52 new synthetic workflow cases and 20 saved-log inputs. They exposed practical limits: blanket abstention can improve an overall score while answering no answerable questions, and unchanged final text can conceal incorrect tool actions. Explicit application checks catch those failures. These were AI-led software tests with zero external-user trials; they do not establish production quality or adoption.
The patch also fixes a numerical boundary: a deterioration exactly at the allowed tolerance now passes, while a recorded value beyond it still fails. Default tolerances and the run-artifact schema are unchanged. 111 source tests and all seven installed recipe checks passed locally on macOS/Python 3.14.
Default comparison covers its declared aggregate checks. Add your critical cohorts, latency and action requirements explicitly. A warm-cache run is a valid experiment, but attributing savings to a changed policy requires controlled starting state. Callable latency from saved-log replay measures lookup, not the original service. Missing required measurements cannot establish a pass.
Download and unzip the trial kit. Open a terminal in its routeval-trial-kit folder, then run:
python -m pip install routeval==0.1.1
routeval --version
routeval run --config examples/local-hosted-cascade/routeval.json
You should see version 0.1.1, five correct answers, 100% escalation recall, $0.011 total synthetic cost, and $0.0022 per correct answer. The report also saves a complete JSON run artifact.
The kit includes three starting points. Each uses a lookup fixture; none calls a model or claims to implement a production integration.
A local fast path, a hosted escalation, and a free local review. Check that price never determines a route's role.
Self-service answers and cases that require a human. Keep policy labels separate from whether the final answer sounds right.
Rules handle ordinary fields; an LLM handles ambiguous layouts. Check whether a policy change skips required fallbacks.
To reproduce all three fixtures and their deliberate routing regressions:
python run_trials.py
For your own pipeline, change one function and replace the five golden examples. Return the answer, actual route, and whole-decision cost, including calls made before escalation:
from routeval import Decision
def evaluate(request) -> Decision:
result = your_pipeline(request)
return Decision(
answer=result.answer,
route=result.route,
cost_usd=result.total_cost_usd,
)
Name the automatic and escalation routes in the supplied config. Add expected_route labels if you want routing metrics. Without those labels, answer quality, per-route cost and latency, and cost per correct answer still work. Missing cost stays unknown; cost gates require complete coverage.
Save a baseline, change one policy, and compare on the same five examples. An unavailable required metric, a changed measurement contract, or a routing regression returns a nonzero exit code.
I'm exploring two uses with equal priority: coding agents integrating the tool into an app, and agents evaluating workflows to recommend policy changes.
A working group of three AI reviewers tested both paths against the published 0.1.1 package. Four integration judgments and six saved-artifact judgments matched expectations fixed before testing. A cheaper policy preserved required review, while other candidates failed critical-case or latency requirements even when native baseline comparison passed.
The result is a practical next step: two linked runnable examples, a short guide, fixed owner rules and explicit accept / reject / invalid recommendations. Those examples are planned work; the study used local prototypes. These were synthetic AI exercises with zero external-user trials.
Explore the agent working-group findings →
In Quorum's verified routing audit, 40 of 184 attempts had correct verdicts with different route labels: 34 missed escalations and six unnecessary escalations. That is 21.7% of all attempts; it is a different quantity from the share of correct verdicts.
The result led me to separate answer quality, policy agreement, and economics. It did not establish that those answers were luck. The Quorum evaluation set is diagnostic evidence, not a verified independent estimate of production performance.
Routeval packages that separation as a small report and versioned artifact. Bring your own scoring function or judge. Existing evaluation tools can also accept custom scoring; Routeval is a focused reporting contract that can sit alongside them.
I'm looking for three independent trials: a cascade, a support handoff, and an extraction flow. The useful result is a repeatable report that changes a decision. Five cases are an onboarding test, not a production-quality estimate.
The kit includes FEEDBACK.md. Record time to your first useful report, wrapper effort, confusing labels, missing costs, and the decision the report changed. If you get stuck, that is useful feedback too.
Email me about a trial.
Full artifacts can contain private answers and errors. The routeval summary command removes item contents and identifiers; review route and stage names before sharing. Nothing in this page or kit sends your results automatically.