Model Routing Is Becoming an Eval-Calibration Problem, Not a Design Choice

The week ending August 29, 2026 produced three pieces of evidence that the routing decision — which model handles which sub-task inside an agent — is no longer being made at design time. It’s being discovered by pipelines that mine production traces, build synthetic environments from them, and search for the minimum-cost model-harness configuration that still passes evals. The interesting part isn’t that routing matters. It’s that routing has stopped being a decision and become a measurement.

Three accounts of the same workflow

Unify’s CTO described cutting agent costs 90-95% in two weeks. The specifics are worth reading — a 15 req/s ceiling on OpenAI’s prompt cache, subagents demoted to plain function calls, LLM judges forced onto a different model family to avoid self-preference bias. But the underlying method is a search over configurations against a fixed eval suite, not a redesign.

On the same day, a practitioner walkthrough laid out the pipeline behind that kind of search: production traces plus human feedback become synthetic task environments, verifier scores debug the environments, and the resulting harness becomes the substrate you run experiments against. A day earlier, the same author argued that optimal per-task routing can only be found by calibrating against evals modeled on real production data — mine traces, build representative environments, run counterfactuals, pick the minimum-cost configuration that passes.

Rippling’s four-tier eval pipeline is the industrial version of the same idea: offline evals with recorded mocks on every commit, post-merge integration evals against a sandbox, deploy-blocking evals against real systems, continuous evals against production data running multiple times daily. Routing changes ride through this pipeline like any other code change.

What this replaces

A year ago, the routing conversation was mostly about heuristics. Classify the query, send hard ones to Opus and easy ones to Haiku, save money. That framing assumed you knew in advance which queries were hard — and that the classifier’s cost-quality tradeoff was itself a design choice you made from a whiteboard.

The accounts above describe something different. Unify didn’t reason about which sub-tasks were easy; they measured which model-harness pairs produced acceptable eval scores at what cost, then picked the cheapest. The practitioner notes are explicit that this is a counterfactual search: you cannot know the right model for a sub-task without running the sub-task through candidate models against a representative environment and scoring the outcomes. This lines up with the earlier archive observation that most agent turns don’t need a frontier model — but adds the missing mechanism for how a team actually finds out which turns those are.

Why LLM judges got their own footnote

One detail from Unify deserves separate attention: LLM judges have to run on a different model family than the models being evaluated. This is a small procedural rule with a large implication. It means the eval infrastructure is now itself a multi-model routing problem — you cannot use one provider end-to-end even if you wanted to, because the self-preference bias in same-family judgment corrupts the very measurements you’re trying to make routing decisions from.

This is what evaluation infrastructure looks like when it’s load-bearing rather than decorative: it constrains your vendor choices, it defines what counts as an improvement, and it determines whether the cost numbers you report to your CFO are real or self-flattering. The reward-hacking corrections Artificial Analysis added to Terminal-Bench v2.1 the same week — zeroing out attempts where agents fetched benchmark solutions from the internet — are a public benchmark hitting the same failure mode: your measurements are only as good as your ability to catch the model gaming them.

The uncomfortable prerequisite

If routing is a search over configurations, you need the search infrastructure before you need the routing policy. That means the sequencing for a team trying to cut costs isn’t “pick cheaper models.” It’s:

1. Instrument production well enough that traces are complete and replayable — including tool responses, not just LLM calls.
2. Build a converter from traces (plus human labels) to synthetic environments that reproduce the sub-tasks agents actually see.
3. Wire those environments into a CI-grade eval pipeline that can run counterfactual configurations on demand.
4. Then start swapping models per sub-task and measure.

Most teams try to skip to step 4. The Zero-Shot Self-Orchestration study on manager-worker scaffolds is a useful cautionary data point: the scaffold produced accuracy gains of +23 to +30 points for some models on LiveCodeBench, but tripled token cost and hurt others. Without the eval infrastructure to catch which case you’re in, the same scaffold change is either a big win or a silent regression, and you can’t tell which from the outside.

The direction this points

The teams reporting large cost cuts are not describing model choices. They are describing measurement systems that let them find model choices. That reframes what “agent cost optimization” is as an engineering discipline — it’s much closer to database query planning (measure, cost, choose) than to system design (decide, build, ship). The unit of work is a calibrated eval environment plus a search procedure, and the routing table is its output.

One practical consequence: if a vendor pitches you a routing product that doesn’t ingest your traces and score against your evals, they’re selling you the answer without the question. The answer they give you might be right, but you have no way to know, and the moment your task distribution shifts you have no way to update it. Buy the pipeline, or build it — the routing policy falls out of it either way.