willianpinho.com Blog
Cover image for Multi-LLM routing in production: the failure modes nobody warns you about
  • llm
  • routing
  • ai
  • production

Multi-LLM routing in production: the failure modes nobody warns you about

Multi-LLM routing looks clean on a whiteboard. In production the real risk is the cost math that hides its own downside, latency treated as a single number instead of a distribution, and the silent failures that return a clean HTTP 200.

The pitch for multi-LLM routing is clean. Route cheap requests to a cheap model, hard requests to a strong one, keep a fallback for when a provider has a bad day, and watch your cost-per-task drop while quality holds. I've built this. It works. But the version that survives contact with production looks very different from the one in the architecture diagram, and the gap is made of failure modes that don't announce themselves.

One of the systems where I learned this the hard way was a KYC pipeline with real compliance stakes. On that build, picking the model was the easy part. The real engineering was everything around the routing decision: knowing when a route was quietly degrading, what a fallback actually cost in latency and correctness, and how to tell a cheap-model success apart from a cheap-model failure that happened to return valid-looking text. Those are the parts nobody warns you about.

The cost optimization that costs more

The first thing teams do is route by cost. Send the simple stuff to the small model, escalate only when needed. The math on the spreadsheet is compelling.

The math in production is different, because routing isn't free and re-routing is expensive. When the small model gets a request wrong in a way you catch, you re-run it on the large model. Now that request cost you both calls plus the latency of two round trips plus whatever your detection step costs. Get your routing threshold slightly too aggressive and you discover you're paying for the cheap model and the expensive one on a meaningful share of traffic, while shipping worse latency than if you'd just used the strong model directly.

The trap is that the per-token price is visible and the re-route tax is not. You see the cheaper bill on the requests that worked. You don't automatically see the compound cost on the ones that didn't, unless you instrumented for it. Most teams don't, so they optimize toward a number that's hiding its own downside.

The fix isn't to abandon cost routing. It's to measure cost-per-successful-task, end to end, including retries and escalations, and route against that. The cheap model only wins if it wins after you account for how often it loses.

Latency is a distribution, not a number

The second failure mode is treating provider latency as a property you can look up once. It isn't. It's a distribution that moves with the provider's load, the time of day, the length of your context, and whether you're on a tier that gets deprioritized when capacity is tight.

A router tuned against median latency will route happily right up until the provider's tail blows out, and then your p99 user is waiting on the slow path you chose because the median looked fine an hour ago. If you've layered a fallback on top, a timeout-triggered fallback adds latency rather than removing it. The user waits for the first model to time out, then waits all over again for the second.

What you actually want is latency budgets per route, measured continuously, with the router making decisions against recent tail behavior rather than a static config. And you want your fallback policy to be honest about the fact that "fall back on timeout" means "sometimes pay both latencies." For a real-time path, that's often the wrong trade, and you're better off routing to the more reliable provider up front and accepting a higher floor for a tighter ceiling.

Dean and Barroso's The Tail at Scale has the sharper version of this move. Instead of waiting for a hard timeout to fire the fallback, you fire it early, around the point where a request has already run longer than most do, say the p90, and take whichever response comes back first. You spend some extra requests to buy a much tighter tail. It only pays off where the second call is genuinely cheap relative to the latency you're saving, so it's a tool for the hot path, not a default to switch on everywhere, but it beats sitting on a slow request until a timeout admits what the latency distribution already told you.

Eval-driven routing, or routing on vibes

Here's the one that separates a system from a demo. How do you decide which model handles which request? Most early routers do it on heuristics: keyword matching, token count, a hand-written rule that "code questions go here." A rule like that encodes a guess you made on day one and never revisited. It drifts the moment your traffic shifts, and you have no way to know it drifted.

Eval-driven routing means you have a measurable definition of success for your task, you score every provider against it on representative traffic, and your routing decisions are downstream of those scores. When you onboard a new model, you don't guess whether it's better — you run the eval and find out. When a provider silently changes a model behind a stable name, your eval catches the regression before your users do.

This is non-negotiable for anything that matters. In the KYC pipeline, "the model returned a confident answer" and "the model returned a correct answer" are entirely different events, and the gap between them is where the compliance risk lives. You cannot route responsibly without a way to distinguish those two, and that way is evaluation, run continuously, not a one-time benchmark you ran during the pilot.

The failures that return HTTP 200

Your alerting watches for exceptions and timeouts. The routing failures that cost you the most return HTTP 200 with a clean, well-formed body and never trip a single one of those alerts.

A provider degrades and starts returning shorter, lazier completions that still parse. Elsewhere, a model update shifts the output format just enough that your downstream extraction silently drops a field. Or a fallback fires, the backup model is genuinely worse at your task, yet every response comes back well-formed, so nothing alarms. These are silent failures, and a router that only watches for errors and timeouts is blind to all of them.

Observability for multi-LLM routing has to go past uptime. You need to track, per route, the quality scores from your evals, the rate of fallback activation, the cost-per-successful-task, and the shape of the outputs over time. A rising fallback rate is a leading indicator that a primary provider is degrading. A quality score sliding while the error rate stays at zero is the signature of a silent regression. If you're not watching those, you'll find out from a customer, which is the most expensive way to learn.

The instinct here comes straight from running payment platforms at 99.9% uptime for ten million users. In that world you learn that the failure that pages you is the lucky one, because at least you know about it. The failures that quietly return success while corrupting your data are the expensive ones, and multi-LLM systems are full of them. Routing is where they hide.

So if you build one of these, instrument for the silent failures first, before you tune a single cost threshold. Cost-per-token is the number that's easy to see. Cost-per-successful-task, fallback rate, and quality drift at a flat error rate are the numbers that tell you whether the system is actually working, and they're the ones you have to go build on purpose. The routing diagram is the part you can draw in an afternoon. The instrumentation is the part that decides whether the thing survives a month in production.

Sources