LLM model routing: a ladder, a cascade, and a kill switch

LLM model routing is usually discussed as one problem. It is two. The first is choosing which tier of model should get a task, decided before anything runs. The second is choosing who answers when the model you wanted is rate limited, retired, or gone. I run both, one in the tool I write code with and one in the assistant on this site, and the work got better the day I stopped treating them as the same mechanism.

What is LLM model routing?

Routing is deciding which model handles a request. The useful question is when that decision gets made. A 2026 survey of the field frames routing systems along three axes: when decisions are made, what information is used, and how they are computed (Moslem and Kelleher, arXiv 2603.04445).

Decide before generation and you are doing model selection: matching the task to a tier, trading quality against price. Decide after a failure and you are doing failover: keeping a feature answering when a provider goes dark. A cost problem and an availability problem. They draw the same on a whiteboard, a list of models with arrows, and they are almost nothing alike.

How do I decide which tier gets a task?

By what a wrong answer costs, not by how hard the task feels. Difficulty is a seductive signal and a bad one. Renaming a field across forty files feels enormous and can be reverted with one git command. Changing how a session cookie is set feels tiny and is a security incident if it goes wrong.

Those prices are Anthropic's list rates per million tokens, read from the pricing page on 3 September 2026. Output on the top tier runs ten times the bottom one, which is the entire reason a ladder is worth maintaining. Anthropic's own cost guidance says it in a line:

Use appropriate models: Choose Haiku for simple tasks, Sonnet for most production workloads, and Opus for the most complex reasoning

I keep the ladder as a roster of standing agents rather than a decision made fresh each time, because a rule you have to remember is a rule you skip at 11pm. Each is a short markdown file whose front matter pins the tier:

---
name: scout
description: Cheap read-only locator. Use to find where something
  lives - which file, which function, which config key, does X exist.
tools: Read, Grep, Glob, Bash
model: haiku
effort: low
---

There are seven. A Haiku locator that returns paths and line numbers. Three Sonnet workers: one reads the long log so my main thread never has to, one applies changes already decided, one runs the live round trips and reports evidence. Two Opus agents: one takes a hard problem on a clean context, one reads a diff adversarially, hunting the case that breaks it. And the top tier, reached for rarely and on purpose.

Notice what the top of that list turns on. The expensive tier is not defined by difficulty in its own description. It is defined by consequence: reach for it when a wrong answer costs a deploy, a data loss, or a client's trust, not merely when a problem is unfamiliar.

When is it worth climbing a rung?

When a wrong answer costs money, data, credentials, or a public correction. In practice that means anything touching authentication, payments, or personal data; migrations and other operations that cannot be undone; a bug whose cause spans more than one layer; and any factual claim that ships where a stranger can read it. Editing public copy is not a rung climb. Asserting a public fact is.

The rule that has paid for itself most often runs the other way. When a cheap tier hands back something hedged or subtly wrong, do not patch it. Re-ask one rung up. Patch it and you pay for the bad answer, then the repair, then the review of the repair, and still end up with something nobody trusts.

Prices also move, which is its own argument against routing from memory. Sonnet 5 launched at $2 and $10 per million tokens as introductory pricing through 31 August 2026, with a rise to $3 and $15 scheduled for 1 September. The pricing page now records that the rise will not happen. Quote a model price from training data and you have a real chance of being wrong in both directions.

Context length is a bigger lever than tier for most sessions anyway. When I measured where my own bill went across seven days of transcripts, 75 percent of cache-read tokens came from requests above 250k of context, 53 percent from requests above 400k, and the ten costliest sessions had spawned no subagents at all. Cache reads bill at a tenth of the base input price on most models. A tenth of a very large number, repeated every turn, is still the bill.

Does a cascade save money?

Mostly no, and there is now a careful answer as to why. The classic cost-routing result is RouteLLM (Ong et al., first posted June 2024, last revised February 2025), which trained routers to choose between a strong and a weak model and reported cost reductions of over 2 times in certain cases with no loss in response quality. That is routing before generation.

Cascading is the other shape: send everything to the cheap model first, escalate when confidence is low. A May 2026 paper by Dylan Bouchard tested that structure across five benchmarks and eight models from five providers, and found something worth sitting with:

A lightweight pre-generation router exceeds the best cascade policy on four of five datasets, mainly because it avoids the cheap model's generation cost on queries sent directly to a larger model rather than because of a stronger routing signal.

The conclusion is that cascade performance is limited by structural cost, because a cascade pays the cheap model before any escalation decision can be made (arXiv 2605.06350). The cheap attempt is sunk. You cannot get it back by escalating well.

That matches what I had arrived at by feel and could not articulate. My ladder is a pre-generation router in which the router is a person applying a written rule. My cascade never fires for quality reasons at all, only after something has already failed, and the day I stopped asking it to save money its design got much simpler.

What does a multi-provider cascade actually look like?

Four providers and, as I write this, ten model ids, because the ids live in settings rows that override the compiled-in defaults. One loop, and one strict rule about what counts as a failure. This is the chain behind the assistant on this site, resolved from the rows this site is running, in the order the code walks it:

groq      openai/gpt-oss-120b
groq      openai/gpt-oss-20b
groq      qwen/qwen3.6-27b
nvidia    nvidia/nemotron-3-super-120b-a12b
nvidia    deepseek-ai/deepseek-v4-flash-0731
cerebras  gpt-oss-120b
cerebras  zai-glm-4.7
cerebras  gemma-4-31b
gemini    gemini-3.5-flash
gemini    gemini-2.5-flash

The code compiles in eight of those. The other two came from the database, where one row hands the Cerebras rung a pair of ids the compiled default never named. Three further rows happen to repeat ids the code already has, so today they change nothing, and the day one of them names something else the deployed source stops describing the chain that runs.

The rule came out of an outage. The Ask feature on a directory site I run went dark, and the cause was that the provider had quietly decommissioned two model ids. My chain at the time only cascaded on 429 and 5xx responses, which was a reasonable-sounding decision that turned out to be exactly wrong: a 404 is not a rate limit, so the chain stopped dead at the first rung with four healthy models sitting behind it. The fix was smaller than the bug. Collapse every failure into one shape.

$j    = json_decode((string) $res, true);
$text = trim($j['choices'][0]['message']['content'] ?? '');
return $text !== '' ? $text : null;

That is the whole error policy for every provider in the chain. The caller never reads the status code, because a retired model id, a 400, a 429, a timeout and a 200 whose content came back empty all mean one thing to the next rung: this model did not answer. The only condition that skips a provider without spending a request is a missing key, which returns null before the connection opens.

The empty-content case is the one people miss. A reasoning model will happily return HTTP 200 with the whole token budget spent in a thinking field and nothing in content. Every reasoning model in my chain has its thinking turned off or pinned to the floor, and an empty string still counts as a failure, because a feature that returns nothing has failed whatever the status line says.

Where should a fallback chain end?

In something that cannot fail, which means it ends below the last model. My chain has two rungs with no model in them: return the best-matching knowledge chunk exactly as written, and if nothing matched, one honest line and an email address. A visitor never sees an error, a dead spinner, or a machine apologising.

Above those sit the models, ordered fastest-adequate-first. Groq leads because it is the fastest thing I have access to, a quarter to half a second on this site's own round trips. What makes crossing vendors worth the trouble, rather than stacking models at one of them, is that the free tiers are scarce in different shapes. Groq publishes 30 requests a minute and 200,000 tokens a day, per model. Cerebras publishes 5 requests a minute on its time-limited free trial but a million tokens a day. Narrow gate and deep well against wide gate and shallow well, and a chain spanning both survives either kind of exhaustion.

My code comment says Gemini Flash sits last because it has the highest free ceiling of the four, and I should flag that this is now the softest claim in the chain. Google no longer publishes per-model free-tier numbers on its rate-limits page:

Rate limits depend on a variety of factors (such as your usage tier) and can be viewed in Google AI Studio.

So that ordering rests on my own measurements and a reading of docs that no longer exist in that form. It has never been wrong in production, and I cannot cite it. Worth saying out loud rather than dressing up.

One constraint only shows up in production: a chain has a time budget. A site of mine sits behind a CDN with a hard request ceiling near 100 seconds, and one trivial prompt took 105 by stacking a 45-second timeout across three API keys. Nothing had failed. The chain worked exactly as designed and the request died anyway. The fix, a short leash inside the chain and a longer one in the lab, is written and waiting on a deploy.

How do you kill a dead model without a deploy?

Put the chain in a database row. The model ids compile into the code as defaults, and a settings row overrides them, so a retired model can be pulled out of the ladder in the time it takes to edit a field. These are the rows in force on this site, and the fourth one accounts for the two ids the source never mentions:

groq_model      openai/gpt-oss-120b
groq_model_alt  openai/gpt-oss-20b
nvidia_model    nvidia/nemotron-3-super-120b-a12b
cerebras_models gpt-oss-120b,zai-glm-4.7,gemma-4-31b
gemini_models   gemini-3.5-flash,gemini-2.5-flash

Clearing a provider's key removes that whole provider from the chain on the next request, with no code change at all. That is the kill switch, and it exists because model ids die on the vendor's schedule, not yours. Groq's deprecation page records llama-3.1-8b-instant and llama-3.3-70b-versatile shutting down on 16 August 2026, and qwen/qwen3-32b and Llama 4 Scout on 17 July. Two of those had been in my chains.

Retirement is rarely loud. In late August an embedding model I depended on began answering HTTP 410 with an end-of-life timestamp in the body, and semantic retrieval on the live site quietly stopped and fell back to keyword matching. No error page, no alert, slightly worse answers. That is the failure mode to design against, and the same reason knowledge has to ship with the feature instead of being refreshed when someone remembers.

The trap in this design is worth naming, because I walked into it twice. Settings rows travel with the database, not with the code deploy. Shipping a corrected chain and forgetting the row leaves the dead model pinned in production while your local copy works perfectly.

How do you know the fallback works?

Poison the primary key and watch. A chain never tested with its first rung broken is a chain you are guessing about, and a happy-path test proves nothing about it, because the happy path never reaches the fallback.

I do it from a command-line harness that boots the application and overwrites the keys in memory, so nothing on disk or in the database changes. The numbers from the last run: normally Groq answers in about 0.3 seconds; with Groq poisoned, Nemotron answers in 0.6; with a dead model id ahead of it, DeepSeek answers in 10. That is a chain I have watched fail over, which is a different object from one I assume will.

Probing also found a failure nobody had reported: a model id in a live chain answering 410 Gone, absorbed silently by the rung below it. Silent absorption is what you want in production and what hides rot, so the ladder needs a window. Mine is an admin page that reads the chain out of the running code and shows the order, each provider's published limits, the last round trip measured per model, and who answered the most recent question. Keys are counted, never printed. Same instinct as the rest of the harness, and the difference between owning this and renting a bot you cannot inspect.

The last time I swapped models, I ran the adversarial reviewer over the diff twice, once after the first round of fixes. The second pass found that the change had left the site's own knowledge base telling visitors a model was retired when it was still perfectly alive. The chain was right and the sentence describing the chain was wrong. There is now a line in the model-swap checklist for it: after you fix the ladder, audit what the site says about itself.