Model Routing and Cascades: Assign Requests of Different Difficulties to Sufficient Capability
From direct routing, small-then-large, and quality-gap prediction, to escalation thresholds, cascade latency, fairness risk, and online recalibration.
- Establish per-sample quality and cost baselines for candidates.
- Define high-risk mandatory rules.
- Train/calibrate strong-model gain scores.
- Select thresholds that meet slice gates.
- Canary-route and randomly audit offloaded samples.
- Recalibrate/roll back after version or distribution changes.
1Routing turns a one-time model selection into a per-request decisionIntuition
In the same refund system, different requests are not equivalent. One user asks “How many days until the refund is credited?” and only needs a simple query against a status table or FAQ library; another user claims the system has already deducted payment but the order status is still unpaid, the amount is abnormal, and the evidence conflicts, which requires reasoning, cross-checking multiple pieces of evidence, and may even involve high-privilege operations. If the entire system pays the same computational cost for these two kinds of requests—sending all of them to the strongest model—then the vast majority of simple queries are subsidizing the pricing of a few difficult cases; conversely, if, to save money, everything is handed to a small model, conflicting cases will fail with higher probability. What model routing aims to solve is precisely to establish a per-request decision between these two extremes: decide who should handle it based on this current request itself, not based on some global constant.
First clarify several selection methods that are easily confused. Static model selection chooses a model when building a component; afterwards all requests within that component are handled by it, and the decision occurs at deployment time, independent of the content of each request. Dynamic routing is the opposite: the component maintains multiple candidate models. Each time a request arrives, the router reads the request content, risk rules, the capability records of each candidate model, and runtime evidence, and decides in real time whether this request should go to a small model, a strong model, a human, or be directly rejected. The difference between the two lies in when the decision occurs and at what granularity. Mixture of Experts (MoE) is different again: it is a model-internal structure that selects which experts to activate per token, and belongs to the model's own structural design. Users generally cannot directly control “whole-model switching” between requests, so it does not replace dynamic routing; the two operate at different levels.
The relationship between routing and capability first needs to clarify a boundary: routing does not create capabilities that the underlying models do not have; it only allocates among existing candidates. If all candidate models fail on a certain input slice, the best the router can do is pick one among several failures; it is impossible to expect routing to fill a capability gap. Therefore, the prerequisite for routing is that model selection and evaluation come first—first confirm for each sample slice that at least one sufficiently good candidate exists, establish a per-sample capability baseline, and then discuss allocation optimization on top of that baseline.
From this we can derive the true goal of routing. It is not “maximizing the small-model invocation rate”: if that were the goal, the system would be induced to incorrectly delegate difficult requests to a small model, trading error rate for cost. The real goal is maximizing safe delegable coverage—under the premise that errors and risks do not cross the red line, send as many requests as possible to lower-cost paths. The meaning of delegating a request is worth emphasizing: it indicates that current evidence shows the small model is sufficient, and absolutely does not mean the request is inherently simple. A request that is delegated today should immediately switch to a strong model or a human when new evidence appears (for example, the amount exceeds a threshold, or the account enters a high-risk list).
Finally, there are the input and output contracts of routing. Inputs include the current request itself, predefined risk rules, capability records of each candidate model, and runtime evidence; output is the handling path for this request: small model, strong model, human, or reject. In the decision process there is a hard priority: high-risk hard rules must take precedence over probabilistic scores. When a hard rule (such as high-privilege operations must be reviewed by a human) is triggered, no matter what score the probabilistic model gives, the path must be human or reject; the rule cannot be overridden by a score.
2Direct Routing, Cascades, and Parallel SelectionArchitecture
When putting routing into engineering practice, the first question is: choose the model before the call, or first let a small model provide an answer and then decide whether to upgrade to a stronger model based on answer quality? Different answers to this question form four basic forms, each differing in process, advantages, and costs. They need to be combined according to request characteristics, rather than using only one threshold across the entire site.
The first is rule-based routing. The router itself is a set of explicit rules: selecting a processing path by domain, by risk level, or by input length. It does not require additional training of any model, and every decision can be directly explained as "because it matched which rule," which is its greatest strength. The cost is that rule boundaries are coarse: overlaps between domains and ambiguous zones in risk judgment require manual rule maintenance; once a request deviates from the preset category divisions, the rules may give an inappropriate path.
The second is direct routing. Before calling a generative model, the system runs a lightweight classifier that predicts the appropriate tier for the current request (for example, small, medium, or large model), and then calls only the selected one. Its advantage is that difficult requests do not have to waste one call on a small model first—if the classifier judges the difficulty as high, the first step goes directly to the large model. The cost is equally clear: this decision can only rely on the input itself before the call, and the difficulty of many requests cannot be judged from input alone; when subsequent signals such as the answer and verification are missing, the classifier's judgment may be insufficient.
The third is cascading. The process first lets the small model S provide a complete answer; if its confidence is low, then it calls the strong model L to reprocess. The advantage of cascading is that it can use signals that are only produced in the first-answer stage: the confidence of the first answer, the verifier's judgment, and whether the retrieval results support that answer. These better reflect the true difficulty than input alone. The cost has two layers: upgraded requests take longer overall because they go through S's complete answer and then L's; moreover, S's first answer may "contaminate" subsequent processing—if L's processing references the first answer or the first answer has already triggered side effects, the error is carried into the strong-model stage.
The fourth is parallel selection. Multiple models run simultaneously on the same request, and after completion a selector chooses which answer to adopt. Its advantage is low wall-clock latency, because all candidates proceed in parallel and multiple candidate answers are obtained simultaneously for easier comparison. The cost is that compute cost grows multiplicatively with the number of candidates, and the selector itself can also make mistakes—the incorrectly chosen answer is not more reliable than a single model answering directly.
These forms suit different signal types. Signals available before the call suit direct selection; first-answer and verification signals can only be obtained after running one round, so they suit cascading; parallel selection trades cost for wall-clock time. Therefore, architecture selection has an input-output structure: the inputs are the decision time point (when signals are available), available signal types, error cost, and latency SLO; the output is a combination of these four forms for different request slices. High-risk requests should use deterministic rules to force the strong model or human review, not leave them to a probabilistic router to "try its luck"; low-risk long-tail requests can safely use cascading. The whole site should combine forms according to different slices, rather than scheduling uniformly by a single threshold.
There is also a boundary that must be observed: when the first stage may produce side effects, you must not cascade without review. If the small model's answer can directly trigger charges, send notifications, or modify state, then "run S first and then decide to upgrade" is itself unacceptable—the side effect of the first stage has already occurred, and subsequent upgrading cannot undo it. Paths with side effects can only be used under the premise of having review and oversight.
| Form | Process | Advantage | Cost |
|---|---|---|---|
| Rule-based routing | Select by domain/risk/length | Interpretable, no extra model | Coarse boundaries, maintain rules |
| Direct routing | Lightweight classifier selects S/M/L first | Difficult requests don't waste the small model first | Input alone may be insufficient to judge |
| Cascade | Run S first, then run L if low confidence | Uses first-answer, verification, and retrieval signals | Upgraded requests are slower and first answer may contaminate |
| Parallel selection | Run multiple models simultaneously, then select | Low wall-clock latency, more candidates | High cost, selector can also err |
3The mathematical objective must include per-slice hard constraintsMechanism
If the objective of routing optimization is only to “minimize average cost”, the router learns a dangerous shortcut: downgrade all requests to the small model. Average cost is indeed minimized, but a few high-risk unauthorized action errors are drowned in the success of many easy requests. This is exactly the structural defect of average constraints—it allows high scores from many easy requests to offset misrouting in a few key slices. Mathematically acceptable, operationally unacceptable.
Therefore, the correct formulation of the routing objective must include per-slice hard constraints. The overall goal is to minimize expected cost while satisfying each key slice’s quality lower bound and risk upper bound. Written as a constrained optimization problem:
Minimize E[ c( m(x) ) ], subject to: for every key slice k, quality q(m(x)) ≥ Q₀ and Riskₖ ≤ Rₖ.
Here x is the request, m(x) is the model the router selects for that request, c is the processing cost, q is the answer quality, Q₀ is the quality lower bound, Riskₖ is the risk exposure of slice k, and Rₖ is the risk upper bound permitted for that slice. A slice refers to a subset along dimensions such as language, domain, risk level, and user group. The constraints are inequalities that hold separately for each slice; if any slice violates its constraint, the entire policy is unacceptable. High quality in one slice cannot be borrowed to fill a gap in another. Slices with small sample sizes require special attention: point estimates are unreliable, so interval estimates should be reported, and a conservative upper bound should be used as the basis for determining whether a constraint is satisfied. It is better to downgrade less than to underestimate risk.
Alongside the objective function is the specification of the training target. If a router learns to “predict the absolute difficulty of a request”, the information it learns is not aligned with the benefit. The correct training target is to predict the quality gain of the strong model relative to the small model: if both answer correctly, upgrading is wasted; if both answer incorrectly, upgrading has no benefit; only on samples where the strong model answers correctly but the small model answers incorrectly does upgrading truly produce value. Hence the most valuable training signal is the samples where “the strong model is correct and the small model is wrong”. Meanwhile, quality labels must come from independent scorers and human annotation. The strong model’s self-ratings cannot be used as unbiased ground truth—letting the strong model be both player and referee would systematically bias labels toward upgrading, because the model tends to endorse its own answers.
There is also a data-level pitfall called selection bias. During online operation, the strong model typically runs only on samples that are upgraded, while downgraded samples are never tried by the strong model. As a result, we can never see the counterfactual: “if these samples had been upgraded originally, would the strong model have done better?”. Training only on upgraded samples biases the gain learned by the router. There are two ways to remedy this: random exploration, upgrading requests that should have been downgraded with a certain probability in exchange for counterfactual observations; or offline dual execution, running the same batch of samples through both the small model and the strong model to directly obtain unbiased paired data.
Putting this all together: routing optimization takes the request x, the selected model m(x), cost c, quality q, quality lower bound Q₀, risk Riskₖ for each key slice, and upper bound Rₖ as inputs, and outputs a minimum expected cost policy that satisfies all per-slice hard constraints. A pretty average-quality number cannot offset misrouting in a few slices; the training target should aim at the strong model’s relative benefit; the missing counterfactual for downgraded samples must be filled through random exploration or offline dual execution. If any of the three is missing, the direction of optimization will deviate from the true goal of “safe to downgrade”.
4How to Calculate the Expected Cost and Latency of CascadesDerivation
The intuition about cascade cost is easy to get wrong. Given a small model at ¥0.02 per call, a strong model at ¥0.10, and an escalation rate of 25%, a common intuition is that “the average cost falls between the two, roughly weighted by the escalation rate.” If all requests first run the small model and only those judged for escalation then run the strong model, and temporarily ignoring other components such as scorers, the expected call cost can be written exactly as:
ECost = Cₛ + r × Cₗ
where Cₛ is the small model’s per-call cost, Cₗ is the strong model’s per-call cost, and r is the escalation rate. The meaning is: every request first pays Cₛ, because the first step of the cascade is executed for everyone; the escalated portion of requests (proportion r) then additionally pays Cₗ. Substituting Cₛ = 0.02, Cₗ = 0.10, r = 0.25 gives 0.02 + 0.25 × 0.10 = 0.045 yuan. So under this calling assumption, the average cost is indeed ¥0.045—consistent with intuition, but the premise must be made clear: this is the cascade accounting for “all requests run the small model first,” not a universal answer for arbitrary routing forms.
Comparing with direct routing makes the differences between the forms clear. If the router, after an upstream classifier, calls only one model, the cost structure becomes Crouter + (1 − r)Cₛ + rCₗ, meaning the classifier cost Crouter is paid first, then requests split into two paths according to the escalation rate: a proportion (1 − r) pays only the small model, and a proportion r pays only the strong model. When the expected costs of the two forms are close, direct routing appears more cost-effective, because the escalated requests in a cascade must bear the serial tail latency of the small model plus the strong model: they first wait for the small model’s full answer, then wait for the strong model’s full answer, and the slowest user experience clearly deteriorates.
Latency accounting is isomorphic to the cost structure. The expected latency ETime = Tₛ + Tjudge + r × Tₗ, where Tₛ is the small model latency, Tjudge is the latency required to decide whether to escalate, and Tₗ is the strong model latency. Everyone first bears Tₛ and Tjudge; the escalated proportion r additionally bears Tₗ, and these delays are serial accumulations with no parallel overlap.
The premise under which these two formulas hold must be clear: they only account for the calls themselves. Real total cost also includes scorers, retries, retrieval, human intervention, and failure handling; each of these changes the final number. The result 0.045 yuan does not hold after adding these components; it is only a value under the “given calling assumptions.” Similarly, the latency formula does not include queuing, retries, or failure degradation paths.
The final structural constraint: the first stage of a cascade should be side-effect-free by default. If the small model’s answer is directly written into the user’s conversation history, or triggers a tool call, then even if it is later escalated to the strong model, the content left by the first answer or the side effects that have already occurred cannot be retracted, constituting contamination. Therefore, the first-stage output of the cascade should only serve as candidate evidence for subsequent stages to reference, not as the source of established facts or external actions.
5Worked example: how thresholds change coverage, errors, and costStep-by-step calculation
The effect of the threshold on a routing system is not abstract; it can be seen directly in a set of numbers. Suppose there are 1000 refund requests, each with an escalation score g(x) for "needs the strong model," and the system sorts them by score from high to low: requests whose score reaches threshold τ are escalated to the strong model, and the rest are delegated to the small model. Three candidate thresholds produce the following results (the latency path for escalated requests is the serial process "small model + decision + strong model"):
At τ = 0.7, 150 requests are escalated, 28 difficult requests are misrouted, the quality pass rate is 91.0%, and the average model cost is 0.02 + 0.15 × 0.10 = ¥0.035. At τ = 0.5, 250 requests are escalated, 12 difficult requests are misrouted, the quality pass rate is 93.1%, and the average model cost is 0.02 + 0.25 × 0.10 = ¥0.045. At τ = 0.3, 420 requests are escalated, 4 difficult requests are misrouted, the quality pass rate is 94.0%, and the average model cost is 0.02 + 0.42 × 0.10 = ¥0.062.
The pattern in the table is clear: lowering τ means more requests are escalated, average cost rises, misrouted difficult requests decrease, and the quality pass rate increases. Raising τ has the opposite effect: it saves money, but more difficult requests are wrongly delegated, and overall quality declines. So the threshold is not just a number being adjusted; it is a common control point moving along three curves: misrouting, quality, and cost.
Now add decision constraints: require a quality pass rate ≥ 93% and misrouted difficult requests ≤ 15. Check each candidate: τ = 0.7 has only 91.0% quality, so it fails; τ = 0.5 has 93.1% quality and 12 misrouted requests, satisfying both constraints; τ = 0.3 also satisfies them, but its cost ¥0.062 is higher than τ = 0.5's ¥0.045. Therefore, under the constraints, τ = 0.5 is the lowest-cost feasible point—threshold selection is about finding the lowest-cost point in the feasible region, not blindly choosing the highest quality or lowest cost.
But passing on global metrics does not mean the decision is complete. Suppose a high-risk slice (such as abnormal amounts or high-privilege refunds) accounts for exactly 5 of the 12 misrouted requests, and this slice has its own independent misrouting cap, say no more than 3. Then τ = 0.5 already exceeds the limit for this slice. In this case, the global average cannot be used to cover it up: the correct approach is to forcibly escalate this high-risk slice separately, even if its escalation score is below the global threshold. The per-slice hard constraint once again overrides the global average.
The inputs to this analysis are the 1000 requests, their escalation scores g(x), threshold τ, and the corresponding quality and cost results. The output is the lowest-cost threshold that simultaneously satisfies the global quality gate, the misrouting gate, and all critical slice constraints. There is also a premise hidden in the feedback loop shown in the diagram: the escalation score must predict "escalation benefit" rather than some other quantity, and samples of delegated requests must flow back through sampled dual-run or human audit; otherwise the system can never see the requests it misroutes, and the misrouting numbers above cannot be measured.
Scroll horizontally to view the full diagram on small screens.
| Threshold | Escalated count | Misrouted difficult requests | Quality pass rate | Average model cost | Escalated request latency |
|---|---|---|---|---|---|
| τ=.7 | 150 | 28 | 91.0% | .02+.15×.10=¥.035 | S+decision+L |
| τ=.5 | 250 | 12 | 93.1% | ¥.045 | S+decision+L |
| τ=.3 | 420 | 4 | 94.0% | ¥.062 | S+decision+L |
6Routing signals must be available and calibratable at decision timeFeatures
The router's entire intelligence comes from the signals it uses, so the choice of signals determines the upper bound of routing. The question to answer is not “which features correlate with difficulty,” but the stricter version: which features can actually predict that the strong model is more valuable than the small model. The answer to this question is first constrained by a timing constraint—the signals must already be available at the moment the decision is made.
By availability timing, signals can be divided into two groups. Available before invocation: the input's domain, input length, language, risk rule matches, retrieval hit status, and degree of evidence conflict. These can be obtained before any generative model has been called, and are suitable for direct routing. The other group can only be obtained after running the first answer: the small model's output probability (logprob), self-consistency disagreement among multiple samples, format validation results, and the judgment of external verifiers. These signals are suitable for cascading, because cascading already runs the small model once. Conversely, when training an offline router, you must never use final human labels, future tool states, or features known only after the strong model has run—if used, the model “sees the future” during training, which is data leakage, and the offline router's evaluation scores will be inflated, but cannot be reproduced after deployment.
Even when signals are available, each has blind spots. Rules and metadata are available before invocation, but new types of difficult requests often do not fall into categories covered by old rules, and historical group features may carry the risk of proxy discrimination. The small model's logprob is available after the first answer, but it is insensitive to fluent but incorrect content, affected by length bias, and often miscalibrated itself. Multi-sample disagreement is powerless against consistent errors—multiple samples can be wrong consistently, and running more samples is costly. Retrieval and verifiers are only available after tool execution, their reliability is limited by verification coverage, and verifiers themselves can also make mistakes. Historical group features are available before invocation, but they involve fairness, privacy, and distribution shift: group behavior in the training distribution changes over time and with policy, and static use introduces discrimination and outdated judgments.
After signals are selected, there is one final step: calibration. The router outputs a score, but the score must be interpreted as “the probability that upgrading can fix this error,” not as the model's self-perceived confidence. A model saying “I am 90% confident” cannot directly become a routing threshold of 0.9, because self-reported confidence is uncalibrated, and systematic overconfidence or underconfidence are the norm. The correct approach is to plot reliability diagrams by slicing the score on held-out independent data: divide the score into intervals, compute the true proportion of cases in each interval where “the strong model actually fixed the small model's error,” and compare it with the interval's average score. If a slice has a high score but a low repair rate, the threshold for that slice must be tightened accordingly. In this way, the routing input is the domain, length, language, risk, retrieval, first-answer, or verification signal already available at decision time, and the output is the “upgrade repair probability” calibrated on an independent set. Both error directions must be blocked: leaking future information during training makes the score untrustworthy, and ignoring slice differences during calibration lets global reliability mask local miscalibration.
| Signal | Stage | Blind spots |
|---|---|---|
| Rules / metadata | Before invocation | Novel difficulties and proxy discrimination |
| Small-model logprob | After first answer | Fluent errors, length bias, miscalibration |
| Multi-sample disagreement | After first answer | High cost, may err consistently |
| Retrieval / verifier | After tool use | Verification coverage and its own errors |
| Historical group features | Before invocation | Fairness, privacy, and distribution shift |
7Where Do Erroneous Downgrades Concentrate?Safety and Fairness
A routing system can meet global quality thresholds while particular groups consistently receive only the weak model. Average quality pass rates mask distributional issues: erroneous downgrades are not spread evenly across all users but concentrated in a few slices. Routers are usually trained on majority languages and common tasks, so they misjudge short inputs as easy—but short does not mean easy, and a short request that condenses complex background actually needs the strong model the most. They may also ignore low confidence on minority languages as noise because there are too few such samples in training data for the model to learn their difficulty. They may further use historical proxy features such as region or account tier to make decisions, creating unfair gaps in service quality along those dimensions.
Fairness evaluation reports slice by slice rather than looking at a single overall number. For each language, domain, user group, and risk slice, report the downgrade rate, strong-vs-small error rate (the proportion of samples where the strong model answers correctly and the small model answers incorrectly), misrouting rate, and upgrade benefit, along with their confidence intervals. When a slice has few samples, the confidence interval will be wide; this itself is information—it tells you the estimate for that slice is unreliable and calls for a more conservative strategy, rather than treating a wide interval as “no problem.” The fairness goal needs to be stated clearly: it is not about requiring the same upgrade proportion across all groups, but about ensuring that errors and service quality do not produce unacceptable gaps due to routing policy. Different upgrade proportions are reasonable—different groups inherently have different problem distributions; what is unacceptable is that some groups continuously receive errors because of the router’s systematic blind spots.
In high-impact domains there is a more direct way out: do not rely on learned routing. Healthcare, financial decisions, permission write operations, and explicitly matched compliance keywords—these scenarios have cost functions that do not allow a probabilistic router to make mistakes, so you can directly use deterministic rules to force a stronger verification or human path. Learned routing is suitable for the long tail where errors are tolerable; deterministic rules handle the head where errors are intolerable. The two divide responsibilities by domain, rather than letting one model decide everything.
There is also a passive feedback loop that quietly contaminates fairness. Users downgraded to the small model are more likely to churn, and their interactions disappear from future logs. If only the interactions of users who stayed are used to train the next router, the system sees only samples where “the user remained after downgrade,” and thus incorrectly concludes that the downgrade was successful. The router learns to downgrade more and more aggressively, while evidence of churned users never appears in the training set. Breaking this loop requires proactively preserving before-and-after churn comparison information, treating “user leaving” itself as an observable negative feedback rather than as missing data.
Therefore, the input to fairness evaluation is language, domain, user group, and risk slices; the output is four sets of metrics plus confidence intervals for each slice. Average pass and slice-level pass are two different things: the former can hide the fact that long-tail users continuously receive the weak model, while the latter determines whether this router can go live.
8Router rollout also requires exploration, monitoring, and rollbackOperations
A router is not a one-time deliverable component. The boundaries it learns rest on a set of specific conditions: routing labels are essentially relative differences between candidate models—which model is stronger on which requests—rather than a static description of a model's absolute capability. Therefore, when the underlying model changes version, the prompt changes, the price changes, retrieval quality fluctuates, or traffic composition drifts, the original optimal boundary immediately becomes invalid. Thresholds that performed well at launch may, after conditions change, simultaneously exhibit increased misrouting and cost waste, without any error signal.
The remedy is to make governance a continuous loop. The first layer is logging: each request leaves a routing score, selection reason, candidate model versions, escalation rate, per-slice quality, cost, and cascade latency. Without these logs, no drift can be located. The second layer is the version change process: new models or new routers first run offline in dual-run mode, compared in parallel with the current solution on the same samples; then go live in shadow mode, only recording decisions without affecting real traffic; finally canary rollout. Skipping any step means the impact of the change can only be revealed by a production incident.
The third layer is active exploration. Selection bias says that downgraded samples have no strong-model counterfactual; the governance loop must create this counterfactual: for a small number of requests that should have been downgraded, randomly run the strong model instead or send them for human audit to estimate the misrouting rate. Exploration has cost and risk, so it must be limited to requests that are safe to run in dual-run mode; irreversible tool calls can only be executed through one approved path, and exploration traffic must not trigger them.
The fourth layer is monitoring. Continuously observe the score distribution, calibration curve, and escalation benefit by time and slice: an overall shift in a slice's score distribution indicates the traffic composition has changed; calibration curve deviation from the diagonal indicates the score semantics are failing; a decline in escalation benefit indicates the gap between the strong and small models is narrowing or widening. Quality labels are often delayed; before real labels are in place, we can first look at the missing-evidence rate, user correction rate, and human override rate as proxy metrics—when these three signals rise, it usually means the router is making errors.
The fifth layer is rollback. When unauthorized action errors or the misrouting rate exceed budget, immediately switch to conservative rules: escalate all traffic to the strong model or switch to human, or simply disable the relevant actions. The order here matters: first stop the bleeding with conservative mode, then recollect dual-run data and recalibrate boundaries. Merely shifting thresholds to "suppress" misrouting numbers without recollecting data uses new numbers to cover up the root cause; drift still exists, and thresholds will continue to fail.
Putting it together, the inputs to online governance are model version, price, traffic, routing score, selection reason, and delayed quality feedback; the outputs are a set of actions: exploration audit, recalibration, canary, or conservative rollback. Random dual-run lets the system continuously see counterfactuals of downgraded samples; drift signals trigger data re-collection rather than threshold fine-tuning; when misrouting exceeds budget, first disable dangerous actions or escalate all. Router quality is not the state at the moment of launch, but whether this loop continues to operate.
10Connecting the causal chainSynthesis
Arrange the constraints from the previous steps in causal order. From routing as “a problem” to “a verifiable practice” there are only six steps, and each step provides the premise for the next. If any step is skipped, the later steps lose their foundation.
Step one: establish a per-sample quality and cost baseline for candidate models. Routing only allocates among existing candidates; if all candidates fail on a slice, the router can at best select a different failure. Therefore, before writing any routing logic, record each candidate model’s performance and cost per sample, and confirm that each slice has at least one adequate candidate. If this step fails, all subsequent optimization allocates on the wrong capability map.
Step two: define high-risk mandatory rules. For medical, financial decisions, permission-write operations, and explicit compliance keywords, the cost on these slices does not allow a probabilistic router to make mistakes; use deterministic rules to force routing to a strong model or human. Hard rules must take effect before any probability score; they are the system's safety baseline, not soft suggestions that can be overridden by scores.
Step three: train and calibrate the “gain of the strong model relative to the small model” score. The training objective is not absolute difficulty but the expected benefit of upgrading: upgrading is valuable only for samples where the strong model is right and the small model is wrong; samples where both are right or both are wrong produce no benefit. Labels come from an independent scorer and humans; counterfactuals missing for downgraded samples are filled by random double runs or offline double runs; on an independent set, calibrate the score by slice as “probability that upgrade can fix the error,” and reject uncalibrated self-reported confidence.
Step four: choose the threshold under constraints that satisfy slice gates. Treat the threshold as a control point moving on the misrouting, quality, and cost curves: a threshold that is too high saves money but causes difficult requests to be misrouted; a threshold that is too low fixes misrouting but pushes up cost and upgrade latency. The feasible region is bounded by the global quality gate, the misrouting gate, and independent upper limits for all key slices; choose the threshold at the point in the feasible region with the lowest cost.
Step five: launch in gray release and randomly audit downgraded samples. Validate routing decisions on small traffic, and for a small number of downgraded requests randomly run the strong model or human audit, so the system continuously sees whether it misroutes. Exploration traffic covers only requests that are safe to run twice; irreversible actions are allowed to execute only through one approved path.
Step six: after version or distribution changes, recalibrate or roll back. Routing labels are relative differences among candidates; any change in model version, prompt, price, retrieval quality, or traffic will shift the optimal boundary. A drift signal triggers collecting double-run data again and recalibrating, not fine-tuning thresholds to hide the root cause; when misrouting exceeds budget, first switch to conservative mode—all requests to strong model, transfer to human, or disable actions—then recover calmly.
The causal direction of this chain is: the baseline determines whether candidates can be allocated; hard rules determine which slices do not participate in probabilistic allocation; the gain score determines whether the basis for allocation is trustworthy; slice gates determine whether the threshold is feasible; random audit determines whether operations can see errors; recalibration and rollback determine whether the system still holds after changes. The output of each step is the input to the next; if any link in the chain breaks, the remaining links merely repeat the same error at a higher level.
- FrugalGPT: Model cascades and budget optimization.
- RouteLLM: Preference-data-driven strong-weak model routing.
- Learning to Route LLMs with Confidence Tokens: Routing based on confidence signals.
- Selective Classification for Deep Neural Networks: Risk-coverage and selective decision-making.