AI Deployment: Turning Offline Capability into a Capacity-Measurable, Canary-Releasable, and Rollbackable Service

Where the same model capability runs is not purely a technical question; it is a question of “who is responsible for it.” The choice of deployment location determines the form of the model service, the benefits the team gains, the costs it bears, and the business stage it suits. Running locations are usually divided into four forms: Cloud API, self-hosting (cloud or on-premises), edge/on-device, and hybrid deployment.

Cloud API is borrowing the inference service that the provider has already built. The team gains elastic scaling, fast model updates, and freedom from GPU operations burden; in exchange, data must leave its own domain, the team must accept the provider’s quotas and unit price constraints, and it becomes dependent on the provider. This form suits products in rapid iteration stages with clearly fluctuating traffic—the team can invest all limited engineering effort into the business rather than into machines.

Self-hosting (your own cloud account or on-premises) trades for control over weights, data, capacity, and versions. Weight files, inference code, and version release cadence are all decided by you, and data does not leave the boundary you have drawn; the cost is that you must bear all operational responsibilities such as resource scheduling, patches, security hardening, and on-call duty. This form is meaningful only when business scale tends to be stable and compliance control requirements are strong, because it requires the team to truly have long-term operational capability.

Edge/on-device deployment places inference on user devices or close to devices. What is gained is offline availability, very low network round-trips, and partial privacy with data not leaving the device; the costs are hard constraints on memory and power consumption, adaptation costs from hardware fragmentation, and difficulty in model updates. It suits scenarios with weak network environments, extremely high real-time requirements, or where data is not allowed to leave the device.

Hybrid deployment is a combination of on-device fast paths and cloud escalation: simple requests are answered locally, while complex requests or new capabilities are routed back to the cloud. This form can layer by risk and capability, letting the device side handle low-risk, high-frequency tasks and the cloud provide high-quality fallback; but it introduces complexity in synchronization, routing, and consistency, requiring the team to be able to clearly design “what goes where.”

Two intuitions often go wrong here. On-device does not automatically mean low latency: if device compute is insufficient, decoding itself will be slow, and the saved network time will be eaten up by computation time. Self-hosting is not automatically cheaper: low utilization, engineering labor, and redundant construction will swallow the apparent unit price advantage. Therefore, the inputs for choosing a running location are actual traffic characteristics, data residency requirements, privacy constraints, latency budget, availability targets, hardware conditions, and team operational capability; the output is one of Cloud API, self-hosting, on-device, or hybrid solutions; each trades off control, elasticity, cost, and responsibility. If the team lacks the ability to bear patches, on-call duty, and disaster recovery, it cannot choose self-hosting merely because of a preference for data control.

Model inference took only 600 ms, but the user waited 2 s—this gap reminds us that treating model weights as the only object of deployment is an illusion. A real request, from being sent to returning, sequentially goes through authentication, rate limiting, queueing, context assembly, retrieval, prefill, token-by-token decoding, tool calls, post-processing, and streaming. Model computation is only one segment of this path, and any segment on the path can become the dominant contributor to tail latency.

The causal chain of this path is: the request first passes through gateway authentication and rate limiting, which determine whether it is allowed to enter; then it waits in the queue for compute resources; after entering the service, the user context and retrieved material need to be assembled into a complete input; prefill processes the entire input at once, while decoding produces tokens one by one; if the model decides to call a tool, it must also wait for the response of the external system; finally, post-processing and streaming send the result back to the user. End-to-end latency is the sum of the time consumed by all these stages, not the model time itself. The 600ms model inference time, plus queue wait and tool round-trips, produces the 2-second experience. When diagnosing tail latency, looking only at model time is meaningless—queue or tool calls on the critical path can completely dominate these 2 seconds.

Tool calls also introduce another layer of responsibility. The model does not directly perform external operations such as transfers and refunds; instead, it does so through tools, and once these operations happen they leave real-world side effects. Therefore tool calls must include permission control, idempotency design, and transaction compensation: repeated calls must not deduct funds repeatedly, and failed operations must be reversible. Model output can be regenerated at any time, but external side effects cannot be undone by regeneration.

Figure 1 shows the deployment path of the refund assistant: the request enters from the gateway, passes through queue and retrieval, reaches the model, and the model then triggers the tool; the entire path is under the coverage of canary rollback. The real conclusion this figure illustrates is: the rollback object is never a single weight, but a version composed of the model, prompt, index, tool schema, policy, and service configuration. Changing any link individually may break the entire chain, so the release unit must cover all dependencies.

Therefore the input of the service path is a request and the authentication, queueing, retrieval, prefill, decoding, tool, and transmission stages it goes through, and the output is end-to-end latency, system state, and external side effects. The 600ms model time is only one segment; queueing or tool calls on the critical path can dominate the 2-second response time. The release unit must cover all dependencies on the path, and external side effects require permissions, idempotency, and compensation; the model alone cannot be rolled back.

If 8 requests arrive per second and each is served in an average of 0.1 s, is a single instance enough? Intuitively, 8 × 0.1 = 0.8, leaving 20% idle, which seems more than sufficient. Queueing theory gives a much more pessimistic answer, because the “average” hides the randomness of arrivals.

The two basic quantities in capacity estimation are utilization and the number of requests in the system. Let the arrival rate be λ, the average service time S, and the number of parallel slots c; then utilization ρ = λ×S/c. Substituting the numbers: c=1, λ=8/s, S=0.1s gives ρ=0.8. That is, the instance is busy 80% of the time. The problem is that requests do not arrive at uniform intervals—random arrival means requests bunch up, so even with an average service time of only 100 ms, requests still queue. As ρ approaches 1, queueing time degrades nonlinearly and tail latency rises sharply; at ρ=0.8 queueing has already occurred in practice, not only when the system reaches full load.

Little’s law L=λ×W provides another way of looking at it: the average number of requests in the system L equals the arrival rate λ multiplied by the end-to-end average residence time W. If W=0.5s on average end-to-end and λ=8/s, then L=8×0.5=4—on average 4 requests are in flight in the system at the same time. This relationship ties latency, throughput, and concurrency together: if users stay in the system longer, the system must be able to hold more in-flight requests; to reduce residence time, either reduce queueing or increase service speed.

LLM services also work in tokens, which adds two more constraints to this ledger. Input length determines the time spent in the prefill phase, and output length determines how long a request occupies compute slots; a long output holds slots for a long time, delaying requests behind it. Continuous batching packs token computation for multiple requests together and improves throughput, but at the cost of more complex scheduling—the scheduler must decide in real time who is served first and how many tokens each gets. Therefore, capacity models cannot be divided only by “requests per second”; they should be sliced by input/output length, model, priority, and tool path, because different slices have completely different service time distributions.

Another fact to watch out for is that averages do not guarantee SLOs. A service with an average latency of 500 ms can have a p99=8s tail, because a small number of users blocked by long requests contribute very large quantiles. Capacity review must look at the tail, and also at two things not covered by formulas: residual computation still running after timeouts—requests abandoned by clients do not automatically release server resources—and burst traffic—bursts raise the instantaneous λ far above the average level, and queues can build up within minutes.

Taken together, the inputs to capacity estimation are arrival rate λ, average service time S, parallel slots c, and end-to-end average time W; the outputs are utilization ρ and average number of requests in the system L. The formulas provide an initial capacity intuition: at ρ=0.8 queueing already occurs, and as it approaches 1, tail latency degrades nonlinearly. But the formulas only cover steady-state average behavior; LLM length differences, batch scheduling, tool round-trips, and burst traffic all need to be validated through stress testing, not just by applying queueing formulas.

The new model has 3 percentage points higher accuracy, halved throughput, and doubled p95; how do we decide whether it can go live? A refund assistant's release process can fully illustrate this decision. The candidate version's performance on five dimensions—task pass rate, unauthorized rate, p95, throughput, and cost per successful task—is as follows:

The release gate is: task pass rate ≥ 87%, unauthorized rate ≤ 0.5%, p95 ≤ 1.8s, peak capacity ≥ 16 req/s. Checking item by item reveals layers of issues. v42 offline evaluation had task pass rate 89% and unauthorized rate 0.3%, indeed better in quality than v41's 86% and 0.4%, but it only proved the model's ability on offline samples, without testing queuing, caching, real input length, and tool timeouts—these are exactly the differences between the serving environment and the offline environment. Shadow testing filled this gap: after placing the same model into the real traffic path, the task pass rate dropped to 88.5% (still meeting the threshold), but p95 became 2.6s and throughput only 11 req/s, both falling below the threshold, and the cost per successful task also rose from ¥0.10 to ¥0.17. Therefore, v42's original serving form could not pass the gate. After optimization, p95 dropped to 1.5s, throughput returned to 17 req/s, task pass rate 88.3% still ≥ 87%, cost ¥0.13, and only then did it have the qualification to enter canary. Offline quality can be used to screen candidates, but cannot replace serving gate—release decisions must be based on actual measurements on the serving path.

Passing the gate is only the first step; next, we need to calculate how many instances are needed. Divide the peak arrival rate λpeak by the per-instance rated throughput qinstance, then round up: peak 24 req/s divided by optimized 17 req/s, 24/17 ≈ 1.41, rounding up gives at least 2 instances. This number is the bare capacity calculated under ideal conditions. After leaving margin for single-instance failure and burst traffic, 3 instances may actually be needed—once 1 of the 2 instances goes down, the remaining instance has only 17 req/s capacity, unable to withstand the 24 req/s peak. Therefore, cost accounting must be based on the actual utilization after redundancy, not on bare provisioning based on average utilization.

The inputs of this release case are v41's and v42's task pass rates, unauthorized rates, p95, throughput, and cost per successful task; the outputs are whether all thresholds are met and the required number of instances. The decision order is to first check the four gates of task ≥ 87%, unauthorized ≤ 0.5%, p95 ≤ 1.8s, capacity ≥ 16 req/s, then divide peak arrival rate by per-instance rated throughput and round up to get the minimum number of instances; redundancy and single-instance failure will raise the actual requirement from 2 to 3, and cost accounting must reflect this redundancy.

Batching, caching, and quantization can all make a service “faster,” but they change different parts of the inference system, and each introduces completely different risks. Treating optimization techniques as the same is the most common attribution error in capacity tuning.

Continuous batching packages token computation for multiple requests into the same hardware execution, improving hardware utilization and overall throughput. The cost lies in scheduling: a request must wait for other requests to form a batch, increasing waiting time; long-output requests occupy slots longer, and short requests may be squeezed out for a long time. Therefore, after launching continuous batching, you must regression-test queueing behavior, fairness between long and short requests, and p99.

Prefix/KV caching reuses already computed attention state: when multiple requests share the same system prompt or document prefix, caching lets subsequent requests skip repeated prefill computation, directly reducing first-token latency (TTFT) and computation cost. Its risk lies in key design: the cache hit key must include model version, prompt version, and permission context at the same time; otherwise, when a different set of prompts is used, it may still hit the old cache, or even worse—users with different permissions may read the same cached content, causing cross-user leakage. Caching also requires a clear invalidation mechanism; once the model or prompt is updated, the old cache must be invalidated.

Quantization changes the representation of the values themselves: compressing weights and activations from high precision to low bit width reduces GPU memory usage and memory bandwidth pressure, thereby speeding things up. But low bit width implies approximation, which may degrade the quality of a few slices—especially inputs that rely on precise numerical patterns. Therefore quantization must be regression-tested on critical slices and long-context scenarios, and the compressed performance must be validated with a calibration set.

Besides these three, speculative decoding, context pruning, and small-model routing are also common techniques, each with its own accuracy boundary. Context pruning reduces the text sent to the model to lower prefill cost, but the pruned content may be exactly the evidence needed to answer the question, requiring regression on evidence recall and the “lost in the middle” phenomenon—the model's ability to use information in the middle of long contexts is already weak, and pruning amplifies this problem. Small-model routing assigns simple requests to a cheap small model to lower average cost, but when classification is wrong, complex requests are delegated to the small model and quality suffers; meanwhile, the latency distribution of routed requests superimposes the tail latencies of the two models, forming a cascaded tail.

The benefits of these techniques and the items that must be regression-tested can be compared as follows:

The deep structure of this table is: each optimization sacrifices some property that was originally guaranteed by default. Batching changes queueing, caching reuses computation, quantization changes numerical representation, pruning changes the evidence visible to the model, and routing changes which model serves the request—the risks are different, so each must be regression-tested separately. The inputs to optimization are the current bottleneck, batching parameters, cache keys, quantization bit width, context, and routing strategy; the outputs are differences in throughput, first-token latency, GPU memory, quality, and isolation. During implementation, change only a small number of variables at a time and keep per-request version records; if all five optimizations are launched at once, after the service becomes faster no one can say which one produced the benefit, and if a problem occurs there is no way to attribute it.

Shadow, canary, and blue-green all convey “don’t go full traffic yet,” but they observe different risks and have different impacts on real users. The choice of release strategy depends on what you most want to verify at this moment and what you most fear breaking.

The first gate of release is the offline gate: exclude known quality, safety, and contract regressions on a frozen evaluation set. Its role is to block obviously unqualified candidates at low cost, but it can only verify the candidate’s performance on offline samples and cannot answer “what will happen when put into real traffic.”

Shadow deployment replicates real traffic to the candidate version; the candidate processes exactly the same inputs as production but its results are not returned to users. What it verifies is the candidate’s capacity and latency performance under the real input distribution, as well as compatibility with read operations; because users cannot see shadow results, observation does not affect the production experience. The boundary of shadow is: real irreversible side effects must be forbidden—tool calls such as refunds and transfers triggered by the candidate are not allowed to actually execute; at the same time, users cannot see the candidate results, so you cannot measure users’ subsequent behavior toward the candidate output, for example whether users will accept a recommendation or continue asking follow-ups.

Canary goes a step further: give a small portion of real traffic to the candidate, and these users see and accept the real candidate results. Its core is staged rollout and observation: traffic expands step by step through 1%→5%→25%→100%, staying at each level for a sufficiently long observation window, waiting for complete labels to come back—for example, the success rate of refund tasks can be confirmed only after the task has truly completed. Canary can measure the blind spot of shadow (users’ real reactions), at the cost of causing real harm itself: the small portion of users in the rollout bear all the candidate’s flaws. Therefore each step of canary must confirm no metric degradation within the observation window before continuing.

Blue-green deployment maintains two complete environments: the blue environment runs the old version and the green environment runs the new version. Switching only relies on changing where traffic points, and rollback is simply pointing traffic back. It solves the risk that “rollback must be fast and certain,” at the cost of double capacity and state synchronization between the two environments—database, cache, and tool states must be aligned; otherwise the moment you switch over, inconsistency arises.

At the intersection of the three strategies’ boundaries there is another common blind spot: no matter how fast blue-green switches back to the blue environment, it cannot automatically roll back refunds already executed in the green environment—side effects executed by external tools are outside the control range of environment switching. Similarly, shadow forbids side effects and canary truly produces side effects, both requiring release strategies to be designed together with side-effect isolation, idempotency, and compensation mechanisms, rather than being a problem that release tooling alone can solve.

Traffic splitting itself has nuances. Do not split randomly per individual request; otherwise multiple requests from the same session will be scattered across different versions, users will alternately face old and new models during a conversation, and behavioral data loses meaning. Use stable hashing by user or session so that the same user unit always lands on the same version, and exclude employee accounts, bots, and traffic with imbalanced regional proportions to avoid contaminating the observation window.

Taken together, the inputs to a release strategy are the candidate version, real traffic, side-effect risk, observation window, and rollback capability; the outputs are shadow, canary, or blue-green plus staged traffic actions. Shadow results do not affect users and prohibit irreversible actions; canary produces real results but is controlled in stages; blue-green keeps two complete environments in exchange for fast switching. Split traffic stably by user or session, and every side-effect path must be equipped with idempotency and compensation.

If you roll the model back to v41, why might the system still answer incorrectly? Because the model is only one component of the release unit: the prompt has already been changed, the index has been rebuilt according to v42 behavior, the tool schema has been migrated, the cache still contains v42 prefixes, and the database has already written records that cannot be undone. Rolling back only the weights file leaves all these components in their new state, so the service naturally continues to answer incorrectly.

The correct way to roll back is to make the release unit atomic. An atomic release includes the model, prompt, decoding parameters, index, tool schema, policy, code, and infrastructure configuration together in a composite version, assigns a releaseId to this composite, and records the dependency compatibility matrix among the components—which model version goes with which prompt set, which index version, and which tool schema constitutes a known working whole. When rolling back, restore all rollbackable components synchronously according to the releaseId, while explicitly listing which states cannot be reversed. The value of the release checklist is precisely this: it forces the team to write down before release, "When rolling back v42, also roll back prompt version A, index snapshot B, cache invalidation scope C," rather than recalling only after an incident.

Irreversible parts need to be handled in advance by design. Database and tool actions use idempotency keys: the same business operation carries the same key, so retries do not duplicate refunds. Schema changes use an expand/contract strategy: first expand, allowing both old and new code to read and write the new structure; then migrate data; finally contract, removing old fields. This way every intermediate state is dual-version compatible, and switching back to old code at any moment will not cause read/write failures. Indexes are built in a blue-green way: after the new index is fully built in the background, atomically switch the alias to point to it; if the switch fails, traffic still points to the old index at that instant. Cache keys must include the releaseId so that when the version switches, the old prefix cache naturally expires and will not be mistakenly hit by the new version.

There is one more layer: drills. Rollback duration must be actually verified through incident drills—really execute a rollback in some drill, time it, and check consistency, rather than trusting the "estimated 2 minutes" written in documents. Numbers in documents are meaningful only if they have been fulfilled in drills.

Therefore, the inputs to an atomic release are the model, prompt, decoding, index, tool schema, policy, code, and infrastructure configuration, and the outputs are a releaseId and a compatibility matrix. Rollback restores reversible components synchronously according to the releaseId; database tools rely on idempotency keys and expand/contract schema evolution; indexes use blue-green alias switching. Rolling back only the model leaves new prompts, new caches, or new states that continue to cause errors. For irreversible side effects that have already occurred, the only option is compensation—refunds can be returned, but it must be explicitly acknowledged as a compensation action, not pretending the system has been rolled back.

When peak load exceeds capacity, optional degradation actions include queueing, rejection, shortening output, switching to a smaller model, delayed processing, or even stopping service, but "which one to choose" cannot be decided by technical convenience; it must be tiered by risk level and task value. Low-risk summarization tasks can be switched to a smaller model or processed with delay; high-privilege refund tasks must not skip the verification step because of congestion—traffic pressure is not a reason to lower the safety threshold. The basis for tiering is the cost of a task going wrong, not the request's type label.

Degradation requires a set of explicit basic mechanisms: set upper limits on input and output lengths to prevent a single long request from consuming all resources; use priority queues to ensure critical requests are served first; use backpressure to make upstream aware of downstream pressure and slow down sending; set timeouts to prevent requests from indefinitely occupying slots; limit the number of retries and introduce circuit breaking so that failed dependencies are temporarily isolated rather than repeatedly hit. The most critical one is reserving capacity for the critical path—the reserved resources may seem idle in normal times, but at peak load they ensure that high-value tasks still have a path.

Degradation itself is a product behavior, not a temporary technical patch, so it must enter the evaluation set. Switching to a smaller model will cause requests that should have been directly processed by the large model to be escalated to humans or a larger model, forming a "small first, large later" secondary traffic that actually worsens the peak; stale cache will return old policy content to users; truncated output may drop safety warnings. Degradation paths also produce errors, so they also require quality regression. All degradation responses should be explicitly marked—users or downstream systems must be able to know "this is a degraded answer"—and observable, so that operations can review the scope of degradation after the fact. During recovery, prevent instantaneous replay: if requests backlogged during degradation are all released at once, it will cause a second avalanche; they should be released in gradients.

Retry storms are the most insidious amplifier in degradation scenarios. A timeout does not mean the upstream stopped computing, nor that the tool did not execute: the moment the client gives up on the request, server-side inference may still be continuing, and a refund may already have occurred. Therefore, only clearly recoverable errors are allowed to be retried—for example, transient failures at the network layer; for side-effect operations where it cannot be determined whether they have already been executed, retrying is equivalent to executing again. Retries must use jittered backoff to avoid synchronized storms, set a total budget to limit the total number of retries, and always carry the same idempotency key so that upstream can identify and merge duplicate requests.

In summary, the inputs to safe degradation are current capacity, risk level, task value, operation reversibility, and dependency status; the outputs are actions such as queueing, rejection, delay, smaller model, truncation, or stopping service. Low-risk tasks can be delayed or switched to another model; high-privilege refunds must not bypass verification due to congestion; every degradation result must be explicitly marked and monitored. A timeout does not prove that upstream stopped or that a tool did not execute; retries must satisfy three constraints—recoverability condition, total budget, and the same idempotency key—so that a congestion incident is not amplified into a retry storm.

The principles of deployment can be reviewed as a continuous causal chain: the output of each step is the input to the next. If any step is skipped, subsequent steps rest on unverified assumptions.

The first step is to define constraints: runtime location, data residency, privacy boundaries, latency budget, and capacity targets. These constraints determine the feasible space for all subsequent decisions—hosted or self-hosted, whether to have an edge fast path, and how strict the SLO is set all establish the baseline here. The second step is to freeze all dependencies into a release unit: model, prompts, indexes, tool schema, policies, code, and infrastructure configuration are packaged into a combined version with a release id, and the compatibility matrix is recorded. Without freezing dependencies, any later traffic ramp-up or rollback cannot find "one complete thing" to operate on. The third step is offline evaluation and stress testing: on the frozen set, rule out known quality, safety, and contract regressions, and use stress testing to validate the capacity model rather than just applying queue formulas. This offline step has the lowest cost and can block clearly unqualified candidates.

The fourth step enters the real world: shadow deployment replicates real traffic to verify the candidate's capacity and read behavior under the real distribution, while prohibiting irreversible side effects. Shadow fills in the queuing, caching, and real input lengths that offline evaluation cannot see. The fifth step is canary: ramp up traffic stepwise by thresholds, 1%→5%→25%→100%, observe the complete label window at each level, and confirm that task pass rate, unauthorized rate, latency, and capacity have not crossed the gates before expanding traffic. The sixth step is the runtime closed loop: when an alert triggers, perform degradation or rollback—degradation protects high-value paths by risk tiering, rollback restores atomically by release id—and feed failure cases back as part of the next round's evaluation set and gates. In this way, an incident is not just extinguished; it makes the next iteration of the entire causal chain stricter.