Inference Optimization: First Locate Whether the Bottleneck Is Prefill, Decode, KV, or Queuing

The first step in optimizing an inference service is not swapping compute kernels, but breaking the time from request entry to exit into segments, because different segments correspond to completely different bottlenecks. The same "slow" can happen in queueing, in input prefill, or in token-by-token generation; without breaking it down, you won't know where to change.

The path of a generative inference request is roughly: after arrival, the request first waits in queue, then the system performs prefill on all input tokens—completing forward computation for all inputs in one pass and establishing the KV (key-value cache). After that it enters the decode phase, where each step generates only one token until output ends; retrieval, tool calling, and network transfer may also be interspersed in between. These stages have different sensitivities to workload shape: long-input, short-output requests place pressure mainly on prefill and KV establishment, while long-output requests occupy decode slots for a long time and continuously read model weights and KV. Therefore, long-input short-answer and short-input long-answer require optimizations in different directions—the former emphasizes batch prefill and cache efficiency, while the latter emphasizes per-step decode latency and concurrency capacity.

To locate problems, you need to slice requests along four dimensions: input length, output length, concurrency, and cache hits. Without first doing this slicing, a single "average slow" metric cannot tell you whether to change the compute kernel, the scheduler, the caching strategy, or the model.

After slicing, you can compute two core metrics. Time to First Token (TTFT) measures the latency from when the request is sent to when the user sees the first token:

TTFT = Q + Ccontext + Tprefill + Tfirst

where Q is queueing wait time, Ccontext is context preparation time (for example, loading conversation history from external storage), Tprefill is the computation time for prefilling all inputs, and Tfirst is the time for the first decode step to produce the first token. Time per Output Token (TPOT) measures the average interval between each new token after the first token, determined mainly by the per-step forward computation in the decode phase and the time to read weights and KV. The end-to-end time for a complete request can be written as:

T_complete ≈ TTFT + (N − 1) × TPOT + Texternal

N is the total number of output tokens, and Texternal is the time taken by tool calling and transfer. The longer the output, the greater the weight of the (N − 1) term, and the more critical the efficiency of TPOT and the decode phase becomes.

These two metrics point to different bottlenecks and correspond to different experience goals. In interactive scenarios, the user stares at the screen waiting for the first token; TTFT and its jitter (high-percentile latency matters more than the average) determine perceived experience. Offline batch tasks, on the other hand, prioritize overall throughput and cost per unit of output. A single tokens/s figure cannot represent user experience—it mixes queueing, first token, and per-token generation into one number and is likely to mask real problems in interactive scenarios. First break down the end-to-end time as described above and record logs for each stage (how much queueing, input length, output length, tools, and transfer each account for), then output TTFT, TPOT, and end-to-end completion time. Subsequent optimization methods each act on only one of these items, and this decomposition is the basis for judging the value of each investment.

The same Transformer model faces completely different hardware constraints in the prefill and decode phases: one consumes compute, the other consumes bandwidth. Understanding this difference is the common prerequisite for judging the effectiveness of various optimization techniques.

In the prefill phase, all input tokens must be processed at once. Tokens at different positions can be computed in parallel during this phase, forming large matrix multiplications; matrix multiplications have high arithmetic intensity—each unit of data movement corresponds to a large number of floating-point operations—and therefore can more easily fully utilize the GPU's compute units. The decode phase is the opposite: each step generates only one new token for the current batch, both the batch dimension and token dimension are small, the computation per step is small, but each step must read all model weights and the ever-growing KV from memory once. The ratio of data movement to computation rises sharply, and at this point the bottleneck is often not compute but the bandwidth of HBM (high-bandwidth memory).

This trade-off can be quantified using the roofline model. For each phase, four inputs are counted separately: FLOPs (the total number of floating-point operations required by that phase), BytesIO (the total number of bytes that need to be moved from memory), ComputePeak (the hardware's peak compute), and Bandwidth (the hardware's memory bandwidth). From these, the arithmetic intensity is calculated:

AI = FLOPs ÷ BytesIO

That is, the average number of floating-point operations completed per byte of data moved. The performance upper bound for that phase given by roofline is:

Perfmax = min(ComputePeak, AI × Bandwidth)

The meaning is: when arithmetic intensity is high enough that AI × Bandwidth exceeds the compute peak, the upper bound is determined by compute, and the phase is compute-limited; when arithmetic intensity is low, the upper bound is determined by AI × Bandwidth, and the phase is bandwidth-limited. The large matrix multiplications of prefill usually fall on the compute-limited side, and the per-token small-step inference of decode usually falls on the bandwidth-limited side. It should be clear that roofline is an intuitive tool for locating bottlenecks, not a precise prediction of the latency of each specific kernel.

Following the same breakdown, the work, bottlenecks, and optimization techniques for each phase of an inference request can be mapped as follows:

This mapping explains why optimization techniques cannot be arbitrarily stacked. Increasing the batch improves reuse of weights in the decode phase—the same weights serve more requests, amortizing the read overhead per token—but a larger batch also leads to longer queueing times and greater KV usage; quantization compresses each value into fewer bytes, reducing the amount of data that needs to be moved, targeting the bandwidth bottleneck; FlashAttention reduces the memory round-trips of intermediate matrices in attention computation, targeting long-sequence I/O. The three act on different bottlenecks; applying a technique to the wrong stage provides almost no benefit to performance.

The inefficiency of static batching comes from an intuitive fact: the GPU computation for each batch cannot finish until every sequence in the batch has produced an end-of-sequence token. Even if a request generates only three tokens and finishes answering, the row of computation slots it occupies remains empty for the rest of the batch's lifetime, and the GPU still reserves that seat for it. The greater the difference in sequence lengths, the more empty slots there are and the more noticeable the throughput loss becomes. Continuous batching replaces the rhythm of "start the whole batch, end the whole batch": the scheduler checks the status of all current requests at finer granularity at each scheduling step, immediately removes finished sequences from computation, and selects new requests from the waiting queue to insert into the freed capacity. Each GPU iteration only faces the requests that are actually still alive, so utilization is no longer held back by the single longest request.

The costs are equally clear. Scheduling frequency changes from "once per batch" to "once per step", and the scheduler itself must be cheap enough; latency is no longer measured uniformly across the batch but fluctuates per request with its own length; when a batch has limited capacity, the scheduler must also make trade-offs between long and short requests, making fairness an objective that must be explicitly designed.

To use continuous batching safely, a set of supporting capacity constraints is required. The maximum number of batch tokens allowed per step provides a hard upper bound, preventing too many requests from being crammed in at once and causing per-step latency to spiral out of control; prefill is executed in chunks so that very long prompt inputs can be digested over multiple steps instead of one prefill monopolizing the GPU while all decode requests wait on the sidelines; priority and preemption mechanisms ensure that high-priority requests can be inserted and low-priority requests can be paused and later resumed. Under mixed load, interactive queues and offline queues are best scheduled in separate pools, with dedicated capacity reserved for critical tenants; otherwise a flood of offline tasks will crowd out interactive requests that need low time-to-first-token latency.

Batching wait time is a tunable knob with a strong directional effect: the longer the scheduler waits to form batches, the more likely it is to assemble a larger effective batch, usually increasing throughput; but the wait itself directly adds to each request's TTFT. Increasing this window improves throughput and worsens time-to-first-token latency; decreasing it does the opposite. There are no free parameters.

Under overload, continuous batching has another danger: if the waiting queue has no upper bound, the scheduler will instinctively fill the GPU, making throughput appear to reach its maximum, but queue length and queuing latency inflate together, and p99 tail latency becomes completely uncontrollable. At this point a throughput optimizer actually turns into a latency amplifier. The real protection comes from admission control, which decides at request entry time whether to accept or reject based on capacity; from backpressure, which sends the full-load signal upstream to cause callers to slow down; and from fairness boundaries between tenants to prevent a single tenant's traffic spike from consuming all shared capacity.

From an interface perspective, the inputs to a continuous batching scheduler are the set of waiting requests, the available token capacity per step, and the priority of each request; the output is the scheduling decision for that step: which sequences are removed, which new requests are inserted, which prefills are chunked, and which low-priority sequences are preempted. Finished sequences immediately yield their slots to new requests, so empty slots are dynamically filled. But it is important to remember what this optimization actually guarantees: GPU utilization and throughput. Higher throughput does not automatically bring improved tail latency; in overload, average latency and p99 can even move in opposite directions. Therefore continuous batching must be deployed together with admission control, backpressure, and tenant fairness boundaries to be a complete solution.

A common confusion is: the model weights clearly fit, so why does OOM occur as soon as concurrent requests arrive? The answer is that GPU memory has a second large expense—KV cache. During attention computation, each token at each layer produces its own K and V vectors for subsequent tokens to use when computing attention. If they are not cached, every newly generated token would require recomputing the entire history, which is prohibitively expensive; therefore these vectors are retained. KV cache memory usage grows approximately linearly with the number of concurrent requests and sequence length, while weights are fixed, so when concurrency increases, the first limit to be hit is often KV cache rather than weights.

Its capacity can be precisely estimated. Let Nlayer be the number of layers, NKVhead be the number of KV heads per layer, d be the head dimension, Ntoken be the number of tokens per request, B be the number of concurrent requests, and b be the number of bytes per element; then the KV cache memory MKV is approximately:

MKV = Nlayer × NKVhead × d × Ntoken × B × b × 2

The final multiplication by 2 is because K and V are each stored once. Taking a configuration with 32 layers, 8 KV heads, 128 dimensions, and FP16 (2 bytes per element) as an example, a single 8k-token request occupies approximately 32 × 8 × 128 × 8000 × 2 × 2 ≈ 1.05 GB; 16 concurrent requests would be about 16.8 GB, not yet counting model weights, intermediate activations, and workspace. This estimate directly answers where OOM comes from and also gives the order of magnitude of the concurrency limit.

PagedAttention is a common means of handling KV memory management. It places the KV cache into fixed-size blocks and allocates by block, reducing GPU memory fragmentation caused by repeated allocation and deallocation, and also supports sharing the same prefix across requests (such as a common system prompt) as well as more flexible eviction policies. But its boundary must be clear: PagedAttention improves the efficiency of memory management; the KV vectors themselves still exist, and it does not eliminate the KV data itself.

The means of truly compressing the KV data are another category. KV quantization replaces cache elements with low-precision representations, reducing capacity and GPU memory bandwidth, but it may hurt quality under long context and requires regression validation; sliding windows retain only the KV within the most recent window, truncating historical context; compressed cache summarizes long history into fewer representations. The latter two change the history actually visible to the model and are essentially algorithmic changes, so they cannot be treated as pure engineering optimizations and must be evaluated together with quality.

So the input for KV cache capacity estimation is the number of layers, number of KV heads, head dimension, number of tokens, number of concurrent requests, and bytes per element; the output is total cache memory. It is used to explain the source of OOM and the concurrency limit, rather than to give a final answer—PagedAttention only reduces fragmentation; quantization or sliding windows still require quality regression, and any means of compressing the KV data must first answer its impact on output quality.

A refund assistant service nearly doubled its throughput after enabling large-batch processing, yet interactive users started complaining. This seemingly contradictory phenomenon is worth breaking down: inference requests start from the queue, pass through prefill and decode stages before returning to the user; batching squeezes more requests into the same GPU iteration, increasing throughput, but individual requests' waiting and execution times can be stretched. Improvements in resource utilization do not automatically guarantee tail latency experience—scheduling parameters must be chosen based on SLO, not on peak throughput.

The comparison table below measures three batching strategies under the same workload:

The order in which you read this table matters. Start with throughput: static small batches get only 40 tok/s, showing low GPU utilization; aggressive continuous batching reaches 78 tok/s, nearly doubling, and this column alone is very tempting. Then look at the latency distribution: its TTFT p50 is 470ms, even better than the static small batch's 520ms—so the average user is indeed faster. But TTFT p99 worsens from 1.8s to 3.2s: a few requests are stuck behind batching waits and long-input prefill, and the first token is slow to appear; this is exactly the source of interactive users' complaints. A TPOT p95 of 58ms indicates that once generation starts, the token rhythm is not bad; the problem is concentrated in queueing and prefill. The approach using separate pools plus chunk prefill achieves 70 tok/s, slightly lower than the aggressive approach, but it brings TTFT p99 back down to 1.6s and p50 down to 430ms, keeping all interactive SLOs within constraints—a truly better choice.

The input to this case is three batching strategies with their throughput, TTFT, and TPOT metrics; the output is the decision of which one meets the interactive SLOs. The judgment method is not to look at averages: first compare throughput, then examine p50 and p99; aggressive continuous batching, although faster on average, fails p99 due to batching waits and long-input blocking. Also note the boundaries of the conclusion: these numbers apply only to the workload distribution at the time; once the request length composition, concurrency pattern, or tenant structure changes, the same parameters may produce completely different results, and the system must be re-tested after the workload changes.

Attention computation has a huge intermediate product: the N×N score matrix. Each of its rows first passes through softmax and then is multiplied with V, where N is the sequence length; for long sequences this matrix is much larger than the input itself. An ordinary implementation writes it completely into high-bandwidth memory HBM, writes it once, reads it once, and then writes the result; the IO round trips consume most of the time. FlashAttention's approach is to split Q, K, and V into small blocks, load only a small block into GPU on-chip SRAM each time, complete the computation needed for that block on-chip, and simultaneously maintain the softmax row maximum and normalization sum online—these two statistics are sufficient to make the block results reassemble into the correct value of the full softmax. The intermediate score matrix never needs to be written back to HBM in its entirety.

It is worth emphasizing that it does not perform approximation. Softmax can inherently be decomposed using the row maximum and normalization sum; FlashAttention only changes the order of computation so that numerical values are iteratively combined across blocks. Within numerical precision, it computes the same attention result as the standard implementation, rather than an approximate target like sparse attention. Therefore, the answer to the question "Can it still compute exactly without storing the full N×N matrix?" is: yes, because what is omitted is only the intermediate copy that travels to and from HBM, not the computation itself.

The magnitude of the benefit depends on the scenario. The longer the sequence and the higher the proportion of attention IO in total time, the more obvious the benefit; with short sequences or when the entire inference is dominated by other operators, the benefit of switching kernels may be negligible, because the bottleneck is not here at all. The benefit also varies with hardware, precision dtype, and implementation details. Upgrading the kernel is a low-level change; numerical, quality, and stability regression must be performed to confirm that the output matches the old implementation and that no edge-case bugs have been introduced.

Its boundaries should also be drawn clearly: FlashAttention reduces the IO of attention computation; it does not change the long-term capacity of the KV cache—what needs to be stored still needs to be stored; nor does it solve queue waiting and the latency of external tool calls—those times occur outside the GPU kernel. To decide whether to prioritize the upgrade, first check whether the attention IO of long prefill actually dominates; if p99 mainly comes from queuing, switching to a faster kernel only treats the symptom, and the metrics will not improve significantly.

In summary, FlashAttention's inputs are the same as ordinary attention—the same set of Q, K, and V; its output is the same attention result within numerical precision, but HBM round trips are greatly reduced. It changes the order of computation rather than the target of approximation. The benefit means attention IO is reduced, not that KV capacity, queue latency, or tool latency has been solved; under short sequences and non-attention bottlenecks there may be no visible benefit.

Changing FP16 weights to INT4 reduces the number of bytes per parameter to one quarter, simultaneously lowering memory usage and weight read bandwidth. This yields two kinds of benefits: the model can fit on smaller devices, decode-stage throughput constrained by memory bandwidth can improve, and the same memory can accommodate a larger batch. But quantization is not as simple as making numbers smaller—low-bit-width tensors usually need to be dequantized back to a higher precision before computation, and this step itself has kernel overhead; if the hardware lacks native support for that bit width, the benefits will be greatly reduced; under small batch sizes, weight bandwidth was not the bottleneck in the first place, so the saved bytes do not translate into speed. Therefore, 'faster after quantization' is not an inevitable conclusion: it may be faster or slower, depending on whether the dequantization kernel, hardware support, and bottleneck under small batch actually match.

You also need to distinguish the targets of quantization: weights, activations, and KV have different effects.

Weight quantization saves model resident memory and read bandwidth; the risk is that a few channels with abnormal value ranges introduce errors, as well as the extra overhead of the dequantization kernel. Activation quantization affects operator execution speed and distributed communication; the risk comes from the large dynamic range of activations and error concentration when calibration data is poorly chosen. KV quantization saves KV cache capacity and bandwidth in long-context scenarios; the risk is reduced retrieval accuracy for distant tokens and error accumulation over the sequence.

Average perplexity is only the coarsest reference line; service quality regressions must be examined separately by task type, language, long context, probability calibration, and tool parameter slices: a drop in a language slice, long-context retrieval failure, model output probabilities that are no longer calibrated (for example, confidence score distortion), or broken tool calling parameter format can all occur on a model whose overall average still looks acceptable. The quantization selection process takes as input the quantization object (weights, activations, or KV), target bit width, hardware kernels, and task slices, and outputs a low-precision model along with its differences in memory, speed, and quality. It is a quality-related change, not a pure engineering optimization—lower bit width reduces storage and bandwidth, while dequantization and calibration add overhead; 'faster' holds only when hardware support is good and bottlenecks match, and before adoption you must regress by language, long context, tool parameters, and high-risk tasks.

A fundamental constraint of autoregressive generation is that it is serial: each time a token is generated, the target model must finish one forward pass. Speculative decoding uses a smaller, faster draft model to break this serial dependency. The draft model first quickly guesses the next k tokens according to its own distribution; the target model takes these k candidate positions, evaluates the probabilities at all positions simultaneously with one parallel forward pass, and then corrects them one by one according to accept/reject rules—if a draft token at a position matches the target model's judgment, it is accepted; otherwise it is rejected and the target model samples its own token from that position onward, and all subsequent draft tokens are discarded. As long as the accept/reject rules are designed correctly, the final produced token sequence still follows the target model's original distribution in probability, so the acceleration does not change the model's behavior itself.

The per-round output can be estimated using the acceptance rate a (the per-token probability that a draft token is accepted by the target model). The first token is always produced, the second is produced with probability a, the third with probability a², and so on. The approximate output of one round is 1 + a + a² + … + a^k. For example, when a = 0.8 and k = 4, it is about 1 + 0.8 + 0.64 + 0.512 + 0.4096 = 3.36 tokens/round. But this is an idealized account: the draft model itself takes time, and the verification batch also has cost, so the actual speedup is always less than 3.36.

The acceptance rate a determines everything. The closer the draft model's distribution is to the target model's distribution, the higher a becomes. Once there is a domain mismatch—such as a specialized task the draft model has never seen—or a very high temperature setting makes sampling more random, or the capability gap between the two models is too large, a drops sharply. Each round then actually produces close to 1 token, while still incurring the double overhead of draft and verification, making it slower than not using speculative decoding.

We should also recognize what it optimizes: speculative decoding improves TPOT and overall throughput by targeting the serial steps in the decode phase; it cannot do anything about the computation of long-input prefill. When combined with streaming output, there is another engineering detail: the verification batch emits multiple tokens at once. If smoothing is not applied, users will see output appear suddenly in chunks, and this visible burst jitter should be avoided.

Let's go through the interface once more: the inputs to speculative decoding are the draft model, the target model, the draft length k, and the per-token acceptance rate a; the output is the token sequence after verification by the target model. The draft first proposes candidates in parallel, and the target model verifies them in batch and accepts or corrects them according to the rules. The approximate per-round output is 1 + a + … + a^k. Keeping the correct target distribution depends on using the correct correction rules, while low acceptance rates or high verification costs will make the overall process slower.

Can a 2x improvement in offline single-request tokens/s show that production has really gotten better? Not directly. The gap between production load and single-request load testing lies in the joint distribution of input/output lengths, concurrency levels, burstiness of request arrivals, KV cache hit rate, and priority composition—single-request load testing flattens all of them. A benchmark that can be transferred to production must replay these real characteristics and report a full set of metrics: p50/p95/p99 for TTFT, TPOT, and end-to-end latency, throughput, timeout rate, KV cache and GPU memory usage, power consumption, and cost per successful task. Reporting only an average tokens/s cannot support any conclusion.

Load scenarios must also be constructed deliberately. Beyond steady state, you also need to run burst traffic, overload, and single-node failure: steady state shows regular performance, bursts show queue absorption capacity, overload shows whether admission control and backpressure take effect, and node failure shows recovery behavior. Any change that can alter output, such as quantization, KV compression, context pruning, or approximate decoding, must run full application evaluation rather than model-level metrics; a claim of 'quality unchanged' must include task slices and confidence intervals, otherwise it is equivalent to no evidence. Kernel and scheduling changes can also go wrong—numerical errors, request starvation, tenant fairness violations—these will not show up in average latency.

Benchmark cheating tricks can be enumerated: fixing short outputs so results always look good; pre-warming caches to hide cold-start costs; ignoring queueing and testing only a single request; reporting only the best batch configuration; not counting failed requests. Any one of these will make benchmark results impossible to transfer to production. Conversely, a usable production benchmark has real length distributions, concurrency, bursts, cache hits, priorities, and fault injection as inputs, and latency percentiles, throughput, resource usage, failure rate, quality, and cost per successful task as outputs. The approach is to compare alternatives under a fixed load and cover three types of scenarios: steady state, overload, and node failure. An average tokens/s improvement alone cannot prove production is better; any algorithmic or low-precision change must pass a quality gate.

String together the previous techniques: inference optimization is a causal chain from problem to verifiable practice, where each step provides the basis for the next; skipping a step leads to choosing the wrong solution.

The first step is measurement, and you must slice by request phase and length. First split end-to-end time into prefill and decode, then stratify by input and output length: long inputs point to prefill, long outputs point to decode, and short requests point to queuing. Only the metric distribution after slicing will tell you where time is being lost.

The second step is to identify the primary cause. High latency may be due to insufficient compute (compute bound), insufficient GPU memory bandwidth (bandwidth bound), KV cache capacity being filled up and limiting concurrency, or queues growing without bound under overload. The four categories of primary causes require completely different solutions, and must be judged from the first step's sliced data rather than intuition.

The third step is to choose a solution, and the solution must match the primary cause. Only long prefill dominated by attention I/O is worth considering a kernel upgrade; only when there are many empty slots and low utilization should you adopt continuous batching; only when GPU memory is squeezed by KV cache should you touch cache management; only when bytes and bandwidth are both tight and you are willing to accept quality risk should you consider low precision. Conversely, if p99 comes from queuing but you go change the kernel, you are applying the solution in the wrong place.

The fourth step is to perform ablation under a controlled load. Fix the same load, change only one variable at a time, and record the net change in throughput and latency percentiles to confirm that the benefit truly comes from that change, not from environmental noise or compounding effects.

The fifth step is to validate under real concurrency. Replay the production length distribution, burst arrivals, and cache hit patterns, and stress-test tail latency and overload behavior; the earlier example has already shown that being faster on average and meeting p99 are two different things, and that performing well in steady state and not collapsing under overload are also two different things.

The final step is a joint quality and cost gate. Any technique that can change the output—quantization, KV compression, context truncation, approximate decoding—must pass quality regression; at the same time, use the cost per successful task to convert throughput, GPU memory, and power consumption into the same scale. Only when both meet the bar is the optimization complete. The output of each step is the input to the next; if any step is skipped, the conclusion loses its basis.