Streaming Output: Turning One Generation into a Cancelable, Resumable Event Protocol

The user sees the first character at 400 ms, yet the server may still have computed for a full 8 seconds—these two facts hold at the same time, and accepting them is the starting point for understanding streaming output. The input received by the streaming protocol is the sequence being generated token by token, together with the lifecycle state of the entire request; the output is incremental events advancing over time, eventually converging into a complete result or a termination signal. What it does is hand over what has already been generated to the client as early as possible, not reduce the total amount of computation.

In non-streaming mode, the client receives a single response after the model has completely generated the entire answer; before that, nothing appears on the screen. Streaming mode changes this shape of waiting: as soon as the next token or text increment is available, it is sent out immediately, so the user can begin reading earlier, can decide to cancel midway, and can see the progress of tool calls. But the model must still generate all tokens for the complete answer, not one less. In addition, each incremental transmission is a network event, and the front end must parse and render each increment—these are additional overheads introduced by streaming. Therefore, streaming can shorten the time from when the request is sent to when the user obtains the first visible result, but it may also make the time from request to complete result longer—the more events there are, the more fixed costs accumulate along each path.

This means that the time spent at different stages must be measured separately, rather than replaced with a blanket conclusion of "it got faster". Time to First Token (TTFT) measures from when the request is sent to when the first visible token arrives; token interval TPOT measures the latency between adjacent tokens, reflecting whether increments are uniform or blocked upstream; end-to-end time (E2E) measures from the start of the request to the end of the entire response, gauging total workload and all overhead; cancellation effective time measures how long it actually takes for a user's stop request to take effect. These four quantities can change in different directions at the same time: streaming usually causes TTFT to drop significantly, but if each token individually triggers a complete event pipeline, E2E may be higher than non-streaming; if an answer is destined to fail midway, the user may well have already seen the erroneous first half.

Promoting streaming only with "fast first token" will obscure these costs. The three boundaries under which it holds should always remain in the judgment: visible is not the same as complete—the first 40 tokens appearing on the screen does not mean the 400th token has already been generated; a valid prefix does not mean the final result is valid—the first half of a sentence being grammatically correct does not mean the entire answer passed validation; a client disconnect does not mean the upstream computation has stopped—after the browser closes the page, the server may still be finishing computing the remaining tokens. Streaming changes the time distribution of the result reaching the user, not the model workload required to generate that result.

A network chunk cannot be directly treated as a character or a model token, because the entire streaming pipeline has at least four mutually independent "boundaries", each determined by a different stage. The input of incremental decoding is model tokens, UTF-8 byte chunks, and protocol frames; the output is complete characters and typed events. From token to character, from character to byte, from byte to network fragment, from fragment to protocol event, each step may split one unit into two halves, or may combine multiple units into one unit.

Start from the source. The model produces tokens, and boundaries are determined by the tokenizer: a token may be only half a word, or may be a structural fragment such as "comma plus space", and may even contain only part of a multibyte character. The server decodes tokens into UTF-8 bytes, thus entering a second boundary system: one Chinese character occupies 3 bytes in UTF-8, and these bytes may be split into two different network chunks; conversely, one byte chunk usually holds many characters. Next, the runtime and network stack split the byte stream into TCP chunks according to buffering policies. This splitting only considers transport convenience and has no correspondence with characters, tokens, or business meaning—a TCP chunk represents only "this transmission fragment" and must never be treated as the boundary of a JSON record or business commit. Finally, the application protocol layer assembles the byte stream into typed frames or events, with boundaries determined by the application's own schema; and JSON escape sequences themselves may span two increments, for example the four bytes of an escaped newline "\n" may arrive in two parts.

Therefore, when parsing a streaming response, the client must maintain state at each layer: use an incremental UTF-8 decoder to combine continuously arriving byte chunks into complete characters, retaining across chunks any multibyte sequence that has not yet been assembled; parse structure according to protocol events instead of TCP chunks, since any network fragment may fall exactly in the middle of a JSON string. Putting the units of each layer together, the boundary relationships and whether they can serve as commit boundaries are as follows:

The causal chain in this table is one-way: the tokenizer's segmentation cannot constrain how the network fragments, and network fragmentation cannot in turn determine token boundaries; only at the outermost layer, after receiving the done event indicating completion and completing validation, does the accumulated content upgrade to a complete result that can be submitted to the business layer.

When the connection drops midway, the client has only a pile of already-received bytes and must be able to answer one question: did this stream end normally, was it cancelled by the user, or did it error? A protocol that only sends strings cannot answer this question, because it represents three completely different terminal states as the same phenomenon—"no more data". What an event protocol solves is exactly this: let the stream itself carry lifecycle information.

Such a protocol takes as input the request state and four kinds of increments: text, tool calls, usage, and errors; its output is an event stream with a request id, an increasing sequence, a protocol version, and terminal-state markers. A typical set of events includes: start marks the beginning of a stream, text_delta carries a text increment, tool_delta and tool_complete describe the progress and completion of a tool call, usage reports token usage, error indicates an error, cancelled indicates the user actively cancelled, and done indicates the generation part completed normally. Each stream is identified by a request id, ordered by a strictly increasing sequence, and carries a protocol version number; an optional event id lets the client deduplicate after reconnection.

The client processes in order and maintains its own state machine: it accepts only an increasing sequence, discards duplicate events, and marks the whole stream as completed only after receiving an explicit done and passing final validation. EOF has no special status here—it only means the connection closed. The reason for the connection closing could be network jitter, proxy timeout, service crash, or user cancellation; none of these can be inferred from the fact that "the connection dropped" alone. Therefore, automatically treating the already-received prefix as the complete answer is misreading a transport-layer phenomenon as an application-layer conclusion. EOF is not done; this is a hard distinction.

There are several transport options for carrying this protocol, each with trade-offs. SSE is simple to implement, has good browser support, and can automatically reconnect after disconnection, but it is a one-way channel, so the client cannot send feedback such as cancellation on the same connection; WebSocket is bidirectional, but brings extra complexity of connection management, heartbeats, and state machines; HTTP chunked transfer is the most universal and can pass through most proxies, but chunks themselves are only byte boundaries, so you need to define your own frame format. Which one to choose depends on the degree of interaction required by the application, and does not change the basic requirements at the protocol layer: no matter which transport it runs over, it must define clear frame boundaries, guarantee ordering, agree on idempotent semantics, and provide a decidable terminal state.

Use a 40-token response to string the preceding concepts into a computable timeline. Given conditions are: the request waits 120ms in the queue, prefilling the input prompt takes 280ms, and each token after the first token decodes in 50ms on average. The case inputs are these four quantities — queue time, prefill time, total number of output tokens, and TPOT — and the outputs are three user-perceivable quantities: TTFT, total completion time, and the compute already wasted at cancellation.

First calculate when the user sees the first character. Only after 120 ms of queueing plus 280 ms of prefilling does decoding of the first token start; the time to first token is visible at around 120 + 280 = 400 ms. This is the experience of streaming rewriting: a single wait of 2.4 seconds is replaced by starting to read at 0.4 seconds, but the full computation has not disappeared. In symbols, time to first token TTFT ≈ Q + P + D, where Q is queue time, P is prefill time for processing the input prefix, and D is the time of the first decoding step itself.

Next look at when it completes. Total time E ≈ TTFT + (N − 1) × TPOT + X, where N is the total number of output tokens, TPOT is the average decoding interval per token after the first token, and X is the time for network transfer and frontend rendering. Substituting into this example, N = 40, TPOT = 50 ms, the 39 tokens after the first token take 39 × 50 = 1950 ms, adding 400 ms gives 2350 ms, plus transfer and rendering overhead, the total time is close to 2.4 seconds. This formula is an approximation for capacity planning: batching, event jitter, and load fluctuations in real services will cause the duration of each step to deviate from these single values.

The cancellation scenario reveals another relationship between streaming and computation. If the user presses cancel at 1 second, the number of tokens already decoded is about (1000 − 400) / 50 = 12 tokens. The user has roughly seen the beginning of a dozen tokens; whether the remaining 28 tokens continue to be computed depends on whether the cancellation signal can reach the model service: if cancellation only stops client rendering and the signal does not propagate upstream, the model will still generate all 28 tokens, and cost continues to be incurred as usual. Therefore the effective cancellation time is not the time the user clicks, but the time the signal passes through proxies, gateways, request queues, and finally stops the model from decoding; only if this propagation path is completed will the "already wasted" portion stop within 28 tokens.

The server sends 100 text deltas per second, while the browser renders frames according to the display refresh rate—the most common refresh rate is 60Hz, and screens at 75/120/144Hz are also common—what happens? If every increment triggers a DOM update, 100 updates per second must squeeze into about 60 frames, so on average each frame must process 100/60≈1.7 deltas; the rate at which updates arrive continuously exceeds the rate at which frames can be drawn, and as a result events pile up in every buffer along the client, gateway, and server: memory keeps growing, queuing at each level increases latency, and eventually the proxy may disconnect the connection. Backpressure control is a feedback mechanism designed for this situation: it takes production rate, consumption rate, and buffer watermarks at each level as inputs, and outputs four types of decisions—batch rendering, pausing, coalescing, or cancellation.

The existence of slow consumers is a reality of the whole chain; it cannot be avoided by assuming consumers will always keep up with production speed. The correct approach is to propagate pressure back layer by layer: if frontend DOM updates are slow, make client reads slower; if client reads are slow, the gateway's send buffer starts to backlog, so the gateway pauses reading from the application; the application then pauses reading from upstream. This propagation chain is DOM slow → client reads slow → gateway sending blocked → application pauses reading from upstream; each layer constrains its upstream by the actual speed of its downstream.

If some upstream cannot be paused—for example, the model API itself does not support mid-stream suspension—then the next best option must be used: set bounded queues at the allowed levels, and when the queue watermark exceeds the threshold, cancel the entire stream rather than letting buffers grow without limit. Text increments can be coalesced, gathering 100 small deltas into a phrase or a frame before rendering; tool events, however, cannot be merged arbitrarily, because they carry order and semantics, and merging could change the parameter boundaries of a tool call. Tool events must preserve event semantics.

Looking at symptoms, causes, and corresponding controls side by side:

This table corresponds to the same causal structure: symptoms are visible results of mismatch between consumption and production rates, causes are lack of rate limiting or unbounded accumulation at some level, and control measures feed the rate difference information back to the production side, or when feedback is impossible, use bounded discarding as a fallback. The sign that backpressure is effective is not any single switch, but that starting from the slowest level, each level only requests data from upstream at the speed it can sustain.

After the user clicks the stop button, why might an already-triggered tool call still complete—for example, a refund is still deducted? Because "cancellation" in a streaming chain is a signal that needs to propagate layer by layer, not an instantaneous global fact. The inputs received by the cancellation and reconnection mechanism are request id, sequence, cancellation signal, idempotency key, and action log; the output is the determined state of each request: cancelled, completed, unknown, or replayable.

The most common way a cancellation signal fails is by stopping only at the outermost layer: the frontend stops displaying, but the AbortSignal does not continue propagating to the model service and already-issued tool requests; the model is still generating remaining tokens, and the tool process is still executing. For cancellation to truly take effect, every layer must receive and propagate the cancellation signal—HTTP request abort, upstream connection close, and the model loop checks the cancellation token before each decoding step—and the tool layer, before initiating an irreversible action, must also check whether the current request is still valid. Even so, cancellation still only applies to things that have not yet happened: side effects that have already been submitted will not be automatically rolled back just because the frontend stops displaying; they require compensating actions or manual intervention. Therefore, "display cancellation" and "transaction rollback" are two different things, and the protocol must truthfully report which state is real.

Timeouts bring another type of problem from the same origin: after a timeout, the result state is unknown—the request may not have reached the server at all, or it may have already completed but the response was lost. Retrying directly at this point may execute an irreversible action repeatedly. The correct order is to first query the action status, and after confirmation decide whether to replay or abandon; for the irreversible action itself, carry an idempotency key and a commit log from the start, so that repeated submissions and repeated completions can be recognized and collapsed into the same execution on the server. The protocol promises only four states for each stream: cancelled, completed, unknown, or replayable, and the client must handle them according to this promise rather than guessing on its own.

Reconnection turns an unknown state into a recoverable state. When SSE automatically reconnects, it can carry Last-Event-ID, and the server uses this to replay events from the short-term event log that occurred during the disconnection; after the client receives them, it deduplicates using request id plus sequence and discards duplicate events that arrive. The key constraint is: reconnection cannot execute tool actions again—tool_complete cannot be executed again because of replay. If the server has no event log and cannot replay, it must explicitly start a new request, and absolutely must not concatenate the prefix of the new request with the suffix of the old request to form an answer that looks continuous. Two different streams each have their own request id; the concatenated text does not belong to any single complete generation in business terms.

The tool argument JSON has already streamed out {"amount":12; why can it still not be executed? Because every prefix in streaming is still changing: after 12, a 0 may follow, turning the amount into 120; the string may not be closed yet; more fields may continue to be appended to the object. At this moment, what looks complete on screen is only an unformed intermediate state. The structured buffer receives block-by-block tool_call_delta as input, and its output is parameters that can be handed to the execution layer only after complete assembly and passing schema, fact, and permission checks.

Therefore, the uses of structured deltas are explicitly divided into two kinds: preview and progress display are allowed, but early execution is not. The server can send dedicated tool_call_delta events for the client to assemble a real-time partial JSON for display, but the client is only responsible for concatenation, not interpretation—making any semantic inference on a half-formed state, such as thinking amount is 12, is guessing a value that might be overturned by the next chunk at any time. The precondition for actual execution is having a complete object and passing three layers of validation: schema validation confirms structural legality, fact validation confirms that the parameters make business sense, and permission validation confirms that the current request has the authority to initiate this action.

The same boundary applies to the streaming body itself. Early citations or assertions may be negated by later text—the first half says "the conclusion is A", and the second half may continue with "but this only holds under condition X" or even directly overturn A. Therefore, when persisting the final answer, you should save the original event sequence and completion state, not a rendering snapshot at some moment; an answer interrupted halfway must be marked as interrupted and must not be mixed into the training set or evaluation set for complete answers, otherwise training and evaluation will treat an intermediate state never endorsed by the model as its final output.

Security review on this chain also has its own time boundary. Checking only a single chunk will miss attack content assembled across chunks—dangerous words split into two deltas, each harmless on its own; waiting until the full text is generated and then reviewing loses the real-time advantage gained from streaming. What works is incremental buffered review: rolling checks on the buffered content already received, while delaying display of high-risk category content until enough context confirms it is safe. The granularity of review is chosen between these two extremes, not by giving up one side.

A single delta looks harmless, but when concatenated it can become a script, an auto-redirect link, or a piece of sensitive information—the risk of incremental rendering is that danger never falls inside any single chunk, but in the seams between chunks. The brackets of Markdown links, the angle brackets of HTML tags, code fences, and sensitive information patterns can all be split in half and arrive sequentially, and each half scans as legitimate on its own. Incremental safe rendering takes as input cross-chunk text and a content risk policy, and outputs an intermediate state accumulated as plain text, Markdown or HTML generated by a safe renderer, and display units that have passed review.

There is only one correct rendering path: first accumulate as plain text, then generate HTML with a safe Markdown renderer. The frontend must never call innerHTML directly on each increment—that is equivalent to injecting unfinished HTML fragments into the DOM as instructions, and attack content will be executed before concatenation is complete. The safe scanner must maintain a cross-chunk sliding window so that the end of the previous event and the start of the next event are inspected in the same window; tool-returned content and external document content are always marked as untrusted sources. This path fixes the order of "accumulate first, interpret later, render later", and any shortcut that treats half markup as code violates it.

There is also a layer of review granularity in display timing. The prefix a user sees will influence their behavior, even if the system eventually deletes or corrects that sentence—misleading guidance that has already been read does not lose its effect because of a later apology. Therefore, medical, financial, and high-risk guidance scenarios can buffer content until sentence, paragraph, or even complete review is finished before display, rather than unconditionally releasing token by token. Streaming is not better by default in all scenarios; its immediacy must be exchanged for delay in domains where the cost of errors is high.

The correspondence of these risks can be aligned into a table:

The last row brings accessible screen readers into the same structure: if a screen reader triggers reading on every token, what is heard is a series of broken sounds rather than sentences; updates should be done by semantic chunks and throttled. All four rows share the same principle—interpretation and display are based on completion, and any handling that treats intermediate states as final states, whether aimed at the DOM, scanners, review, or screen readers, turns the incremental nature of streaming into a vulnerability.

Average TTFT is attractive, but it alone cannot prove streaming is reliable—because most of the evidence for reliability hides in the completion path, cancellation path, and failure path, and these paths are invisible in the average. The input to streaming evaluation should be end-to-end event logs, concurrent scenarios partitioned by network and device, and systematic fault injection; the output should be a full set of metrics: TTFT, TPOT, completion time, cancellation propagation, wasted tokens, recovery capability, buffer high-water mark, frontend frame rate, and final answer quality—not a single number.

First, consider what the metrics need to cover. Latency metrics are recorded by quantiles: p50 describes the typical experience, p95 and p99 expose tail latency, because the average can be made to look good by a few fast requests while what users feel is precisely the slow part. Also measure TTFT, TPOT, full completion time, the gap between adjacent events, and the interruption rate. On the cancellation path, measure two quantities: cancellation propagation delay—the time from the user click to the model actually stopping—and the number of tokens still wasted after cancellation; the latter directly measures whether cancellation truly saved subsequent computation. On the recovery path, measure the reconnection recovery rate and duplicate event count; on the backpressure path, measure peak buffer usage and frontend frame rate. Finally, above all paths, measure final answer quality: a stream that displays smoothly but is wrong is not a good stream. All these metrics should be sliced by output length, network conditions, proxy presence or absence, device type, and concurrency, because each is an independent source of stress.

Testing only the normal path is not enough; you must also actively inject faults. The injection checklist covers the boundaries discussed in each preceding section: a UTF-8 character split across two byte chunks, an event truncated in transit, proxy buffering introducing long pauses, out-of-order events, duplicate events, a model error midway, a tool call timeout, client network disconnect, and server restart. After each fault is injected, what is accepted is not "the text looks approximately fully displayed" but the final state and side effects: what terminal state the request ends in, whether there was any unwanted tool execution, and whether a failure was disguised as completion.

Bring these acceptance conditions together, and a single streaming evaluation must simultaneously prove four things: the user saw safe increments in a timely manner; cancellation actually saved subsequent work; the complete result is committed only after done and verification passes; and failures are never silently disguised as completion. If any of the four fails, no matter how good the latency numbers look, this streaming implementation is unreliable.

From the initial problem all the way to verifiable practice, the entire causal chain of response streaming is made up of six links that interlock in sequence.

First link: assign a request ID and an increasing sequence number to each stream. This is the foundation for all later behavior: without a request ID, after reconnecting you cannot determine whether two byte chunks belong to the same stream; without a sequence number, you cannot detect out-of-order delivery or duplicates. Second link: decode tokens and byte increments into typed events. The tokens produced by the model and the byte chunks cut by the network are not units that an application can directly consume; an incremental UTF-8 decoder and protocol frame parsing turn them into semantic events such as text_delta, tool_delta, and usage. From then on, clients work by events rather than by transport fragments. Third link: the client renders safely under backpressure constraints. When consumption rate is lower than production rate, render in batches by frame instead of reordering token by token; interpretation and display should be based on completeness, and plain text should be accumulated before safe rendering, to prevent scripts or links assembled across chunks from being executed as code.

Fourth link: propagate the cancellation signal along the chain while protecting side effects. The frontend stopping the display is only the starting point; cancellation must pass through the gateway to reach the model and tools, so that subsequent tokens stop being generated. Already-submitted irreversible actions are handled through idempotency keys, status queries, and compensation, rather than pretending they never happened. Fifth link: submit the final result only after done and after validation passes. EOF does not equal completion; a visible prefix does not equal a valid answer; incomplete content must remain in the interrupted state. Any business submission presupposes an explicit terminal state plus schema, fact, and permission checks. Sixth link: verify the previous five links with fault injection and end-to-end metric regression. Inject UTF-8 cross-chunk splits, event truncation, out-of-order delivery, duplication, tool timeout, and network disconnection restart; measure p50/p95/p99 latency, cancellation propagation delay, wasted tokens, recovery rate, peak buffering, and final quality; confirm that safe increments arrive in time, cancellation truly saves work, and failures never masquerade as completion.

Among the six links, the output of each link is the input to the next: without identity and sequence numbers, events have no ownership; without typed events, rendering and backpressure have no actionable granularity; without safe rendering and cancellation propagation, there are no side-effect semantics that can be promised; without the submission gate of done plus validation, evaluation has no acceptable final state. Once the chain breaks at any link, all links after the break continue running on a wrong basis—and this is exactly why the entire chain must be measured end to end.