Assistant Response Prefilling: Letting the Model Continue from a Given Answer Prefix

Response prefilling solves the following problem: making the model begin with a structure or content we have predetermined, instead of letting the model freely choose the starting point of its answer. Its inputs include the user and system context, plus an assistant prefix that has already been confirmed to appear in the answer; the output is the token sequence the model continues to generate after that prefix.

Mechanistically, prefilling does not make the model "re-answer" all over again. When we put {"action":" a fragment like this into the assistant-side history messages, the model sees this text already present in its context. During autoregressive generation, the only thing the model has to do is predict the next token after this prefix. In other words, the prefix is placed in the conditioning context; the model does not select it again, but continues writing conditioned on it.

This mechanism has two direct consequences. First, content that is incompatible with continuation coherence will have its probability suppressed. Opening pleasantries, Markdown code fences, or other structures that are discontinuous with the given prefix will all have their probability lowered because they cannot connect naturally to the prefix. Therefore, a brief prefill prefix is often more directly effective than long instructions such as "don't greet, just give the conclusion"—instructions only describe expectations, whereas the prefix directly turns the expectation into a reality condition the model must face. Second, the prefix also locks in assumptions. If the prefill is "refund approved, because", the model's task changes from "determine whether a refund should be issued" to "find reasons for the already announced conclusion"; even if the order does not actually meet refund conditions, the model tends to follow the prefix and fabricate reasons. Prefilling is therefore not a neutral formatting trick; it has real binding force on the direction of the conclusion.

Interpretation of the results must also be limited to this mechanism: the generated result only proves that the continuation is coherent with the prefix, not that the facts in the prefix themselves are correct. The model accepting the prefix does not mean that it has verified the prefix; the model writing out reasons along the prefix does not mean that the reasons hold.

Therefore the usage boundary is very clear: only prefill the minimal text that you can independently guarantee is correct and that you want to fix in place, such as a definite JSON starting fragment or a verified factual opening; leave the rest to subsequent constraints and business validation to decide. The strength of prefilling should match the correctness guarantee you can bear.

The mechanism by which prefilling changes the generated result can be explained using conditional probability. Let x be the user and system context and p the already supplied response prefix; the probability of the model generating the full continuation is written P(continuation | x, p). The model weights do not change at all during this process—prefilling involves no fine-tuning and modifies no parameters. The change occurs on the conditioning side: at each step of autoregressive generation, the attention mechanism reads p, so the space of possible tokens and the probability distribution at each step are reshaped.

From this we can derive the causal chain of prefix strength. The more specific the prefix, the fewer allowed paths remain. Prefilling only a "{" commits merely to a JSON-style opening; prefilling the complete {"action":"refund" has already made the business decision for the model—the refund action is hard-coded, and the model can only continue writing under that already-established conclusion. The risks of the two prefixes are completely different: the former only constrains the format, while the latter locks in the content.

If p conflicts with the model's own template habits or with the task itself, the continuation is not rejected; instead it is pushed into a low-probability region and proceeds only with difficulty. The model is forced to search for a coherent but low-probability path along the track set by the prefix, and this "strained" progression is precisely evidence that a conflict exists.

It should also be noted that probability bias is not a hard guarantee. Even if the opening is fixed, the model can still append an explanation after closing the object, omit fields, or generate factual errors. What the prefix changes is the tendency of the path, not the correctness of the result.

The complete picture of conditional generation is therefore as follows: the inputs are the context x, the prefix p, and the continuation generated so far, and the outputs are the probability distribution over the next token and the final continuation text. Understand the prefix as a pruning of the event space—more specific prefixes reduce the available paths, while conflicting or incorrect prefixes anchor the continuation in the wrong region.

The name "prefill" refers to several completely different things in inference services, and mixing them up directly leads to implementation errors. Four concepts need to be distinguished: Assistant Response Prefilling, the input prefill phase, Prompt Caching, and, as a reference, Constrained Decoding.

The differences between the four concepts can be seen from the "Target" column. Assistant Response Prefilling handles the text prefix on the assistant side, directly shaping the response history that the model sees, and therefore changes output conditions. The input prefill phase processes all input tokens, with the goal of encoding the context into a KV representation; it is an essential internal part of inference computation and is not a product-level control. Prompt Caching saves the KV results of the common input prefix, allowing multiple requests to reuse the same computation; under a correct implementation, the output conditions for a cache hit and recomputation should be identical, so it only reduces computation and does not change the content. Constrained Decoding provides a valid token set at each generation step and uses hard masking to ensure the output conforms to a formal language, such as forcing valid JSON; it also changes output conditions, but in a different way from Assistant Response Prefilling—it excludes invalid paths rather than giving a continuation starting point.

There is another layer of common confusion: "prefill acceleration" usually refers to computational optimization of the input prefill phase, while "Assistant Response Prefilling" is handing the beginning of the answer to the model to continue writing. The two share the word prefill, but the concept and performance goals are completely different.

Therefore, concept disambiguation can follow a fixed procedure: the input is a feature named prefill or cache and its target, and the output is one of the categories "response prefix, input computation, cache reuse, or hard constraint". First see whether it handles assistant text, input tokens, KV results, or a valid token set, and then judge whether it changes content conditions or merely reduces computation. Similarity of names does not imply semantic or interface compatibility; final behavior should be based on service documentation and contract tests.

A concrete worked example shows how "distributional mismatch" occurs. Without a prefix, the model's first candidate when answering the task is often a pleasantry such as "of course"; after prefilling a "{", the candidate set faced by the model is entirely different.

Figure 1 compares the probability distribution of the first continuation token under two conditions: no response prefix and prefilled left curly brace. The two distributions have different event spaces—the prefilled "{" itself is no longer in the candidate set; it has become part of the conditioning context. Therefore, the 0.25 for "{" in the no-prefix distribution and the 0.60 for "action" in the prefilled distribution cannot be described as "the probability of the same token rising from 0.25 to 0.60": the former predicts the first token of the answer, while the latter predicts the token immediately after the curly brace; the two are not the same event at all. The correct evaluation approach is to compare the full output's structural pass rate, content quality, and failure types.

*Depends on support scope.

The three-row comparison shows the layer at which improvement occurs: from prompt-only to prefilled "{", the first-attempt JSON syntax pass rate rises from 94% to 98%, and the Schema compliance rate rises from 86% to 89%; constrained decoding can push these two to 100% and 99%, but the latter depends on the service's supported range for the required grammar. The rightmost column, factual correctness, remains 80% in every row, and no row improves it. The point of this example is that improvements in the first two layers (syntax and structure) do not automatically raise factual correctness.

Returning to the mechanism level, the inputs of this example are the two conditions of an empty prefix or "{" together with their respective first-step candidates, and the outputs are probability distributions at two different positions and complete structural metrics. When interpreting results, one must acknowledge that 0.25 and 0.60 are not the same event and cannot be directly subtracted as an improvement; what needs to be explained is the structure, facts, and failure types of the complete output, not the numerical change of a single token.

The risk of prefilling depends on whether the prefix falls under "format" or "content". More suitable for prefilling are low-risk format skeletons: JSON opening braces, fixed headings, already verified code preliminaries, language or tone openings, and common prefixes for enumerated labels. These texts only specify the form of expression and do not inject any judgment that has not yet been established. High-risk prefixes are the opposite: success/failure conclusions, user identity, amounts, citations, tool-executed status, and any facts that have not been externally verified. They hard-code content that should be determined by evidence or an authoritative system into the condition.

Look at the causal logic of these four types of prefixes line by line. "{" only commits to a JSON-style opening; the risk is low, but it cannot guarantee that the output is complete and valid JSON, so structural validation must still be left to schema constraints. "According to policy evidence:" assumes that policy evidence exists; if the evidence is actually missing, the model will follow this opening and fabricate citation sources, disguising misleading content as well-documented. The alternative is to use a conditional template that explicitly outputs a refusal when evidence is insufficient. "Refund approved" directly anchors an unauthorized conclusion in the context; the alternative is to have the server fill in this sentence after tool confirmation. Pasting existing code files as preliminary context is equally dangerous: old code may contain vulnerabilities or injection flaws, and the alternative is to verify the version and tests before deciding whether to use it as context.

Therefore, prefix selection can be described as a decision function: input the text to be fixed, its evidence status, and the task risk, and output a neutral skeleton or a decision of "do not use prefilling". Only content such as JSON opening braces, headings, or verified code is suitable for fixing; conclusions, identities, amounts, citations, and tool states must be confirmed by an authoritative system. The longer the prefix, the more semantics it covers, and the higher the probability of anchoring unknown facts.

The API integration layer is often where prefilling fails. Why do some services not allow the last assistant message to be non-empty? Because chat templates and security policies are implemented by the service provider: some APIs natively support an assistant prefix, some reject this kind of request, automatically close the message, or treat the prefix as a historical response. The semantics of the same concept differ across services. Therefore, before integrating, you must consult the API documentation and test three questions with a real model: whether the returned delta includes the prefix itself, whether billing counts the prefix, and where the first event of the streaming response begins. These three answers determine how the client should assemble the result.

Client-side concatenation rules also have hidden pitfalls. The prefix can only be concatenated once, and then each streaming chunk is incrementally decoded as UTF-8; stop strings may span token or event boundaries, so you cannot simply trim the string. If the prefix ends with half an escape sequence or an incomplete Unicode character, the model is forced to start from a position that cannot legally continue, and the resulting path may not be recoverable. Logs should record version information about the prefix, but avoid writing sensitive source text.

Template duplication is a typical symptom of misunderstanding the contract. If the server echoes the prefix in the output, and the client manually prepends it again, the final text will contain double opening braces such as "{{" or duplicated headings. This kind of issue must be covered in contract tests, not discovered only when malformed output appears in production.

Abstract the entire integration flow: the input is service capabilities, the assistant prefix, streaming events, and stop rules, and the output is one complete result that is correctly concatenated. The server decides whether the prefix is treated as a continuation condition, a historical response, or an invalid message; the client completes concatenation, decoding, and stopping according to the contract. As soon as you see an echo or a repeated beginning, it means the contract is misunderstood. Half an escape, incomplete Unicode, and untested interfaces are not suitable for direct entry into production.

Why does forcing the answer to start with "Yes" make the model more confidently fabricate reasons? The way controlled generation systems (LMQL, Guidance) work illustrates this: the system places the given partial response into the context as if it had already been generated, and the model's only task is to condition on it and continue completing it. Even if internal evidence leans toward "No", an already supplied "Yes, because" makes contradiction linguistically abrupt—denying it means directly conflicting with the prefix, while continuing along the prefix only requires finding supporting reasons. So the model is more likely to search for material that argues for "Yes". This is a natural result of conditional generation, not the model actively lying: the model is only maintaining coherence, and coherence itself has been hijacked by the wrong prefix.

You can expose this anchoring with a positive/negative control test. For the same question, prefill three versions: "Yes", "No", and an empty prefix, and observe how the conclusion changes with the prefix. If the factual judgment flips with the prefix, it indicates that the system is replacing evidence with format control—the output looks confident and well-supported, but it is actually just an echo of the prefix. High-risk tasks therefore can only prefill a neutral skeleton, and no word that carries a conclusion direction should appear in the prefix.

There is also a layer of security bypass risk. Prefilling may put the model in a state that is uncommon in safety templates: conventional safety training usually assumes that the beginning of the answer is generated by the model itself, while an externally supplied beginning may bypass some protective paths. The range of prefixes a provider allows and the safety behavior must be separately subjected to AI red teaming; you cannot assume safety just because the interface supports it.

The complete form of the anchoring test is: input three versions of the same question with an empty prefix, "Yes", and "No", and output the respective changes in conclusions and evidence for each. The model continues the existing opening according to coherence, so when the rationale flips with the prefix, it indicates that format is replacing evidence, rather than the answer becoming more reliable. High-risk tasks can only use a neutral skeleton, and they also require separate AI red team validation.

Output already starting with "{" does not mean it has passed any substantive checks. Prefilling only shapes the form; what truly determines safety is a series of independent business validation gates. The system's checks must proceed in order: wait for complete output and an explicit end marker—a truncated stream is not completion, and half JSON must not enter subsequent processes; parse the JSON and perform schema validation to confirm that the structure and field types are legal; validate amounts, dates, citations, and cross-field invariants to prevent fields that are individually legal but contradictory when combined; query authoritative databases for the entities involved, and unknown entities should ask the user rather than guess; tool execution must re-authenticate with the current subject and submit idempotently, avoiding side effects from repeated execution.

When structural validation fails, allow a limited number of retries; if the same type of failure keeps recurring, it means the path of prefilling plus prompts has reached its limit, and you should switch to constrained decoding or go straight to a form. This points to a more general conclusion: if hard structure is a core requirement, directly using supported constrained decoding is more reliable than constantly lengthening prefixes and repeatedly patching prompts. Prefilling is suitable as a compatibility or experience optimization and should not be packaged as any form of guarantee.

The complete picture of business validation can be described as follows: the input is the complete model output, schema, authoritative data, and current permissions; the output is one of three verdicts: "acceptable, reject, or re-inquire". The system checks completeness, structure, field facts, entity permissions, and idempotent submission in order. Passing JSON parsing only shows formal correctness, not that facts are established or actions are authorized—formal correctness, content correctness, and action legality are three mutually independent issues.

After the structural pass rate improves, a harder question must be answered: whether factuality, refusal behavior, and safety are being quietly harmed by the prefix. The answer is paired evaluation—the same batch of inputs must be run with both the empty-prefix and candidate-prefix versions, comparing item by item. Metrics to measure include first-try syntax/schema pass, field accuracy, task success, factual correctness, refusal, overreach, output length, time to first token (TTFT), and token cost; and they must be reported separately for four slices—missing information, high risk, language, and long input—to avoid the masking effect of averages.

Paired evaluation includes a dedicated metric: prefix reversal rate. Add a prefix with the opposite conclusion to the same question, and observe whether the answer flips along without evidence. A prefix with a high reversal rate indicates that it is making decisions in place of evidence, rather than only guiding format. Additionally, fault injection tests must be run: stream interruption, server-side echo, repeated concatenation, Unicode boundaries, stop strings, and API upgrades, to confirm that the prefix does not produce unrecoverable errors on exceptional paths.

A candidate prefix can be adopted only if two conditions are met simultaneously: format metrics improve, and content and risk metrics do not degrade. The success criterion can be stated in one sentence—the prefix should reduce low-value openings and format deviations, but it must not decide unknown facts for the model, and it must not cause downstream systems to relax validation because "the beginning looks right".

The input to paired evaluation is the results of the same samples under the empty prefix and candidate prefix, and the output is the differences across six dimensions: format, factuality, refusal, safety, latency, and cost. After any change to the model, API, or template version, the full paired evaluation must be run again, because the effectiveness and risks of the prefix are tied to the specific implementation version.

Link the preceding steps in causal order: the complete response prefill chain starts from choosing the prefix and ends with monitoring anchoring.

First, choose the smallest prefix that can be independently guaranteed correct. This choice determines the risk ceiling of the entire chain: the prefix carries only the part you can endorse with evidence, and the rest is left blank. Second, send the assistant prefix according to the real API contract—the service may support, reject, or echo it, so the integration method must be verified through actual testing. Third, the model continues writing conditioned on the prefix: the weights do not change, but at each step it reads the prefix, the event space is pruned, and path probabilities are reshaped. This step is the core of the mechanism and the source of all anchoring effects. Fourth, concatenate and stop correctly and wait for the complete result; stream interruption, duplicate prefixes, half an escape sequence, and incomplete Unicode will turn previous work into unrecoverable errors at this step. Fifth, run schema, factual, and permission validation on the complete output; formal correctness does not constitute any guarantee, and facts and action authorization must be confirmed by an independent system. Sixth, pair with an empty prefix for evaluation and continuously monitor anchoring—only when formatting improves and content and risk do not degrade is the prefix worth keeping.

Each link in this chain takes the previous link's output as its input; relaxing any link passes risk downstream: the more specific the prefix, the stronger the anchoring; the vaguer the contract, the easier it is for concatenation to go wrong; the weaker the validation, the easier it is for formal correctness to be mistaken for content correctness. Conversely, the chain's end also constrains its beginning—only content that you can independently guarantee to be correct belongs in the prefix of the first step.