Sampling and Decoding Parameters: From Logits to Final Sequence

# Generation is an iterative closed loop, not a one-shot classification

The process by which large language models generate text is often mistaken for a one-time “read input, output answer.” In reality, the model does not write the entire response at once; it writes step by step. Each step adds only one new token, then appends that token after the existing text as the input for the next step. Therefore, generation is an iteratively executed closed loop: the input is the current prompt plus the prefix already generated, and the output is a new token, or a stop signal meaning “that’s all.”

Every step of this loop follows the same process. The model first outputs a set of logits over the entire vocabulary—that is, relative scores for each candidate token. The decoder then transforms and filters these scores (temperature, top-k, top-p, repetition penalty, and so on all belong to this step), chooses one token from them, and appends it to the end of the generated sequence. The appended sequence becomes the input for the next step, and the loop continues until a stop signal is selected or the length limit is reached.

The fact that every step is conditioned on the previous step has an important corollary: a small difference at one step changes the conditional distribution that all later steps face. A single random difference in the first token can send the entire subsequent answer in a completely different direction. The second step’s logits are computed under the condition that the first step chose word A; if the first step actually chose word B, then the score table facing the second step is already different. The difference propagates and amplifies step by step and is not automatically corrected in later steps.

Therefore, one call to the model produces only one of many possible sampling paths. Seeing a single output does not allow you to infer that other sampling paths will give similar content. A candidate ranked second in probability at a given step, although just slightly behind at that moment, may open a later branch completely different from the first-ranked one, ultimately leading to a very different paragraph.

The overall probability of a sequence can also be understood from this perspective. Suppose the model generates the sequence x₁, x₂, …, x_T word by word; then the joint probability of the entire sequence equals the product of the conditional probabilities along the path:

P(x₁, …, x_T) = P(x₁) × P(x₂ | x₁) × P(x₃ | x₁, x₂) × … × P(x_T | x₁, …, x_{T−1})

Each factor at every step is the probability the model assigns to that token under the current prefix. A small probability difference at any position enters this product directly, and all later factors depend on earlier choices, so “which step chose which token” determines the shape of the whole path. Understanding this also explains why sampling parameters must act individually on each step of the loop, rather than acting once on the final answer.

# Logits are only relative scores; softmax is what gives probabilities.

At each step, a language model outputs a set of logits over the entire vocabulary: each candidate token corresponds to a real-valued score. These scores have only relative meaning—a higher score indicates that the model relatively prefers that token, but they are neither probabilities nor constrained to lie between 0 and 1. Turning logits into a probability distribution that can be sampled from and interpreted is what softmax does.

Softmax has two steps. First, exponentiate each logit. The exponential function is monotonically increasing and grows faster and faster, so it amplifies score differences between candidates: a candidate with a slightly higher logit receives substantially greater weight after exponentiation. Second, add up all the exponentiated values as a normalization factor, and divide each candidate's exponentiated value by this sum to obtain a set of probabilities that sum to 1. For candidate i:

pᵢ = e^(zᵢ) / Σⱼ e^(zⱼ)

Here e is the base of the natural exponential function, zᵢ is the raw logit of candidate token i, pᵢ is the probability of drawing that candidate after softmax, and j ranges over all candidates in the vocabulary. This transformation ensures that every pᵢ is between 0 and 1 and that they sum to 1, so it can be used as the distribution for “which token to choose next.”

Softmax has an often-underestimated property: it depends only on the differences between logits, not on their absolute values. If you add the same constant c to all logits at once, the numerator and denominator are both multiplied by e^c, so the ratio is unchanged:

pᵢ = e^(zᵢ − c) / Σⱼ e^(zⱼ − c)

Therefore, adding 100 to all logits at once leaves each token's probability completely unchanged. Numerical implementations take advantage of this: they typically subtract the maximum of all logits before exponentiating, so the largest exponent is exactly 1, avoiding numerical overflow from exploding exponentials, and the result is exactly the same as direct computation.

After obtaining the probability distribution, the decoder still needs to decide how to use it. Greedy decoding directly selects the candidate with the largest logit, which is equivalent to choosing the token with the highest softmax probability. Random sampling draws according to the distribution: if the distribution is [0.7, 0.2, 0.1], then after many draws the frequency of each candidate will approach this proportion, but a single draw can absolutely select the candidate with probability only 0.1—low probability does not mean impossible.

Finally, it is important to clarify the boundary of what this probability means. pᵢ measures “which token the model relatively prefers given the current prompt and the already generated prefix”; it does not indicate that the token is factually correct, nor can it be taken directly as confidence in the entire answer. The denominator of the probability is normalized only over the vocabulary at the current step, so it cannot measure the quality of the entire generation path.

# Manual temperature calculation: how the same ranking becomes sharper or flatter

Temperature is the first scaling parameter applied before softmax. Its inputs are the original logits and a positive number T, and its output is a new probability distribution with altered sharpness. The specific method is to first divide each logit by T, then apply softmax to the result:

pᵢ(T) = e^(zᵢ / T) / Σⱼ e^(zⱼ / T)

T must be greater than 0. When T is less than 1, each logit is amplified, and the differences between logits are also amplified synchronously. After exponentiation, the gaps widen further, so the share of high-scoring candidates increases and the distribution becomes sharper. When T is greater than 1, logit differences are compressed, probabilities move toward the middle, the share of tail candidates increases, and the distribution becomes flatter. Temperature does not inject any new knowledge: it does not change what the model has learned, nor does it change the ranking of candidates—dividing all logits by the same positive number does not change their ordering. What it changes is “how much the first place wins” and “how much chance the tail gets.”

A set of concrete numbers can make this process clear. Suppose the original logits of three candidates are [2, 1, 0], and calculate item by item under different T values:

At T=1, the distribution faithfully reflects the original logit differences. At T=0.5, the first place rises from 66.5% to 86.7%, almost monopolizing the sampling. At T=2, the first place drops to 50.6%, and the third place's probability rises from 9% to 18.6%; its chance of being selected clearly increases. The ranking is always “candidate one > candidate two > candidate three”; what changes is the gap between them.

Temperature interacts with subsequent filtering parameters. Because division by a positive temperature does not change the logit ranking, the set of members selected by pure top-k filtering is unaffected by temperature; however, temperature changes the rate at which candidate probabilities accumulate, so a top-p set that truncates by cumulative mass may change accordingly. In addition, the limit as T approaches 0 cannot be directly computed in the formula—the divisor cannot be 0, so services usually treat temperature=0 as greedy decoding, or internally use their own minimum temperature value, rather than actually “dividing by zero.” Understanding this helps avoid the mistaken imagination of “what probabilities look like at temperature 0”: that is not an infinitely sharp distribution, but a branch that bypasses sampling.

# top-k: Fixed Candidate Count, Ignoring Probability Gaps

top-k is a truncation filter. Its inputs are candidate scores and an integer k, and its output is the support set containing only the top-k candidates, along with the probabilities renormalized over this support set. In implementation, it sets all logits after rank k to negative infinity. Since e^(−∞) = 0, the contribution of these candidates in the softmax is completely zeroed out, and the remaining k candidates are summed and renormalized to obtain a distribution that still sums to 1. After that, whether using greedy selection or random sampling, the result can only come from these k candidates.

The key to understanding top-k is that the values after truncation and renormalization are the actual sampling probabilities. Suppose the original distribution is [0.665, 0.245, 0.090], with k=2:

The third-ranked candidate originally had a 9% probability, which becomes zero after truncation; the probabilities of the top two are rescaled according to their original ratio to sum to 1, so the top candidate rises from 66.5% to 73.1%. If you read logprobs at this point for analysis or recording, you must indicate which stage these values come from: the original model distribution, the temperature-scaled distribution, or the truncated and renormalized distribution. The three have different values, and mixing them up will lead to incorrect conclusions.

The advantage of top-k is that it is easy to explain, and its implementation and computation are stable: no matter how sharp the distribution is, the number of candidates is clamped to at most k. But its disadvantage also comes from this 'fixed number' property—k is completely decoupled from the shape of the distribution. It does not look at probability gaps, only at ranks. On an extremely sharp distribution, for example when the top candidate has probability 0.99, top-k still retains k−1 extremely weak candidates that are almost impossible to be chosen; this is not actually harmful, but it is also meaningless. On an extremely flat distribution, for example when the top 100 candidates have almost uniform probabilities, k=3 will arbitrarily delete many options that are equally reasonable, forcibly narrowing the space that could otherwise be freely explored. In other words, the same k imposes completely different constraint strength when the model is extremely certain versus extremely hesitant. Whether this is acceptable depends on how much flexibility you want sampling to preserve.

# top-p: fixed cumulative mass, candidate count varies with uncertainty

top-p (also known as nucleus sampling) addresses top-k's 'fixed count' problem. Its input is a sorted probability distribution and a cumulative threshold p, and its output is the minimal prefix of candidates whose cumulative probability first reaches p. The procedure is to accumulate probabilities from high to low, keeping the fewest candidates that first reach the threshold. In contrast to top-k, top-p fixes not the number of candidates but the retained total probability mass; the candidate count automatically expands or contracts with the shape of the distribution.

Using the previous logits [2, 1, 0], let's examine the behavior of top-p=0.8 at different temperatures:

When the distribution is sharp, the set shrinks automatically: at low temperature, the top candidate alone accounts for 86.7%, so a single candidate crosses the threshold. When the distribution is flat, the set expands: at T=1 the top candidate alone is not enough, and adding the second candidate brings the cumulative total to 0.910, meeting the threshold. This contrasts with top-k—with a fixed k, the same k keeps too many candidates on a sharp distribution and removes too many on a flat distribution; top-p narrows when the model is confident and widens when it is uncertain, letting the retained range follow the model's current degree of confidence.

The retained probabilities also need to be renormalized. After truncation, the remaining probabilities usually sum to less than 1 (e.g., a cumulative total of 0.910); dividing by this sum again makes the sampling probabilities sum back to 1. This is the same as with top-k.

When using top-p, also pay attention to its implementation boundaries. Which candidates are counted in the cumulative sum, whether the token that exactly 'reaches the threshold' is included in the set, how tied probabilities are ordered, and whether a minimum number of candidates is forcibly retained all vary by implementation; you cannot assume consistent behavior from the parameter name alone. For example, some implementations guarantee at least one candidate is retained, some include the item that just crosses the threshold while others do not. When reproducing behavior across services or frameworks, these differences directly affect the token that is ultimately sampled.

# The order of parameter combinations changes the final distribution

Temperature, repetition penalty, top-k, and top-p have each been explained earlier, but they rarely appear alone. In actual decoding, a pipeline simultaneously receives the raw logits, generation history, and all parameters, and outputs the final candidate support set and sampling distribution. These transforms are not commutative: the penalty directly modifies scores and can even change candidate rankings; temperature does not change rankings but changes how quickly probability accumulates; truncation removes candidates. Applying temperature first and then truncation, versus truncation first and then temperature, operates on different objects, so results may differ.

A simple derivation can reveal the impact of order. Suppose the penalty lifts a candidate originally ranked third to second, then the membership set of top-k=2 changes—this is the consequence of 'penalty changing rankings.' Suppose further that temperature flattens the distribution, the accumulation for top-p=0.8 may go from 'first 2 reach the threshold' to 'first 3 reach the threshold'—this is the consequence of 'temperature changing accumulation speed.' If the penalty is applied only after truncation, deleted candidates have no chance to be revived by the penalty. These differences all propagate to the final sampling distribution.

Therefore, the specific meaning of a parameter combination can only be interpreted according to the service implementation. Many APIs do not promise a unified execution order, and the pipeline orchestration inside different frameworks also differs. When top-k and top-p are set together, the common practice is to take their intersection—first remove down to only the top k, then continue narrowing by cumulative mass, or the reverse; either way, the intersection is usually much stricter than either single strategy. In addition, default values may implicitly enable some kind of truncation: you think only temperature is set, but the service may also apply its own top-p or minimum temperature in the background.

When conducting attributable experiments, the correct approach is to start with a single mechanism and add parameters one by one. First change only temperature and confirm the behavioral change; then add top-p and observe the additional impact; finally stack the penalty or enable two kinds of truncation simultaneously. Adjusting multiple parameters at once makes it impossible to determine which one the output change comes from.

# Repetition penalty modifies history-related scores

Repetition penalty is the only score correction that depends on generation history. Its input is the history of generated tokens and current logits, and its output is scores adjusted for “tokens that have appeared.” The intent is straightforward: to discourage the model from repeatedly saying the same word or phrase. However, its implementations have several variants with quite different behavior:

presence only considers “whether it has appeared,” frequency considers “how many times it has appeared,” and both are soft penalties: they only lower scores and do not guarantee non-reappearance. no-repeat n-gram is hard filtering: it directly removes continuations that match the rule from the candidate set. The implementation of repetition penalty is the least uniform—some implementations divide the logits of seen tokens by a coefficient, others multiply by a coefficient; and logits can be negative, so the direction of effect for multiplication vs division and for positive vs negative logits must be confirmed according to the specific implementation.

“Lowering repetition rate” does not automatically equal “better output.” Many texts require precise reuse: variable names in code, fixed terms in papers, and proper nouns in citations must appear repeatedly to be correct. The penalty mechanism does not distinguish between “harmful mechanical repetition” and “necessary precise reuse,” and may suppress the only correct candidate to the point that it cannot be selected. Hard n-gram filtering is more direct: when sentence structure is forced to need a phrase that was just used, the candidate set may be emptied, and the model can only reroute or stop.

Also note the complexity at the token layer. A “word” in the user's eyes may span multiple token fragments, and capitalization, space prefixes, and morphological changes (singular/plural, tense) may correspond to completely different tokens. Penalties are recorded per token or n-gram and may be inconsistent with the user's perceived “repetition.” Therefore repetition penalty is not a semantic deduplicator. When configuring it, you should check three things in light of the task: which repetitions are necessary in the text, whether long loops may occur, and whether factual consistency is harmed.

# Stop conditions determine the boundary, not content completeness

When generation stops and whether the generated content is good are two independent things. At each step, the terminator receives the newly sampled token, the decoded text, the grammar state, and the remaining budget, and outputs one of two results: continue generating, or stop, with a finish reason. Different termination mechanisms trigger at different positions and have completely different failure modes:

EOS is a dedicated token in the vocabulary. The model stops only when it samples this token; if the prompt template writes the ID incorrectly, or the filtering step accidentally removes it, the model will keep generating until other conditions force a truncation. The stop string, by contrast, operates at the text layer: the decoded string is matched against the configured sequence, and generation stops on a hit. Its difficulty is that a stop string can be composed of multiple tokens and can also appear in the body text quoted by the user — when the content in the model output that should have been preserved happens to contain the stop string, it will be mistakenly treated as a termination signal and truncated. Streaming services must also preserve matching state across chunks, because a stop string may be split across the boundary of two pushes, and must at the same time clarify whether the returned content includes the stop string itself.

max tokens is the most straightforward hard budget: no matter where the output is, it is forcibly stopped when the budget is exhausted. A JSON object being written, an unclosed block of code, or a half-finished sentence will all be truncated midway. The grammar accepting state, by contrast, is provided by the constrained decoder: it allows termination only when the generated structure reaches an acceptable state. What it proves is “structural completeness,” not “semantic correctness”—a syntactically valid JSON can perfectly well contain wrong data.

Therefore, successful stopping only shows that the generation process ended according to some rule; it cannot be interpreted as content completeness or semantic correctness. Structured tasks should record three things separately: finish reason (which mechanism triggered the stop), whether parsing succeeded, and whether content validation passed. These three are independent of each other; conflating them will cause downstream systems to treat “a truncated valid prefix” as a complete result.

# What Greedy, Sampling, and Beam Search Each Optimize

After scaling, penalties, and truncation, one question remains: how do we actually choose from the candidate distribution? A sequence strategy takes the candidate distribution at each step and outputs one or more complete sequences. The three commonly used strategies optimize three different objectives and cannot replace one another.

Greedy decoding selects only the highest-probability token at each step. It optimizes the local probability at each step; it is fast and has low randomness, making it suitable for simple extraction and baseline comparison. However, choosing the highest probability at each step does not guarantee the highest probability for the entire sequence: the highest-probability token early on may lead to very low-probability subsequent branches, while the slightly lower second-best token at that time leads to a more complete path. Greedy cannot see this because it discards all other branches at the first step. Repetitive loops are also often related to this local optimum.

Beam search is a fix for the limited horizon of greedy decoding. It keeps several prefixes with high cumulative scores, and after each step of expansion it retains only the B prefixes with the best cumulative scores (B is the beam width), finally outputting the sequence with the highest overall cumulative score. This approximates the search for the solution with the highest probability for the entire sequence, closer to the global objective than greedy; tasks with a narrow target space, such as traditional translation, benefit significantly from it. But its costs and boundaries are also clear: computation grows with beam width; the model's estimate of sequence probability itself has a length bias, and short sequences are easily overestimated, requiring specific correction; in open-ended generation, beam search often degenerates into monotonous text lacking variation. Beam width, length bias, and degeneration in open-ended generation together limit its applicable scope.

Random sampling (usually with top-k/top-p truncation) does not pursue the highest-probability sequence at all. It draws according to the distribution given by the model, aiming to preserve diverse paths while using truncation to avoid the noise of the very low-probability tail. Dialogue, creation, and scenarios that require generating multiple candidates and then selecting one are suitable for this strategy. Its cost is that the result is itself a distribution: a single draw is only one realization, and evaluation must repeat many times, using statistics rather than a single output to draw conclusions.

Whichever strategy is used, the output should ultimately be interpreted according to task quality. A high probability assigned by the model expresses the model's preference, not factual correctness; a sequence with higher probability only means the model considers it more likely, not better.

# Choose Parameters by Task, Not by Seeking a Universal Recipe

Rules such as “use temperature=0 for code and 1 for creative writing” are widely circulated, but they are only rough heuristics. Parameter selection is not a table lookup; it is a decision process whose inputs include task success criteria, risk, cost, and service implementation, and whose outputs are candidate decoding configurations plus an external acceptance plan. The same problem may have different optimal parameters on different models and versions; the same parameter also involves different trade-offs among quality, cost, and latency.

A reliable approach is to first establish a low-randomness baseline, then sweep a single mechanism around the target metric, and finally re-test combinations. Generate the baseline with greedy decoding or very low temperature, first holding the “randomness” variable fixed so that the real differences caused by parameter adjustment can be seen; then change one parameter at a time and record metric changes; after confirming the direction of each mechanism, test combination configurations and pay attention to order effects between combinations.

Different tasks differ greatly in starting point and validation approach:

The validation items in each table row are as important as the parameters: for classification tasks, you need to test not only temperature but also the calibration of label tokens and the refusal rate; for JSON tasks, count “schema validity” and “field semantic correctness” separately; code tasks are ultimately judged by compilation, testing, and security scanning, not by output fluency.

Finally, a common misconception should be clarified: lowering temperature only makes the model more stable in choosing the high-scoring paths it already prefers; it does not repair any knowledge deficiencies. If the highest-probability path itself is a hallucination, bias, or incorrect algorithm, temperature=0 will only reliably reproduce the same error. What parameter adjustment changes is the stability of “which existing path is chosen,” not the model's knowledge itself. Knowledge-level problems must be addressed through retrieval, tools, constraints, and external verification.

# Why a fixed seed is still not necessarily fully reproducible

A fixed seed is often treated as a switch that 'guarantees word-for-word identical output', but seed actually does only one thing: fix the sequence of a certain random number generator. The inputs needed to reproduce a generation are far more than this one number, and also include the full request, model and tokenizer versions, decoding implementation, and execution environment; the resulting output can only be promised as a 'comparable token path', not verbatim identical.

Why can output still differ even when the random numbers are the same? Because the essence of sampling extraction is: take a random number and compare it against cumulative probability boundaries, see which interval it falls into, and choose that token. Any change in temperature, top-p, or penalties moves these boundaries; once the boundaries move, the same random number may fall into another candidate's interval. Once the selected token differs, the conditional distribution at each subsequent step changes entirely, and the whole path diverges. So 'same random number' only guarantees that the number used for comparison has not changed, not that the boundaries of comparison have not changed.

And the sources of boundary changes are far more numerous than parameter adjustments. Model weight updates, tokenizer version changes, byte-level differences in prompts, replacement of floating-point computation kernels, parallel reduction order, influence of other requests in a batch, decoding implementation changes, and even silent server-side updates can all cause the same request to compute slightly different probabilities. Some hardware operators themselves do not provide bit-level determinism; the results of two computations on the same batch of data may differ in the last significant bit—this difference is enough to make the random number cross a boundary and select an adjacent candidate.

Therefore, the fixed seed provided by a managed service is usually only a promise of approximate reproduction and does not constitute a guarantee of verbatim identity. When actually doing a reproducibility experiment, you should completely save the model or service version, the full request (including all default parameters), the seed, the output token IDs and logprobs, so that at least you can determine whether the 'inconsistency' occurred in the probability computation stage or the sampling stage. If a managed API only provides a 'best effort' seed, then the two results should be regarded as approximate reproduction, not as provable deterministic evidence for any one result.

# How Should Random Systems Be Evaluated

Once sampling is enabled, the system outputs a distribution rather than a deterministic path. Evaluation of random decoding therefore differs from deterministic systems: its inputs are a fixed input set, system version, and parameter candidates, and its outputs are the quality, diversity, variance, cost, and failure distribution obtained from multiple independent samples. The quality of a single generation cannot represent the overall quality of a set of sampling parameters—that one run may be just one lucky or unlucky realization of the distribution.

A reliable evaluation process can be laid out in eight steps.

First step, fix the input set and version: save the exact prompts, templates, model, and tokenizer versions to ensure the evaluation target does not drift. Second step, establish a greedy baseline: first look at the quality and failure modes of the path the model most prefers; this is the reference point for all parameter comparisons. Third step, sweep only one mechanism at a time: sweep the curves for temperature, top-p, top-k, and penalty separately, rather than changing multiple parameters at once. Fourth step, repeat independent sampling: run each configuration enough times and report the mean, variance, quantiles, worst failure, and success rate. The mean hides tail risk—a 95% success rate and “5% of requests fail completely” are two numbers that must be seen separately.

Fifth step, measure quality and diversity simultaneously: high string difference does not automatically equal valuable diversity; two completely different wrong answers are also “highly different”. Diversity must be weighed together with quality. Sixth step, record the end reason: EOS, stop strings, length truncation, and errors must be counted separately; why a sample stops is itself evaluation information. Seventh step, perform external validation per task: code must actually compile and run tests, facts must be checked against evidence, structure must pass a parser; do not judge based only on model probability or surface fluency. Eighth step, perform interaction ablations: after measuring the benefit of a parameter combination, remove one parameter at a time and measure again to confirm the benefit really comes from that factor and not from chance.

The value of this process lies in turning randomness from “noise” into a measurable object: through multiple sampling and quantile reporting, you can judge how stable a set of parameters is across what range, and under what conditions it goes out of control.

# Concept Dependencies and Further Learning

The mechanisms discussed on this page all build on several lower-level concepts, each of which leads to deeper topics. The table below gives directions for further reading and the key question each direction should answer:

Passing criterion: be able to compute by hand from a row of logits the candidate distribution after temperature, top-k, and top-p, identify the implementation dependencies of parameter combinations, and design evaluations for specific tasks that include multiple samples, end reasons, and external verification.