Large Language Model (LLM)

Before large language models appeared, natural language processing typically used a “one task, one model” approach: sentiment classification, machine translation, question answering, summarization, and named entity recognition were each trained separately, each relying on large amounts of manually annotated task-specific data. Such systems could work on a single task, but whenever a task was added or changed, it was often necessary to prepare data and train a model again, and the capabilities of existing models were difficult to transfer directly. The result was high development costs, siloed systems, and each model usually handled only the small slice of problems it was responsible for.

The key shift that large language models rely on is to unify seemingly different language tasks into “continue generating text based on the existing text.” For example, translation can be expressed as “English: ... Chinese: ___”, question answering as “Question: ... Answer: ___”, summarization as “Source: ... Summary: ___”, and sentiment classification can also be expressed as “The sentiment of this review is: ___”. Although these input forms express different tasks, the output required from the model is the same: complete the next appropriate text segment.

Unifying the task format changes how models are built. The input is text that contains task instructions and the content to be processed, and the output is a continuation that meets the task's requirements; there is no longer any need to design a completely separate model for each type of task. As long as the same model can judge from context “what should be written next here,” the same set of parameters has the opportunity to perform translation, question answering, summarization, or classification. The problem therefore shifts from “how to model each task separately” to “how to train a general model that is sufficiently good at text continuation.” Large language models are the answer to this question.

Here, “continuation” does not mean mechanically adding a sentence. The model must identify the task type from the input text, understand the existing content, and generate output that matches the context and task requirements. The task is encoded in the text, so changing the prompt text can change the work the model needs to do without immediately retraining a task-specific model. This is also the core value of general language models relative to older task-specific models: using a unified input-output interface and the same set of parameters to cover multiple language tasks, reducing the need to build a system from scratch for each task.

This unification also has clear boundaries. Large language models are suitable for open-ended tasks centered on language expression and generation, but “being able to generate text that looks appropriate” does not mean “being able to guarantee precise computation, factual reliability, or strictly reproducible results.” When a task requires exact numerical values, verifiable facts, or stable and consistent output, relying solely on model continuation is not appropriate. These limitations come from its text-continuation-centric mechanism and cannot be simply understood as something that will disappear automatically as long as the model gets larger.

Given a sequence of tokens that have already appeared, the large language model does not directly output a specific word in a single computation; instead, for each candidate token in the vocabulary, it computes the conditional probability that it is the next token. This can be written as:

P(next token = w │ context) = softmax(z)_w

Here, w is a candidate token in the vocabulary, “context” is the current context, and z is the real-valued vector computed by the model’s final layer for all candidates in the vocabulary, also called logits. A logit itself is not yet a probability; it can be any real number. Softmax converts the entire set of logits into a probability distribution that is nonnegative and sums to 1. softmax(z)_w is the entry corresponding to candidate w. Therefore, what the model actually gives at each step is a probability table over the entire vocabulary, not a single word.

To generate text from this probability table, a selection rule is also needed to pick a token from it. The chosen token is appended to the end of the context, and the new complete prefix is fed into the model again, which computes the next probability table. For example, after selecting “Paris” after “The capital of France is”, the input becomes “The capital of France is Paris”; the next step the model might select “.” or “,”. This rolling process of “generate one, append it back to the input, and generate another” is called autoregressive generation. Long-form text is not a complete draft written all at once, but the result of this loop repeated many times until the model generates an end-of-sequence token.

This also means that at a given step the model determines the next step only based on the prefix available at that time; it does not have an immutable global answer prepared in advance. Each newly generated token changes the conditions for subsequent steps, and early choices steer the generation onto different paths. As a result, the model may produce locally fluent text continuously but enter an incorrect path at some step, and then continue plausibly along that incorrect prefix.

Mathematically, the probability of an entire text can be decomposed into the product of step-by-step conditional probabilities:

P(x₁…xₙ) = ∏ₜ₌₁ⁿ P(xₜ │ x₁…xₜ₋₁)

xₜ denotes the t-th token, x₁…xₜ₋₁ is the entire prefix before it, and n is the sequence length. This decomposition shows that a “language model” models the probability of a complete text sequence, and the sequence probability is jointly determined by the probability of “generating the next token given the current prefix” at each step.

For example, given some context, the conditional probability that the model chooses “Paris” in the first step is 0.62; after appending “Paris” to the context, the conditional probability that it chooses a period in the second step is 0.80. Then the conditional probability of the two-step path that generates “Paris.” is:

P(Paris.│context) = 0.62 × 0.80 = 0.496

If another path has a probability of 0.09 for choosing “Lyon” in the first step and then a probability of 0.90 for choosing a period, then the path probability for “Lyon.” is 0.09 × 0.90 = 0.081. Although the second path has a higher second-step probability, the entire path is still far lower than the first, because the sequence probability depends on the product of all conditional probabilities along the way, and a high probability at a later step cannot fully compensate for a large gap at an earlier step.

For real text there are exponentially many candidate paths, and the probability distribution given by the model is not the same concept as which path is finally taken. Greedy decoding keeps only the token with the highest current probability at each step; beam search keeps a limited number of candidate paths with higher cumulative probabilities; random sampling selects and explores according to the probability distribution at each step. They use the same model distribution but may produce different results, so one should distinguish between “probabilities computed by the model” and “text selected by the decoding rule”.

Text cannot directly enter a neural network, because a neural network performs weighted sums and nonlinear transformations of numbers. To turn a string like “the capital of France is” into a computable input, it must first go through tokenization and embedding: tokenization converts text into discrete IDs, and embedding then converts IDs into continuous vectors.

Tokenization first splits text into a sequence of tokens according to a vocabulary. A token is usually a subword piece: common words may correspond to a single token, while rare words may be split into multiple pieces. Each token has an integer ID in the vocabulary, so “the capital of France is” can ultimately be represented as a token ID sequence like [121, 340, 88, 502]. These integers only serve as indices; the numerical difference between ID 121 and ID 340 does not indicate how far apart the two tokens are in meaning, nor can it be used directly as a semantic relationship.

Embedding is responsible for mapping each token ID to a vector of hundreds to thousands of dimensions. It can be understood as using the ID to look up a corresponding row in a large table: the input is a discrete ID, and the output is a vector composed of many continuous numbers. The embedding space can carry “near/far” relationships, so that semantically similar tokens land in relatively close positions. In this way, when the later network performs mathematical operations on the vectors, it can exploit the semantic relationships encoded in them, rather than mistakenly treating vocabulary IDs as numbers with meaningful magnitude.

The meaning of a token alone is not enough; its position in the sequence also affects understanding. The same token at the beginning and at the end of a sentence can play different roles in context. Therefore, each token’s embedding must also add a positional encoding that represents “which position it occupies.” After this step, the input is no longer a string of text or integers, but a stack of vectors carrying both token information and order information; this is the representation that the model’s subsequent computation actually receives.

This transformation chain can be written as:

text → token sequence → token ID sequence → token embedding + positional encoding → vector sequence

Each step changes the representation. Tokenization decides which fragments of the original text serve as basic processing units; embedding places these units into a continuous space; and positional encoding adds the ordering. The model processes the final vector sequence and does not directly operate on the characters that people see.

This mechanism has two practical consequences. First, billing and context length are usually counted in tokens, not in characters; visible text of the same length may occupy different numbers of tokens because of differences in splitting. Second, character information can be packed into larger tokens during tokenization, and the model cannot always inspect text letter by letter. For example, when asked “how many r’s are there in strawberry,” the model may make mistakes, because the basic units it receives are not necessarily individual letters. Embedding is good at providing continuous representations for later semantic computation, but it cannot guarantee that character-level details are always preserved in a directly countable form.

Transformer receives a sequence of token vectors with positional information. Its task is not to translate these vectors directly back into text, but to repeatedly process the representation at each position so that that position gradually absorbs the contextual information needed for prediction. After multiple layers of processing, the final representation contains enough information to compute the probability of the next token.

The key mechanism in it is attention. When processing a position, the model computes relevance weights between it and earlier positions, then aggregates information from the preceding text according to those weights. The higher the weight, the greater the influence of the corresponding position's information on the current representation. For example, when making a prediction in “France's capital is ___”, the current position can place more attention on “France”, thereby linking the country to the content to be filled in, rather than relying only on the token immediately adjacent to the blank. Attention solves the problem of “which positions in the preceding text the current prediction should take information from, and how much from each”.

The context obtained by one layer of attention is further processed; after many layers are stacked, information can be combined layer by layer. Thus, each position starts out as only a local token vector, but after repeated updates it becomes a more abstract contextual representation that incorporates relevant preceding text. The core role of the Transformer in a Large Language Model (LLM) can be summarized as: input the vectors of the preceding positions, output context-adapted vectors, and thereby support the prediction of the next token.

When used in an autoregressive language model, attention must also satisfy a key constraint: the t-th position cannot access information after the t-th position. During training, to improve computational efficiency, the entire text can be fed into the model at once; but if a position can see the token it is supposed to predict or later content, the model is effectively using the answer to predict the answer, and the training objective loses its meaning.

The causal mask enforces this restriction in the attention computation. It allows each position to attend only to positions already appearing to its left, and prevents the use of future information on the right. If the “can it view” relationships between all positions are drawn as a matrix, the result is a lower-triangular matrix: the beginning of the sequence has the least available information, and positions further along can see a longer prefix. This is exactly the implementation in the Transformer's computation of “predicting the next token based only on the preceding text”.

The causal mask also brings significant training efficiency. A text of length n needs only one parallel input to construct training signals at all positions simultaneously: each position uses the prefix it is permitted to see to predict the token that follows immediately. Although these positions are processed together in a single computation, they remain strictly constrained by the mask and cannot leak future answers to each other. Thus, one piece of text can simultaneously provide n position-by-position prediction samples, allowing vast amounts of text to be used efficiently.

The boundaries of this mechanism also stem from the same constraint. A position can only aggregate content that has already appeared, and cannot use future tokens that have not yet been generated during prediction; the model's representation changes as the prefix grows. Therefore, what the Transformer forms is a contextual representation “conditioned on the current preceding text,” not a global draft that knows the entire subsequent text in advance.

The problem pre-training aims to solve is how to provide learning signals for a large number of model parameters from massive amounts of ordinary text. Next-token prediction is suited to this task because the supervised answer is already contained in the text itself: given a prefix x₁…xₜ₋₁, the token xₜ that actually appears at position t is the correct target. The training data does not require humans to label categories or answers item by item; the text can be automatically broken into many “prefix—actual successor” samples. This way of producing labels from the data itself is called self-supervised learning.

For each prefix, the model outputs a probability distribution over all candidate tokens in the vocabulary. Training needs to measure whether this distribution assigns a sufficiently high probability to the actual successor, and the loss used is cross-entropy. Over a piece of text, the total loss can be written as:

L = −∑ₜ log P(xₜ │ x₁…xₜ₋₁)

L is the sum of the losses at all prediction positions; t denotes the current position to be predicted; xₜ is the token that actually appears at that position in the text; x₁…xₜ₋₁ is the prefix before it; P(xₜ │ x₁…xₜ₋₁) is the conditional probability that the model assigns to the actual token after seeing this prefix. ∑ₜ means summing the losses over all positions in the piece of text.

The negative logarithm determines how the model is penalized. If the model assigns a high probability to the actual token, log P is close to 0, and the corresponding −log P is small; if the model assigns a low probability to the actual token, −log P is large, and the model receives a heavier penalty. Therefore, the loss can be understood as the model’s “degree of surprise” when facing the actual continuation: the less it believes the token that actually appeared, the greater the loss.

After the loss is computed, gradient descent adjusts the model parameters according to the direction in which the loss changes with respect to the parameters, so that later, when facing similar prefixes, the actual token receives a higher probability. This process is repeated over large amounts of text and at many positions. The inputs are the text prefix and the actual successor token in the text; the model directly outputs a probability distribution over the vocabulary, and the training system then collects the probabilities corresponding to the actual successors into the loss. Pre-training is essentially the continuous reduction of this cross-entropy loss on extremely large-scale data.

A decrease in loss only means that the model is less surprised by the actual continuations of this kind of text in the training distribution; it cannot be interpreted as the text content having been fact-checked. The model learns statistical regularities that appear in the corpus, and both its capabilities and limitations are affected by the time period, language, sources, noise, and biases of the training corpus. Adding more data and parameters can expand the scale of training, but it does not automatically eliminate these limitations.

The importance of self-supervised learning is that it removes the throughput bottleneck of manual labeling. Older task setups relied on people to create labels, and the more data there was, the higher the cost and time usually were; next-token prediction, by contrast, can obtain the answer directly from ordinary text, so data, parameters, and compute can be continuously scaled up. This empirical regularity that model performance improves as scale increases is known as scaling laws, and one of the foundations that makes it possible is precisely that the pre-training objective does not require people to label the correct successor for every piece of text.

“Predicting the next token” looks like input-method suggestions, but when this objective is applied to large amounts of diverse text and required to achieve very high accuracy, the task itself forces the model to learn many reusable structures. The next token often depends on the preceding grammar, semantics, factual relations, speaker intent, or derivation process; if the model has not formed internal patterns sufficient to represent these relations, it is difficult to continuously reduce prediction loss.

For example, to continue “because A and B, therefore ___”, the correct continuation depends on the relationship between A, B, and the conclusion; the model needs to learn reasoning patterns or intermediate representations that can support this prediction. To continue code, the next token must conform to syntax and semantics, otherwise the program will have errors; to continue a dialogue, it is necessary to choose an appropriate expression based on the other party's intent and tone; to complete “1234 × 5678 = ___”, it is necessary to master the multiplication rules sufficient to produce the result. On the surface, these are all next-token predictions, but the information and regularities they need to use are different.

The direct input of training is large and diverse text and a model with sufficient capacity; the direct optimization result is lower prediction loss and a set of internal representations. Translation, code generation, or reasoning performance is not taught item by item by separate task category labels, but rather is the ability transferred from the representations formed by the model to complete broad prediction tasks under new prompts. The more diverse the contexts covered by the prediction objective, the more valuable the grammar, facts, code, and partial reasoning representations that can be reused in different scenarios.

This transfer can be understood from the perspective of “compression pressure.” The model must use limited parameters to cope with many different contexts; memorizing all situations one by one in isolation is not the only way it operates. Learning reusable grammar, semantics, and reasoning patterns can help it predict the continuation for more prefixes. This explains why the model may generalize to inputs it has not seen verbatim. But the parameters also memorize training fragments, so the final behavior is typically a mixture of memorization, template imitation, and compositional generalization. Regular performance cannot be directly equated with full understanding.

To judge whether the model has truly acquired transferable reasoning ability, one cannot only look at whether it can fluently write correct steps. An answer may come from reproducing common templates in training, or from internal representations that can generalize to new problems. To distinguish the two, it is necessary to use new combinations, counterfactuals, and out-of-distribution problems: if the form of the problem or the combination of elements changes and the ability remains stable, this more strongly supports the explanation that the model has “formed generalizable representations.” Even if a certain benchmark score improves, it only shows that the model performs better on that test; it cannot be used to assert that it has human understanding or that it is executing a reliable algorithm every time.

The development of these abilities also depends on scale. When parameters, data, and compute increase proportionally, the model's predictive ability improves, and many specific behaviors emerge along with it. A common claim is that certain abilities “emerge suddenly” after the scale crosses a threshold, but this phenomenon remains controversial: when evaluation metrics are black-and-white, smooth improvement may look like a sudden jump; after switching to continuous metrics, the curve may also show a smooth rise. Therefore, a more cautious conclusion is that increasing scale can strengthen predictive ability and give rise to a variety of transferable behaviors, but when abilities appear, whether they are truly sudden, and whether they are supported by memory or compositional generalization all require specific evaluation, and cannot be judged only from fluent output.

At each step, the model produces logits over the vocabulary or the probability distribution obtained by normalizing them. The final text also depends on how the decoding rule selects a token from the distribution. The inputs to decoding include the model logits, the currently visible prefix, and sampling parameters; the output is the token selected at this step. Appending this token back to the prefix and looping forms the complete text. Changing the decoding method only changes how the probability distribution of the same model is used; it does not retrain parameters or add new knowledge to the model.

The simplest rule is greedy decoding: at each step, select the token with the highest current probability. Its output is more deterministic, but looking only at the current best choice step by step tends to produce dull or repetitive text. Random sampling selects according to the probability distribution; high-probability tokens are more likely to be chosen, while low-probability tokens still retain some chance. This randomness leads to different generation paths and higher diversity, and it also means the same prefix may produce different results.

Temperature T changes the shape of the probability distribution before sampling. The calculation first divides each logit by T, then applies softmax:

P(w) = softmax(z / T)_w

z denotes the logits of each candidate token in the vocabulary, w is one of the candidates, and P(w) is the probability of selecting it after temperature scaling. Temperature does not change the knowledge source of candidates; it only readjusts the relative probabilities among them.

When T → 0, differences between logits are amplified, the distribution becomes very sharp, and probability is almost concentrated on the highest candidate; the output is more deterministic, more conservative, and may be more repetitive. When T = 1, the model's original distribution is used directly. When T > 1, logit differences are compressed, the distribution is flatter, and tokens with originally lower probability gain more chances of being selected; the text is therefore more diverse but also more likely to deviate from an appropriate path.

From an information-theoretic perspective, temperature adjusts the uncertainty of the output distribution, that is, entropy. Low temperature corresponds to low entropy, with a few candidates occupying most of the probability; high temperature corresponds to high entropy, with probability spread across more candidates. The so-called 'increasing creativity' in this mechanism does not mean the model suddenly gains new ideas; rather, sampling allows more low-probability paths to be explored. Higher temperature does not necessarily mean better quality; it adjusts the trade-off between stability and diversity.

How much preceding text can be used during generation is limited by the context window. The context window specifies the maximum number of tokens that the model can directly condition on in a single computation; content outside the window does not automatically participate in the current prediction. If an application wants to use earlier information, it needs to retrieve it again or summarize it and inject it into the current context. The window is measured in tokens rather than the number of visible characters, so how the text is segmented also affects how much content can actually fit.

Expanding the usable window is constrained by both computation and model architecture. Standard global attention requires computing attention scores between every pair of positions, and its matrix size grows quadratically with sequence length. In addition, the KV cache, hardware resources, position representations, and the sequence length used during training also affect the actual usable window. The context window is therefore only the range that the model can "directly see" at the moment; it is not equivalent to permanent memory. A longer window also does not mean that every piece of information within it will be used equally effectively.

After pre-training is completed, what you get is a base model. It can already continue writing based on the language, knowledge, and patterns learned from large amounts of text, but the pre-training objective only requires it to judge "how text usually continues" and does not specifically require it to recognize user input as an instruction and give a helpful answer. Therefore, when a question is given directly to a base model, it may continue to generate more questions, imitate web page paragraphs, or adopt other continuation styles common in training text, rather than necessarily answering like an assistant.

Going from a base model to an everyday conversational assistant usually involves three stages. The first stage is pre-training: input massive raw text, and learn general language, knowledge, and patterns through self-supervised next-token prediction. It produces a broadly capable base model, but "able to continue writing" does not equal "able to follow instructions".

The second stage is instruction fine-tuning, also called SFT. The training data is no longer just raw web page text, but demonstrations of "instruction → ideal answer". The model continues to update its parameters, learning what answer format to adopt after seeing an instruction, how to organize content around the question, and the conversational habit of "answering when asked". SFT mainly adds instruction-following behavior: it guides the existing capabilities of the base model into assistant-style interaction.

The third stage is preference alignment, with typical methods including RLHF and DPO. The training signal comes from human preference comparisons about "which answer is better" or corresponding feedback, and the model adjusts response tendencies accordingly, making outputs more useful, safer, and closer to the way people expect them to be expressed. What is learned here is not a single standard answer, but relative preferences among multiple candidate answers.

The inputs and goals of the three stages differ:

Pre-training: raw massive text → learn general continuation ability → base model Instruction fine-tuning: instruction and ideal answer demonstrations → learn to follow instructions → conversational behavior takes shape Preference alignment: answer comparisons or feedback data → adjust response tendencies → better conform to helpfulness and safety standards

When evaluating the three stages, different metrics should also be distinguished. Changes after pre-training are better suited for observing knowledge and continuation ability; instruction fine-tuning should be assessed by instruction-following rate; preference alignment should be assessed by preference and safety metrics. A model becoming more obedient and more aligned with expression norms does not necessarily mean it has mastered more reliable facts, and improvements in answer style cannot be directly equated with higher factual accuracy.

Post-training changes the model's behavioral tendencies, not providing an absolute guarantee for every output. An assistant that has completed instruction fine-tuning and preference alignment may still answer incorrectly and cannot be guaranteed to be safe every time. When the latest or verifiable facts are needed, retrieval, tools, and external verification must still be relied upon. The reason everyday conversational models can both converse and be relatively obedient is the combined result of pre-training capabilities and subsequent behavioral training; the later two stages make capabilities easier to invoke according to human intent, but they cannot eliminate the boundaries of the generation mechanism itself.

The core sentence for understanding the behavior of a Large Language Model (LLM) is: it outputs “the content most likely to come next under the statistical patterns of the training data,” not verified “facts.”

Given a prompt and the current context, the model computes probabilities for candidate tokens, then a decoding process gradually forms the continuation. Here, language likelihood is the relative possibility the model assigns to different continuations based on statistical patterns in the training data; it answers “which text is more like a plausible continuation in the training data,” not “which text is true in reality.” What the model optimizes is whether the next token resembles plausible human text, and the training objective itself has no independent truth constraint.

This distinction directly explains hallucination risk. An incorrect passage can also have high language likelihood: as long as the wording, sentence patterns, and contextual relationships are natural enough, the model may generate it fluently and confidently. High probability only shows that the continuation matches the textual patterns the model has learned; it cannot serve as proof of fact. If a task requires a factual conclusion, the generated result must still be verified through retrieval, citations, or tools.

The model’s knowledge comes from the data available at training time, so it has a knowledge cutoff date and does not automatically know the latest information or private information beyond the training data. This temporal boundary is not the same as hallucination: the former limits the range of information the model can possess, while the latter means that even if the text is fluent, it cannot be used to determine that the content is true.

“The most likely text to appear” is also not the same as “what people want the model to say.” Common continuations in the training text are not necessarily useful, safe, or aligned with user intent, so alignment training is also needed to adjust the model’s behavioral preferences. Alignment addresses whether answers better match human preferences and safety goals; it is related to fact-checking but cannot replace fact-checking.

Autoregressive generation decides only one token at a time, with no pre-fixed global draft. Once a step enters an incorrect path, subsequent tokens are conditioned on the already erroneous prefix and continue to generate locally coherent content, so even a wrong answer may be explained very completely. Prompt techniques such as having the model first write out the reasoning process can sometimes improve the generation path, but they cannot provide a guarantee of correctness.

Hallucination, knowledge cutoff, and alignment needs are related to each other but are not exactly the same single problem: hallucination involves the difference between language likelihood and factual truth; knowledge cutoff limits the temporal range of information obtained from training; alignment adjusts preferences and safety behavior, but does not turn language probability into proof of fact. When using the model, the most important boundary is to always separate “seems true” from “is proven true by evidence.”

We must also avoid compressing these boundaries into a single universal causal chain. What can be inferred from the preceding text is: language probability is not proof of fact, training data has a temporal boundary, and alignment changes behavioral tendencies. As for how to connect external materials and how to delineate trust boundaries in input, each requires its own mechanisms and evidence; conclusions cannot be drawn solely from the “next token” objective. Therefore, Retrieval-Augmented Generation (RAG) and prompt injection in the table below serve only as an index of extended topics, and do not carry explanations of specific mechanisms or evidentiary boundaries.

The complete mechanism of Large Language Models (LLMs) can be understood by starting from “unifying language tasks into continuation.” Translation, question answering, summarization, or classification originally required different specialized systems, but as long as task requirements and input are written into text, their outputs can all be represented as “what should appear next.” The unified continuation interface enables the same model and the same set of parameters to cover many language tasks.

Computationally, “continuation” is not generating the entire answer at once; at each step it outputs a probability distribution over the entire vocabulary and then selects a token according to decoding rules. The chosen token is appended to the preceding text, and the model continues to predict the next step with the new prefix. Long text is the result of this autoregressive loop continuously rolling forward, so the current choice both determines the current step's output and becomes the condition for subsequent predictions.

Text itself cannot directly participate in neural network computation, so the input must first be split into tokens and converted to token IDs. Each ID is then mapped to a continuous embedding vector, and positional information is added. In this way, discrete text becomes a sequence of vectors that the model can process through weighted sums and nonlinear transformations.

Transformer receives this set of vectors and uses attention to aggregate relevant information from different positions in the preceding text, repeatedly processing each position into a contextual representation suitable for predicting the next token. Causal masking restricts each position to using only the left-side preceding text and prevents it from peeking at future answers. This restriction maintains the task definition of “predicting subsequent tokens based only on preceding text” while also allowing an entire text segment to produce multiple position-wise prediction signals in a single training computation.

During training, the target token is the actual successor that appeared in the original text at that position, so text can provide labels by itself without requiring manual annotation one by one. Cross-entropy penalizes cases where the model does not assign a high enough probability to the true token, and gradient descent continuously adjusts parameters to reduce this loss. The self-supervised objective removes the bottleneck of manual annotation, making scaling up data, parameters, and compute a sustainable training path.

The reason next-token prediction can produce translation, code, or some reasoning performance is that accurately predicting continuations of different types of text requires forming reusable syntactic, semantic, factual, and reasoning patterns. When the prediction task covers sufficiently diverse data and the model has sufficient capacity, these internal representations can be transferred to specific tasks under prompting. However, such performance may mix memorization, imitation, and genuine compositional generalization; fluent output alone cannot prove reliable reasoning.

The generation stage no longer learns parameters; instead, it decides how to use the model's existing probability distribution. Greedy decoding tends toward stability, while random sampling introduces diversity; temperature adjusts between conservative and divergent by changing the sharpness or flatness of the distribution. The context window limits how many tokens can be directly used for a single prediction, and information outside the window does not automatically participate in the computation.

Pre-training only yields a base model that is good at continuation. To turn it into an assistant that can follow instructions, it is also necessary to perform instruction fine-tuning using demonstrations of “instruction—ideal response” and then use response preferences or feedback for preference alignment. Post-training makes behavior more obedient, more useful, and more aligned with safety goals, but it cannot guarantee that facts are correct or eliminate other risks of the generation mechanism.

The entire chain ultimately rests on an important boundary: the model optimizes language likelihood under the statistical regularities of the training data, not factual truth. Without truth constraints, incorrect content may also be generated fluently and convincingly, so hallucination is structural, and fluent answers alone cannot serve as proof of fact. At the same time, alignment deals with target specifications and behavioral preferences; it changes how the model tends to answer but cannot guarantee that the answer has undergone fact-checking.

Understanding large language models requires advancing along a clear chain of concept dependencies. The lowest layer is the neural network and its numerical computation foundation: models can only process continuous numbers, so text must first go through tokenization and word segmentation, be mapped to IDs, and then become vectors through embeddings. Softmax is responsible for turning the logits output by the model into a probability distribution over the vocabulary, and cross-entropy measures whether the probability the model assigns to the true next token is high enough. Self-supervised learning explains why real text can directly provide training targets without needing additional human labels.

Transformer and attention form the core structure for processing vector sequences. Attention allows the current position to aggregate information from relevant preceding positions, and Transformer forms contextualized representations through multi-layer processing. In language models, causal masking further stipulates that each position can only use the preceding context on the left, thereby turning Transformer into a structure suitable for "predicting the next token based on the preceding text". Neural networks, tokens, embeddings, attention, softmax, cross-entropy, and self-supervised learning are therefore not isolated terms, but a continuous mechanism from text input to training signal.

After establishing these prerequisite concepts, the core of this page can be strung together into one main thread: next-token prediction defines the problem to solve at each step; autoregressive generation loops one prediction into complete text; causal masking ensures each step uses only the preceding text that has already appeared; the pre-training objective uses the true successor in the text to compute the loss; sampling and temperature determine how to select a token from the probability distribution; the context window specifies the range of preceding text that can be directly used in a single computation; the base model then goes through instruction fine-tuning SFT and preference alignment in turn, becoming an assistant that better follows instructions.

The extended topics immediately adjacent to this main thread respectively answer questions about scale, facts, and control. Scaling laws discuss how capabilities change as data, parameters, and compute expand; hallucination discusses errors arising from the mismatch between language likelihood and factual truth; alignment and RLHF adjust the model's responses to human preferences and safety goals. RAG, prompt engineering, and prompt injection also belong to further-learning extended topics, but based only on the generation main thread of this page, one cannot determine their specific working methods or risk boundaries.

Expanding further outward, one can learn about multimodal models, Mixture of Experts (MoE), reasoning models, AI Agent, fine-tuning, and quantization and deployment. These topics respectively connect language models to more input forms, different model structures and computation methods, more complex reasoning and action processes, parameter adjustments for specific tasks, and resource constraints in actual operation. They are built on the same foundation: the model first converts input into a computable representation, then produces conditional probabilities according to the parameters obtained from training, and forms final behavior through generation or external systems.

The two most critical causal relationships in this dependency graph are: the answer for predicting the next token comes from the text itself, so no manual item-by-item labeling is needed; and to continuously predict accurately across diverse texts, the model will form transferable representations of syntax, semantics, and partial reasoning. At the same time, the training objective optimizes language likelihood rather than factual truth, so erroneous content may also receive high probability and be generated fluently; hallucination is therefore a structural risk of the generation mechanism.