Model Families: Understanding Capability Boundaries from Information Flow and Training Objectives

When people say “this is a Transformer,” what they get is often just the name of an information-flow skeleton, not a description of model capabilities. “Model family” here is not a collection of brands, but a coordinate system used to describe model mechanisms and delivery methods: the input is a specific model version or a task requirement, and the output is four mutually independent descriptions—architecture, training objective, modality interface, and post-training/product layer.

The four axes each answer a different question. The architecture axis answers “how is information read, stored, and mixed,” corresponding to skeletons such as encoders, causal decoders, diffusion models, and state space models (SSMs). The training objective axis answers “what is the model rewarded to predict,” corresponding to objectives such as masked word prediction, next-token prediction, denoising, and contrastive learning. The modality/interface axis answers “what is the structure of the inputs and outputs,” corresponding to text, images, audio, actions, and even multimodal combinations. The post-training/product axis answers “what is added on top of the original model,” corresponding to instruction alignment, retrieval-augmented generation (RAG), tool use, memory mechanisms, and safety policies.

These four axes are mutually independent, meaning that a label on any one axis does not entail the content of the other three axes. The most common confusion is to treat “Transformer,” “large language model,” and “chat product” as things on the same layer. Transformer is only an information-flow skeleton; a large language model includes, in addition to this skeleton, the training objective and data; a chat product further adds post-training, retrieval, tools, and permissions on top of the model. Seeing one of these names does not automatically lead to the capabilities and boundaries of the other two layers. The same Transformer skeleton can be trained into a classification encoder, a text generator, or a vision model; the same generative model can also be packaged into completely different products.

The value of the four-axis coordinate system lies in explaining “why it excels”: when a model performs strongly on a certain type of task, you can locate the reason along these four axes instead of stopping at the family label. But it also delineates the boundary: the family label itself does not guarantee quality, privacy, or security. To judge whether a specific model meets the needs, you still need to examine the specific version and actually test it; the statement “it belongs to a certain family” alone is far from enough.

Even though they all use attention, the encoder, decoder, and encoder–decoder families are suited to very different tasks. Distinguishing them is not about whether they have attention; it is about what attention can see, what the training objective rewards, and the resulting computational path.

Encoder-only models see the full input bidirectionally at every layer: the representation at each position can simultaneously absorb all tokens before and after it. Their common training objective is masked token prediction: randomly mask some words in the input and have the model reconstruct them from context. This “read the whole passage and fill in the blanks” training makes encoders naturally good at classification, representation retrieval, and extraction-style tasks—given a complete text as input, it outputs a judgment about the whole passage or about local positions. Causal decoder-only models, by contrast, can see only tokens before the current position; later content is masked. Their training objective is next-token prediction: given a prefix, output the most likely continuation word. Precisely because each output step depends only on the already generated prefix, decoders are naturally suited to open-ended generation, continuation, and in-context learning. Encoder–decoder models combine the two: the encoder side reads the full input bidirectionally, and the decoder side generates output step by step in causal order; each position on the input side can reference the entire text, while each step on the output side depends only on previously generated content. This information flow matches the conditional sequence generation objective and is naturally suited to translation, summarization, and structured transformation—the input and output are two different sequences, and the output must respect the full meaning of the input.

“Natural strengths” do not mean exclusive capability. All three architectures can in principle be forced to do tasks from other families. The difference is whether the training signal and computational path fit: when a task’s natural form aligns with a family’s information flow, the model learns more directly and with higher sample efficiency.

The probability decomposition for conditional generation reveals why information flow direction matters so much. Given input x, the probability of output sequence y is written as

P(y | x) = ∏ₜ P(yₜ | x, y<ₜ)

Here x is the conditional input, such as the source text to be translated; y is the complete output sequence; t is the current generation position; and y<ₜ is the token already generated before position t. The meaning of the expression is that the probability of the whole output equals the product of the conditional probabilities at each step. At each step when generating yₜ, the model sees not isolated information but the input x plus the entire prefix y<ₜ generated so far. Therefore early generation choices change the conditions for every later step—once the previous token is fixed, the probability distribution for the rest is re-constrained. This also explains why causal generation must proceed sequentially: y<ₜ exists only after the earlier steps are complete. Decoder-family tasks naturally unfold along this chain; encoder-family models do not need this step-by-step dependency because they produce the complete output at once.

Why does image generation always start from a lump of noise and require many steps to produce an image? This is related to how diffusion models learn: they do not learn “to continue writing from left to right” but learn “to push noisy samples one step back toward clean samples.”

Training consists of two processes. In the forward process, noise is gradually added to clean data: starting from real samples, random perturbations are mixed in repeatedly until the samples almost become pure noise. The model’s training input is the noisy sample at a given timestep, the timestep index, and optional conditions such as text; the training output is the model’s prediction of this noise—which can be the noise itself, velocity, or the clean sample, depending on the specific parameterization. During generation, the direction is reversed: starting from pure noise, the learned predictions are used repeatedly to gradually restore structure, removing some noise at each step rather than continuing pixel by pixel from left to right.

The core formula shows how the noisy sample is constructed:

xₜ = $\sqrt{\text{ᾱₜ}}$ · x₀ + $\sqrt{\text{1 − ᾱₜ}}$ · ε

where x₀ is the clean sample, xₜ is the noisy sample at timestep t, and ε is the added random noise. The coefficient ᾱₜ controls how much of the original signal remains: when t is small and near the start of adding noise, ᾱₜ is close to 1, and xₜ consists mainly of the real sample x₀; when t is large, ᾱₜ approaches 0,$\sqrt{\text{1 − ᾱₜ}}$ approaches 1, and xₜ is almost only noise. The denoising network is denoted εθ, where θ is its parameter, and c represents conditions such as text. Training teaches the network to predict the actual added noise ε from xₜ, t, and c; during generation, the scheduler then works backward step by step, and each step uses the current noisy sample and the network’s estimate of the noise.

Understanding “high dimensionality” is key to understanding why diffusion is a multi-step process. High dimensionality means that a sample must be described by many numerical coordinates together: image latents contain a large number of spatial positions and channels, and each coordinate carries noise. Therefore, denoising is not about changing a single number but about simultaneously coordinating thousands of coordinates so that they become cleaner overall while maintaining a consistent structure. Getting there in one step is almost impossible; multi-step iteration allows the structure to be corrected slightly at each step, which is exactly why traditional sampling requires many steps.

Diffusion models can correct an entire latent representation at once and are especially well suited to continuous high-dimensional signals, such as image pixels or latent variables. But more dimensions do not automatically mean richer information: if the data consist mainly of discrete rules, or require character-by-character precision—such as exact text, strict logical operations, or latency-sensitive real-time scenarios—the advantages of continuous denoising may not apply, and additional design is often needed. Diffusion itself is a generation mechanism, not something exclusive to images: the same step-by-step noise-adding—denoising framework can also be applied to audio, video, motion sequences, and even discrete sequences.

Attention lets every position see every other position directly, at the cost that the number of pairwise comparisons grows quadratically with sequence length. The state-space and recurrent families take another path: instead of comparing every position with every other, they compress history into a fixed-size state, updating it while scanning, so that computation grows linearly with length.

The minimal information flow can be described by two formulas:

hₜ = A · hₜ₋₁ + B · xₜ yₜ = C · hₜ

xₜ is the input at the current position; hₜ₋₁ is the state vector into which all prior history has been compressed; matrix A determines how the old state evolves—which information is retained and which decays; B writes the current input into the state; adding the two gives the new state hₜ, which both serves as the basis for this step's output and is passed to the next step for continued updating; matrix C then reads the state out as the output yₜ at the current position. So the whole process only needs to maintain a single state vector: each time a new token arrives, update it with one multiplication and one addition, then read out the output. History is not discarded; rather, it is continuously rewritten into the same compressed representation.

This formula is only the skeleton of the minimal information flow. An actual RNN or SSM adds nonlinearities, gating mechanisms, or input-dependent parameters on top of it—gating decides which information at this step is worth writing and which old information should be forgotten, and input-dependent parameters let the state update vary with content. But the skeleton remains the same: a serially scanned chain in which the state is updated step by step.

Working out the complexity makes this clearer. For a sequence of length n, the attention score matrix has n² position pairs: when n = 1000 it has about one million pairs, and when n = 4000 it has about sixteen million pairs—length increases only fourfold, but the number of pairwise comparisons increases sixteenfold. In contrast, a recurrent or state-space model updates only a fixed-size state at each step, and the number of scanning steps grows linearly with n; when length increases fourfold, the number of steps also increases only about fourfold. That is what “trading compressed state for linear scanning” means.

But compressed state comes at a cost: forcing all history into a fixed dimension inevitably loses some detail. When a task requires precise random access—for example, jumping back to the beginning at any time to find a particular word—or requires copying an early long segment verbatim, compressed state may be inadequate; full attention or external memory is more suitable in those cases. Modern systems therefore often use hybrid designs: use state-space layers for coarse long-range scanning, and use attention layers or external retrieval for the parts that require precise localization, rather than choosing one or the other. Actual speed is also not determined only by asymptotic complexity; it also depends on hardware parallelism, state dimension, and the specific kernel implementation.

If you feed a model images and text at the same time, does that mean it has formed a unified understanding? Not necessarily. The essence of multimodal fusion is to place representations of modalities such as images and audio together with text representations into computational paths where they can influence each other. The input can be two or more modalities, and the output can be a shared representation, text, images, or actions; the key to evaluating a multimodal model is not whether the interface can accept files, but which layer begins to allow cross-modal information interaction.

There are three common fusion approaches. The first is the "frozen encoder + projector": a visual or audio encoder first extracts the input into vectors, and the projector then converts these vectors into tokens readable by the language model and feeds them into an existing text model. The second is cross-attention fusion: each modality is encoded separately, preserving its specialized representations, and cross-attention then lets the two modalities query and influence each other. The third is a unified token space: from the beginning, all modalities are mapped to the same token space or trained on the same backbone. The trade-offs differ among the three. The frozen encoder plus projector has the lowest training cost and can reuse mature vision and language models, but the underlying representations are learned separately within each modality, making it hard for them to adapt well to each other. Cross-attention fusion preserves the specialized representation of each modality at the cost of a more complex interface and training pipeline. A unified token or unified backbone enables deep cross-modal interaction and a unified interface, but requires large-scale alignment data and greater computational investment.

Fusion location and data pairing quality determine the actual capability of a multimodal model. Fusion location determines which layers can engage in cross-modal interaction: if concatenation happens only at the topmost layer, the deep representations remain separate; if parameters are shared from the lower layers, the representations of the two modalities are forced to align with each other early in learning. Data pairing quality determines whether these representations are truly aligned: the more accurate the correspondence between images and text descriptions in the training pairs, the more reliable the learned shared space.

Testing for these models must also be separated. A model that can fluently describe an image only proves that the "image in, text out" generation path is usable; it does not automatically prove that its OCR recognition, spatial relationship judgment, counting, or fine-grained localization are reliable. These capabilities need to be verified one by one with tests that have clear inputs and outputs—capability boundaries can only be measured item by item and cannot be inferred as a whole from "being able to describe images."

When designing a refund assistant, it is natural to first ask “which brand of large model should I use,” but the correct order is to decompose tasks first: classify tickets, read receipts, generate replies, and execute refunds. These subtasks have completely different information flows, risk levels, and cost structures, and they do not need to be handled by the same model.

The starting point of the decision map is task decomposition; model family selection occurs after decomposition. Each subtask carries its own constraints when looking for a matching family: constraints include input-output form, accuracy requirements, latency budget, cost, and impact scope of failures.

First, routing. A small encoder model can handle low-cost routing well: when a ticket arrives, it determines the intent and decides which processing branch to take. The encoder reads the complete input bidirectionally and outputs a whole-sequence judgment, which naturally aligns with the information flow of classification tasks; it is also small, low-latency, and inexpensive, making it suitable to serve as the first gate for all requests.

Next, reading receipts. Only requests with attachments need the vision model; tickets without attachments do not need to go through it at all. Restricting visual processing to the subtasks that really need it saves not only compute but also reduces the error surface—the vision model is only invoked where it is needed, and its output affects only this branch.

Generating replies is typical open-ended text generation, so a decoder drafts the explanation. The real change in refund status—from “pending” to “refunded”—does not go through a generative model. The refund API handles the real state change; the model only produces text that a person reviews or sends. High-risk actions are handed to tools with permission boundaries, rather than letting a generative model “execute through language”: the model can say “I have refunded you,” but it has no permission and should not have the ability to actually alter the account status; only that API call can change the state, and its permissions, audit logs, and rollback mechanisms are all independent of the model.

A system composed in this way is usually cheaper, more testable, and more controllable than “the largest model handling everything.” It is cheaper because each subtask uses only the model that is just sufficient; it is more testable because each stage has clear inputs and outputs, so routing accuracy, receipt recognition rate, and reply quality can be evaluated separately; it is more controllable because high-risk actions are isolated in a tool layer with clear permissions, and when the model makes an error its impact scope is limited to the segment it is responsible for. The causal chain of family selection is: task constraints determine subtask decomposition, the subtask's information flow and risk determine the matching family, and only then come the specific version and price.

The top-ranked model is not necessarily the best choice for you, because leaderboard scores are computed on general-purpose evaluation sets, while your task has its own failure costs, hard constraints, and cost structure. The correct screening order is: first define the task, then filter by hard constraints, then compare on real data, and finally validate in canary deployment.

The first step is to define the task unit and failure cost. In the same system, “one processing” might be each email, each receipt, or a complete session; the cost of one misjudgment might be missing a ten-yuan receipt, or incorrectly executing a refund. When the unit and cost differ, the meaning of the same set of metrics is completely different; if this step is not defined clearly, subsequent comparisons are meaningless.

The second step is to filter by hard constraints. These constraints are “fail if not met” conditions; they do not require comparing quality: data residency (which region data must remain in), licensing (the authorized scope of model weights or APIs), modality (whether image support is required), context length, available hardware, and maximum latency. First use these conditions to shrink the candidate set, to avoid wasting evaluation costs on options that are fundamentally unusable.

The third step is to evaluate the remaining candidate families and scales on real data slices, and the evaluation must include refusal and fallback behavior—when the model says “I cannot handle this” and escalates to a human, it is usually better than hard-coding an answer; the evaluation set must cover this situation.

The fourth step is to draw the Pareto frontier, rather than arbitrarily adding heterogeneous metrics into a total score. A teaching example can illustrate this comparison method: on a receipt field extraction task, a small encoder plus OCR has a field F1 of 0.94, P95 latency of 180 ms, and cost of about ¥2 per thousand items, suitable for standard receipts; a multimodal generator has F1 of 0.96, but latency of 1.8 s and cost of about ¥28 per thousand items, suitable for complex long-tail receipts; a rule-based template has F1 of only 0.88, but latency of 25 ms and about ¥0.2 per thousand items, covering only fixed layouts. No option wins on all dimensions: who is on the frontier of “cannot improve quality without spending more, cannot reduce cost without sacrificing quality” depends on your latency budget and failure cost. These numbers are only a teaching example and do not represent the performance of any real vendor; what really matters is to measure with the same data and the same definition of success, otherwise metrics are not comparable.

The last step is canary rollout, and record the model version, prompts, retrieval index, and tool versions. Model behavior is jointly determined by these four; any change to any of them can change the result; if you do not record after go-live, you cannot locate which layer caused a problem when one occurs.

The names “reasoning models,” “chat models,” and “tool models” sound like different architectures, but in most cases they refer to the same underlying skeleton that has undergone different post-training. Supervised fine-tuning, preference optimization, reinforcement learning, and distillation can significantly change the model’s response style, instruction following, inference-time compute, and tool-calling behavior, yet the underlying architecture may still belong to the same decoder family. The skeleton of information flow and training objective has not changed; what changed is the behavioral layer layered on top of that skeleton.

The product layer can continue to stack outside the model: system prompts, Retrieval-Augmented Generation (RAG), output filters, caching, and workflows. Therefore, be especially careful when inferring model mechanisms from product behavior. Observing that a product can cite sources does not directly imply that the base model “has memorized a database”—it most likely just performed retrieval at runtime and injected the retrieved content into the context. Observing that a product calls tools also does not imply that the model has permissions: the model is only responsible for generating the call request; the actual permissions are in the execution layer, where the system decides whether this request can be executed and whom it acts on. The language-level “can call” and the execution-level “has permission to call” are two different things.

This distinction directly determines the order of troubleshooting. When a system fails, first locate which layer the error occurred in: did the base generation itself go wrong, did the post-training strategy teach the wrong behavior, did context retrieval bring back the wrong material, were the tool parameters filled in incorrectly, or was the execution permission configured incorrectly. Each layer has a completely different repair method—base generation issues may require switching models or changing prompts, retrieval issues require changing the index and queries, and permission issues require changing the policies in the execution layer. First locate the layer, then choose the remediation method; otherwise you are likely to keep patching the wrong layer repeatedly while the problem is actually in another layer. Post-training can change behavior, but it does not erase the underlying mechanisms: the information-flow constraints of the underlying family are always there, determining the boundaries of what the model can and cannot do.

“Open-source models are more private” and “closed-source models are definitely stronger” are both invalid judgments, because they conflate several independent delivery-layer issues into one label. Weight licensing, the openness of training code, the openness of data, hosting location, and API data policy are five different questions; the answer to any one cannot be inferred from the others.

Open weights allow self-hosting: running the model on your own servers means data indeed does not have to leave your own data center. But if you deploy an open-weight model on a third-party cloud, data may still be sent out to the cloud provider; conversely, API services can also offer zero retention and regional options, contractually promising not to save requests and to process only in specified regions. “Open source” describes whether the weights are available, not where data flows; “closed source” describes whether the weights are unavailable, and also does not describe the service provider's logging policy. Privacy depends on deployment topology, contract, and logging policy, not the name.

A brand is also not a precise identifier. Under the same brand there can simultaneously be multiple generations of models, multiple sizes, multimodal and text-only versions, and versions with different context lengths, and their capabilities and licenses can all differ. Therefore you cannot directly infer any key attribute from a name; you must verify item by item: for privacy and data retention, check deployment topology, contract, and logging policy; for commercial usability, check weight, code, and data licenses; for tool reliability, check function-calling evaluations, permissions, and retry behavior; for knowledge freshness, check knowledge cutoff, retrieval source, and update time; for the actual version, check model ID, date, parameter scale, and service tier.

This also explains why “choosing a brand first” is the wrong starting point when selecting a model. A brand identifies only the delivery party, not the architecture family, training objective, or the boundary of a specific version. Only by checking the four dimensions of brand, license, hosting, and version separately can you give an actionable answer to “what does this model actually mean in my scenario?”

Connect the entire causal chain, from “I have a task” to “I have a verifiable system”; each step is driven by the conclusion of the previous step.

The starting point is to decompose the task and the cost of failure. A requirement is first broken down into several subtasks, and it is made clear what cost is paid when each subtask goes wrong: misclassifying an order, misreading a receipt, or saying the wrong sentence—each has a completely different cost. This step determines the weight of all subsequent trade-offs.

The second step is to list the hard constraints: modality, latency, licensing, context length, and data residency. These are “fail and you're out” conditions; filter them first to avoid wasting time on unusable options.

The third step uses information flow and training objectives to screen model families. Classification and extraction lean toward encoders that read the full text bidirectionally; continuation and dialogue lean toward causal decoders; continuous high-dimensional signals lean toward diffusion; long-sequence scanning leans toward state space models. Families match the natural form of the task, not the brand.

The fourth step is to compare candidates on real data slices. Use the same data and the same definition of success to evaluate the candidate families and sizes that pass the hard constraints, draw a quality–cost–latency Pareto frontier, rather than adding heterogeneous metrics into a single total score.

The fifth step is to isolate generation from high-risk execution. The model produces text and suggestions; the execution that actually changes state—refunds, charges, sending—is handed to the tool layer with permission boundaries. The model's output can be “request execution,” but permission always remains at the execution layer.

The sixth step is to record the complete version and continuously run regression. Model versions, prompts, retrieval databases, and tool versions jointly determine system behavior; any change may alter the results. Record them, and continuously run regression evaluation when data drifts or versions update.

Each step of this chain produces verifiable intermediate artifacts: the task list, constraint table, family candidate set, slice evaluation results, permission boundary diagram, and version record. The significance of the causal chain is that any error can be traced to a specific link: if routing is wrong, revisit task decomposition; if metrics are distorted, revisit evaluation data; if execution oversteps, revisit the permission layer. Model families turn from abstract coordinates into a traceable decision-making process.