Tokens and Tokenization: The Discrete Language That Models Actually Read and Write

A piece of text entered by the user passes through a fixed conversion pipeline in sequence before entering the model:

Unicode text → normalization → pre-tokenization → subword combination → vocabulary lookup → template insertion → token ID → embedding vector

The input side is the Unicode text that the user sees and inputs; the output side is a sequence of discrete token IDs. The model then uses these IDs to look up the corresponding embedding vectors. Strings that look the same or similar on screen are not guaranteed to yield the same ID sequence after going through this pipeline.

Normalization first handles character representation. Some characters that are visually identical or semantically equivalent may have different Unicode representations, and normalization rules can unify some of them. What it changes is the character sequence that subsequent steps actually receive, so it must be consistent with the rules used during model training.

Pre-tokenization then establishes candidate boundaries, commonly based on spaces and regular-expression rules. It is not equivalent to final tokenization; instead, it first restricts the ranges within which the subword algorithm continues processing. The subword algorithm then combines characters or smaller units within these boundaries, turning the text into a sequence of symbols that the vocabulary can represent.

Vocabulary lookup maps each symbol to a unique integer ID. After that, the chat template or task protocol may also insert BOS, EOS, role markers, or tool tokens into the sequence. The complete ID sequence finally obtained is the input that the model actually receives; each ID is then converted to a vector through the embedding table and fed into the neural network.

Every step in this chain affects the next step. As long as any boundary or rule differs, the subsequent symbol combinations, vocabulary lookup results, and the number and position of IDs may change accordingly. Therefore, the normalization rules, pre-tokenization rules, subword model, vocabulary, and template of the tokenization pipeline must all be versioned as a whole. A “roughly the same” implementation is not precise enough.

Different models may use different pipelines, and the same visible text may produce tokens of different counts, different boundaries, and different values across models. The character count is only a display-level length and cannot be used to directly infer the number of tokens; to know what a model actually reads, you must run a conversion with the complete tokenization pipeline that matches that model.

“Character,” “byte,” “subword,” and “token ID” belong to different levels. Conflating them is what leads to the mistaken judgment that “one character is one token.” This statement holds only when certain inputs and a particular tokenizer happen to produce single-character boundaries; it is not a universal rule.

Unicode code points describe abstract characters, but a glyph that a user sees is not necessarily composed of only one code point. Combining diacritical marks display a base character together with additional symbols; emoji skin tone modifiers change the preceding emoji; zero-width joiners can also connect multiple code points into one visual whole. Therefore, the number of visible characters is not equal to the number of code points.

UTF-8 in turn encodes code points into byte sequences. For example, “猫” occupies 3 bytes after encoding. Byte encoding guarantees that any Unicode text has a representation that can be stored and transmitted, but it does not guarantee that one byte corresponds to one character, nor that byte boundaries are the character boundaries as understood by users.

Subword pieces are the compositional units that a tokenizer uses to represent open-ended text with a finite vocabulary.tokenization can be represented as token and ization, but the actual boundary depends on the specific tokenizer. The role of subwords is to participate in composition; each piece is not required to have complete semantics when taken out of context.

A token ID is an integer index within a vocabulary. For example, the meaning of ID 31415 comes from the current tokenizer’s vocabulary: the model uses it to index a particular row in the embedding matrix. This number has no fixed meaning across tokenizers; 31415 in another vocabulary may point to a completely different piece.

Therefore, a piece of text goes through different representation layers: “visible glyph—code point—byte—subword—ID.” The boundaries of any one layer cannot be directly passed off as the boundaries of another. A reliable system should preserve the original text and maintain a mapping between tokens and character offsets in the original text; when displaying selections, annotations, or locating errors, it should return to the text coordinates that the user sees, rather than directly treating token boundaries as character boundaries.

What BPE learns is not a human-defined dictionary but an ordered set of adjacent-symbol merge rules. Training starts from smaller basic units, repeatedly counts how often pairs of adjacent symbols appear in the current representation, and merges the most frequent pair into a new symbol. This process lets frequently co-occurring fragments gradually become larger tokens.

Suppose the toy corpus contains only low 5 times, lower 2 times. Initially, split each word into characters and add an end-of-word symbol </w>:

• l o w </w>, with weight 5;
• l o w e r </w>, with weight 2.

When counting adjacent pairs, you must multiply by word frequency. Thus l + o appears in both words, giving a total frequency of 5 + 2 = 7. Merge it into lo, then recount adjacent pairs in the current representation.

The first round yields the rule l + o → lo. In the second round, lo + w also appears in all 7 word occurrences, so we get lo + w → low. The third round requires distinguishing the end-of-word position: low + </w> only in the standalone low appears 5 times, while lower in low is followed by e. Therefore, the complete end-of-word form low</w> is merged into one token, lower is still composed of low, e, r and other units.

This example shows the causal chain of BPE: corpus word frequency determines adjacent-pair frequency, frequency determines the next merge rule, and a rule that has already been applied changes the next round's representation and statistics. The larger symbols in the final vocabulary come from this continuous merging process, not from the system knowing in advance whether a fragment is a “word”.

Real training is also affected by pre-tokenization, tie-breaking for equal frequencies, and implementation-specific rules, so the same corpus may show small differences across implementations. However, the core mechanism remains the same: repeatedly adding merges that cover high-frequency adjacent combinations, allowing common fragments to be represented with fewer symbols while retaining the ability to compose other text from smaller units.

BPE, Unigram, and byte-level subword schemes can all represent open text as subword sequences from a finite vocabulary, but they differ in how they construct the vocabulary and choose segmentations. When comparing them, it is not enough to look only at "whether subwords are used"; one must also examine what the training process searches for, how choices are made during encoding, and how they degrade when a segment of text cannot be directly covered.

BPE starts from a small set of symbols. During training, it repeatedly selects adjacent symbols to merge, producing a sequence of ordered rules; at encoding time, it combines input according to these rules, so it can produce deterministic segmentations. What it directly favors is high-frequency local combinations in the training corpus, but this greedy choice itself is not equivalent to directly optimizing the objective of the downstream language model.

Unigram goes in the opposite direction: it first prepares a larger set of candidate pieces, then evaluates segmentations according to a probabilistic model, and gradually removes candidates that contribute less. At encoding time, a piece of text may have multiple valid segmentations; the system selects a high-likelihood scheme based on piece probabilities, and it can also sample from these schemes. The cost is that it needs to maintain a probabilistic model and handle pruning of the candidate vocabulary, making the training and implementation logic more complex.

Byte-level subword approaches first build a complete base using 256 byte values, and then merge common byte sequences into larger tokens. Because any Unicode text can ultimately be encoded as bytes, they never encounter characters that truly cannot be represented. However, "being encodable" and "encoding efficiently" are two different things: when a certain language, rare character, or special string is not covered by larger vocabulary pieces, it falls back to multiple byte tokens.

This fallback has two layers of impact. First, the same visible text consumes more tokens, and therefore occupies more context positions; second, the model receives more fragmented evidence and must learn how to reassemble it from multiple smaller units. Therefore, when evaluating a tokenizer, whether unknown tokens disappear is only the lowest-level coverage metric; one must also observe whether sequences are excessively lengthened, and whether the target text receives sufficiently complete and stable piece representations.

Vocabulary size is not simply bigger-is-better or smaller-is-better; it involves trading off model parameters, sequence length, and training sparsity. Let the vocabulary size be |V|, the embedding dimension be d, and the post-tokenization sequence length be L. Two approximate relationships can first make the main costs clear:

Embedding parameter count ≈ |V| × d

Attention pair count ≈ L²

As |V| increases, the vocabulary can directly accommodate more common segments. Fragments that originally required multiple tokens may be merged into a single token, so the average sequence length L of the same corpus may decrease. Because attention needs to form pairs between sequence positions, a decrease in L makes the approximately L² pair count drop significantly, potentially saving sequence processing cost.

The cost occurs in vocabulary-related parameters. The input embedding matrix needs to store a d-dimensional vector for each token, so its size grows linearly with |V|; the output-side matrix used to classify over the entire vocabulary also expands as the vocabulary grows. If the input embedding and output weights are shared, the exact parameter accounting differs, but the coverage benefit of a larger vocabulary and the training sparsity problem still exist.

Expanding the vocabulary also distributes training evidence across more tokens. High-frequency fragments may thereby receive compact representations, but rare tokens appear less often and receive fewer parameter updates. Thus, a shorter sequence does not automatically mean better learning: the model may also pay a larger vocabulary parameter cost and need to learn some vocabulary entries well from sparser data.

Conversely, a smaller vocabulary can reduce vocabulary parameters and give basic units denser updates, but it may split text into more pieces, increasing L, so that the same content occupies more context positions and computation. When choosing vocabulary size, the real question is: does the sequence shortening gained by increasing |V| sufficiently offset the parameter growth and insufficient training of rare items?

Therefore, one cannot just compare “tokens per character” or a single compression ratio. Effective evaluation should report, at the same time, vocabulary-related parameter counts, the distribution of sequence lengths on actual corpora, processing throughput, downstream task quality, and results for different language slices. A higher compression ratio only indicates that the text uses fewer tokens; it does not mean the model necessarily learns better semantic representations.

Tokenization efficiency determines how many tokens a piece of information occupies, while context window, inference cost, and truncation are usually counted in tokens. Therefore, semantically similar content that produces token sequences of different lengths in different languages may consume different context budgets and face different truncation risks.

This difference first comes from the vocabulary's training data. The vocabulary forms from frequency patterns in the training corpus: common word forms that occur repeatedly in high-resource languages are more likely to be learned as longer, more complete tokens; languages with less coverage may fall back more to character- or byte-level units. For visible text of the same length, the latter may be split into more tokens.

Fertility can be counted by language, that is, the number of tokens per word or per character, or compression rate can be compared. They can describe how a tokenizer allocates sequence length, but they cannot alone give a conclusion about fairness. Different languages have different morphological structures and writing systems; a language producing more tokens cannot be directly interpreted as discrimination. It is also necessary to observe whether this length difference actually translates into differences in cost, truncation, or task quality.

Per-language length distributions are more informative than a single average, because the average may hide long-tail inputs. If a language more often produces long sequences for the same content, it will use up a fixed context window more quickly and may also more easily lose tail information at the length limit.

Numbers, names, and code need to be tested separately, because they are especially sensitive to exact boundaries and copying. Testing only fluent natural passages cannot show whether the tokenizer splits sequences too finely on such structured or rare inputs, nor can it reflect whether the model can stably preserve the original form.

Finally, task accuracy and token length should be observed together. If quality declines as sequences become longer, it is necessary to further distinguish whether language-specific data and task factors are at work, or whether the extra length caused by the tokenizer is. The goal of fairness evaluation is not to eliminate all language differences, but to measure segment by segment the causal chain "corpus frequency → tokenization granularity → sequence budget → truncation and task performance", avoiding conclusions based only on token counts or a single quality metric.

Special tokens are not merely abbreviations for ordinary strings; they are control symbols in the model's input protocol. During pre-training or instruction fine-tuning, certain fixed IDs appear repeatedly in specific positions, thereby taking on structural meanings such as “sequence start,” “user start,” “assistant start,” or “tool result.” What the model learns is the pattern formed by these IDs and their contextual positions.

Chat templates are responsible for converting human-readable conversation structure into the token sequences the model saw during training. They determine the order in which role markers appear, the line breaks between segments, the message termination position, and whether an assistant prefix is needed before generation begins. As a result, the same user question, if wrapped with different role markers or boundaries, can produce a different sequence of IDs actually received by the model, and generation behavior may change accordingly.

This difference comes from a clear causal chain:

Conversation messages → template arrangement → special token and text token sequence → roles and boundaries recognized by the model → generation result

If the assistant prefix is omitted, the model may not recognize “now the assistant should continue” at the familiar position; repeatedly inserting BOS creates a structure at the beginning of the sequence that is uncommon in training; treating EOS as ordinary text confuses content with the end signal. Even if the message text itself does not change, these errors put the model into an input state that is unfamiliar in the training distribution.

Adding a new special token also cannot merely register a string in the vocabulary. After the vocabulary gains an entry, the input embedding matrix and output classification matrix need to be extended with corresponding new rows; these new parameters must also be trained before they can acquire representations and output behaviors consistent with the intended control meaning. Having only the string name does not automatically make the model understand that it represents a role, tool, or concept.

Therefore, chat templates, tokenizer files, vocabulary hashes, and model weights should be treated as a single release unit. Replacing only one of them, even if the interface can still output integer IDs, may break the protocol correspondence established during model training. During deployment and reproduction, the versions of these components need to be checked together to ensure the same message is always encoded into the complete control sequence the model expects.

Tokenization alone does not determine whether a model can do arithmetic, write code, or copy text, but it changes the representation pathways these capabilities must learn. If boundaries are not aligned with the basic structure of the task, the model must first recover structure from inside tokens before performing the task; overly fine boundaries, in turn, can make sequences significantly longer.

Long numbers are a typical example. If a digit string is split into fragments of unstable length, the same decimal place can end up at different internal positions within tokens in different inputs. When a model performs digit-by-digit arithmetic or exact copying, it must learn not only the relationships among digits but also how to recover the digit-position structure from different fragments. Tokenization does not make the operation impossible, but it adds boundary variation that must be handled at the same time.

Code is equally sensitive to indentation, line breaks, and operator boundaries. If these structures are merged with nearby content into unstable fragments, the model must simultaneously understand the program content and infer the formatting boundaries hidden inside tokens. During exact generation, missing a single space, newline, or operator character can change the result. However, unconditionally splitting everything into single characters is not a universal solution either, because sequence length increases markedly and consumes more context positions.

A reasonable tokenization scheme is a trade-off between structural visibility and sequence length. Whether a scheme is good cannot be judged solely by whether tokens look like "human words"; it should be measured on real tasks: whether numeric operations preserve digit relationships, whether code preserves formatting and operators, whether copying tasks can reproduce targets byte by byte or character by character, and how long a token sequence these capabilities require.

The generation phase also faces token boundaries. The model generates one token at a time, but when the service outputs to the user it must convert the data corresponding to the token back into text. A streaming block may contain only part of the bytes of a UTF-8 character and cannot yet be decoded independently; a stop string may also span two or more tokens, or even across adjacent output blocks.

Therefore, the server side cannot treat each token or network block as an independent, complete text. The byte decoder must preserve cross-block state and wait until the bytes needed for a character are complete before outputting; the stop matcher must also keep the tail of the previous block and judge together with the new block whether a stop string has occurred. This is how to avoid garbled text, missed stops, or mistakenly sending part of the stop string to the user.

Text that looks the same on screen is not necessarily the same sequence of Unicode code points underneath. For example, a precomposed character é can be represented by one code point, or by e combining accent marks; the two may look the same visually but differ in the underlying sequence. Fullwidth characters, invisible control characters, and characters that look similar across different writing systems can also create “looks the same, compares different” situations.

This difference propagates along the processing chain. String matching, regular-expression rules, pre-tokenization, and vocabulary lookup actually receive the underlying sequence, not the screen rendering. As a result, visually identical inputs may follow different token boundaries, producing different tokens and IDs; look-alike or invisible characters may also bypass rules written only for literal strings.

Unicode normalization can unify some of these equivalent representations, reducing unnecessary divergence. However, normalization itself changes the underlying text, and this change may be unacceptable in tasks that require verbatim fidelity, exact reproduction, or preservation of original evidence. A system must explicitly adopt a specific Unicode form and distinguish “normalized text for model processing” from “original text that must be preserved as-is”; it cannot assume normalization is always lossless.

Offsets further amplify the problem. Different components may record positions in different units:

• An annotation tool may count by user-visible characters;
• Some systems count by UTF-16 code units;
• Storage or network components may count by UTF-8 bytes;
• Tokenizer also outputs its own offset mapping, mapping tokens back to input ranges.

These units often coincide exactly in pure ASCII text, so errors remain hidden for a long time; they diverge only when encountering multibyte characters, combining characters, or symbols that require multiple UTF-16 code units. If start and end positions in one unit are used as if they were in another, entity highlighting shifts, training labels fall on the wrong span, and audit evidence may point to the wrong text. Because the numbers are still valid integers, such failures usually do not trigger exceptions; they silently produce incorrect results.

A reliable approach is to explicitly annotate each offset with its unit and the corresponding text version, and to perform explicit conversions at component boundaries. The relationships among the original text, normalized text, and token offset mapping must be saved together; during highlighting, training, and auditing, the coordinate system should be confirmed first, and then ranges should be mapped to the target text. This avoids failures where “the string displays correctly, but labels and evidence are silently misaligned.”

The goal of a reproducible tokenizer acceptance is not to prove that it "can run," but to prove that the same artifact produces expected results across different times, implementations, and input boundaries. Being able to encode and then decode ordinary English only covers the simplest path and cannot show whether Unicode, templates, offsets, and streaming handling are reliable.

First, pin down the complete artifact. You need to save the vocabulary, merge rules or tokenization model, normalization rules, pre-tokenization rules, special token definitions, chat template, and corresponding hashes. The tokenizer's behavior is jointly determined by these components; recording only the vocabulary name or version number cannot guarantee that the same processing pipeline is reconstructed later.

Round-trip tests should cover boundary inputs beyond ordinary text, including empty strings, newlines, combining characters, emoji, CJK, right-to-left text, code, and random bytes. The tests should observe whether encoding and then decoding restores the input according to the interface contract, and distinguish allowed normalization changes from unexpected data loss.

Checking only the decoded text is still not enough, because different ID sequences can sometimes be restored to the same displayed text. You need to establish golden ID tests: fix several representative strings and their exact token ID sequences, and compare them item by item across different implementation languages or deployment environments. Any difference in ID, order, or special markers means that the actual input fed to the model has changed.

Offset mapping should also be verified separately. Each token's offset should be mapped back to the original text, checking whether the resulting range covers the correct segment, with particular attention to combining characters and surrogate pairs. Here you must also confirm both the offset unit and the text version at the same time, to avoid results whose values are legal but that land in the wrong position.

Chat template verification should directly print the complete ID sequence after applying the template. Role markers, BOS, EOS, and tool boundaries should appear only at the agreed positions and the expected number of times. This can reveal duplicate BOS, a missing assistant prefix, an incorrect end token, or misplaced tool message boundaries, without waiting until the model behavior becomes abnormal and then working backwards.

The acceptance should also report actual distributions, not just a few examples. Statistics on token length, truncation rate, unknown character or byte fallback, and latency should be collected by language and task in order to determine whether an implementation significantly lengthens sequences, increases truncation, or reduces processing efficiency on particular inputs.

Finally, streaming output should pass random chunking tests. Cut the same output into network chunks at different positions, maintain UTF-8 decoding and stop-string matching state across chunks, and ensure the final decoded text and stop positions remain unchanged. If changing the chunking changes the result, the service is incorrectly treating the transport chunks as complete character or token boundaries.

Together, these tests establish reproducibility: the fixed artifact defines "which tokenizer is being tested," golden ID and template tests define what the model actually receives, offset and round-trip tests verify text mapping, distribution statistics verify real workloads, and random chunking verifies service boundaries. Only when all these layers are stable does the interface not only handle ordinary examples but also provide deployable deterministic behavior.

The complete causal chain of text processing by a model begins with a Unicode string produced by the user and ends with the visible text that the service outputs again. Each intermediate representation conversion constrains the next step, and any change in artifacts or boundary rules can alter the discrete sequences that the model actually receives and generates.

User input is first a sequence of Unicode data, which may contain combining characters, invisible symbols, or multiple underlying representations. Normalization rules decide whether to unify certain representations, and pre-tokenization rules then establish candidate boundaries based on whitespace, regular expressions, or other conventions. Together these two steps determine the input units and the combinable range that the subword algorithm actually faces.

Then BPE, Unigram, or byte-level rules cover the text with segments from a finite vocabulary. The algorithm and the vocabulary together determine which common sequences can be a single token and which inputs must be split into multiple smaller units. This segmentation granularity directly changes sequence length, and also changes the frequency of each token in the training corpus, as well as how many compositional steps the model must learn to recover numbers, word forms, code boundaries, or other structures.

The vocabulary maps each segment to an integer ID, and the chat template inserts role, start, end, or tool control tokens according to the protocol. At this point the model no longer directly sees the characters in the user interface, but only receives this complete sequence of IDs. The embedding matrix looks up the corresponding row for each ID, converting discrete symbols into vectors; subsequent neural network computation is built entirely on this sequence of positions and vectors.

Therefore, even a seemingly small change at the front end propagates into later stages:

Unicode input → normalization and pre-tokenization → subword coverage → vocabulary ID → template control token → embedding vector → model computation

If normalization changes the code point, the pre-tokenization boundaries may change; boundary changes alter the subword combinations; different subwords change the IDs, sequence length, and embedding rows; the model thus computes different representations at different positions. Even if the displayed text appears nearly identical, the final behavior may differ.

The generation direction returns along the opposite path. The model first predicts token IDs, the tokenizer restores them to segments and bytes, and the streaming service then performs UTF-8 decoding and stop-string detection across output chunks, ultimately forming the text the user sees. Here token boundaries, byte boundaries, and display character boundaries likewise are not guaranteed to coincide, and the service must preserve cross-chunk state.

A tokenizer is therefore not a preprocessing tool that can be arbitrarily replaced apart from the weights. Normalization, pre-tokenization, the subword model, the vocabulary, special tokens, the chat template, decoding, and streaming boundary handling together define the model's text protocol. If any part changes, it should be validated together with the model weights and delivered as a consistent release unit.

After understanding the tokenizer, subsequent concepts can be developed along the path of “how discrete IDs enter the model, how they interact inside the model, and how they are reliably returned to text.” Each direction picks up one interface in the tokenization pipeline.

The vocabulary maps fragments to IDs, but the integers themselves serve only as indices. Embeddings explain how the model uses IDs to look up vectors and, during training, allow these vectors to form geometric relationships usable in computation. Here it is necessary to distinguish between “the vocabulary specifies which fragment an ID points to” and “the model weights determine what that ID’s vector learns.”

The tokenizer outputs an ordered ID sequence, but the token content alone is not enough to express position. Positional encoding further explains how the model adds positional information such as order and distance to the representation, so that the same token can participate in different computations when it appears at different positions.

Sequence length is directly affected by segmentation granularity, and self-attention explains why this length translates into time and memory costs. Only by understanding the connection between the two can changes in the tokenizer’s compression ratio be interpreted as actual computational changes, rather than stopping at “more or fewer tokens.”

Role markers, assistant prefixes, and response boundaries become meaningful protocols only through training. Instruction tuning shows how these control tokens co-occur with inputs, target responses, and supervision signals, so that the model learns to perform the corresponding role after specific boundaries.

Finally, model serving is responsible for keeping these agreements at the deployment boundary. Streaming decoding needs to handle cross-chunk byte state, truncation must be performed according to actual token length, and version management must ensure consistency of weights, vocabulary, normalization, and templates. Online monitoring should be able to detect whether these interfaces drift.

Meeting the standard of practical applicability means that, when faced with a multilingual text and a model package, you can trace each step from raw bytes to token IDs, manually compute a BPE merge based on corpus frequency, and point out the interfaces that must be revalidated after changing the vocabulary, chat template, or normalization rules. In this way, tokenization is no longer just a function call, but a model input-output protocol that can be explained, reproduced, and validated.