Attention Mechanism

Understanding a word in a sequence depends not only on the word itself but also on which words in the context it is related to. For example, in a sentence containing "Xiaohong" and "she", to correctly understand "she", it must be linked to the preceding "Xiaohong". This associated word may be several positions away, and what needs attention changes with the sentence content, so it cannot be resolved by fixed rules such as "only look at the previous few words".

The attention mechanism enables the model to dynamically decide, based on the current content, which positions in the sequence should be attended to for each position. The input is a set of representations for all words in the sequence. When processing a word, the model first compares its representation with those of other words one by one to obtain content relevance, then converts these relevance scores into weights and aggregates the information carried by each position according to the weights. Positions with higher relevance contribute more to the aggregated result; the final output is a new representation for each word that combines information from the entire sequence.

Therefore, a higher weight for a word indicates that it is more relevant to the current word in the current context, not that it is closer in position. The causal chain can be summarized as: word representation → content comparison with other positions → different attention weights → information aggregation weighted by these weights → a new context-aware representation. This process simultaneously addresses two issues: "distant information may be important" and "the target of attention must change with the content."

Before attention mechanisms became mainstream, recurrent networks typically processed words one by one in order and passed memory forward along time steps. The longer the chain through which information passes, the harder it is to retain earlier content; different time steps depend on each other, making parallelization difficult. Gated recurrent networks can alleviate the information-retention problem, but cannot eliminate the two structural constraints of long propagation paths and sequential computation.

Attention provides a mechanism for learning how to associate content and aggregate information, not a guarantee of correctness. That weights are produced by content comparison does not mean the model will necessarily place the highest weight on the truly correct position; whether the model can learn appropriate attention relationships still depends on training outcomes and input conditions.

To let the model decide which positions the current word should attend to, each word first generates three vectors with different purposes from its own input representation: Query, Key, and Value. Query represents "What am I looking for?", Key represents "What can I be matched by?", and Value is the content that actually participates in aggregation after a successful match. Taking the pronoun "she" as an example, its Query can look for possible referents; each word in the sentence provides its own Key for comparison and simultaneously prepares a corresponding Value as information that can be taken away.

Attention computation consists of three steps: matching, normalization, and aggregation. In the first step, the current word's Query is dotted with the Keys of all visible words, producing a row of matching scores. The larger the dot product, the better this Query–Key pair matches in the current attention computation. In the second step, the scores are passed through softmax, converting them into a set of non-negative weights that sum to 1; positions with larger scores receive larger weights. In the third step, these weights are used to compute a weighted sum of the Values at all positions, producing a new representation of the current word that incorporates global information.

Arrange the vectors of all visible tokens into matrices Q, K, and V respectively. The whole process can be written as:

Attention(Q, K, V) = softmax(Q·Kᵀ / $\sqrt{\text{d}}$) · V

Q·Kᵀ computes the dot products between every query and every key simultaneously. The T in Kᵀ denotes transpose, so that the matrix dimensions correspond to pairwise comparisons of "each Query against all Keys". d is the dimensionality of the Query and Key vectors; dividing by $\sqrt{\text{d}}$ is to prevent the dot-product values from becoming too large as vector dimensionality increases. Softmax normalizes the row of scores corresponding to each Query, producing attention weights; multiplying by V then completes the weighted sum over all Values. The inputs are Q, K, and V, and the output is a new representation for each Query after aggregating its visible Values.

When processing "she", its Query is compared with the Keys of "Xiaohong" and every other word in the sentence. If "Xiaohong" receives the highest weight after softmax, the new representation of "she" will absorb more of "Xiaohong"'s Value. The thickness of the lines in the figure corresponds to the weight magnitude: the line to "Xiaohong" is thicker, indicating that it contributes more information in this aggregation.

The final scores are not determined solely by distance between words. Query–Key content matching is the core source; if the model also uses masking, positional bias, RoPE, or local windows, these mechanisms affect the scores or the visible range before softmax, allowing position and distance to contribute to the result. Masking can make certain positions invisible, local windows restrict the comparable range, and positional mechanisms change the matching conditions at different relative positions.

A word receiving a high weight only means that it is a better match under the current attention head, current input, and current visibility conditions; it is not an explanation probability, nor can it alone prove that it is the only cause of the model's final conclusion. Attention weights describe how the Value aggregation proportions are distributed in this step, and their explanatory scope should be limited to the interior of this computation.

Suppose a query q = [1, 0], two keys k₁ = [1, 0], k₂ = [0, 1], key dimension d = 2; the values corresponding to the two keys are v₁ = [2, 0] and v₂ = [0, 4], respectively. This set of inputs can fully demonstrate how scaled dot-product attention goes from "matching degree" to "information mixing result".

First, compute the dot product of the query with each key:

[q·k₁, q·k₂] = [1×1 + 0×0, 1×0 + 0×1] = [1, 0]

The first score is higher, indicating that q matches k₁ better. Next, use $\sqrt{\text{d}}$ = $\sqrt{\text{2}}$ scale these two scores:

[1, 0] / $\sqrt{\text{2}}$ ≈ [0.707, 0]

Scaling does not change the relative order of the two keys; it only controls the numerical scale of the scores. Then apply softmax to the scaled scores. Softmax first exponentiates each score, then divides by the sum of all exponentials:

softmax([0.707, 0]) = [exp(0.707), exp(0)] / [exp(0.707) + exp(0)] ≈ [0.670, 0.330]

In this way, the original matching scores are converted into two weights that sum to 1. Since k₁ is the better match, the corresponding v₁ receives the larger weight of 0.670; v₂ receives the smaller weight of 0.330. Finally, use these weights to compute a weighted sum of the Values:

0.670v₁ + 0.330v₂ = 0.670[2, 0] + 0.330[0, 4] = [1.340, 0] + [0, 1.320] = [1.340, 1.320]

The final output is not a direct copy of the best-matching v₁; instead, it mixes the two Values according to the matching weights. This clarifies the roles of Key and Value: comparing the Query with the Keys determines "how much to take" from each position, while the Value determines "what to actually take" from that position. The complete chain from input to output is: dot product of query and key → scale by dimension → softmax normalization → mix the Values using the resulting weights → output a new vector representation.

The 0.670 and 0.330 values here are only coefficients for this one information-mixing step in the current layer and current attention head. A higher weight indicates that the corresponding key matches the query better, but it is not an explanatory probability and cannot prove that the corresponding token is the sole reason for the model's conclusion; residual connections, other attention heads, and later layers will continue to change the representation.

In self-attention, the Query, Key, and Value are all produced from the word representations of the same sequence, so each word in the sequence can both issue a query and provide keys for other words to match against and values to be aggregated. It addresses how the words within a sentence refer to one another, thereby forming contextual understanding of roles and relationships.

The input is the representations of all words in a sentence. For each word, the model generates its Query and compares relevance with the Keys of all words in the same sentence; after obtaining weights, it then computes a weighted sum of those words' Values. Because this process is carried out at every position, the output still corresponds to each word in the sentence, but each output is no longer just the isolated representation of that word; instead, it is a new representation that incorporates the information visible within the sentence.

For example, when "Xiaohong" and "she" appear in the same sentence, the Query of "she" will match the various Keys in the sentence, including "Xiaohong". If "Xiaohong" receives a larger weight, the output representation of "she" will absorb more of the Value of "Xiaohong", thereby establishing the association between the two in the current sentence. When a word mainly looks toward another word, this indicates that they are more related in this within-sentence attention computation; this describes an information aggregation relationship, not a fixed lexical relation independent of context.

The "self" limits the source of information: Query, Key, and Value come from the same sequence. If the Query comes from one sequence while the Key and Value come from another sequence, this variant is called cross-attention. Its matching, normalization, and weighted-sum mechanisms remain unchanged; what changes is the source of the querying side and the information being queried. For example, in translation, the Query of each word in the target sequence can match the Keys of the source sequence and aggregate information from the corresponding Values in the source.

Therefore, self-attention is suitable for modeling cross-references within a sequence, while cross-attention is suitable for allowing one sequence to selectively read another sequence. The boundary between the two is determined by where Q, K, and V come from, not by the computational form of dot product, softmax, or weighted sum.

A classic RNN processes sequences serially across time steps: step t+1 must wait for the state from step t to be produced, and information from earlier positions can only reach later positions along a chain of step-by-step propagation. Attention changes both of these structural constraints. It receives representations of all tokens in the entire sequence, allows each position to directly match all visible positions, and simultaneously outputs new representations for each position that fuse context.

For information separated by n positions, the interaction path in a classic RNN usually grows with the distance, requiring propagation over about n time steps. Global self-attention allows any two visible positions to interact directly within the same layer: one position's Query can directly match another position's Key, and its Value is weighted and aggregated. In the figure, information in the RNN propagates step by step along positions, while attention establishes direct connections between visible positions, reflecting exactly this path difference.

Shorter interaction paths are beneficial for learning long-range dependencies, because distant information does not have to pass through a series of intermediate states before it can contribute to the current representation. But a “shorter path” does not mean that information is unchanged, nor does it guarantee that the model necessarily learns the correct associations. Information still goes through vector projection, score scaling, softmax, Value aggregation, and subsequent layers; attention weights may also fail to land on truly useful positions. Therefore, what it improves is the structural condition for long-range interaction, not an automatic guarantee of correct understanding.

Another turning point comes from parallelism. RNN time steps have dependencies from earlier to later, making it impossible to complete computations for all positions simultaneously in the same layer; in attention, matching and aggregation for each position can be computed in parallel, without waiting for the previous position to complete. This allows hardware to process many positions in a sequence at the same time, making fuller use of GPUs.

This parallelism directly affects whether model and data sizes can continue to scale up in engineering terms. If training were still constrained by step-by-step serial dependencies, the larger the model and the more data, the more likely training time would become impractical. Attention transforms the main interactions within a sequence into parallel whole-sequence computation, providing a key condition for training today's large models along the path of scaling up model and data size.

Attention is a turning point not merely because it can represent relationships between words, but because the same mechanism simultaneously alleviates two bottlenecks: using direct interaction within a layer to shorten long-range information paths, and enabling computation across all positions to be parallel. The former improves the conditions for learning long-range dependencies, while the latter makes large-scale training feasible from an engineering standpoint.

A single attention head uses one set of Query, Key, and Value projections to measure correlation between tokens, but the same pair of words may simultaneously have grammatical, referential, positional, and other relationships. Multi-head attention runs multiple attention heads in parallel within the same layer, providing multiple representation subspaces and multiple information routes for these different matching patterns, without having to squeeze all relationships into the same set of scores.

The input is the same set of word representations. Each attention head uses its own projections to transform the input into that head's Q, K, and V, and computes matching scores, softmax weights, and a weighted sum of Value in its own subspace. Because different heads use different projections, even with the same input they can form different attention distributions and extract different content from different positions. The outputs of the heads are then concatenated and passed through another projection, producing a new representation that fuses multiple perspectives.

This computational chain can be summarized as: the same set of input representations → each head projects into a different subspace → each head independently computes attention and aggregates information → concatenate the results of all heads → project into the layer's combined output. Multi-head mechanisms do not solve the problem of 'repeating the same result several times,' but rather give the model multiple simultaneous opportunities to learn relationships and transmit information.

After training, some heads may show fairly clear grammatical, referential, or positional patterns. For example, one head may often link pronouns to their referents, while another may pay more attention to neighboring positions. However, the architecture does not pre-specify 'which head handles which type of relationship'; if such specialization emerges, it is learned during training rather than being a rule written in by humans.

Multi-head attention also does not guarantee that every head forms a stable, independent, and easily interpretable role. Many heads may be redundant with each other, and the same head may not express only one relationship. Therefore, observing a certain attention pattern can serve as an analytical clue, but one cannot infer the function of each head merely from the architecture's name. What multi-head attention provides is the capacity for multiple subspaces and parallel routing; whether a clear division of labor actually forms must be determined through empirical analysis.

Global attention allows every token to interact directly with all tokens, and the cost comes precisely from this pairwise comparison. Let the sequence length be n; each position must compute matching scores against n positions, so the attention score matrix contains n×n = n² elements. The model takes representations of all positions as input, performs pairwise scoring, normalization, and Value weighting in turn, and finally outputs a new representation for each position; the score computation—and score storage in a naive implementation—both grow with n².

Quadratic growth means that when the sequence length doubles, this part of the cost becomes roughly four times as large. For 1 thousand tokens, about 1,000×1,000 = 1 million position pairs must be computed; for 100 thousand tokens, about 100,000×100,000 = 10 billion position pairs are required. This explains why long-context inference quickly becomes slower and more expensive: each time the context increases by one level, attention does not pay a linearly increasing cost.

n² describes the core cost of attention's pairwise interactions, not the model's total cost. Actual computation also includes Q, K, V projections, output projection, and feed-forward layers; specific memory usage and runtime are also affected by implementation details. FlashAttention can significantly reduce memory traffic and change the actual memory constant, but it does not reduce the computational complexity of global pairwise attention from the quadratic level.

Quadratic overhead directly limits the context window from being extended arbitrarily. The longer the window, the more position pairs can be compared, and the harder computation and naive memory requirements become to sustain. Long windows may also expose positional biases such as the “lost in the middle” effect, where the model makes better use of information at the beginning and end but underutilizes information in the middle. This phenomenon is related to multiple factors such as training distribution, position representation, and attention patterns, and cannot simply be reduced to “attention weights being diluted.”

Different optimizations make different trade-offs around this bottleneck. Sparse attention computes only some position pairs; sliding-window attention restricts each position's visible range to nearby regions, both starting by reducing the number of interactions; FlashAttention, by contrast, optimizes the execution process of global attention and reduces data movement. The first two may change visibility relationships and effective complexity, while the latter mainly improves engineering efficiency. They can all alleviate long-context costs, but they cannot be broadly understood as both preserving arbitrary global interaction and completely eliminating the quadratic cost.

To verify whether an optimization is truly effective, you need to check both efficiency and task capability. You should fix model weights, numerical precision, batch, and hardware, and record task accuracy, peak memory usage, throughput, and p95 latency at input lengths such as 1K, 4K, and 16K. You should also place key evidence at different positions in the sequence to check whether the speedup damages long-distance information utilization through truncation, local windows, or changes in visibility range.

If speed improves but accuracy on long-distance tasks decreases, you should first check the visibility range and mask to confirm whether key evidence can still be accessed by the query position, then compare the underlying kernel implementation. An attention heatmap can only show a particular weight assignment and cannot replace joint verification of accuracy, memory usage, throughput, and latency.

The starting point of sequence understanding is that a word's meaning often depends on other words in the sentence, and the truly relevant words may be far apart, while what is attended to also changes with content. Classical RNNs rely on state being passed successively along time steps; distant information requires a longer path, and computation is also constrained by dependencies between successive time steps. Attention reframes the problem as “how much information should the current word take from each visible position?”

Each position first produces Query, Key, and Value. The current Query takes dot products with all Keys, producing content-based matching scores; after these scores are scaled by dimension and passed through softmax, they become weights that sum to 1; these weights are then used to compute a weighted sum over all Values, generating a new representation for the current position. Thus, content matching determines “how much to take,” and Value determines “what to take,” closing the causal chain from original representation to contextual representation.

The role of softmax is not to select a single position, but to turn a row of scaled scores into weights that can be used to mix information. The position with the highest weight usually contributes the most, but the output can still be a mixture of multiple Values. The weights only describe the proportions of information aggregation in the current head and current layer, not causal proof that a particular token caused the final conclusion.

When Q, K, and V all come from the same sentence, this mechanism is self-attention. Every position in the sentence can match all visible positions in the same sentence, so relationships between words can be established directly within a single layer. Positional mechanisms and masking can still affect the final scores or the visible range, so “content-determined” means the core matching of the weights comes from Query–Key, rather than adopting the rule that “the closer the distance, the larger the fixed weight.”

This direct interaction simultaneously changes the information path and the computation method. Two visible positions far apart need not be passed successively along RNN time steps; they can interact directly in one attention layer. Computation at each position in the same layer also need not wait for the previous time step to complete, so it can be parallelized. The shorter path improves the structural conditions for learning long-distance dependencies, while parallelism makes scaling up models and training data more feasible in engineering terms. Together, these two points explain why attention could replace classical RNNs as the core mechanism for large-scale sequence modeling.

Multi-head attention further provides multiple sets of projection subspaces in the same layer. Different heads can learn different matching methods and information routing, and the results of each head are then concatenated and projected into a combined representation. Some heads may form specialized patterns for syntax, coreference, or position, but this division of labor comes from training, not from architectural presets; there may also be redundancy among multiple heads.

Directly looking at all positions also brings clear costs. Standard global attention of length n needs to form a score matrix containing n² elements, so computation and naive memory usage grow quadratically with length. This limits expansion of the context window, and long contexts may also exhibit position biases such as “Lost in the Middle”; the latter is also influenced by factors such as training distribution, position representation, and attention patterns, and cannot be attributed solely to quadratic complexity.

The entire mechanism ultimately forms a set of mutually constraining results: content-based dynamic matching lets the model select relevant information, short paths help distant positions interact, and parallel computation supports scaling; multi-head attention brings multiple representational perspectives, while global pairwise matching incurs quadratic cost. Transformer further packages this mechanism into a standard layer that can be repeatedly stacked.

Understanding attention mechanisms requires first mastering vectors and dot products, softmax, and neural networks. Vectors and dot products are the basis for computing matching scores between Query and Key, softmax normalizes a row of scores into weights, and neural networks provide the overall background of representation learning and parameter projection. Recurrent neural networks (RNNs) are also an important prerequisite, because the changes that attention brings to path length and parallelism become clear only when compared with RNNs' step-by-step passing.

The core concepts of attention mechanisms themselves, in order, include Query, Key, Value; self-attention and cross-attention; information path length and parallel computation; multi-head attention; and the quadratic cost caused by global pairwise matching. These concepts connect into a single logical chain: Q/K/V define how information is matched and aggregated; the sources of Q, K, and V determine whether it is self-attention within a sequence or cross-attention across sequences; direct interaction changes path length and supports parallelism; multiple heads provide multiple projection subspaces; and global interaction brings cost that grows quadratically with sequence length.

After mastering these core concepts, the closely adjacent extension directions are Transformer, positional encoding, vanishing gradients, context window, and lost in the middle. Transformer shows how attention composes into a stackable standard structure; positional encoding adds sequence position information; vanishing gradients help understand why long paths are difficult to train; the context window corresponds to the range of sequence a model can process at one time; and lost in the middle focuses on whether information at different positions in a long sequence is used equally effectively.

Further learning directions include scaling laws, large language models, and inference optimization represented by FlashAttention. Scaling laws and large language models examine attention's parallel capability in the context of increasing model and data scale, while inference optimization starts from practical execution cost and studies how to alleviate the efficiency pressure of attention in long contexts.