Positional Encoding and RoPE: Making Attention Know Order and Distance

Self-attention receives a set of token representations and generates query Q, key K, and value V from each token's content separately. The dot product between query and key determines how attention is distributed among positions, and then these weights are used to mix V; the output is still a set of representations that correspond one-to-one with the input positions. The problem is that if the entire process only looks at content, the model has no independent coordinates to distinguish “this token is at position 1” or “it is at position 7”.

Imagine simultaneously rearranging all input tokens with the same permutation. The Q, K, and V generated from content will have their rows rearranged in the same way; the rows and columns of the attention score matrix will also be rearranged accordingly; and the final weighted result on V will appear in the output according to the same permutation. In other words, however the input is synchronously permuted, the output is synchronously permuted. This property is called permutation equivariance: the model can still maintain the correspondence between each output and the corresponding input, but by this mechanism alone, the position number itself does not become usable information.

Permutation equivariance is different from permutation invariance. Permutation invariance means that no matter how the input is rearranged, the overall output is exactly the same, commonly seen when all positions are aggregated into a single result; permutation equivariance, on the other hand, allows the output to be rearranged along with the input. Calling position-less attention permutation invariant ignores the per-position correspondence that the sequence output still retains.

Position vectors, position biases, and geometric position transformations can all add coordinate information to attention, making the same content at different positions interact differently. Causal masking also introduces directional structure: a position can only look at the past, not the future, so autoregressive attention is not a completely unordered system in the strict sense. However, “can only look at the past” mainly specifies the visible range and is not sufficient by itself to independently express precise distance, periodicity, or finer sequential patterns; these still require dedicated position mechanisms.

“Positional encoding” does not refer only to a single additive vector, nor is it a single encoding format. More accurately, it represents a class of designs that inject token coordinates, position differences, or distance relationships into attention computation. Different methods choose different injection locations, and therefore differ in how they change attention, the relationships they are good at expressing, and the boundaries when extending beyond the training range.

When position is added to token representations, attention receives a “combination of content representation and position representation”. A learned absolute position table directly stores a vector for each index; expressive power comes from the independent parameters of each index. Once an index exceeds the table's range, there are no corresponding trained parameters. Sinusoidal absolute encoding likewise injects position at the representation layer, but the coordinates are computed by a deterministic multi-frequency function, so it is not subject to the hard boundary of a finite parameter table. Computable does not mean usable: the model has only learned how to interpret these coordinates within the range it experienced during training; the function values outside the table exist, but that does not guarantee the model can use them correctly.

When position is added to attention scores, positional relationships do not need to be mixed into token content first. Relative bias can adjust logits based on the difference between two positions, while ALiBi directly changes the tendency for different keys to be attended through a distance penalty. Their input is the positional relationship, and their output is an increase or decrease to the original content scores, so how distance affects attention is structurally explicit and more naturally extends to larger position differences.

RoPE takes a third path: it does not add to embeddings, nor does it add a separate bias to logits, but instead rotates Q and K by position. When the rotated vectors undergo dot product, the content similarity naturally carries the relative phase between the two positions. Its rotation formula can be applied to positions outside the training range, but at distant positions untrained phase combinations will appear, and limited frequency resolution also restricts distinguishable distances.

Therefore, when evaluating a position mechanism, one needs to look at both where it injects coordinates and how that injection propagates to the attention result. Representation-layer injection, score-layer injection, and Q/K geometric transformation can all make attention aware of position, but “where the formula can extend” and “what the model can reliably interpret within the extended range” are two different things.

Sinusoidal positional encoding converts a position index into a multidimensional vector that can be added to the token representation. For the i-th group of dimensions, it uses a pair of sine and cosine coordinates:

PE(pos, 2i) = sin(pos / 10000^(2i/d))

PE(pos, 2i + 1) = cos(pos / 10000^(2i/d))

Here, PE is the position vector, pos is the token position index starting from 0, d is the total dimension of the position vector, and i indicates which sine–cosine dimension group; 2i and 2i + 1 are the two coordinates of the same group. The constant 10000 is used to control the range of frequencies covered from fast to slow across the whole set, and does not represent the maximum sequence length.

When i is fixed, sin and cos can be regarded as the two coordinates of the same circular pointer. Each time pos increases by 1, the pointer rotates through a fixed angle around the circle. Lower dimensions have smaller denominators, so the angle changes rapidly with position and is therefore more sensitive to local displacement; higher dimensions have larger denominators and change more slowly, providing coordinate information over longer scales. The final position vector is not the reading of a single pointer, but a “position signature” jointly given by a set of pointers with different rotational speeds.

When only a single frequency is used, the pointer returns to the same coordinates after completing one full circle, and positions separated by one or more periods become confused. Combining multiple frequencies makes different dimensions repeat at different speeds: even if a fast dimension has already returned to its original value, other slower dimensions are usually still at different phases. The combined repeat condition is harder to satisfy simultaneously, so positional confusion is pushed farther away. Fast dimensions are responsible for fine-grained discrimination, slow dimensions provide large-scale reference, and together they handle localization.

This construction also has an exploitable translation structure. According to the sum and difference identities for sine and cosine, after shifting a position from pos to pos + Δ, the new sin/cos coordinates at the same frequency can be obtained as a linear combination of the original two coordinates; the combination coefficients depend only on the shift Δ and the frequency of that dimension. Thus, the absolute coordinates contain an implicit regular structure for relative displacement, giving the model an opportunity to learn “how far apart two positions are.”

The boundary is that being able to compute the encoding for any pos mathematically only shows that the function is still defined beyond the training length. It does not guarantee that a finite-layer network can stably extract the required multi-frequency relationships, nor does it guarantee that finite training data has already taught the model to interpret unseen position ranges. Computable out-of-table coordinates are not equivalent to reliable length extrapolation.

RoPE does not add position as an independent vector to the content; instead, it rotates the query Q and key K according to position. First consider only one pair of dimensions in two-dimensional space: q is the query's content vector, k is the key's content vector; m and n are the position indices of the query and key, respectively; with each step forward in position, the vector rotates by θ. Using R(α) to denote the two-dimensional rotation matrix, the query at position m is R(mθ)q, and the key at position n is R(nθ)k.

Their dot product can be written as:

(R(mθ)q)ᵀ(R(nθ)k) = qᵀR(mθ)ᵀR(nθ)k = qᵀR(−mθ)R(nθ)k = qᵀR((n − m)θ)k

The first step applies the transpose to the product; the transpose of a rotation matrix equals the reverse rotation, so R(mθ)ᵀ = R(−mθ); angles of successive rotations can be added, so −mθ and nθ combine into (n − m)θ. The part of the absolute rotation shared by the query and key cancels out as a result, and the only remaining position factor in the dot product is how much more the key has rotated relative to the query.

Take q = k = (1, 0), and let θ = 60°. Then the two content vectors originally have the same direction, so the dot product simplifies to:

qᵀR((n − m)θ)k = cos((n − m)θ)

When m = n = 0, the relative rotation is 0°, the two vectors are exactly aligned, and the dot product is 1. After moving the key to n = 1, the relative rotation is 60°, and the dot product drops to 0.5. Shift both the query and key later by two positions, from (m, n) = (0, 1) to (2, 3); although the absolute positions have changed, the position difference is still 1, so the dot product remains 0.5. If the position difference increases to 2, the relative angle becomes 120°, and the dot product becomes −0.5. These numbers show how the same pair of dimensions modulates the content score according to relative displacement.

This example uses q = k = (1, 0) only to make the result directly equal to the cosine value. Real RoPE groups the high-dimensional q and k into pairs, rotates each group at its own frequency, and preserves the original directional relationship of the content vectors within each group. Therefore, the actual dot product is not determined solely by the position difference; more precisely, the match between the contents undergoes a rotation controlled by the relative position, and the shared absolute displacement cancels out in the dot product of each group.

Draw a pair of dimensions as vectors in a two-dimensional plane, and RoPE appears as a pointer rotating around the center. The position index determines the number of rotations: moving from one position to the next, the vector continues rotating at the fixed angular velocity of that dimension pair. If the figure draws three positions successively, you will see three arrows for the same content vector lying on the same circle, with directions changing as the position advances.

Rotation preserves vector length. If the original vector is (x, y), after rotation both coordinates change, but the distance to the origin is still $\sqrt{\text{x² + y²}}$Therefore, when RoPE injects position, it changes the vector's phase or direction, not the magnitude of the vector in that pair of dimensions. For attention, the relative orientation of Q and K affects the dot product, and the common rotation does not magically enlarge or shrink the vectors themselves.

A high-dimensional vector is split into multiple pairs of two-dimensional coordinates, and each pair can be viewed as a pointer. Different dimension pairs use different frequencies: fast pointers produce noticeable phase differences within relatively short position changes, while slow pointers change slowly over longer distances. When attention compares Q and K, it actually uses the phase relationships formed by many pointers rotating at different speeds, so it can simultaneously carry information at different distance scales.

RoPE is usually applied to some or all head dimensions of Q and K, but does not rotate V. In this way, position affects “who should be attended to” through the Q/K dot product, while V remains the content that is weighted and aggregated. Exactly how many dimensions are rotated and how the high-dimensional coordinates are paired into two-dimensional pairs are part of the model definition.

At implementation time, it is not enough to ensure that the rotation formula looks equivalent; it must also be completely consistent with the conventions used by the training weights. The rotation ratio determines which head dimensions participate in position transformation, the base frequency determines the speed range of each pointer, the dimension pairing order determines which coordinates together form a two-dimensional plane, and the complex or real layout determines how these coordinates correspond in memory and computation. Any mismatch may cause the phase applied during inference to differ from that during training; even if the vector length is preserved, the positional relationships interpreted by attention have changed.

When a task directly cares about how far apart two positions are, you can add the distance relation to the attention's raw scores without first mixing position into the token representation. A common form is:

score(i, j) = qᵢ · kⱼ / $\sqrt{\text{d}}$ + b(i − j)

i is the query position that initiates attention, j is the key position being looked at; qᵢ and kⱼ are the query vector and key vector for these two positions, respectively; d is the key vector dimensionality. qᵢ · kⱼ measures content matching, divided by $\sqrt{\text{d}}$ is used to control the dot-product scale. b(i − j) is determined only by relative position, used to increase or decrease the score of a position pair beyond the content score.

score(i, j) is the attention logits before entering softmax. The bias takes effect before softmax, so it directly changes the relative competition among different keys: a larger bias raises the tendency for the corresponding position to receive attention weight, and a smaller or more negative bias lowers it. The content term answers “do these two representations match”, and the position term answers “how should this relative distance affect the match”; the two are added together to jointly determine attention allocation.

Relative position representations can learn a bias for each distance, or they can learn relative position vectors that participate in computation. To keep parameters and the usable distance range under control, distances can also be clipped: position differences beyond a certain range share the same class of representation. This uses i − j, not two independent absolute indices, so when the query and key are shifted together as a whole, the relative position term remains unchanged.

ALiBi further expresses this relation with a linear distance penalty. Each attention head uses a different slope; in causal attention, more distant historical positions typically get a more negative bias. A steeper slope biases a head more strongly toward nearby positions, while a gentler slope lets more distant positions keep more competition opportunity, so different heads cover different distance scales. It does not need to maintain a separate vector table for every absolute position, and the distance rule can continue to apply to larger position differences.

This design also introduces an explicit inductive preference: the farther the distance, the less relevant they usually are. If the task structure matches this tendency, the linear penalty can give attention a direct and stable distance signal; if distant relationships should not be naturally penalized, or if relevance has a different structure over distance, this preference may not be appropriate. The value of relative bias and ALiBi is not to eliminate content matching, but to explicitly specify on the content logits how distance should participate in the decision.

What long-context scaling needs to solve is not “can the positional formula compute 64K”, but how to ensure that signals generated by 64K positions still fall within the range the model can interpret. If the model only saw an 8K range during training, directly pushing positions to 64K will cause RoPE’s phases to enter many untrained regions. Positional interpolation chooses to compress coordinates: in the simplest correspondence, the new 64K range is mapped to the original 8K range, equivalent to compressing the new positions by a factor of 8 before computing the rotation phases.

The benefit of compression is coverage. The farthest new positions no longer correspond to phases entirely outside the training range; the positional changes the model sees are closer to the regions familiar from training. The cost is resolution: the limited angular range originally allocated to 8K positions now needs to accommodate 64K positions, so the phase difference between adjacent tokens shrinks. The denser the positions, the harder it is to distinguish different positions in rotation space, so there is a direct trade-off between short-distance discrimination and long-distance coverage.

Not all scaling methods uniformly compress all coordinates. Some methods adjust RoPE’s base frequency, changing the rotation speed of each dimension’s pointer at the source; some methods scale per-dimension in segments, letting different frequencies bear different degrees of extension. What they change at the core is still “how positional indices are converted into rotation phases for each pair of dimensions.” No matter which form is used, covering longer indices is not a free increase in coordinate capacity, but a redistribution among untrained phases, frequency range, and positional resolution.

To judge whether scaling is successful, we need to distinguish evidence at different levels:

Perplexity measures the model’s overall uncertainty about the next token, and generally lower indicates better language modeling. But in long texts many tokens can be predicted using only nearby content; these easy positions participate in the average, making overall perplexity look normal even if the model is not effectively using distant information. Therefore, being able to run at the target length and having no obvious deterioration in perplexity only indicate that scaling has crossed a relatively low-level threshold.

Stronger evidence requires progressively ruling out shortcuts. Retrieval tests should vary the depth at which the target information is placed, to avoid the model only adapting to a fixed template; reasoning tests should require the model to combine multiple pieces of evidence and use order, so that single-point copying is insufficient to complete the task. Only when consistent results are obtained from runnable, language modeling, retrieval, and reasoning utilization can we distinguish “the coordinate range was expanded” from “the model truly gained usable long-context capability.”

The position function can compute values for any integer position, which only shows that the coordinate system does not immediately break down mathematically; it does not show that long-range relationships remain distinguishable, or that attention continues to work in the way it was trained. One root cause of long-range quality degradation is that a finite set of frequencies must simultaneously handle local discrimination and long-range localization.

Fast frequencies produce a large phase change with each position step, making them suitable for distinguishing adjacent or nearby positions; but as distance grows, they wrap around many times, and different displacements can land on similar phases, creating ambiguity similar to periodic aliasing. Slow frequencies rotate less and can cover farther distances without repeating as often, but they are not sensitive enough to adjacent positions: the phase difference between two neighboring positions in a slow-frequency dimension can be very small. Combining multiple frequencies lets fast and slow components complement each other, making it harder for multiple displacements to be confused across all frequencies at once, but it cannot infinitely eliminate the resolution limits of a finite frequency representation.

The model must also learn how to combine these frequencies. Training covers only a limited distribution of lengths, so attention layers form their interpretation of phase combinations, content signals, and distance patterns within a limited range. Beyond this distribution, even though every frequency can still be computed, new phase combinations may not have been seen during training; attention patterns, normalization scales, and the behavior of retrieving targets from a large set of candidate content may all change as a result. The result may appear as insufficient scores for relevant distant content, similar scores assigned to wrong positions, or multi-frequency combinations that were stable within training length becoming unstable.

Therefore, “how long RoPE supports” is not a constant determined only by the rotation formula. The base frequency determines the range of rotation speeds across dimensions, scaling rules change the mapping from position to phase, the maximum training length and length sampling determine which phase and distance combinations the model has seen, fine-tuning data affects whether it learns to use information at the target length, and the KV cache implementation must ensure that the phases of historical keys are consistent with the current position.

The criterion for what counts as “supported” also changes the answer. Merely requiring that the target length can run sets the lowest bar; requiring information retrieval in long text also requires examining long-range attention and content retrieval; requiring reasoning that depends on multiple distant pieces of evidence requires more stable position discrimination and combination abilities. When discussing context length limits, one must specify model configuration, training and scaling conditions, cache implementation, and evaluation goals, because an isolated length number cannot represent actual usable capability.

Autoregressive generation does not recompute all historical tokens at each step; instead, it caches the already obtained K and V in the KV cache. When generating the next token, it only computes the representation of the new token, then lets its Q attend to the historical K in the cache, and uses the weights to read the historical V. In this way, whether the cache contents correspond to the correct positions becomes part of the attention computation.

The Q and K of a new token must use its position index in the full sequence, not directly adopt the local index in the current mini-batch tensor. Even if the prefix has already occupied several positions and the input tensor in the incremental step may contain only one new token, its position should not restart from 0. If the position ID is off by one, RoPE rotates the new Q/K by the wrong angle; the relative phase with the historical K in the cache shifts accordingly, and the distance relationship expressed by the attention logits no longer corresponds to the real sequence.

The danger of this error is that it usually does not break tensor shapes. The number of cached K/V entries, head dimensions, and matrix multiplications remain valid, and the program can continue generating; only the positional relationships are silently rewritten, ultimately manifesting as quality degradation. It is hard to detect the problem just by looking at whether it runs, cache length, or output shape.

Prefix reuse makes a new sequence continue counting from an existing cache length; sliding windows may retain only part of the history, but cannot arbitrarily change the positional semantics of retained tokens to local numbering; paged caching splits logically contiguous history into different physical blocks; left padding makes the start of valid tokens in a batch differ from tensor indices; when sample lengths within a batch differ, the next position for each sample also differs. These situations all separate the "index in the tensor" from the "position ID in the full sequence," so they are most likely to produce a starting-point offset.

A reliable acceptance method is to compare two equivalent computation paths: one computes the full sequence forward in one pass, and the other computes forward token by token using the cache. Under the same model and input, the logits at corresponding positions should be numerically close. Tests should not cover only a single short sequence; they should also vary sequence length, padding direction, and cache chunking so that prefix reuse, different batch lengths, and paged or chunked paths are all checked.

When reproducing and troubleshooting, it is also necessary to record simultaneously the RoPE base frequency, scaling type, dimensions involved in rotation, and the position ID generation rule. As long as any one of these is inconsistent between the full forward pass and the cached forward pass, you may get results that have the correct shape but wrong positional semantics. The correctness of the KV cache is therefore not just "caching the same values"; it also requires that every cached item and every new Q/K be in the same coordinate system as training and the full sequence.

Evaluating positional ability cannot just ask "Has the model ever answered correctly on the longest input?" Instead, it must separate content capability, changes in position, and reasoning requirements. A valid evaluation uses the results of the same task under different lengths, different evidence positions, and different computation paths as input, observing how answers, success rates, or logits change with these factors. The goal is to confirm that the model's output is indeed driven by positional and sequential information, rather than by fixed templates, local cues, or accidental hits.

First, establish a short-length baseline. Verify that the model can already complete the task in a short context with which it is familiar; this avoids misdiagnosing deficiencies in reading, retrieval, or reasoning as failures of length extrapolation. If short-length performance is also poor, increasing the length alone cannot indicate that there is a problem with the positional mechanism.

Next, scan total length and evidence depth simultaneously. For the same content, place key evidence at the beginning, middle, end, and random positions separately; total length also varies across tests. This makes it possible to see whether failures occur as context length grows or are concentrated in specific positional regions. Fixing evidence at only one depth would mistake the model's adaptation to that template or position for a general capability.

Order comparisons are used to verify whether the model actually reads relations. Shuffle the order of multiple pieces of evidence, or swap the subject and object in a statement; the correct answer should change correspondingly as the semantic relation changes. If the input order or roles have changed but the model still gives the original answer, it may be relying on content keywords or answer preferences instead of using the structure carried by position.

Task difficulty should also be increased step by step from copying to composition. Single-needle retrieval only requires retrieving one distant piece of information; multi-needle aggregation requires simultaneously extracting evidence from multiple positions; temporal order tasks require recognizing earlier versus later; cross-paragraph reasoning requires connecting scattered content. Step-by-step testing can locate at which level the model fails: being able to copy a single point does not mean being able to use long contexts for order-sensitive or multi-evidence reasoning.

To isolate positional factors, control content difficulty. Keep the core content and question unchanged, and change only the filler length or the position of the evidence; then output differences are more likely to come from positional processing rather than from differences in sample difficulty. At the same time, compare full forward passes with incremental forward passes that use KV cache, aligning logits at corresponding positions of the same sequence; if the two paths disagree, first investigate the cache and position IDs rather than directly interpreting it as a change in model capability.

The final report should present a distribution, not just announce a maximum usable length. Compute success rates separately by position, total length, and task type, and provide confidence intervals, to show where performance degrades and how stable the results are. Only when the short-length baseline holds, position and order comparisons produce expected changes, compositional tasks remain reliable as length increases, and full forward and cache paths agree is there evidence that the model not only can accept long inputs but actually uses the positional relationships within them.

Positional capability from input to verifiable behavior must pass through a causal chain with no skipped steps:

Discrete sequence → coordinate injection → position-aware attention scores → learning to interpret position during training → length extension and correct inference implementation → stable performance on long-distance tasks

The tokenizer first converts text into a discrete token sequence with an order. The order exists in the sequence arrangement at this point, but each token's content embedding does not automatically carry a coordinate of 'which position I am.' If self-attention generates Q, K, and V from content alone, then synchronously rearranging the input only synchronously rearranges the output; the model can match content, but it lacks an independent structure for distinguishing absolute positions and precise distances.

The positional mechanism adds this structure to the computation. Absolute encoding incorporates position coordinates into token representations; relative bias directly modifies attention logits based on position differences; RoPE rotates Q and K according to position. For RoPE, the rotated dot product is affected by both content direction and relative phase, the common absolute rotation cancels out, making the displacement between two positions enter the content matching process. At this step, the model has obtained a computational structure that can express position, not a guarantee that it has already mastered position regularities.

The training length distribution determines which frequency, phase, and distance combinations the model has actually seen. The model needs to learn through training how to use these signals for local order, long-distance retrieval, and cross-position relationships. The formula remaining computable at farther indices does not mean the model has learned to interpret those regions; the potential expressive power of a mechanism must be converted into usable behavior through the corresponding training distribution.

When context extends beyond the training range, scaling can map more position indices to a computable or more familiar phase range, but it redistributes positional resolution. Compressing coordinates increases coverage while reducing the phase difference between adjacent positions; adjusting base frequency or piecewise scaling also changes the distance scale that different frequencies are responsible for. Therefore, extending the window is not simply enlarging an upper limit but a rebalancing among local discrimination, long-range coverage, and untrained phases.

The inference implementation must keep this coordinate system unchanged. When using KV cache, historical K/V are reused, and a new token's Q/K must continue the full sequence's position ID and use the same base frequency, scaling, rotation dimensions, and layout as in training and full forward. As long as the position start or rotation configuration is misaligned, tensor computations may still run normally, but the relative phase in attention is already distorted.

The final evidence comes from an evaluation matrix of length × position × task, not the maximum window declared in the configuration. Total length should vary, evidence should be placed at different depths, tasks should extend from single-point retrieval to multi-evidence, order-sensitive, and cross-segment reasoning, and at the same time verify that full forward and cache forward logits are consistent. Only when coordinates are correctly injected, training has learned to use them, scaling has not destroyed key resolution, inference implementation keeps positions consistent, and multiple long-distance tasks show stable performance can we say the model really uses distant information.

Positional encoding is situated in a larger knowledge chain: it receives the discrete sequence produced by the tokenizer, plays a role in the Q/K score computation of self-attention, is embedded in the overall Transformer architecture, and undergoes implementation and capability validation in KV cache and long-context evaluation. Continuing along these dependencies allows you to connect the "position formula" to the model's complete behavior from input to output.

"Tokens and Tokenization" determines the basic unit on the position axis. A position index does not directly count characters, words, or sentences in natural language; instead, it advances one discrete token at a time as the tokenizer outputs tokens. Only after clarifying how the sequence is segmented and numbered can you explain what a given position difference actually spans in the model input.

"Self-attention" explains which part of the computation position relationships enter. The QK dot product produces content-matching scores, masking limits which keys each query can see, and positional encoding, relative bias, or rotation then add coordinate structure to these scores. Understanding the division of labor among these three allows you to distinguish where content similarity, visible direction, and precise distance each come from.

"Transformer" puts attention back into the complete network. The position mechanism influences the information selection of attention, but a layer's output also passes through residual connections and feed-forward layers and continues to subsequent layers. The model's use of order and distance is the result of the position structure working together with the entire network; you cannot infer final capability from a single position formula alone.

"KV cache" connects position semantics with inference cost. Caching historical K/V avoids repeatedly computing existing tokens during autoregressive generation, but reusing the cache requires continuing the position IDs of the full sequence and maintaining consistent rotation configuration. The parts whose computation is omitted still belong to the original coordinate system; they must not be renumbered just because tensors are chunked, paged, or reused.

"Model Evaluation" is used to determine whether these mechanisms ultimately translate into capability. Long-context tests must control content difficulty while varying length, evidence position, and task structure to avoid mistaking sample content differences for position effects. Runnable, retrievable, order-sensitive, and multi-evidence reasoning are progressively stronger requirements; no single metric is sufficient to replace complete validation.

After mastering these connections, you should be able to accurately explain why positionless self-attention is permutation-equivariant, compute RoPE dot products by hand in two dimensions based on Q/K rotation, and design a validation matrix for models that claim to extend context. This matrix must simultaneously cover whether the target length can run, whether distant content can be retrieved, whether answers change accordingly when evidence order changes, and whether the model can combine multiple distant pieces of evidence to complete reasoning.