Context Window

Each time the model generates the next token, the total amount of input it can directly “see” is finite; this upper limit is the context window. Window size is usually expressed in tokens—16K is about 16,000 tokens—and it defines which information in a single call can directly affect subsequent generation: as long as it is still within the window, the model can use it; once it slides out, this generation can never directly use it again.

The window contains far more than the sentence the user just typed; it is all the content that will be fed to the model together in this call. Specifically, it includes: system prompts and instructions (for example, rules like “You are a customer service assistant…”), conversation history (every sentence said in previous exchanges), retrieved material (document snippets fetched by RAG), and the model’s own response being generated—output also occupies the window.

Therefore, the context window can be understood as the budget of content the model can directly see in a single call. It receives system rules, history, retrieved evidence, tool results, and output reservation; it is first counted by the tokenizer, then accepted, trimmed, or compressed according to product policy, ultimately forming a layout that does not exceed the upper limit.

When the sum of the parts exceeds the upper limit, the system must reject, trim, or compress. If the application adopts a “drop the earliest content” strategy, the information at the beginning will disappear first, but the specific behavior depends on the product implementation; not all services do this. Window size only indicates the capacity upper limit, not that information at every position in the window can be used with equal accuracy.

Figure 1 shows the budget allocation and over-budget rearrangement of a 16K token window. It emphasizes: the context window is a shared budget, not “up to 16K for input, output always counted separately”; actual counting rules vary by service, so when designing, input, output reservation, and tool round trips should be accounted for together.

Let’s make this account clear with a numerical example: in a 16K budget, rules 1K + history 7K + evidence 6K + planned output 4K = 18K, exceeding by 2K. If you bluntly cut off the earliest 2K, you might just drop the user’s name or a key constraint; a more robust plan is to compress history to 3K and retrieve only 4K of high-value evidence, total occupancy 12K, leaving 4K of margin for tool-result fluctuations or longer answers.

The real reason behind the opening example lies in a common misconception: does the model “remember” the conversation? The answer is no—basic inference calls are typically stateless. The model does not automatically retain the previous request; the application needs to put the relevant history, summary, or external memory back into the current input before the model can use them in this generation.

A chat product can save sessions on the server side, so the user experience feels like it “remembers,” but what actually participates in the current generation is still the content assembled into this context. If a particular sentence was neither retained, summarized, nor retrieved back, the model cannot use it this time.

Statelessness can be understood as: the model does not automatically retain the previous content on each call. It explains why the model appears to remember the conversation yet can suddenly forget—the so-called “remembering” is actually the application reassembling the relevant history, summary, or retrieved results into the input on each call, and then generating a response conditioned only on that content. If the response can mention old information, it means that information is still in the window this turn; if it was neither retained nor retrieved back, the model is not truly remembering it this time.

This clarification also explains two things in passing. First, why when using an API for multi-turn conversations you have to send the history each time—because the model does not store it itself. Second, why long conversations become increasingly expensive as they go on—the longer the history, the more tokens are resent each time, and billing is calculated based on tokens.

Since larger windows are more convenient, why don't vendors simply make them infinite? The most direct constraint comes from how attention is computed. For standard global self-attention, n positions form n² attention scores, so the associated compute and memory of a naive implementation therefore grow quadratically with length.

The source of the quadratic overhead is: every position must compute attention pairwise with every other position. Its input is the sequence length: first every position computes scores pairwise with all positions, then this score table is normalized, so the output computation and memory grow quadratically with length. If the length increases tenfold, the number of position pairs increases about a hundredfold, so the cost is superlinear—each time the window is stretched by one step, this cost must be paid. This is precisely the root cause of long contexts being slow and expensive.

We can do a quick calculation: with an input of 1 thousand tokens, attention computes about 1 million pairs; with 100 thousand tokens, it's about 10 billion pairs.

However, quadratic overhead is an important constraint, but it is not the only reason. Actual inference is also affected by the KV cache, bandwidth, positional extrapolation, and training distribution; sliding windows, sparse attention, and efficient kernels can change the complexity or constants. Therefore, the cost of making the window larger cannot be attributed solely to quadratic growth in attention; it is the result of multiple factors combined.

Models that also claim to support 16K—why are some requests slow, some hit OOM as soon as concurrency rises, and others finish but answer incorrectly? Because behind the same claimed length lie four different constraints. The inputs for diagnosing such problems are request length, concurrency, hardware, evidence position, and task result: first account for computation and I/O, cache capacity, training length, and assembly separately, then locate the actual bottleneck before giving separate judgments for the four types of problems. A successful API response only means the input was accepted; it does not mean the model can effectively use the entire length, so one type of metric cannot substitute for another.

The four constraints each constrain different things:

KV cache capacity can be estimated with an approximation: B_KV ≈ 2 × L × H_KV × d_h × n × b. Here B_KV is the KV cache size in bytes for a single request; the factor 2 means each layer stores both Key and Value; L is the number of layers, H_KV is the number of KV heads per layer, d_h is the dimension of each head, n is the number of tokens already cached, and b is the number of bytes per numeric element. This approximation is only for estimating capacity and does not include model weights, temporary workspaces, or memory fragmentation.

Plugging in a concrete configuration: 32 layers, 8 KV heads, head dimension 128, FP16, 16K tokens, gives 2×32×8×128×16000×2 ≈ 2.10×10⁹ bytes, about 1.95 GiB per request—and this still does not count model weights, activation workspace, or fragmentation. Using grouped-query attention, KV quantization, or paged management changes the numbers, but the relationship that “context length also consumes concurrency capacity” remains.

This leads to a core distinction: declared window ≠ effective context. The API being able to accept a certain length only proves that the input was not immediately rejected; to truly evaluate it, you need to plot accuracy curves by length, evidence position, task type, and language, and at the same time record first-token latency, KV usage, and failure rate.

Even if a model supports a very long context window, does cramming all the material in at once mean everything will be fine? Not at all. Long windows have two pitfalls.

First, Lost-in-the-middle Effect: information placed in the middle of the context is used much less than information placed at the beginning or end; the model tends to “miss” the middle. Second, noise dilution: cramming in a lot of irrelevant content dilutes attention and mixes in noise, making the answer worse instead.

Thus we can define effective context—the information that the model can actually access and use correctly in the current task. Its input is the candidate materials and their order: first, attention picks out clues from a large amount of content, then it answers based on that, outputting an answer with evidence. Placing the same material at the beginning, end, or middle significantly changes the result, showing that both position and noise are at play; a single correct answer does not mean the entire window is used reliably.

In one sentence: long context is a capability, not a license to “mindlessly cram things in”. What you put in, how much you put in, and where you put it matter more than “how much can fit”—put key material at the beginning or end as much as possible.

When the content to process exceeds the window, or a long conversation starts to lose memory, a set of methods can be used for window governance. Window governance is a set of methods for deciding what to keep, retrieve, compress, and order when content exceeds budget: its inputs are over-budget materials, task goals, and failure cost. First protect irreplaceable rules and the current task, then retrieve high-value evidence, compress recoverable history, and leave room for output, ultimately producing a usable window layout. Only when answer quality, evidence coverage, and cost improve together does it indicate that governance is effective; at the same time, be aware of its boundaries—summaries may miss details, and retrieval may miss recall.

There are four specific methods, mutually complementary.

Only include relevant content (RAG): do not cram the entire knowledge base in; instead, retrieve a small number of fragments most relevant to the current question and feed them to the model. It turns the problem of 'whether it fits' into the problem of 'which few to include.'

Carefully allocate budget (context engineering): decide what exactly to put into the window for each call and in what order. This is higher-level than polishing a single prompt; it is the design of the overall context layout.

Compression and summarization: compress overly long history into a summary and bring it along, freeing up window space for the current task and new evidence.

External memory: let the Agent store information that cannot fit in the context window into external storage and retrieve it when needed, trading storage for window space.

String together the previous steps and you get a causal chain that runs from the shared budget of a single call to statelessness, computational cost, effective utilization, and external memory.

The context window is the upper limit of tokens that can be seen in a single call. It contains the prompt, history, retrieved material, and output; content outside the window cannot be seen. The model itself has no memory: "remembering earlier parts" is only because those earlier parts remain in the window and are fed back in by the application; once they are squeezed out of the window, they are truly forgotten.

The window cannot be infinitely large: the quadratic cost of standard global attention, KV cache, hardware, positional representations, and training length together form constraints. Looking further, computation, KV capacity, effective utilization, and application assembly are four distinct bottlenecks; a claimed length cannot substitute for them.

Even if the window is long enough, filling it up does not mean using it well: the lost-in-the-middle effect and noise dilution can both make long context "look like it can fit, but actually be unusable." The responses therefore split into four paths: RAG includes only the relevant chunks, context engineering allocates budget, compression and summarization free up space, and external memory stores what cannot fit outside and retrieves it on demand.

This chain answers two key questions: why the model actually has no memory yet seems to remember the conversation—because the history is constantly reassembled into the window; and why the window cannot be infinitely large, and why filling it up does not mean using it well—because constraints such as quadratic cost limit capacity, while position and noise limit utilization.

Before studying this page, you first need to master the basic concepts of tokens and tokenization, attention mechanisms, and large language models—know how tokens are counted, how attention is computed pairwise, and how the model generates token by token according to context, so that you can truly understand where the context window limit comes from.

The core concepts of this page are token budget, no memory, quadratic cost, Lost-in-the-middle Effect, and “cramming full ≠ using well.”

The adjacent extended concepts include: Lost-in-the-middle Effect, RAG, context engineering, context compaction, and Agent memory—they are the elaboration of the coping strategies in this section and connect directly to one another. Further extensions are inference optimization, prompt caching, and chain of thought, which discuss how to further improve long-context reasoning in both engineering and effectiveness.