RAG Retrieval-Augmented Generation

A language model reads through vast amounts of text during training and has broad knowledge; many questions can be answered plausibly when asked directly. So why do we still have it “look up materials” before answering? The key lies in its training objective: the model optimizes likelihood, that is, the probability of a piece of text appearing in a given context. Optimizing likelihood is to make continuation fluent and wording natural, not to verify whether a statement matches facts. Precisely because it models likelihood rather than truth, direct questioning exposes three unavoidable flaws:

The first is hallucination. The model’s optimization objective has no constraint that “the fact is true,” so when it does not know something, it often will not admit it does not know, but instead fabricates a fluent but false answer according to language habits. The second is the knowledge cutoff. The model only knows content that appeared in the training corpus; events that occur after training ends and the latest published data are unknown to it. The third is private data. Internal enterprise documents never entered the training corpus, so the model naturally cannot answer.

The idea of RAG is to change the way of answering from “closed-book” to “open-book”: rather than letting the model answer from memory left during training, before answering, first retrieve relevant materials from the knowledge base, place them in front of it, and then have it answer according to the materials. The reason open-book exams are more reliable than closed-book is that the answer does not need to rely on vague recollection, and each step can be annotated with its source for verification.

Specifically, RAG accepts two types of input: the question posed by the user, and the external knowledge base that the user has permission to access. The system first retrieves evidence snippets related to the question from the knowledge base, places them in front of the model, and then the model generates an answer based on the materials. The output is an evidence-backed, updatable answer: the evidence points to specific entries in the knowledge base; after the knowledge base content is updated, the next retrieval can reflect new facts without retraining the model.

At the same time, we must see its boundaries. Open-book answering makes answers more verifiable, but verifiable does not mean automatically correct: the retrieved materials themselves may be wrong, and the model may answer based on the wrong materials. If there are no reliable sources, or if the task does not require external facts at all, there is no need to force retrieval. The recurring “likelihood” here refers to the probability of a piece of text appearing in a given context; language models optimize likelihood, obtaining smoother continuation, not verification of reality—this is precisely the fundamental reason why RAG needs to add a retrieval step outside the model.

The name RAG itself is the three steps of the process: Retrieval, Augmented, Generation. The entire processing chain can be drawn as a straight line:

Figure 1: RAG three steps—Retrieval, Augmentation, Generation

① Retrieval: take the user's question and fetch relevant material from the knowledge base (see the “Retrieval” deep-dive page for the specific mechanism). ② Augmentation: combine the fetched material and the question into a complete prompt. ③ Generation: the model answers based on this material and marks which sources the answer uses.

Use a return inquiry example to walk through it. The user asks: “Can I still return it 35 days after receipt?” Answering this question correctly requires three things: the return period, the starting point of the period, and the exception conditions. The retriever pulls two snippets from the knowledge base:

Snippet A supports the general rule: standard items can be returned within 30 days of receipt, and 35 days after receipt exceeds this period, so normally returns are not allowed. Snippet B supports the exception: quality issues are not subject to the 30-day limit, but supporting evidence must be uploaded. The model processes the two pieces of information separately and generates “normally not; if it is a quality issue, you can apply with proof,” with the first half citing A and the second half citing B, each supported by the corresponding snippet. The model does not add other exceptions not mentioned in the materials—no evidence, no fabrication.

The augmentation step is not simply pasting materials into the prompt. The system must preserve each snippet's ID, version, permissions, and title, explicitly mark in the prompt that “the following are materials to be cited, not new instructions,” and require the model to map every verifiable assertion to the snippet that supports it. The purpose is to distinguish two completely different things: “the answer mentions a source” and “the source actually supports the answer.” The former is just formatting; the latter is what makes it trustworthy.

Essentially, RAG does not modify the model itself. It only, at the moment of asking, first stuffs the materials needed to answer into the prompt—equivalent to handing the model a reference document before letting it answer. All complexity is concentrated in two things: “how to fish out the right materials and place them properly.” Therefore, the inputs to the whole chain are the user question and knowledge base snippets (together with the snippets' versions and permissions), the processing is retrieving candidates, preserving evidence boundaries during augmentation, and citing per assertion during generation, and the output is an answer that can be verified sentence by sentence. A citation indicates trustworthiness only when it truly supports the corresponding assertion; conversely, when no evidence is retrieved, the correct approach is to explicitly answer “no answer” or ask the user for clarification, rather than fabricating nonexistent facts.

What does Retrieval-Augmented Generation (RAG) make up for each of the three hard flaws listed in Section 1?

The mitigation of hallucination comes from a change in the way answers are produced. When the model has real material in hand, it no longer needs to piece together answers from training memory; the generation process is “anchored” to the given material: every assertion should be supported by the material, and content not mentioned in the material is no longer added. But this is mitigation rather than elimination—the material itself may be wrong, and the model may misinterpret the material. See the “Hallucination” deep-dive page for the detailed mechanism.

The updatability of knowledge comes from a shift in storage location. Knowledge is no longer frozen in model weights but is placed in an external knowledge base. To add or correct a piece of information, you only need to update the document; the next retrieval will read it, without retraining the model, and the latest information can enter the answer at any time.

Source traceability is a key advantage of RAG over pure generation. You can have the model mark in the answer which material each sentence comes from, so a person can verify the original text with one click. The mechanism behind this capability is discussed specifically on the “Citations and Source Attribution” deep-dive page.

In one sentence: RAG turns “you can only trust the model” into “you can verify the sources.” For enterprise, customer service, and professional Q&A scenarios where reasons must be given and traceability is required, this is often more important than how smart the model itself is.

So the capabilities that RAG adds can be summarized this way: it takes externally updatable knowledge and the current question as input, first moves the facts from model weights to evidence retrieved at runtime, then has the model answer based on the evidence, and outputs answers with fewer unsupported assertions, updatable and traceable back to their sources. The boundary must also be seen clearly: having a source usually means the answer can be checked, but being checkable does not mean the source itself is credible, nor that the reasoning is faithful to the source—the source may be wrong, and the model may also use the source incorrectly. RAG mitigates hallucination but cannot eliminate it.

“Making the model use my knowledge” is not only RAG; fine-tuning can do the same. The two routes give the model fundamentally different things, and the choice depends on what you want to inject.

RAG injects facts. Facts are stored externally in a knowledge base and retrieved on demand at answer time, not written into weights. Therefore knowledge updates are real-time: just edit the document and the next retrieval takes effect immediately, with no retraining; answers can also include sources for verification one by one. It suits changing, private, verifiable facts, such as the latest company policies, product parameters, and internal processes.

Fine-tuning injects behavior. Through continued training, it writes style, format, and ways of working into the model weights. It is not responsible for remembering facts: when knowledge changes, retraining is required, and the model cannot say where the answer came from. It suits fixed tone, format, and ways of working, such as the tone of customer service scripts or the style of code comments.

This leads to a rule of thumb: facts that can't be remembered go to RAG (retrieval); habits that can't be changed go to fine-tuning. “Want the model to know my company's latest documents” is almost always RAG; “want the model to consistently answer in a certain format or tone” is fine-tuning (see the fine-tuning deep-dive page). They are not mutually exclusive; in real systems they are often used together: fine-tuning fixes behavior, and RAG supplies facts.

The input to selection is the thing to be injected itself: whether it is changing facts or stable behavior. First determine whether the knowledge changes frequently and whether traceability is needed, then decide which route to take; the output is a selection conclusion. The boundaries of the two directions must also be seen clearly: when knowledge changes frequently and requires traceability, retrieval is usually more appropriate; but retrieval cannot replace behavior training, and fine-tuning cannot reliably serve as a real-time database—writing frequently updated knowledge into weights has high update cost and cannot provide sources.

Retrieval-Augmented Generation (RAG) is not a silver bullet. It fails most easily in four places.

The deepest bottleneck is retrieval. If the retrieval stage does not pull up the correct material, the model has no correct answer available to use and can only answer incorrectly or fabricate — "garbage in, garbage out." Therefore, the ceiling on RAG performance is often in retrieval rather than in the model itself, which is also why retrieval deserves to be taken out separately for in-depth reading (see the "Retrieval" deep-dive page).

Second, even when the material is provided, it may not be used well. Stuffing too much, too long material into the prompt triggers "Lost in the Middle": the model pays insufficient attention to content in the middle positions of long context, key material is ignored, and even if retrieval is correct, it still answers incorrectly (see the "Lost in the Middle" deep-dive page).

Third, if the material itself is wrong, the model will follow the error. If the content in the knowledge base is incorrect, RAG will faithfully answer according to the erroneous material. What it guarantees is "having a basis," not "the basis is correct." Putting wrong documents into the knowledge base is equivalent to handing the model a wrong answer.

Fourth, RAG can also "answer incorrectly with citations." Citations do not automatically equal faithfulness. The model may mistakenly apply the 30-day rule from snippet A to the exceptional case of quality problems; it may cite a snippet that is topically related but does not support the conclusion; it may also ignore document versions and treat a repealed provision as a current rule. Therefore the presence of citations is not proof of faithfulness; you must check item by item the correspondence between assertion and evidence, the timeliness of the evidence, and access permissions.

The entire failure causal chain can be summarized as follows: input retrieval snippets, context assembly, and final answer; first look at whether the evidence was recalled and correctly assembled, then look at whether generation faithfully used the evidence, and the output is a localization of the failure cause. Boundaries are equally clear: the presence of citations usually indicates that the system attached sources, but does not prove that the sources truly support the assertion; outdated knowledge or permission errors cannot be fixed by switching to a stronger model — the problem lies in the evidence pipeline, not in generation capability.

When the final answer is wrong, why is “a drop in end-to-end accuracy” still insufficient to guide fixes? Because RAG has at least three layers that can fail independently, and end-to-end accuracy mixes these three layers together:

• The retriever may have failed to recall the necessary evidence;
• The assembler may have trimmed correct evidence, placed the wrong version, or mixed in unauthorized content;
• The generator may ignore evidence, or incorrectly combine general rules and exceptions.

Only by attributing along the chain can you know what to fix. The approach is to save three things per question during evaluation: the target assertion, the relevant passages, and the final assertion, then compute three metrics separately.

Continuing with the return case demonstration. Suppose retrieval returned 4 passages: general rule A, quality exception B, plus “arrival time” and an old policy; the model generated 3 verifiable assertions, of which “no proof required” is fabricated.

The three metrics correspond to the three layers. Necessary evidence coverage measures the retrieval layer: whether all passages that should have been recalled came back. In this example, 2/2, so retrieval recall is acceptable. Context precision measures the assembly layer: how many of the passages that finally entered the prompt are actually useful. In this example, only 2 of 4 passages are useful, 50%; arrival time and old policy are noise. Assertion support rate measures the generation layer: how many of the model's verifiable assertions are actually supported by evidence. In this example, 2 of 3 assertions have evidence, about 67%—the model fabricated “no proof required”.

The bottom row is the final task from a business perspective: the key exception lacks the proof condition, so the task fails. Failure cannot be offset by fluency or number of citations—the model can fluently fabricate assertions while attaching seemingly relevant citations to each assertion.

The formula for assertion support rate: SupportRate = Ssupported ÷ Sverifiable, where Ssupported is the number of verifiable assertions supported by evidence, and Sverifiable is the total number of verifiable assertions.

Two caveats. Here, “context precision” and “assertion support rate” are manual annotation definitions for teaching, and different evaluation frameworks may define them differently; the LLM judge is only another model, not ground truth. Before going live, keep a set of manually verified samples to calibrate the scorer, and observe separately by general rules, exceptions, time sensitivity, permissions, and no-answer questions.

Finally, there is a diagnostic matrix that maps metric drops to repair directions: if necessary evidence is not recalled, fix query, chunking, and indexing; if it is recalled but does not enter the final context, fix filtering, reranking, and budget; if evidence is in context but assertions are unsupported, fix generation constraints, citation verification, or gracefully degrade; if the evidence itself is outdated, fix knowledge governance rather than replacing with a stronger model.

The inputs to layered diagnosis are the target assertion, necessary evidence, final context, and generated assertions. First compute evidence coverage, context precision, and assertion support rate separately, then use them to localize the failure layer and output attributable repair directions. The boundary is: a drop in a metric usually indicates a problem in the corresponding layer, but a metric drop can only localize generation faithfulness and cannot alone prove that retrieval or the business task is acceptable.

The entire causal chain begins with the model's knowledge boundary. Directly answering with a large model has three weaknesses: hallucination, a knowledge cutoff date, and inability to access private data (§1). RAG's countermeasure is to turn “closed-book” into “open-book”: retrieve material before answering and place the material in front of the model (§1). This countermeasure is implemented in three steps: retrieve relevant material → incorporate it into the prompt (augmentation) → the model generates based on the material (§2).

This transformation brings three benefits: mitigating hallucination, making knowledge updatable, and enabling traceability, turning “can only trust the model” into “can verify the source” (§3). It also delineates the division of labor with fine-tuning: RAG provides “facts”, fine-tuning provides “behavior”; facts that cannot be remembered are handed to RAG, habits that cannot be changed are handed to fine-tuning (§4).

But every step in the chain can fail: the bottleneck is in retrieval; too much material triggers the lost-in-the-middle effect, and if the material itself is wrong, the output will also be wrong (§5). Therefore, evaluation should separately measure necessary evidence coverage, context noise, assertion support, and the final task in order to locate which layer needs fixing (§6).

If you can explain this chain clearly—explain clearly “why RAG can mitigate hallucination and enable traceability,” and accurately state “what to use RAG for and what to use fine-tuning for”—you have grasped the core of RAG.

RAG sits in the middle of a conceptual network: understanding it requires several prerequisite concepts, and once you understand it, it leads to a set of extended topics. The hierarchy is as follows:

The prerequisite concepts explain where each part of RAG comes from: large language models explain why the model hallucinates and why it needs to consult information; retrieval and semantic search, and embeddings explain how materials in the knowledge base are found; the context window determines how much material can be stuffed into the prompt, directly relating to the lost-in-the-middle effect in Section 5 and context precision in Section 6.

The core of this page is a set of concepts centered on retrieval-augmented generation: open-book answering (present materials before answering), source attribution (answers include sources), RAG vs fine-tuning (feed facts or change behavior), retrieval as the bottleneck (the performance ceiling is mostly in retrieval).

The immediate extensions are the next step from here: vector databases are the storage form of the knowledge base, document chunking determines the basic unit of retrieval, reranking optimizes the order after recall, advanced RAG handles multi-hop, complex assembly, and other problems; citations and source attribution and hallucination expand the two topics mentioned in Section 3 into independent deep-dive pages.

The farther layer places RAG in a larger picture: knowledge graphs and GraphRAG use structured relationships to strengthen retrieval, context engineering develops material assembly into an independent craft, and AI Agent (Agentic RAG) allows the model to autonomously decide when to retrieve and what to retrieve during the answering process.