What is RAG (retrieval-augmented generation)?

A pre-trained language model knows only what was in its training data, frozen at the moment training stopped. Ask it about something newer, or something private that never appeared in that data, and it has two options: decline, or guess. Retrieval-augmented generation is the technique that gives it a third option. Let the model look something up first, then answer from what it found.

What RAG actually is

The idea is to bolt an external, searchable knowledge source onto a model that would otherwise answer only from memory. The term and the recipe come from Lewis et al., who described “models which combine pre-trained parametric and non-parametric memory for language generation,” where the parametric part is a pre-trained sequence-to-sequence model and the non-parametric part is “a dense vector index of Wikipedia, accessed with a pre-trained neural retriever” (Lewis et al., 2020).

Retrieval-augmented generation (RAG): a technique that pairs a pre-trained language model with an external knowledge source it retrieves from at inference time. A retriever fetches relevant passages for the query; a generator produces the answer conditioned on both the query and those passages. The model’s weights are never modified.

The defining property is that last line. The knowledge lives outside the model, so it can be updated or cited without touching the weights. You change what the model can read, not what it knows how to do.

The two parts: retriever and generator

RAG has exactly two moving pieces, and they run in sequence for every query.

The retriever takes the query and searches an external corpus, returning the passages most relevant to it. In the original work that corpus was a dense vector index of Wikipedia, searched by a neural retriever rather than keyword match. This external, swappable store is the model’s non-parametric memory: it holds knowledge as data you can edit, not as weights you would have to retrain.

The generator is the pre-trained language model. It receives the query plus the retrieved passages and writes an answer conditioned on both. The knowledge encoded in its weights is its parametric memory. The passages handed to it at inference are the non-parametric memory. The answer is produced from the two together.

That split is why the retriever is a component you can improve on its own. Work on dense retrieval showed a learned dense retriever can “outperform a strong Lucene-BM25 system largely by 9%-19% absolute in terms of top-20 passage retrieval accuracy” (Karpukhin et al., 2020). Better retrieval means better passages reach the generator, with no change to the generator itself.

Parametric vs non-parametric memory

This is the distinction the term was built on, and it is the one worth holding onto.

Parametric memory is everything a model absorbed during training and stored in its weights. It is broad and fast to draw on, but it is fixed. It has a cutoff you cannot move without retraining, and it keeps no record of where any given fact came from.

Non-parametric memory is knowledge kept outside the model, in a store the retriever reads at inference. Add a document and the system can use it on the next query. Remove one and it is gone. Every answer can be traced to the passage it was drawn from.

RAG’s contribution was to run both at once: keep the fluent, general model, and give it a live, inspectable knowledge source alongside. The tension RAG resolves is the same one that separates it from fine-tuning, which changes the weights instead of the reading material. If you have not met the parametric side yet, start with what an LLM is.

Why RAG was invented

Here is the part most explainers skip. RAG was not built to make models reason better. It was built to fix two specific failures of large pre-trained models that have nothing to do with intelligence.

Lewis et al. named them directly. Large pre-trained models’ “ability to access and precisely manipulate knowledge is still limited,” they wrote, and “providing provenance for their decisions and updating their world knowledge remain open research problems” (Lewis et al., 2020). Two problems: you cannot see why a model said what it said, and you cannot keep its knowledge current. Retrieval addresses both at once. The answer points back to a retrieved passage, which is provenance, and the passage comes from a store you control, which is updatable knowledge.

What the paper claimed on quality was narrower and specific: RAG models “generate more specific, diverse and factual language than a state-of-the-art parametric-only seq2seq baseline” (Lewis et al., 2020). That is the kind of gain that comes from better source material reaching the model, rather than from a better mind doing the reasoning.

Does RAG reduce hallucination?

Partly, and it helps to be precise about how. Grounding an answer in retrieved text gives the generator real passages to draw from instead of relying on whatever the weights half-remember, and it lets a system show the source behind a claim. Both make fabrication less likely and make an answer checkable.

It is not a correctness guarantee. If the retriever returns the wrong passage, or a passage that is itself wrong, the generator can still produce a confident, wrong answer. RAG changes where the knowledge comes from. It does not verify that the knowledge is true.

The mental model that holds up

Think of RAG as an open-book exam for a model that would otherwise sit the test from memory. The book is separate from the student, so you can revise it without re-teaching the student, and every answer can name the page it came from. The point was never to make the student cleverer. It was to let the student cite a source and study from a book you can keep up to date.

Sources: Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv:2005.11401, 2020); Karpukhin et al., Dense Passage Retrieval for Open-Domain Question Answering (arXiv:2004.04906, 2020).