Ask “RAG or fine-tuning?” and you have already narrowed to the wrong choice. The two are not competing answers to one question. They act on different parts of the system, and plenty of production setups run both at once.
What each one actually changes
The cleanest way to tell them apart is to ask what you want to move: what the model knows, or how the model behaves.
RAG (retrieval-augmented generation) changes what the model can see at the moment it answers. Fine-tuning changes the model itself. The distinction is old enough to be named in the original RAG paper, which framed a language model as parametric memory (facts baked into weights) paired with non-parametric memory (an external index the model reads from) Lewis et al., 2020. RAG is the non-parametric side. Fine-tuning works on the parametric side.
How RAG changes what a model knows
RAG leaves the model’s weights untouched. At answer time, the system searches an external store (often a vector index of your own documents) for relevant passages and hands them to the model as context. The model reasons over text it was given, not text it memorised.
Two things follow. Knowledge stays current because you update the index, not the model. And answers can carry attribution, since the retrieved passages are real documents you can surface as sources. The original paper introduced RAG precisely because a pretrained model “stores factual knowledge in its parameters” yet its “ability to access and precisely manipulate knowledge is still limited” Lewis et al., 2020. External, updatable memory was the fix, and the method set the state of the art on three open-domain question-answering tasks.
If you are new to the retrieval side, what RAG is covers the mechanics in full.
How fine-tuning changes how a model behaves
Fine-tuning continues training a pretrained model on curated examples, updating its weights so a pattern becomes the model’s default. It is the tool for shaping form: tone, output format, adherence to a task, instruction-following. InstructGPT is the canonical demonstration. Tuning a base model on human demonstrations and feedback produced “improvements in truthfulness and reductions in toxic output generation” while making the model follow user intent more reliably Ouyang et al., 2022. The change was behavioural. The same work found a 1.3-billion-parameter tuned model whose outputs people preferred over the raw 175-billion-parameter GPT-3, so behaviour gains do not track raw size.
What fine-tuning does not do well is inject specific, fresh facts. Knowledge learned this way is frozen at training time and spread across the weights with no source trail, so refreshing it means training again.
Cost used to be the sharpest edge here. Full fine-tuning retrains every parameter, which the authors of LoRA called “prohibitively expensive” for models at the 175-billion-parameter scale Hu et al., 2021. Parameter-efficient methods have cut that sharply: the same work reports reducing trainable parameters by several orders of magnitude and GPU memory by roughly threefold while matching full fine-tuning quality. The expense is real but no longer the wall it once was.
How they compare
| Axis | RAG (non-parametric) | Fine-tuning (parametric) |
|---|---|---|
| What it changes | What the model can see at answer time | The model’s own default behaviour |
| Knowledge freshness | Update the index and it is current | Frozen at training time, needs a retrain |
| Attribution | Retrieved passages can be shown as sources | Diffused into weights, no source trail |
| Behaviour and format control | Steered via prompt and context | Baked into the weights |
| Cost shape | No training run, ongoing retrieval per query | Upfront training compute, cheap at inference |
| Data needed | A document corpus | Curated examples of the target behaviour |
| Typical failure | Retrieves the wrong context | Drifts from base skills, or holds stale facts |
When to reach for which
The decision rule falls out of the axis. Need current, proprietary, or citable knowledge, and RAG is the fit. Need a consistent voice, a strict output format, or fluency on a narrow task, and fine-tuning earns its cost. A support assistant that must quote today’s pricing wants retrieval. A model that must always reply as valid JSON in a house style wants tuning.
Both approaches act on a base model, so it helps to be clear on what a large language model is before choosing how to adapt one.
Why production systems often use both
The combined case is common because the two solve non-overlapping problems. Fine-tune a model so it answers in the right form for your domain, then wrap it in RAG so every answer draws on current, verifiable facts. One governs how it responds, the other governs what it responds with. Asking “which is better” hides that split; asking “am I changing knowledge or behaviour” points straight at the lever to pull, and often the honest answer is both.
Questions, answered
Is RAG or fine-tuning better?
Neither, because they do different jobs. RAG supplies knowledge at answer time; fine-tuning shapes behaviour by updating weights. The better question is whether you need to change what the model knows or how it responds. Many production systems use both together.
Can you use RAG and fine-tuning together?
Yes, and it is common. Fine-tune a model so it answers in the right format and voice for your domain, then wrap it in RAG so each answer draws on current, verifiable facts. One controls how it responds, the other controls what it responds with.
Does RAG change the model's weights?
No. RAG is non-parametric: it retrieves relevant documents and hands them to the model as context at answer time. The weights are untouched, so you update knowledge by updating the index rather than retraining.
Does fine-tuning teach the model new facts?
Not reliably. Fine-tuning bakes patterns of behaviour into the weights, but knowledge learned this way is frozen at training time and carries no source trail. For fresh or citable facts, retrieval is the better fit.
Brief is a team of AI associates you direct in plain language. Opening to a small group at a time.
Request access