Denshin / Blog / AI
RAG vs fine-tuning: how to choose
RAG changes what a model knows, fine-tuning changes how it behaves. A practical decision framework covering data freshness, citations, per-user permissions on source documents, tone consistency, latency and cost, and the ops burden of a training pipeline, plus the hybrid most production systems actually land on.
Denshin Engineering · Engineering Team · 27 August 2026 · 7 min read
Most teams who ask "should we use RAG or fine-tuning?" have quietly already decided on fine-tuning, because it sounds like the serious engineering answer. It usually is not. The two techniques solve different problems, and picking the wrong one costs a small team months it cannot spare. Retrieval changes what the model knows at the moment it answers. Fine-tuning changes how the model behaves across every answer. Hold those two sentences in your head and most of this decision makes itself.
RAG vs fine-tuning: what each one actually changes
Retrieval augmented generation (RAG) fetches relevant documents at request time and places them in the prompt, so the model answers from text it can see and you can cite. Fine-tuning continues training a model on your own examples so it internalises a task, a tone or an output format, and it does not reliably add new facts you can point at. If the problem is "the model does not know our material", reach for retrieval. If the problem is "the model knows enough but answers in the wrong shape", consider fine-tuning.
That distinction is not a technicality. It predicts almost every failure we see. Teams fine-tune a model on last quarter's documentation, ship it, and then discover that the answers are confidently out of date with no way to update them short of another training run. Other teams bolt retrieval onto a product that needed a consistent output format, and end up with correct facts wrapped in prose that their front end cannot parse.
Try the boring baseline first: a longer, better prompt
Before either option, spend a day on the baseline almost everyone skips. Put the relevant material directly in the prompt. Add three or four worked examples of the output you want. Write the instructions as rules rather than vibes. Context windows on current mainstream models are large enough that a policy document, a product catalogue extract or a style guide often just fits, and for a surprising number of internal tools that is the whole solution.
The baseline is worth running properly because it tells you which of the two real options you need. If a stuffed prompt produces correct answers, your problem is retrieval scale, not model behaviour: you need a way to select the right slice of text, which is RAG. If a stuffed prompt still produces answers in the wrong register, wrong format or wrong length even with examples, you have a behaviour problem, and that is the narrow slice where fine-tuning earns its keep. Getting the prompt itself right is a discipline of its own, and it is a subject in its own right, usually filed under context engineering.
Six decision rules that settle most cases
- Freshness. If the correct answer changes weekly, daily or per transaction, use retrieval. A fine-tuned model is a snapshot. Updating it means running the pipeline again, and nobody schedules that as often as the data changes.
- Citations. If a user needs to see where an answer came from, or if being wrong has consequences worse than mild embarrassment, use retrieval. Only retrieval gives you a document, a section and a link to show. A fine-tuned model can produce a citation-shaped string with no source behind it, which is worse than no citation at all.
- Per-user permissions. This one is decisive and often forgotten. If different users are allowed to see different source documents, retrieval is the only sane architecture, because you filter the corpus by the caller's permissions before anything reaches the model. Training on the union of everyone's documents bakes restricted content into the weights, and no prompt instruction reliably unbakes it. Treat that as a hard rule.
- Tone and format consistency. If you need every output to follow one house voice or one rigid structure, and prompting plus examples keeps drifting, that is the honest case for a light fine-tune. It is also the case where structured output modes often solve the problem more cheaply than training.
- Latency and cost profile. RAG adds a retrieval hop and a larger prompt on every single request, so it costs input tokens and milliseconds forever. A fine-tuned smaller model can run with a shorter prompt, which at high volume can be cheaper per call, but you pay the training cost up front and again on every refresh. Which side wins depends on your request volume, so do the arithmetic with your own numbers rather than borrowing anyone else's.
- Ops burden. Fine-tuning is not an afternoon. It is a dataset you have to curate and keep clean, an evaluation set, a training run, versioning of model artefacts, a rollback story and a person who owns all of it. For a small team, that is a permanent tax on a fixed headcount.
A trade-off table you can put in front of a stakeholder
| Concern | RAG | Fine-tuning |
|---|
| Adds new facts | Yes, at request time | Not reliably |
| Data changes often | Update the index | Retrain |
| Citations and provenance | Natural fit | Not supported |
| Per-user access control | Filter before retrieval | Unsafe by design |
| Consistent tone or format | Prompt dependent | Strongest case |
| Per-request cost | Higher, bigger prompts | Can be lower |
| Up-front effort | Moderate | High |
| Time to first useful version | Days | Weeks |
Read the last two rows together. For a studio-sized team, time to a version real users can criticise is the most valuable thing on the table, because the first version is always wrong in ways no planning session predicted.
The hybrid that most mature systems land on
The honest end state for a lot of production systems is not one or the other. It is retrieval for facts, plus something light for format. Retrieval handles the knowledge, so answers stay current and citable. Format is handled by a constrained output schema, a handful of few-shot examples, or, if those genuinely fail at your scale, a small fine-tune whose only job is shape and tone.
Build it in that order. Get retrieval working and measured first, because retrieval quality dominates the final answer quality by a wide margin. Most systems that get labelled "the model is not good enough" are actually pulling the wrong three paragraphs out of the corpus, which is a problem you fix with better chunking and reranking, not with a bigger model or a training run. Only after retrieval is solid should you decide whether the remaining errors are behaviour errors worth training away.
When fine-tuning really is the right move
There are real cases, and it is worth naming them so this does not read as blanket discouragement. Fine-tuning fits when you have a narrow, high-volume, repetitive task with a stable definition: classifying support tickets into a fixed taxonomy, extracting the same fields from the same kind of document, rewriting text into one strict house format. It fits when you want to run a much smaller and cheaper model at production volume and need it to punch above its weight on that one task, which is the thread we pull on in when smaller models win. And it fits when you already have a large body of genuinely good examples, because the dataset is the expensive part and you cannot fake your way past it.
The prerequisite in all three cases is an evaluation set built before you train. Without it you cannot tell whether the fine-tune improved anything or simply overfitted to the examples you happened to collect. If you do not have the appetite to build that evaluation set, you do not have the appetite to run a training pipeline, and that is useful information to discover before you start rather than after.
Cost and ops honesty for a small team
Prices and available fine-tuning options change constantly, so check current vendor documentation rather than trusting any figure written down in August 2026, including ours. What does not change is the shape of the commitment. RAG costs you an ingestion pipeline, an index to keep in sync with the source of truth, and a slightly larger prompt on every call. Fine-tuning costs you a dataset, a repeatable training job, artefact versioning and a rollback plan, and it costs you all of that again the next time the requirements move.
For a team of a handful of engineers with client deadlines, the second commitment usually loses on staffing grounds alone, regardless of what the token maths says. That is the real reason fine-tuning is rarely the first move: not that it does not work, but that it asks for ongoing attention that a small team has already spent elsewhere. Keeping that spend visible is the same instinct behind adding AI without setting money on fire.
What to do next
- Write down the ten questions your users most need answered, with the ideal answer next to each. This is your evaluation set, and it takes an afternoon.
- Run those ten through a plain prompt with the relevant material pasted in. Note what breaks: missing facts, or wrong shape.
- Missing facts means retrieval. Wrong shape means constrained output first, examples second, and training only if both fail.
- If any source document is restricted to some users, stop and design permission-filtered retrieval before writing anything else.
- Only after retrieval is measured and stable should fine-tuning appear on the roadmap, with a named owner and a refresh cadence.
If you are weighing this decision for a real product and want a second opinion from a team that has shipped both patterns on AWS with small budgets and real deadlines, talk to us. We would rather talk you out of a training pipeline you do not need than watch you build one.
Tags: RAG, Fine Tuning, LLM, AI Architecture, Retrieval
All posts · Work with Denshin