Fine tuning and RAG are two answers to two different problems. Confusing them is the most frequent cause of expensive, low-value projects.
The confusion has an understandable origin. Both techniques are used to adapt a general-purpose model to an organization’s context, and in sales presentations they’re often pitched as equivalent alternatives, to be chosen based on budget.
They aren’t equivalent. They solve distinct problems, and in many projects they coexist.
RAG solves access to up-to-date knowledge
The problem: the model doesn’t know your documents and isn’t up to date.
The solution: at the moment of the question, the system retrieves the relevant content from your archives and provides it to the model along with the request. The answer is built on that material, with the source indicated.
Four properties follow from this, and they’re the reason this is the reference architecture in regulated contexts.
Updates are immediate. A revised circular is available as soon as it’s reindexed, without touching the model.
The answer is verifiable. Every statement points back to a document and a specific location.
Permissions are enforceable. Retrieval can be filtered based on the user’s rights, which is impossible with knowledge embedded in a model’s parameters.
Errors are diagnosable. You can tell whether the problem lies in retrieval, in generation, or in the source document.
Fine-tuning solves style, format and very specific tasks
The problem: the model gets the substance right, but not the required way of expressing it.
Fine-tuning is additional training on selected examples that changes the model’s behavior. It’s not meant to teach it new facts — or rather, it can, but it’s the worst way to do it, because those facts become non-updatable, non-citable and non-verifiable.
It’s useful instead in three situations.
Output format. When the answer has to follow a rigid, recurring structure: a classification into a closed set of internal categories, a data layout, a document template.
Specialized language. When the domain vocabulary is far enough from everyday language that the general-purpose model misreads the terms. This happens in highly vertical technical and regulatory domains.
High-volume repetitive tasks. When the same operation is performed millions of times, a smaller specialized model can end up faster and cheaper than a large general-purpose model, for the same result on that specific task.
Outside these three cases, experience suggests that almost everything you’d get from fine-tuning can be obtained first with well-designed instructions, in-context examples and good document retrieval.
Comparing costs, timelines and maintenance
Comparing upfront cost is misleading. Comparing maintenance is decisive.
RAG. The upfront cost is concentrated in document preparation: acquisition, cleanup, metadata, indexing, permission management. This is the line item that’s regularly underestimated. Maintenance consists of keeping the index up to date, a predictable activity that’s largely automatable. The per-query execution cost is higher, because every request also carries the retrieved documents.
Fine-tuning. The upfront cost is concentrated in building the example set. You need correct, consistent and representative examples, produced by competent people: it’s skilled human work, not a technical task. Maintenance is the critical point. Every time the desired behavior changes, or the base model is updated or discontinued by the vendor, the process has to be redone. Execution cost is lower and latency is lower.
A point that’s often overlooked: fine-tuning ties you to a model. When that model is superseded, which happens regularly, the work has to be replicated. An architecture based on document retrieval, by contrast, allows the model to be replaced with limited impact.
A four-question decision tree
Is the problem that the model doesn’t know specific, up-to-date or confidential information? If yes, you need document retrieval. Fine-tuning on this content would make it non-updatable and non-citable.
Is the problem that the answer has to follow a rigid format or a very specific vocabulary, and instructions aren’t enough? If yes, fine-tuning is a candidate. Before proceeding, though, you should verify that structured instructions and in-context examples have actually been tried: in most cases they solve it.
Is the task repetitive, extremely high-volume, and is the per-operation cost a constraint? If yes, a smaller specialized model may be justified by the operational savings.
Do the answers need to be verifiable and traced back to a source? If yes, document retrieval is mandatory, possibly combined with a specialized model that structures its output.
In our experience, the most frequent combination in regulated contexts is document retrieval for knowledge, designed instructions for behavior, and fine-tuning reserved for high-volume classification tasks.
What neither technique solves
Neither compensates for a disorganized document base. Retrieval would return unreliable sources; fine-tuning would learn from inconsistent examples.
Neither eliminates the need for evaluation. A system has to be measured on real cases before release, with a test set that includes the hard cases, regardless of the architecture chosen.
And neither replaces deterministic rules where those are applicable. If a check can be expressed as a verifiable condition, it should be implemented as one: more reliable, cheaper, and simpler to explain during an audit.
In closing
The choice has to be made after defining the problem, not before. In most of the cases we analyze, the initial question about fine-tuning is hiding a need for access to company knowledge, which is solved differently and at a lower cost.
If you’re evaluating the architecture of a system, we’re available for a technical conversation about your case: which problem needs solving, what data is available, and which combination would be sustainable over time.
