Fine-tuning or RAG?

Imagine a support assistant that confidently approves a refund under last month's policy. Its answer arrives in perfect JSON. You could train it on more examples, give it access to the current policy, or write a longer prompt. Only one of those directly supplies the fact it was missing.

Now imagine that the current policy is already in the prompt and the assistant still applies the wrong exception. Same unhappy customer, different engineering problem. Choosing between fine-tuning and RAG starts with finding out which failure you have. These are hypothetical cases, but they expose a distinction that matters before you spend time building either system.

Find what the model was missing

I would start with a small set of representative questions and their expected answers. Include the policy version and passage that justify each answer. Keep some cases separate for evaluation, so an improvement means more than remembering the examples used to develop the system.

Run a baseline with clear instructions and a few worked examples. For each failure, inspect what the model actually received. Then give it the correct source passage directly and ask again. If that fixes the answer across your test cases, you have evidence that access to information is the bottleneck. If it still gets the decision wrong, investigate the instructions, the examples, and the model's ability to perform the task.

Check output validity separately. A missing JSON field and an incorrect refund decision can both break an application, but they need different repairs. A model can follow the schema perfectly while approving the wrong refund.

This test gives the comparison a useful starting point: what must change for the answer to become correct?

RAG puts the reference material within reach

Retrieval-Augmented Generation, or RAG, finds relevant material and puts it into the model's context before it answers. The model's weights do not change. Think of giving a colleague the current policy manual with the relevant page marked. They still have to read it correctly, but at least they have the right edition.

A basic vector retrieval pipeline looks like this:

Query → Embedding → Vector Search → Top-K Chunks → LLM + Context → Response

The embedding represents the query numerically so the search can find passages with related meaning. Top-K means selecting a fixed number of results. Those passages become part of the input to the model. This is one common design; exact identifiers and product codes may also need keyword search.

For the refund assistant, retrieval provides the current policy and any applicable exceptions. When the policy changes, you update the source and its searchable representation. You do not have to retrain the answering model to supply the new text. That makes retrieval a good fit for changing documentation and large reference collections. AWS's comparison of RAG and fine-tuning describes this advantage for document question answering.

The database work still matters. A new file sitting in storage is useless if the index serves its predecessor. Version handling, ingestion, and deletion need to be part of the update process. Retrieval from a large collection is practical, but search quality and response time depend on the index and workload.

RAG also gives you source passages that can accompany the answer. That makes verification possible; it does not make every citation correct. Check that the cited passage supports the actual claim. A link to a refund policy is weak evidence if the answer ignores the exception two paragraphs below it.

For multiple tenants, filter retrieval using permissions enforced by the application. The model should receive only material the user may access. Fine-tuning is not an access-control mechanism, and a separate model per tenant is not the only alternative. Shared models can use authorized context without learning each tenant's private documents through training.

Fine-tuning teaches a recurring task

Fine-tuning updates model parameters through additional training. With supervised examples, you show the model the responses you want it to produce for particular inputs. The useful target might be applying your classification taxonomy, extracting entities according to your conventions, using domain terminology, or writing reports in a consistent style.

Return to the refund assistant. Suppose it receives the correct policy but repeatedly confuses the ordinary return window with an exception for damaged goods. Better instructions and examples are the first experiment. If a persistent pattern remains, a training set can demonstrate how to apply those distinctions across varied cases. The evaluation set must then test cases the model did not train on.

Training can improve consistency, but it does not pin behavior in place. A fine-tuned model can still misread a question, make an unsupported claim, or fail on a case unlike its training examples. It can also learn facts, but updating weights is a poor substitute for an inspectable, regularly updated reference store.

The number of training examples depends on the task, base model, coverage, and quality of the demonstrations. OpenAI's accuracy optimization guidance recommends starting an experiment with 50 or more examples and evaluating before expanding the set. That is a starting point for testing, not a reliability guarantee.

Fine-tuning may also reduce the examples and instructions you send with every request. Test that reduction separately: removing useful instructions too early can hurt quality. The fine-tuning best practices recommend retaining effective instructions in training examples, especially with small datasets.

Fix the output contract before training for it

If the failure is invalid JSON, investigate schema-constrained generation before collecting a training dataset. A schema specifies the required fields, types, and allowed values. Supported structured-output mechanisms constrain the response to that structure, subject to the provider's supported schema features and completion conditions. Your application still needs to handle refusals and incomplete responses.

The distinction is easy to see with a refund field. A schema can require approved to be a Boolean. It cannot establish whether true is justified by the customer's situation. OpenAI's structured-output documentation explicitly warns that structurally valid responses can contain mistakes. The site's structured-output article develops this separation between valid structure and correct content.

Fine-tuning remains useful when the model needs to learn which values belong in those fields, or how to write the explanation inside them. Training solely to stop a missing brace is an expensive way to approach a problem the generation interface may already handle.

What breaks, and what to inspect

With RAG, inspect the retrieved passages before blaming the answering model. The right policy might be absent because the query uses different language, a chunk split an exception from its rule, or an old version remained in the index. Adding more passages can bury useful evidence in unrelated text. AWS's retrieval guidance recommends evaluating retrieval relevance and measuring embedding, search, and reranking separately.

With fine-tuning, compare performance on held-out cases and other tasks you still need the model to handle. Better training scores with worse evaluation results suggest overfitting. A narrow training set can teach a brittle rule; poor or overly aggressive training can also degrade capabilities outside the target task. More examples help only if they address the missing coverage and have consistent answers.

These checks also explain why retrieval and training can belong in the same system. You may need the current policy and better application of it. Improve the component implicated by the failure, then rerun the same evaluation before adding another.

When both earn their place

A hybrid system uses retrieval to supply reference material and a fine-tuned model to perform the task with that material. The architecture remains straightforward. This illustrative pseudocode uses a medical assistant to show the separation; it is not a validated clinical application or a complete SDK example:

The five retrieved results are an example setting, not an established optimum. The comments describe the intended division of work, not a guarantee that training produces reliable reasoning or citations. The application still has to supply authorized, relevant sources and check the generated answer. The instruction to cite sources is not itself a citation validator.

The hybrid approach is justified when evaluation shows both information-access failures and persistent task errors with correct context. It also brings both maintenance burdens: keeping the reference collection current and checking the trained model when the task changes. You do not need a hybrid system merely because both techniques are available.

Compare complete costs

RAG requires ingestion, embeddings where used, indexing, storage, retrieval, and generation with the retrieved context. Changed documents may need processing again. Fine-tuning requires dataset preparation, training, evaluation, and serving the resulting model. Further training runs and model versions bring additional work.

Neither architecture has a universal dollar range or latency penalty. Provider, model, dataset size, training method, traffic, and hosting arrangements change the bill. For example, Amazon Bedrock pricing separates model customization from the cost of using customized models. Training does not make inference free, and it does not guarantee the same serving price as the base model.

Compare cost per accepted answer at the same quality target. Include retries and the effort needed to maintain each system. Shorter prompts can make a fine-tuned model cheaper to use, but a lower token count is no bargain if the answers fail more often.

Measure latency across the full request too. RAG adds lookup work and often input tokens. Fine-tuning may allow shorter prompts or a smaller model, but the resulting response time depends on how that model is served. Measure embedding, search, reranking, and generation separately to see which stage accounts for the wait.

Choose the next experiment

For the hypothetical refund assistant, the next step follows directly from the failure. If supplying the current policy fixes its answers, test whether retrieval can consistently deliver that policy with the relevant exceptions. If the facts are present but the decision remains wrong, improve the task examples and evaluate whether training adds value. If the decision is correct and the JSON is broken, test schema enforcement.

Use the same questions and acceptance criteria as you make those changes. That lets you distinguish an actual improvement from a more impressive demonstration. Combine retrieval and fine-tuning when each solves a failure you can identify, and keep checking both as the policy and workload change. You should be able to explain what each component contributes to a correct answer before taking on the cost of maintaining it.