How RAG works
Three steps run every time somebody asks a question. Your documents are split into passages and indexed, usually in a vector database, so they can be searched by meaning rather than by keyword. When a question arrives, the system retrieves the passages most relevant to it. Those passages are placed in the prompt alongside the question, and the model answers from them.
The consequence worth understanding is that the model is not remembering your content. It is reading it, each time, in the moment.
Why RAG is used
- Answers stay current. Editing a document changes the next answer. There is no retraining step between the two.
- Answers can be cited. Because the model answered from specific passages, it can name them, and a reader can check.
- Information can be removed. Deleting a document removes the fact, which matters wherever retention rules apply.
- Hallucination drops. A model given the right passage is far less likely to invent one.
What RAG does not solve
RAG governs what a model knows, not how it behaves. If the output format is wrong, the tone is off, or the model will not follow a structure reliably, retrieval will not fix any of it, because none of those depend on what it can see. That is what fine-tuning is for.
It also inherits the quality of what it retrieves. Contradictory or out of date documents produce contradictory or out of date answers, delivered with a citation that makes them look more trustworthy rather than less.
What does RAG cost to run?
Retrieved passages are tokens, so a RAG answer costs more per request than an ungrounded one, and retrieval adds a step to the latency. Against that, it is far cheaper to build and change than training, and the cost is predictable per request rather than concentrated in a training run you have to repeat.
Related
- RAG vs fine-tuning: which one your problem actually calls for.
- Generative AI development: retrieval grounded features built into your product.