What is RAG (retrieval-augmented generation)?
RAG lets a language model find and read the relevant documents before it answers, so the answer rests on current information you control.
What is RAG?
RAG stands for retrieval-augmented generation. What a language model knows is limited to its training data. It cannot know a regulation published after training, your company's internal procedures or today's stock levels. RAG closes that gap: when a question comes in, the relevant documents are found first, and the model writes its answer by reading them.
Think of a closed book exam versus an open book exam. A model without RAG answers from memory; a model with RAG has the right pages open in front of it.
How does it work?
- Preparation: documents (PDFs, web pages, support records) are split into small chunks, each chunk is turned into an embedding vector and stored in a vector database.
- Retrieval: the user's question is embedded too, and the chunks closest in meaning are found.
- Generation: the retrieved chunks are given to the model along with the question. The model answers from them and can show which document each part came from.
Why use it?
With RAG a model can work with company knowledge that was never in its training data. When documents change, answers change with them, without retraining. Because each answer rests on a document, it can be checked, and the risk of hallucination goes down. It goes down but does not disappear: if retrieval brings back the wrong passage, the answer can still be wrong.
RAG or fine-tuning?
If the model needs knowledge (products, prices, policies), RAG is usually the right tool, because you only update documents as things change. If you want to change the model's behavior (a specific writing style or a fixed output format), fine-tuning may fit better. Many systems use both.
Building a good RAG system
Most of the quality is decided at the retrieval step. Chunks that are too small lose context, chunks that are too large fill the context window with noise. Combining vector search with keyword search (hybrid search) and re-ranking the results with a second model noticeably improve accuracy. Testing the system regularly with real user questions is the most reliable way to see where the wrong document comes back.
An assistant that works from your own documents is also a core part of an AI agent setup.