RAG (retrieval-augmented generation) is a way of letting a language model search your documents before it answers, pick out the relevant passages and write its answer from those passages, citing where each point came from. The model does not rely on what it memorised during training; it works from the text placed in front of it at that moment. That makes it possible to get answers that are current, specific to your company and easy to check.
The approach became widely known through a 2020 paper by Patrick Lewis and colleagues, presented at NeurIPS 2020. The paper combined the knowledge stored in a model’s weights with a searchable index built from Wikipedia, and highlighted two advantages: showing where an answer came from, and updating knowledge without retraining the model.
Why RAG, and where is it used?
A language model knows what it saw in its training data. It does not know your internal procedures, the returns policy you changed last month or the clauses of a contract you signed with a customer. Faced with a question it cannot answer, it may produce something plausible but wrong instead of saying “I don’t know”.
RAG addresses both problems. It gives the model the current, company-specific information it needs to write the answer, and it makes the source of that answer visible. OpenAI’s guide to optimising LLM accuracy recommends this kind of approach when a model’s knowledge is out of date or when it needs proprietary information.
Typical use cases:
- Customer support assistant: answers from help-centre articles, user manuals and returns policies. It can run on your website or over WhatsApp, and hands the conversation to a human agent when it is unsure.
- Internal knowledge search: answers questions across procedures, HR policies, technical documentation and contracts, with sources, within each user’s permissions.
- Field and technical teams: searches maintenance manuals and past fault reports.
AI search engines work the same way: ChatGPT’s web search, Perplexity and Google’s AI Overviews search the web before writing an answer and cite the pages they used. We cover how to get your site cited in those answers in our guide to GEO.
You do not always need RAG. Anthropic notes that if a knowledge base is smaller than 200,000 tokens (roughly 500 pages), you can simply include all of it in the prompt. RAG starts to pay off when the material is larger than that, changes often, or must be filtered differently for different users.
How does RAG work?
A RAG system works in five steps. The first three run when documents are added or changed; the last two run for every question.
- Ingestion. Sources such as PDFs, Word files, web pages and support tickets are converted to plain text. Headings, tables and page numbers are kept; scanned documents need text recognition (OCR) first.
- Chunking. Long documents are split into smaller pieces. According to Anthropic’s write-up, these chunks are usually no more than a few hundred tokens. Each chunk is tagged with the document name, section heading, date and access permissions.
- Embedding. An embedding model turns each chunk into a list of numbers (a vector) that represents its meaning, and the vectors are stored in a database. This is why a user searching for “refund deadline” can also find a passage about the “cancellation period”.
- Retrieval. The question is turned into a vector too, and the chunks closest in meaning are found. Because semantic search can miss exact matches such as product codes or clause numbers, it is usually combined with classic keyword search (BM25); this is called hybrid search. In Anthropic’s own tests, combining contextualised chunks, hybrid search and reranking reduced the cases where the right information was missing from the top 20 results by 67%.
- Answering with citations. The retrieved chunks are passed to the language model together with the question. The model is instructed to answer only from those chunks, to say which part of which document it used, and to say so plainly when the answer is not there.
RAG or fine-tuning?
Fine-tuning means training a model further on your own examples. The same OpenAI guide positions fine-tuning for problems of tone, format and consistent behaviour, and RAG for missing or outdated knowledge, and notes that the two can be combined.
| Aspect | RAG | Fine-tuning |
|---|---|---|
| Solves | Missing, proprietary or outdated knowledge | Consistent tone, format and behaviour |
| When knowledge changes | Update the document and re-index | Retrain the model |
| Citing sources | Answers can link to the chunk they used | Knowledge sits in the weights, so there is no source to cite |
| Access control | Applied per document at retrieval time | Hard to restrict what the model has learned by user |
| Data needed | The documents you already have | Many carefully prepared examples |
For an assistant that answers from company knowledge, RAG is usually the better starting point. Fine-tuning comes in when a tone or format problem remains that good instructions and RAG cannot fix.
Access control and data privacy
The most overlooked risk in RAG is a user seeing, through the assistant, a document they could not open directly. LLM08:2025 in the OWASP Top 10 for LLM applications describes how weak access controls can expose sensitive information held in a vector database to unauthorised users. Its recommendations include permission-aware vector stores, logical partitioning of datasets, tagging data by access level and keeping logs of retrieval activity.
In practice, every chunk carries the permissions of the document it came from, and filtering happens during retrieval. Telling the model “don’t show this to that user” is not enough; a chunk the user may not see should never reach the model.
The second risk is instructions hidden inside documents. LLM01:2025 explains that instructions in external content the model processes, such as a web page or a file, can change its behaviour, and that techniques like RAG and fine-tuning do not fully remove this risk. Only feed the knowledge base from sources you trust, and keep the actions the assistant can take narrow.
Finally, decide where the data will be processed. Your documents are sent to a model at both the embedding and the answering stage. Whether you use a cloud-hosted model or a setup in which data never leaves your company depends on how sensitive the data is. In Turkey, where we are based, personal data falls under KVKK, the Turkish Personal Data Protection Law. As of October 2026, its Article 9, amended in 2024, allows transfers abroad on the basis of an adequacy decision, appropriate safeguards such as a standard contract, or a limited set of incidental cases. Under GDPR or any other regime the question is the same: know where the model runs and what the provider may do with your data.
Answer quality and common pitfalls
What to measure
Measuring quality starts with a test set built from real user questions, noting for each one which document holds the correct answer. Then measure three things separately: Did retrieval find the right chunk? Is the answer faithful to the retrieved chunks? Does it actually address the question? The Ragas paper by Es and colleagues (2023) proposes a way to score these dimensions automatically, without reference answers. Once live, review user feedback, unanswered questions and conversations handed to humans regularly.
Stale and conflicting documents
If three versions of the same procedure sit in the knowledge base, the model may use an old one. Each piece of information needs a single current source, and when a document is updated or withdrawn, the index should be updated or cleaned automatically.
Poor chunking
A table split in half or a paragraph cut off from its heading loses meaning. Anthropic’s example is a chunk that says “The company’s revenue grew by 3% over the previous quarter”, which on its own does not say which company or period it refers to. Chunk in a way that keeps headings intact and, where needed, add a short piece of context to each chunk.
Unsourced answers and hallucination
RAG reduces made-up answers but does not eliminate them. In a 2024 study by Magesh and colleagues, AI legal research tools from LexisNexis and Thomson Reuters hallucinated between 17% and 33% of the time; the authors note that these tools use RAG and that it reduced hallucination without removing it. If an assistant does not cite its sources, users cannot spot these errors. Make citations mandatory, let the assistant say “I couldn’t find this”, and keep a human in the loop for critical topics.
Frequently asked questions
Does RAG teach our documents to the model?
No. Documents are given to the model as context each time a question comes in; the model itself is not retrained. Remove a document from the knowledge base and the assistant stops using it. With a cloud-hosted model, still check the provider’s terms on retaining and using the data you send.
Which language model should we use?
There is no single right answer. Data sensitivity, answer quality in your users’ languages, cost and speed all matter. The most reliable approach is to compare a few models against a test set built from your own questions.
Conclusion
RAG connects a language model to your own documents so that it can give answers that are current, specific to your business and traceable to a source. Good results depend on clean ingestion and chunking, hybrid search, access control applied at retrieval time and regular quality measurement.
We use AI in our own product too: Dynamic Translator combines AI translation with human editing. If you are considering an assistant that answers from your documents, take a look at our AI solutions page or get in touch, and we can look at the structure and sensitivity of your data together.
