Contents
RAG (retrieval-augmented generation) is a technique in which an AI language model retrieves relevant information from your own documents before each answer and bases its response on it. This lets an app answer questions about your products, policies or manuals with up-to-date company knowledge – with sources and without retraining the model. RAG is best explained through its building blocks, which we walk through step by step below.
Why language models alone are not enough
Large language models are trained on publicly available text. They do not know your internal price lists, maintenance instructions, contract terms or process descriptions. Ask them anyway and they often produce plausible-sounding but wrong answers – so-called hallucinations. In addition, a model's knowledge stops at a certain point in time, while your documents change continuously.
RAG solves both problems by combining search and text generation: first search, then answer based on what was found. The model does not need to "memorise" your knowledge; it receives the relevant excerpts with every question.
How RAG works: the two phases
A RAG system works in two phases – preparing your data and answering questions.
Phase 1: Preparing knowledge (indexing)
- Connect sources: PDFs, Word files, wiki pages, ticket systems, product databases or websites are ingested.
- Extract and clean text: headers, navigation and duplicates are removed; tables and headings are kept as structured as possible.
- Split into chunks: long documents are divided into meaningful sections, for example along chapters or paragraphs.
- Compute embeddings: each chunk is converted into a numeric vector that represents its meaning. Similar content ends up close together in vector space.
- Store: vectors, original text and metadata (source, date, access rights) go into a vector database or a database with vector search.
Phase 2: Answering questions (retrieval and generation)
- The user's question is also converted into a vector.
- The system finds the most similar chunks, often combined with classic keyword search (hybrid search).
- A re-ranking step sorts the hits by relevance; only the best are used.
- The question, the chunks and an instruction ("Answer only based on the following sources") are sent to the language model.
- The answer is returned to the app with references to the sources used.
RAG, fine-tuning or long context?
RAG is not the only way to give a model your own knowledge. The alternatives compared:
| Approach | Principle | Strengths | Weaknesses |
|---|---|---|---|
| RAG | Retrieve relevant chunks per question | Current, with sources, controllable permissions, scales to large collections | Quality depends heavily on search and data preparation |
| Fine-tuning | Retrain the model with your own examples | Learns style, format and terminology | Poor for changing facts, no sources, effort for every update |
| Long context | Put entire documents straight into the request | Simple, good for a few documents | Expensive and slower at scale, no fine-grained permissions |
For factual knowledge from company sources, RAG is the first choice in most cases. Fine-tuning can be a useful addition when the model needs to learn a particular tone or a fixed output format.
Typical use cases in apps
- Service and technician apps: field staff ask about error codes or maintenance steps and get the relevant passage from the manual.
- Customer support: an assistant answers product questions using FAQs, documentation and warranty terms. Our article on integrating an AI chatbot into your app covers the app side.
- Employee apps: questions about travel expenses, holidays or IT policies are answered from the intranet.
- Sales: product data, price lists and reference information are available on the tablet during customer meetings.
- Documentation-heavy industries: legal, insurance, construction or medical technology benefit especially, because knowledge there lives in long texts.
The most common mistakes in RAG projects
A first RAG prototype is quick to build. The gap between demo and reliable product almost always opens in the same places:
- Poor data: outdated, contradictory or duplicate documents lead to contradictory answers. Data hygiene is the biggest quality lever.
- Unsuitable chunking: chunks that are too small lose context; chunks that are too large dilute search. Tables and scanned PDFs need special handling.
- Missing access control: if all documents sit in one index, a user may receive information they should not see. Permissions must be filtered during retrieval, not only in the answer.
- Vector search only: article numbers, names and abbreviations are often found better by classic keyword search. Hybrid search is therefore usually superior.
- No measurement: without test questions with known correct answers, you cannot judge whether a change helps or hurts.
- Manipulated content: documents can contain hidden instructions aimed at the AI. Sources should be trustworthy, and the assistant must not trigger critical actions based on document text.
Further techniques for reducing wrong answers are described in avoiding hallucinations.
Data protection and operations
With RAG, excerpts of your documents leave your system with every request on their way to the language model – unless you host it yourself. Check the provider, server location, data processing terms and whether inputs are used for training. For particularly sensitive data, open-source models and vector databases on your own servers in the EU are an option. Our article on implementing AI features in a GDPR-compliant way summarises the basics (this is not legal advice).
In operation, a RAG system needs automatic index updates when sources change, monitoring of answer quality and an easy way for business departments to add or remove documents. As a guide figure, a RAG solution with app interface, data preparation and admin area is often in the range of mid-complexity projects (EUR 15,000–50,000); with many data sources and fine-grained permissions, it can be more.
Agentic coding workshop for your team? From €799 net – request a quote by e-mailConclusion
RAG makes company knowledge usable with AI without retraining a model: your documents are prepared, made searchable and supplied as the basis for every answer. Success depends less on the language model than on data quality, search strategy, permissions and consistent measurement. Starting with a clearly bounded knowledge area and expanding step by step is the fastest route to reliable results. For a specific project, you can request a fixed-price offer through our offer wizard.
Frequently Asked Questions
What is RAG in simple terms?
RAG stands for retrieval-augmented generation: before answering, the AI searches your documents for relevant passages and bases its answer on them. It therefore responds with your current knowledge rather than general model knowledge alone.
Does the AI model need to be trained for RAG?
No. The model stays unchanged and receives the relevant information with each request. New or changed documents are simply re-indexed.
Which documents are suitable for RAG?
Essentially any text source such as PDFs, wikis, manuals, FAQs or database texts. Well-structured, current documents work best; scanned files and complex tables require additional preparation.
Can RAG completely prevent false statements?
Not completely, but it reduces them significantly and makes answers verifiable through source citations. Clear instructions, test questions and the ability to decline honestly when information is missing help further.