Retrieval-augmented generation is moving from experimental to production for many enterprise teams. After building RAG systems for clients in legal, finance, and healthcare, we have learned that the architecture decisions you make early have an outsized impact on accuracy and cost.
If you are new to the idea, read what RAG is and why enterprises use it first. This article is about the engineering.
Ingestion comes before everything
Answers can only be as good as the text you extract. Scanned PDFs, tables, headers and footers, and multi-column layouts all break naive extraction. Spend time on parsing, and keep metadata with every piece of text: the source document, section, date and who is allowed to see it.
Chunking is the foundation
How you split documents determines what the model can retrieve. Fixed-size chunks are simple but break context; structure-aware chunking that respects headings and sections consistently produces better answers.
Keep tables and clauses whole, and attach the section heading to each chunk so a passage still makes sense on its own.
Choose your embedding model deliberately
The default embedding model is rarely the right one. Domain, language, and latency budget all matter. Evaluate two or three candidates against your own data before committing.
Hybrid retrieval and reranking
Dense vector search alone misses exact-match terms; sparse keyword search alone misses meaning. Combining both, then reranking the results, is the single biggest accuracy lever we have found.
Exact terms matter more than people expect in enterprise data: product codes, clause numbers, names and abbreviations are exactly what keyword search finds and vector search blurs.
Shape the question before you search
Users ask short, ambiguous questions, often referring to earlier messages. Rewriting the question into a complete, standalone query before retrieval improves results noticeably, particularly in a chat interface.
Permissions belong in retrieval
The system must never retrieve a document the user is not allowed to read. Filter by the user’s permissions at search time, using the metadata stored with each chunk. Relying on the model to withhold information it has already been given is not access control.
Make it say “I don’t know”
When retrieval returns nothing relevant, the right behaviour is to say so. Set a relevance threshold, instruct the model to answer only from the supplied passages, and show citations so users can check the source.
Measure everything
Without an evaluation set, you are guessing. Build one early so every change is a measured improvement, not a hopeful one.
Measure retrieval and generation separately. If the right passage was not retrieved, no prompt will fix the answer. If it was retrieved and the answer is still wrong, the problem is in generation.
Plan for cost, speed and change
- Cost. Fewer, better passages are cheaper and more accurate than many mediocre ones.
- Speed. Stream the answer and cache frequent questions.
- Freshness. Re-index documents when they change, and remove the old version.
- Monitoring. Log questions, retrieved passages and user feedback so you can see where the system fails.
Planning a RAG or AI build? See our AI development services, the industries we serve, or hire dedicated AI engineers.
