Almost every internal RAG project starts the same way: point an embedding model at the company document store, wire up a vector database, put a chat box on top. The demo is impressive. Then domain experts use it, and it confidently returns the maintenance procedure for the wrong equipment class, or quotes a specification that was superseded eighteen months ago.
The instinct at that point is to blame the model and reach for a bigger one. In my experience the model is rarely the bottleneck. The bottleneck is that the system was built as if the company had one vocabulary, when it actually has one vocabulary per department — and a flat index quietly destroys that distinction.
The failure mode: semantic collision
In any organisation with real domain depth, two things are simultaneously true: the same concept has different names in different departments, and the same word means different things in different departments. Engineering says "unit," operations says "asset," finance says "line item" — sometimes for the same physical object. Meanwhile "cycle" means one thing to a maintenance planner and something entirely different to a billing analyst.
An embedding model has no idea which meaning applies. It maps everything into one shared space, so documents from unrelated departments end up as near neighbours purely because they share surface vocabulary. Retrieval then returns a plausible-looking chunk from the wrong world, and the language model — being a good writer — turns it into a fluent, confident, wrong answer.
A retrieval bug and a hallucination look identical to the user. The difference is that only one of them is actually the model's fault.
The reframe: a bounded context is a retrieval namespace
If you have done domain-driven design, this problem should feel familiar — because it is the exact problem DDD already solved for code. A bounded context is a boundary within which one model, and one ubiquitous language, applies consistently. DDD's core insight is that trying to force one canonical model across the whole enterprise fails; instead you draw explicit boundaries and define how they relate.
The same idea transfers directly to retrieval. A department with its own vocabulary, its own document conventions and its own subject-matter experts is a bounded context. So:
- One namespace per bounded context. Documents are indexed with a context tag, and retrieval is filtered to the relevant context rather than searching one global pool.
- The retrieval router is a context map. Deciding which context (or contexts) a question belongs to is exactly the routing decision a context map describes.
- The glossary is a published language. A shared, curated term mapping is what lets a question phrased in one department's language reach another department's documents — deliberately, not accidentally.
- Cross-context lookup is an anti-corruption layer. Translation happens at the boundary, so neither context has to adopt the other's vocabulary internally.
This is not a metaphor stretched for the sake of it. It has a concrete consequence: the moment you tag chunks with a context and filter on it, an entire class of "confidently wrong" answers disappears — without changing the model at all.
Ingestion: where retrieval quality is actually decided
Most of the quality of a RAG system is determined before a single query is served. Two decisions matter more than the rest.
Chunk on structure, not on character count. Fixed-size chunking splits a procedure in half and severs a table from the heading that gives it meaning. Technical documents already carry structure — sections, numbered procedures, tables, revision blocks — and splitting along those seams keeps each chunk independently meaningful. Where a chunk is too small to stand alone, retrieve the small chunk but pass its parent section to the model.
Metadata is not optional bookkeeping — it is a retrieval feature. Context, document type, effective date, superseded-by, and access level all belong on the chunk, because each one becomes a pre-filter that removes wrong answers before ranking even begins. The effective-date field in particular is what stops the system quoting a specification that was withdrawn last year.
Retrieval: route first, then search hybrid, then rerank
With contexts in place, the query path becomes a short pipeline where each stage has one job.
Route. Classify the question into one or more contexts, using the asker's own department as a prior. Ambiguity is a legitimate outcome here — if a question genuinely spans two contexts, search both and say so in the answer, rather than silently picking one.
Search hybrid, not pure vector. This is the single most under-appreciated point in technical RAG. Dense embeddings are good at paraphrase and terrible at exact identifiers — part numbers, error codes, clause references, revision tags. Those are precisely what domain experts search for. Lexical search (BM25) handles them perfectly. Running both and fusing the result sets recovers the cases each one alone would miss.
Rerank. First-stage retrieval optimises for recall; a cross-encoder reranker over the top candidates optimises for precision. Cheap to add, and usually the largest single quality jump after hybrid search.
One rule worth enforcing at the boundary: filter by permission during retrieval, never after generation. If restricted content reaches the model, it will influence the answer even when the citation is stripped out. Access control belongs in the query filter, not in post-processing.
Linking departments without merging them
Partitioning solves wrong-context answers, but it creates a new question: what about the genuinely cross-functional query? A compliance officer asking about a maintenance interval needs engineering's documents, phrased in compliance's language.
The answer is not to merge the indexes back together. It is to translate at the boundary. A curated glossary maps each context's local term to a canonical concept, so a query can be expanded into the target context's vocabulary before searching it. Crucially, this mapping is owned by domain experts, not inferred by a model — it is a small, high-value artefact that a subject-matter expert can review in an afternoon, and it encodes knowledge no embedding will discover on its own.
Measuring it, so "better" isn't a vibe
A RAG system without an evaluation set is not an engineering artefact, it is a demo. The minimum useful setup is a golden set of real questions with the passages that should have been retrieved, collected from the domain experts who will actually use the system.
Measure retrieval and generation separately, because they fail for different reasons and have different fixes:
- Retrieval — recall@k and MRR, tracked per context. An aggregate number hides the one department where the system is useless.
- Groundedness — is every claim in the answer supported by a retrieved passage? This is the metric that catches confident fabrication.
- Routing accuracy — how often the router picks the right context. Cheap to measure and the first thing to check when quality drops.
- Refusal rate — a system that never says "I don't know" is not more capable, it is less honest.
Treat these as a regression suite. Every change — a new chunking strategy, a new embedding model, an expanded glossary — gets scored against the same golden set before it ships. Without that, "the new model felt better" is the only evidence you will ever have.
What I'd tell someone starting this today
- Draw the context boundaries before you write ingestion code. They are an organisational fact you are discovering, not a technical choice you are making.
- Hybrid retrieval is not an optimisation. In a domain full of identifiers and codes, pure vector search will fail on the exact queries experts care about most.
- Build the glossary with domain experts early. It is the highest-leverage artefact in the system and the only part a model cannot produce for you.
- Filter permissions in the query, not after generation. Anything the model sees, the model uses.
- Ship the evaluation set before the chat interface. Otherwise you cannot tell improvement from novelty.
The pattern underneath all of this is unglamorous and familiar: most of what makes an AI feature trustworthy in production is ordinary systems engineering — boundaries, metadata, filters, measurement — applied to a new kind of component. The model is the part everyone looks at. The retrieval design is the part that decides whether anyone keeps using it after the first week.