If you've spent any time in enterprise search over the last few years, you've watched the ground shift beneath you. Keyword indices and even semantic search were once the finish line. Now they're the first stage of a longer pipeline, one that ends with a large language model synthesizing an answer, not just a ranked list of documents. That pipeline is called Retrieval-Augmented Generation, or RAG.
This post breaks down what RAG actually is, how the architecture fits together, and where it delivers real value versus where it's overhyped. Written for people who already understand search, and want the LLM half of the equation explained clearly.
RAG is an architectural pattern that combines two things that are individually well understood:
The "augmented" part is the key idea: instead of asking an LLM to answer purely from what it learned during training, you inject retrieved, up-to-date, organization-specific content directly into its prompt at query time. The model then generates its answer grounded in that retrieved content, rather than relying solely on parametric memory.
It's worth being precise about what RAG is not:
A production RAG system typically has four layers. Understanding each one matters because failures tend to cluster at specific layers, and diagnosing "why did RAG give a wrong answer" means knowing which layer to inspect.
Source documents (PDFs, wikis, ticketing systems, SharePoint, Confluence, code repos) are pulled in, cleaned, and indexed. There are many different types of search engine, these are explained in another post; Keywords to Meaning. Metadata (source, date, access permissions, document type) is typically indexed alongside the content for filtering, relevance tuning etc.
At query time, the user's question is processed to generate a search query. There are many strategies for this from simply passing the question as it is to the search engine to formulating a custom search payload with preset filters and weights. The search platform returns the most relevent document extracts that are sorted with the most relevant at the top.
The retrieved search results are assembled into a prompt alongside the user's original question and system instructions, then sent to the LLM. This is the "R" meeting the "G" — the model now has grounded context it didn't have during training.
The LLM produces a response conditioned on the retrieved content. Well-built systems also return citations, mapping specific claims in the generated answer back to source documents, which is critical for enterprise trust and auditability.
A fair question, given how large context windows have grown. Three reasons RAG still matters even with million-token contexts:
For an enterprise search practice, these are the patterns that tend to show up in real engagements:
RAG isn't a departure from enterprise search, it's the next layer built on top of it. Every skill that matters in traditional search (relevance tuning, access control, metadata modeling, query understanding) still matters in RAG; it's just that the output is now a synthesized answer instead of a ranked list, and the failure modes are more subtle because a bad retrieval doesn't look like an empty results page, it looks like a confident, wrong answer.
That's exactly why this is such fertile ground for search practitioners moving into this space: the hard problems were never really about the LLM. They are about the search.