Skip to content

Enterprise Search Data RAG

What is RAG?

Steve Sale
Steve Sale

If you've spent any time in enterprise search over the last few years, you've watched the ground shift beneath you. Keyword indices and even semantic search were once the finish line. Now they're the first stage of a longer pipeline, one that ends with a large language model synthesizing an answer, not just a ranked list of documents. That pipeline is called Retrieval-Augmented Generation, or RAG.

This post breaks down what RAG actually is, how the architecture fits together, and where it delivers real value versus where it's overhyped. Written for people who already understand search, and want the LLM half of the equation explained clearly.

What RAG Is (and Isn't)

RAG is an architectural pattern that combines two things that are individually well understood:

  1. Retrieval: Finding relevant information from a knowledge base (this is nearly always search.  While search is usually the best solution it, could also be knowledge graphs or even very well architected databases)
  2. Generation: Using a large language model to produce a natural-language answer.

The "augmented" part is the key idea: instead of asking an LLM to answer purely from what it learned during training, you inject retrieved, up-to-date, organization-specific content directly into its prompt at query time. The model then generates its answer grounded in that retrieved content, rather than relying solely on parametric memory.

It's worth being precise about what RAG is not:

  • It is not fine-tuning. You're not changing model weights; you're changing what the model sees at inference time.
  • It is not a replacement for a good search system, it's built on top of one. Garbage retrieval produces garbage generation.
  • It is not a single product. It's an architecture pattern implemented differently by every vendor and every in-house build.

The Core Architecture

A production RAG system typically has four layers. Understanding each one matters because failures tend to cluster at specific layers, and diagnosing "why did RAG give a wrong answer" means knowing which layer to inspect.

1. Ingestion and Indexing

Source documents (PDFs, wikis, ticketing systems, SharePoint, Confluence, code repos) are pulled in, cleaned, and indexed.  There are many different types of search engine, these are explained in another post; Keywords to Meaning. Metadata (source, date, access permissions, document type) is typically indexed alongside the content for filtering, relevance tuning etc.

2. Retrieval

At query time, the user's question is processed to generate a search query. There are many strategies for this from simply passing the question as it is to the search engine to formulating a custom search payload with preset filters and weights.  The search platform returns the most relevent document extracts that are sorted with the most relevant at the top.

3. Augmentation

The retrieved search results are assembled into a prompt alongside the user's original question and system instructions, then sent to the LLM. This is the "R" meeting the "G" — the model now has grounded context it didn't have during training.

4. Generation

The LLM produces a response conditioned on the retrieved content. Well-built systems also return citations, mapping specific claims in the generated answer back to source documents, which is critical for enterprise trust and auditability.

Why Not Just Use a Bigger LLM Context Window?

A fair question, given how large context windows have grown. Three reasons RAG still matters even with million-token contexts:

  • Cost and latency: Stuffing an entire knowledge base into every prompt is expensive and slow, retrieval narrows this to what's actually relevant.
  • Freshness: Retrieval pulls from a live index that can be updated continuously, without retraining or re-uploading a static context blob.
  • Precision: LLMs demonstrably perform worse when relevant information is buried in a sea of irrelevant context ("lost in the middle" effects). Retrieval curates signal from noise before generation even starts.

Common Use Cases

For an enterprise search practice, these are the patterns that tend to show up in real engagements:

  • Internal knowledge assistants: Employees asking natural-language questions across policy docs, HR systems, engineering wikis, and support tickets, with a synthesized answer instead of a document list.
  • Customer support augmentation: Grounding a support chatbot in product documentation and past resolved tickets to reduce hallucination and improve first-contact resolution.
  • Contract and compliance review: Retrieving relevant clauses across large legal or regulatory document sets and summarising obligations or risks.
  • Technical documentation search: Letting engineers query across codebases, API docs, and architecture decision records conversationally.
  • Sales and pre-sales enablement: Surfacing the right case study, pricing detail, or competitive positioning from a scattered content library in response to a rep's question

Where RAG Breaks Down

  • Retrieval quality caps generation quality: No amount of prompt engineering fixes a system that retrieves the wrong chunks. Most "RAG isn't working" complaints are retrieval problems, not generation problems.
  • Access control is easy to get wrong: If your search index doesn't enforce the same permissions as your source systems, you risk surfacing content a user shouldn't see. Conversly, the system may miss key facts the user does have access to see. 
  • Indexing and metadata design require real domain work: This isn't a plug-and-play layer, it's where most of the actual consulting effort goes.
  • Data Quality: RAG is not the silver bullet to resolving an organisations bad data.  Garbage in, garbage out still applies.  Ensuring documentation exists in the first place, thaat it is acurate, up to date & correctly stored is where most of the business effort is required for a good RAG solution.
  • Evaluation is still immature: Measuring whether a RAG system is "good" requires more than accuracy on a test set; it needs retrieval metrics, generation metrics, and end-to-end human evaluation.

The Takeaway

RAG isn't a departure from enterprise search, it's the next layer built on top of it. Every skill that matters in traditional search (relevance tuning, access control, metadata modeling, query understanding) still matters in RAG; it's just that the output is now a synthesized answer instead of a ranked list, and the failure modes are more subtle because a bad retrieval doesn't look like an empty results page, it looks like a confident, wrong answer.

That's exactly why this is such fertile ground for search practitioners moving into this space: the hard problems were never really about the LLM. They are about the search.

Share this post