Why Most Enterprise RAG Implementations Fail (And How to Fix Them)

The Problem With Off-the-Shelf RAG

Every enterprise RAG proof-of-concept looks good in a demo. You upload a few PDFs, wire up an embedding model, point it at GPT-4, and it answers questions. Then you move it to production — real documents, real users, real edge cases — and the cracks appear fast.
The failure modes are consistent. We've seen them across financial services, healthcare, and logistics clients. They're almost never about the LLM itself.

Failure Mode 1 — Naive Chunking

Fixed-size chunking (e.g., 512 tokens, hard split) destroys context. A contract clause that spans a page boundary gets split mid-sentence. The retriever fetches half a clause, the LLM hallucinates the rest.
What works: semantic chunking based on document structure. For contracts — split by clause. For technical manuals — split by section heading. For financial reports — split by table + surrounding paragraph as a unit.

def chunk_by_structure(doc: Document) -> list[Chunk]: 
""" Chunk based on detected document structure, not fixed token counts. """ 
sections = detect_sections(doc) # heading-aware parser 
return [ Chunk(text=section.text, 
               metadata={"heading": section.heading, "page": section.page}, ) for section in sections if len(section.text.strip()) > 50 ]

Failure Mode 2 — No Retrieval Evals

Teams obsess over generation quality (is the answer good?) but ignore retrieval quality (did we fetch the right chunks?). If retrieval is wrong, no LLM will save you.
Measure recall@k on a labelled eval set before optimising anything else. If recall@5 is below 0.7, the problem is in your embedding model, chunking strategy, or index configuration — not your prompt.

Failure Mode 3 — Single-Stage Retrieval

Dense vector search alone misses exact keyword matches. A user searching for "Section 4.2(b)" or a specific product SKU gets semantically similar chunks, not the exact clause. Hybrid retrieval (dense + BM25 sparse) consistently outperforms either approach alone. For most production systems we set alpha = 0.6 (dense) / 0.4 (sparse) as a starting point, then tune from eval results.

What Actually Works

The pattern that consistently ships to production:

1. Structure-aware chunking with overlap

2. Hybrid retrieval (dense + sparse)

3. Re-ranking with a cross-encoder before passing to the LLM

4. A retrieval eval suite run on every chunking or indexing change

5. A human-in-the-loop review path for low-confidence answers

None of this is exotic. It's engineering discipline applied to a new problem class.