3 min read

RAG in Production: Grounding LLM Answers in Your Own Data

Retrieval-augmented generation is easy to demo and hard to run. This is a practical guide to chunking, embeddings, hybrid retrieval, re-ranking, citations, and the evaluation loop that keeps a production RAG system honest.

AIRAGLLMPython

Retrieval-augmented generation is the most practical way to make a language model useful over private data. It is also routinely done badly. Teams wire an embedding model to a vector store, get impressive results on three test questions, ship it, and then discover that real users ask things the retrieval layer never anticipated. The fix is almost never a better model. It is better retrieval.

Chunking decides your ceiling

The way you split documents determines what is even retrievable. Fixed-size chunks cut sentences in half and separate a heading from the paragraph that explains it. Structure-aware chunking respects headings, tables, and code blocks, and carries a small amount of parent context with each chunk.

  • Chunk on semantic boundaries — headings, sections, list items — not on a fixed character count.
  • Overlap adjacent chunks slightly so a boundary does not hide the answer.
  • Attach source metadata to every chunk: document, section, URL, and last-updated date.
  • Keep tables intact. Splitting a table across chunks destroys its meaning.

Retrieval should be hybrid, not purely semantic

Embeddings are excellent at meaning and terrible at exact identifiers. If a user searches for an error code, a product SKU, or a person's name, lexical search wins. Run both and merge the results.

  1. Run a keyword search (BM25 or a database full-text index) for exact terms.
  2. Run a vector search for semantic matches.
  3. Fuse the two ranked lists, for example with reciprocal rank fusion.
  4. Re-rank the merged candidates with a cross-encoder before they reach the model.
fused = rrf([bm25_results, vector_results], k=60)
top = cross_encoder_rerank(fused, query)[:8]

Re-ranking is the highest-value upgrade

If you can only change one thing in an existing RAG system, add a re-ranker. Retrieving fifty candidates and re-ranking down to eight almost always beats tuning the embedding model.

The prompt is the smallest part

Once retrieval is good, the generation prompt can be short. Its job is to constrain the model to the retrieved context, require citations, and admit ignorance when the context does not contain the answer.

  • Instruct the model to answer only from the provided context.
  • Require a citation for every factual claim, pointing at a chunk identifier.
  • Tell it to say 'I could not find that' rather than guessing.
  • Keep the temperature low for factual tasks.

Stop hallucinations at the boundary

The most valuable safety feature is a refusal path. If retrieval returns nothing above a similarity threshold, do not call the model with an empty context and hope for the best — return a clear 'no answer found' and offer a way to escalate. A confident wrong answer costs far more trust than an honest miss.

Evaluate or drift

RAG systems degrade quietly as documents change and user questions drift. Build an evaluation set of real questions with known-good answers, and re-run it on every change to the pipeline.

  • Track retrieval recall: was the correct chunk in the top-k?
  • Track answer faithfulness: does every claim trace to a retrieved chunk?
  • Track refusal rate: are you refusing questions you should answer?
  • Log the exact chunks used for every answer so failures are debuggable.

Treat RAG as a search problem. Get retrieval right and the language model looks far smarter than it is. Get it wrong and no model will save you.

Ground every answer in a citation, refuse when you have nothing, and measure retrieval independently of generation. That combination is what turns an impressive demo into a system people actually rely on.

More writing

All writing