Production-Ready RAG Architecture

Chunking, done right

Chunking is just cutting long documents into smaller pieces so retrieval can find the right piece instead of returning an entire 40-page PDF. Cut wrong and you split a sentence in half — or separate a table from its own header.

• Cut on natural boundaries — headings, paragraphs — not a fixed character count
• Overlap chunks by ~10–15% so a fact split across a cut isn't lost
• Attach metadata to every chunk (title, section, date, source, permissions) — this is what filtering and access control run on later
• For big documents, cut twice: one broad summary chunk, plus smaller detail chunks underneath it, so retrieval can start broad and drill down

Hybrid search & reranking

Vector search alone is bad at exact things — a policy number, a product SKU, a legal clause quoted word for word. Classic keyword search (BM25) is bad at understanding what you meant if you didn't use the exact words. Hybrid search runs both and merges the results.

Reranking is the step after: hybrid search might hand back 40 "maybe relevant" chunks. A reranker — a smaller, cheaper model whose only job is judging relevance — re-sorts them so only the best 5–8 reach the expensive LLM call. This one step usually improves answer quality more than switching to a bigger model.

Avoiding hallucinations

A hallucination isn't the model "lying" — it's the model generating something plausible when the retrieved context didn't actually contain the answer. Fight this at the retrieval step, not the prompt:

• Set a minimum relevance score — if nothing retrieved clears the bar, say "I don't have that" instead of guessing
• Require every claim in the answer to point back to a specific retrieved chunk
• If two retrieved sources disagree, flag it instead of silently picking one
• Route high-stakes answers (medical, legal, financial) to a human reviewer instead of auto-sending

Backpressure, retries, background jobs, fallbacks

Backpressure: if ingestion can process 100 documents a second and a source system dumps 50,000 changed files at once, queue that work and process it at a sustainable rate instead of falling over — the same idea as a message queue absorbing a spike.

Retries: an embedding call fails maybe 1 in 500 times. Retry with backoff and a cap, and make retries idempotent — retrying the same job twice should never create duplicate data.

Background jobs: indexing a document should never happen inside a request a user is waiting on. A source changes, an event fires, a worker indexes it — the user's next query just sees the updated result.

Fallbacks: reranker down? Skip it, use raw hybrid results. Primary LLM down or rate-limited? Fall back to a secondary model. Retrieval times out? Say "let me get back to you" instead of hanging the request. Every stage needs a defined "what if this fails," not just a happy path.

Evals — how to actually test this

You can't eyeball-test a RAG system the way you'd test a normal feature — the same question can get a slightly different answer every run. You need a fixed set of test questions with known-correct answers (or known-correct source documents), run automatically every time chunking, the embedding model, or a prompt changes.

• Retrieval precision/recall — did it fetch the right chunks, and how much noise came with them
• Answer correctness — does the generated answer match the expected one
• Groundedness — is every claim actually backed by a retrieved chunk
• Hallucination rate — how often it says something no retrieved chunk supports
• Latency — end-to-end, not just the LLM call

Tools people actually use: Ragas and TruLens (open-source RAG eval frameworks that automate most of the above), LangSmith and Langfuse (tracing, eval, and prompt versioning together) — and for a lot of teams, a small custom eval harness, because off-the-shelf tools don't always match your exact definition of "correct."

Checking output quality in production

Beyond automated evals, three checks teams routinely skip:

• Show the user which sources the answer came from — if you can't show them, you probably shouldn't trust the answer either
• Log every query, its retrieved chunks, and the final answer — when someone reports a wrong answer, that log is the only way to tell whether retrieval or generation was at fault
• Sample real production conversations weekly and have an actual human read them, not just watch a dashboard number