RAG without the hype: when it helps and how not to break retrieval

RAG sounds like a technology of its own, but it’s really two steps in a row: find the passages in your documents that relate to the question, then hand them to the model along with the question and ask it to answer from them. The model doesn’t learn anything new. It reads what you put in front of it.

That leads to the conclusion that saves weeks: answer quality is almost entirely retrieval quality. If the right paragraph didn’t make it into the results, no model will come up with it. Well, it will come up with something, which is worse.

When RAG helps

Knowledge changes faster than you’d want to retrain anything: docs, prices, policies, a ticket history.
You need to show where the answer came from: legal, support, internal reference.
Different people should see different things. Retrieval can respect permissions; knowledge baked into a model can’t.

When you don’t need it

There isn’t much content. If the whole base fits comfortably in the model’s context, just send all of it. Fewer moving parts, fewer ways to miss.
People search for specific things. If queries are SKUs, names or contract numbers, plain full-text search with filters will do better and faster. Sometimes “chat with your docs” is a search box that just needed decent ranking.
You need style, not facts. If the model must write in a certain voice or reliably follow a complex format, that’s a job for examples in the prompt, or fine-tuning as a last resort. RAG supplies facts, not manners.

Chunking: boring, but decisive

Documents are cut into chunks, and each chunk is searched separately. The usual mistakes:

cutting by character count: sentences break mid-way, tables fall apart, a heading lands in one chunk and its content in the next;
too small: the chunk is found but useless without context;
too large: a whole page comes back with one relevant line, and the model loses it in the noise.

Cut along the document’s structure: sections, subsections, paragraphs. Prepend each chunk with its section heading and document title. It’s cheap and helps both retrieval and the model. A small overlap between neighbors prevents breaks at the boundary.

Test retrieval separately from the model

The most common mistake is judging the system only by its answers. An answer is bad and you can’t tell why: nothing was found, the wrong thing was found, or the model misread it.

Collect a couple dozen real questions and mark the correct chunk for each. Then look at retrieval alone: did the right chunk land in the top few results? It takes minutes and shows you exactly where to fix.

What usually helps:

hybrid search: vector search catches meaning but misses exact strings like codes and names; full-text search is the opposite. Together they beat either one;
reranking: fetch extra results and re-sort them with a more precise model;
metadata filters: date, product, language, access rights. Don’t search the whole base when the question is clearly about one section.

Cite the source

Ask the model to answer only from the provided chunks and to say which chunk each claim comes from. In the UI, keep those sources one click away. This isn’t decoration: the user can check the answer, and you can spot the model citing a chunk that doesn’t say what it claims.

Also agree on what happens when the answer isn’t in the documents. “I couldn’t find this in the knowledge base” is a fine answer. A confident summary of something the base doesn’t contain is not.

The upshot

RAG is search first and a model second. Start by asking whether you need it at all, chunk documents by meaning, test retrieval on its own and always show the source. Then “chat with your docs” will answer from the docs, not from its mood.

SUBXCAT can measure retrieval quality in your RAG setup and find the step where the right answers get lost.