Retrieval that survives real documents

When a retrieval-augmented system gives a wrong answer, the instinct is to blame the model. Usually the model was never given the right passage. It answered exactly as well as the context allowed, and the context was bad.

Retrieval quality is measurable and fixable. It is also where the majority of the engineering effort belongs.

The demo corpus lies

Prototypes are usually built against a clean corpus: a few dozen well-structured documents, one language, consistent formatting. Real corpora are not that.

Real corpora are scanned PDFs where the text layer is missing or wrong. Tables that mean nothing once flattened into a paragraph. Documents where the same term is used three ways across five years. Ten versions of the same policy, only one of which is current, with no field saying which. Mixed languages inside one page.

Every one of these is an ingestion problem, not a model problem — and none of them appear in a clean demo.

Where retrieval quality is actually won

Parsing. If your parser turns a table into an unreadable string, no amount of embedding quality recovers the meaning. Layout-aware parsing, OCR fallback for image-only pages, and preservation of table structure change results more than any reranker.

Chunking. Fixed-size chunking splits sentences, separates a heading from the clause it governs, and orphans a figure from its caption. Structure-aware chunking — splitting on document sections, keeping headings attached, overlapping enough to preserve context — is unglamorous and consistently effective.

Metadata. Effective date, document version, jurisdiction, department, status. Without these you cannot filter, and without filtering the system will confidently cite a superseded document. Most “hallucination” complaints we investigate are actually a correct quote from a document that should have been excluded.

Hybrid search. Dense embeddings miss exact identifiers — part numbers, statute references, error codes, names. Keyword search misses paraphrase. Running both and fusing the results catches what either alone drops.

Reranking. Retrieve broadly, then let a cross-encoder or a small model rank the candidates by actual relevance to the question. This is the cheapest large improvement available in most systems.

Measure retrieval separately

The critical discipline: evaluate retrieval on its own, before the generation step.

Take a set of real questions. For each, mark which passages a competent human would need in order to answer. Then measure whether your retriever returns them — recall at five, recall at twenty, position of the first relevant hit.

If recall at twenty is 60%, no prompt engineering will save the answer quality. You are debugging the wrong layer. Fixing retrieval first is faster, cheaper, and the improvement is permanent rather than model-dependent.

Grounding and refusal

Two behaviours separate a system people trust from one they stop using.

Citations that resolve. Every claim links to the passage it came from, and the link opens to the right place in the source document. Users verify one or two, find them accurate, and calibrate their trust correctly.

A working “I don’t know”. When retrieval returns nothing relevant, the correct output is an explicit no-answer, not a fluent guess assembled from adjacent material. This has to be tested deliberately: put questions in your evaluation set that the corpus genuinely cannot answer, and score the refusal.

The order of operations

For a new system, the sequence that consistently works:

1. Ingest and parse properly, with metadata, before touching prompts.
2. Build a retrieval evaluation set from real questions.
3. Add hybrid search and a reranker; measure the delta.
4. Only then tune generation — and re-measure retrieval after every corpus change.

It is slower for the first two weeks and considerably faster after that. The teams that skip to step four spend months tuning prompts against a retrieval layer that was never going to hand them the right passage.