Your RAG Isn't Dumb, Your Chunks Are: Fixing Long-Document Retrieval

This is a build-in-public post about Labiba AI, a RAG assistant I built solo for Zain's Legal Affairs department.

The bug report that taught me the most about RAG didn't come from a log file. It came from a legal officer who said, politely but firmly: "I know the answer is in this document. I can open it and point at the paragraph. Why can't Labiba find it?"

They were right. And once I dug in, the pattern was unmistakable: the failures clustered on long, multi-page documents where the relevant information sat near the end. Ask about something on page one of a letter and Labiba nailed it. Ask about the commitment buried in the closing paragraphs of page nine, and it either retrieved the wrong document or answered from insufficient context.

Why late pages lose

The standard RAG recipe splits every document into small chunks and embeds each one independently. That works better than it has any right to — until document structure starts to matter. Legal letters front-load their identity: the subject line, the parties, the reference numbers all live on page one. A chunk from page nine is often a fragment of mid-argument prose with none of that identity attached. Embedded in isolation, it doesn't look like it belongs to the document it came from.

So when a user asks a question that mentions the case, the counterparty, or the subject, the chunks that match are the early ones. The late-page chunk holding the actual answer scores poorly and never makes the top-k. The retriever isn't broken; it's doing exactly what it was told with the information it was given. The information was the problem.

The fix: match small, answer big

What worked for Labiba was parent-document retrieval — decoupling what you search from what you read:

• Index small. Documents are still split into small chunks for embedding, because small chunks are precise matchers — they keep vector search sharp.

• Retrieve big. When a chunk matches, I don't hand the model that fragment. I resolve it back to its parent document and hand the model the whole thing.

• Let the model do what models are good at. Finding a needle inside one document that's actually in front of it is something a capable model does reliably. Finding that needle across an entire archive through a keyhole of disconnected fragments is not.

The chunks became signposts instead of source material. Once any part of the right document matched, the model saw pages one through nine together — the identity from the front and the answer from the back, in the same context.

What it cost

Nothing is free. Whole documents mean far more tokens per query, and on locally hosted models — Labiba runs gemma4:26b via Ollama behind a LiteLLM proxy — context space is a real budget, not an abstract one. Long documents needed guardrails, and latency per question went up. But for a legal department, the trade was obvious: a slower correct answer beats a fast wrong one, every single time.

There's also a second path that sidesteps retrieval entirely: Labiba lets users upload a specific document and interrogate it directly. When someone already knows which letter they're holding, forcing that flow through vector search is pure downside — just give the model the document.

What I'd tell past me

Evaluate retrieval separately from generation, and build the eval set from real user questions — not the questions you'd ask your own system. Mine were unconsciously shaped by how I knew the index worked, so I asked page-one questions and everything looked great. My users asked page-nine questions and found the truth in a week. They always do.