Direct answer
Five recurring causes. Documents are split by character count instead of by meaning, so answers arrive in fragments. No evaluation set is built, so nobody can tell whether the system works until staff stop trusting it. Retrieval accuracy and answer accuracy are treated as one number, hiding which part is broken. No one owns the content after launch, so it drifts out of date. And the corpus is assembled from what was easy to upload rather than from the questions people actually ask.
1. Chunking by character count
The most common failure, and the hardest to notice, because it does not produce errors — it produces plausible answers built from half a document.
A fixed-length split cuts a policy in the middle of a clause. Retrieval then returns a fragment, the model answers from the fragment, and the answer is confidently incomplete. The system looks like it works, which is why this failure survives so long.
2. No evaluation set
Without a set of real questions with known correct sources, there is no way to tell a working system from a broken one except by waiting for a colleague to complain. By then the complaint is about trust, not about a bug.
The evaluation set is the cheapest insurance in the project and it is almost always skipped, because building it feels like work that does not produce a feature.
3. One number for two problems
Retrieval and answering are separate stages with separate failure modes. If retrieval fails to find the right document, no model can answer correctly. If retrieval succeeds and the answer is still wrong, the problem is the answering step.
Track them separately. A single overall accuracy figure cannot tell you which stage to fix, and teams routinely spend effort tuning the model when the real problem was the split rule.
4. Nobody owns the content
A knowledge base is a living system. Prices change, policies are revised, products are discontinued. Without a named owner and a review rhythm, the base decays — and a knowledge base that is 90 per cent right is worse than one that is obviously out of date, because the errors are invisible.
5. The wrong corpus
Projects often start by uploading whatever was easiest to find. The result is a system that answers questions nobody asks and fails on the questions everybody asks.
Start from questions, not from folders. Collect the twenty questions your team actually asks each other, then find the documents that answer them.
Key facts
| Most common failure | Character-count chunking producing fragment-based answers |
| Most expensive omission | No evaluation set built before launch |
| Metric discipline | Retrieval accuracy and answer accuracy tracked separately |
| Operational requirement | A named content owner and a review rhythm |
| Correct starting point | The questions people actually ask, not the folders that were easiest to upload |
Sources
- Retrieval-augmented generation deployment practice
- Knowledge base chunking and evaluation methodology
Follow-up questions
Which of the five do I fix first?
The evaluation set. It is the only one that makes the other four visible. Build it, run it, and the ranking of your real problems usually reorganises itself.
How many documents do I need for a useful evaluation set?
The set should cover the questions staff genuinely ask, with the correct source document recorded for each. A set of fifty real questions is far more useful than a large synthetic one.
Can a good model compensate for bad chunking?
No. If the correct passage never reaches the model, no amount of model quality recovers it. Chunking and retrieval set the ceiling; the model works within it.