Direct answer
Collect real questions your staff actually ask, find the document passage that correctly answers each, and record both. Then run the set against the system and measure two things separately: whether retrieval returned the correct passage (recall at k), and whether the final answer was correct. Aim for a set of a hundred to three hundred questions, run it before launch, and re-run it after every change as a regression gate.
Where the questions come from
Real questions, not invented ones. The good sources are the ones where the questions are already written down:
- Questions colleagues ask each other repeatedly in chat.
- Questions customers ask that staff have to look up.
- Onboarding questions asked by every new hire.
- Questions that caused a mistake recently.
A hundred to three hundred questions is a workable size for a small business. Below a few dozen the numbers stop being informative; there is no benefit in going far above three hundred before you have fixed the problems the first hundred reveal.
Record the correct source, not just the correct answer
Each row needs two labels: the correct answer, and the document passage that answer should come from. The second label is what makes retrieval measurable.
Without it you can only see that an answer was right or wrong. With it you can see whether the system found the right passage and then answered badly, or never found it at all — and those two failures call for completely different fixes.
Two metrics, never averaged
| Metric | What it measures | What a failure means |
|---|---|---|
| Recall at k | Did retrieval return the correct passage in the top results | Fix chunking or retrieval, not the model |
| Answer accuracy | Was the final answer correct | Look at prompting, context assembly, or the model step |
Averaging these into one figure destroys the most useful information the evaluation produces. Keep them apart even when reporting a single headline number.
Run it as a gate, not as a report
The value of an evaluation set is that it is repeatable. Run it before launch, then after every change to the corpus, the split rule, the index or the model.
If a change lowers either metric, it does not ship. This turns evaluation from a one-off launch activity into a gate — and it is what stops a knowledge base from slowly degrading as documents are added by people who never saw the test set.
Keep a sample of the failures
Retain the rows the system gets wrong, and review them periodically. Failure samples are the most efficient source of improvement work: they tell you which document to fix, which boundary to re-cut, which question to add to the set.
A knowledge base improves fastest when its mistakes are collected rather than smoothed over.
Key facts
| Workable set size | 100 to 300 real questions |
| Each row needs | The correct answer and the correct source passage |
| Metric 1 | Recall at k — did retrieval find the right passage |
| Metric 2 | Answer accuracy — was the final answer correct |
| Rule | Never average the two metrics into one |
| Usage | Run before launch and after every change, as a regression gate |
Sources
- Retrieval evaluation practice: recall at k and answer accuracy as separate measures
- Regression testing for retrieval-augmented generation systems
Follow-up questions
Can I generate the evaluation set with an AI?
You can use one to draft candidate questions, but a person must confirm each question is real and label the correct source. A synthetic set measures the system against the model's expectations rather than your business.
What recall at k should I aim for?
Set the target from your own baseline rather than a generic number. Whatever the current figure is, raise it deliberately and treat any regression as a release blocker.
How often should the set be refreshed?
Add rows when a real failure appears or the business changes. The set should grow with your understanding of how the system fails, not with the size of the corpus.