Direct answer
Split by structure, not by length. An FAQ entry becomes one chunk — question and answer together. A policy document splits at clause boundaries, and each chunk carries its clause number back into the index so the citation is meaningful. Structured documents with fields split along their fields. Where structure genuinely cannot be determined, fail loudly rather than falling back to a fixed character length, because a silent bad split is far more expensive than an error you can see.
The rule: split where the document already splits
Documents carry their own boundaries. Your job is to find them and cut there, rather than imposing a length from outside.
| Document type | Split rule | What travels with the chunk |
|---|---|---|
| FAQ entries | One question-and-answer pair per chunk | The question itself |
| Policy and procedure documents | At clause boundaries | The clause number |
| Structured records | Along the field structure | The field names |
| Product specifications | Per specification item | Product identifier and item name |
The pattern is consistent: the chunk keeps whatever identifies it, so that a citation points somewhere real.
Why the clause number must travel with the chunk
If the answer to a question is that a policy says a thing, the useful citation is not the document title — it is the clause. Storing the clause number on the chunk means the retrieved passage can be cited precisely, and it gives you a way to check the answer independently.
It also creates a second benefit: chunks from the same clause can be grouped, so a question that needs two related passages can retrieve both.
Two index views, not one
Different document types are searched differently, which argues for more than one index view over the same corpus:
- A question-to-answer index for FAQ-type content, where the user question is matched against a stored question.
- A clause index for policy content, where the search is for a topic and the clause number comes back as part of the result.
Running both against one naive index is where a lot of accuracy quietly leaks away.
The fallback rule
Where structure cannot be determined, the tempting fallback is a fixed character split. Resist it. A fixed split produces chunks that are retrievable but wrong, and nothing downstream flags them.
Failing on an unparseable document produces an error, an error produces a fix, and the fix improves the corpus. A silent bad split produces a wrong answer and no signal at all. Given a choice between a visible failure and an invisible one, choose the visible one.
Key facts
| Core rule | Split where the document already splits |
| FAQ content | One question-and-answer pair per chunk |
| Policy content | Split at clause boundaries, carrying the clause number |
| Index design | Separate question-to-answer and clause-based index views |
| Fallback policy | Fail loudly; never fall back to a fixed character split |
Sources
- Retrieval-augmented generation chunking practice
- Document structure parsing and citation design
Follow-up questions
Is overlap between chunks useful?
Overlap is a patch for bad boundaries. If you are splitting by structure, most of the need for overlap disappears. Reach for overlap only after you know which boundary is causing the miss.
What about tables?
Keep a table with its header row. A table split from its headers becomes a set of numbers with no meaning, which is one of the more damaging ways a chunk can fail.
How do I check whether my splitting is any good?
Take your evaluation questions and look at what retrieval returns. If the returned passage cannot answer the question on its own, the split rule is the problem, not the model.