Direct answer
Yes. On 61 real spoken-language questions (31 answerable, 24 guidance-type, 6 that must transfer), the DeepSeek route reached 100% (61/61) on the should-this-transfer-to-a-human binary decision with zero mis-transfers; a local small-model route reached 96.7% (59/61) with 2 mis-transfers. Both used the same production routing prompt. We kept DeepSeek for the decision layer — one of the local model's two mis-transfers was a customer saying 'I can't explain it, just handle it for me', which is exactly the case that must not be transferred.
How the test was set up
- Regression set: 61 real spoken-language questions with expected routing labels (31 answerable from the knowledge base / 24 guidance / 6 that must transfer).
- Two routes compared: ① cloud DeepSeek generating the decision directly ② a local small model reading logits.
- Same prompt: both routes used the production triage prompt, so we were not comparing different questions.
Results
| Metric | DeepSeek route | Local small model |
|---|---|---|
| Transfer binary accuracy | 100% (61/61) | 96.7% (59/61) |
| Must-transfer cases transferred | 6/6 | 6/6 |
| Mis-transfers | 0 | 2 |
| Three-way classification | 80.3% | 70.5% |
| Average latency | 1.14 s | 0.46 s |
| Cost | Pay per use | 0 (local compute) |
Our conclusion
We did not switch; the decision layer stays on DeepSeek. The reason is the standard's explicit requirement not to over-use transfer to humans: one of the two local-model mis-transfers was a customer saying 'I can't explain it, just handle it for me' — precisely the case that must not be escalated.
A methodology lesson worth repeating
When evaluating a local instruction model, you must apply the chat template and give examples. The same model, called as a bare completion, had a transfer recall of 0%; with the standard chat template plus three examples it rose to 100%. Bad methodology produces the false conclusion that 'the model can't do it'.
Key facts
| k | v |
| k | v |
| k | v |
| k | v |
| k | v |
Sources
- Our 2026-09 measurement: 61-question spoken regression set, both routes, identical prompt
- GB/T 47746—2026 chapter 5 (transfer-to-human requirements)
Follow-up questions
Is 61 questions enough?
Enough as a gate, not enough as a statistical conclusion. It detects regression before and after a change — small sample, sensitive to direction. For statistical significance you would expand the set.
Can I see the raw data?
Yes. On delivery the regression set and per-question results are handed over together, and you can re-run them yourself.