71%
of lines contain at least one model mistake (114 and 111 of 161 lines)
Method case study · Human-verified eval · Historical Urdu Nastaliq
Frontier models cannot read historical Urdu yet.
We ran a blind test of Claude Opus 5.5 and Claude Fable 5.1 on a 1908 lithographed Urdu dictionary, scored word by word against ground truth typed by a native expert. Every disagreement was ruled against the scan. The sample, the method and the word-level results are published here. This is one example of our method: the same blind, expert-written, adjudicated harness applies in any domain where a vetted expert can write the truth, from clinical medicine and law to physics and engineering.
55 · 50
words each model got wrong with no doubt flagged (Opus · Fable)
90 · 91%
strict word accuracy (Opus 89.95% · Fable 91.08%): about 1 word in 10 not reproduced exactly
130 · 113
diacritics written by the katib that each model dropped
What the test shows
The errors are silent
Both models were told to flag any word they were unsure of. They still misread 55 and 50 words without a flag. Nothing in the output marks them as wrong.
The historical hand is erased
The katib wrote pre-Partition Urdu in full rasm-ul-khat, with shadd, jazm, zer and pesh. The models dropped 130 and 113 of these marks, rewriting 1908 as modern print.
Why it matters
Urdu has 246 million speakers (Ethnologue 2025), yet it appears in none of the 52 disclosed AI data-licensing deals compiled by Neudata. Verified ground truth for historical Nastaliq is scarce, so there is little to train on and nothing standard to measure against.
misread, no doubt flagged
misread, doubt flagged
diacritic differs
Results
Word by word, page by page
Strict word accuracy is the share of the human truth's words reproduced exactly: a word is wrong if it is misread, missing, loses or changes a diacritic, or differs in punctuation, spacing or order. Reading accuracy is a lenient secondary view that counts only misread, missing and invented words. Silent errors are reading errors with no doubt flag.
| Page | Opus 5.5 strict word accuracy | Reading accuracy | Misread | Silent | Fable 5.1 strict word accuracy | Reading accuracy | Misread | Silent |
|---|---|---|---|---|---|---|---|---|
| Page 12 | 93.10% | 98.71% | 6 | 4 | 94.83% | 97.20% | 13 | 6 |
| Page 13 | 88.44% | 96.15% | 37 | 22 | 89.58% | 95.00% | 48 | 26 |
| Page 14 | 89.94% | 95.70% | 41 | 29 | 90.78% | 96.12% | 37 | 18 |
| All 3 pages | 89.95% | 96.47% | 84 | 55 | 91.08% | 95.88% | 98 | 50 |
Human truth: 2,378 words and marks over 161 lines, typed blind in 166 minutes.
The eval harness
Built so the truth can't inherit a model's mistakes
- Native-resolution cropsEach page is cut into a header strip and 8 overlapping bands per column at the scan's native 400 DPI. Every model sees identical images.
- Blind model readsEach model reads in a fresh context with a fixed protocol: type exactly what is printed, one line per printed line, and mark every doubt.
- Blind human truthA vetted native expert types the page from the scan on a screen that shows no machine text.
- AdjudicationEvery spot where the models agree against the expert is re-checked against the scan, and each decision is logged.
- ScoringUnicode NFC, lookalike-letter folding, word alignment by edit distance, and a type for every difference: misread, missing, invented, diacritic, punctuation, spacing or order.
Provenance
Source
Farhang-i-Asifiyah, vol. 4 (1908), compiled by Syed Ahmad Dehlvi (1846–1918). Public domain in the US and in its countries of origin. Pages 12–14.
Human-only truth
The ground truth is typed by people from the scan. Model outputs appear only as scored evaluation evidence.
Behind the sample
124 titles sourced (49,421 pages) in Urdu, Punjabi, Sindhi, Pashto, Balochi and Saraiki, selected as public-domain works published before 1931.
Data format
One record per aligned word, per model
- page
- Page of the source volume
- model
- Model ID that produced the reading
- band
- Scan band: header, R1–R8 (right column) or L1–L8 (left column)
- truth
- Human-verified word or mark
- model_output
- What the model wrote in that position
- result
- ok, wrong_word, missing, invented, diacritic, punctuation, spacing or order
- model_flagged_doubt
- Whether the model marked the word as uncertain
- reading_error
- True for misread, missing and invented words
- scan_crop
- The crop the word was read from
A real record
{
"page": 13,
"model": "claude-fable-5-1",
"band": "R3",
"truth": "مدھ",
"model_output": "مدہ",
"result": "wrong_word",
"model_flagged_doubt": false,
"reading_error": true,
"scan_crop": "p13_R3.png"
}
Work with us
Measure your model on pages it has never seen.
Pilot: a 250-page held-out set of historical Nastaliq, scored against your model with this harness. We send the scans, you send your model's output, and we return the scorecard and full error log. Your model never sees the truth.
- Held out: never published, never online, never in a training crawl
- Every word typed blind by a vetted native expert and adjudicated against the scan
- Word-level error log you can audit, in the JSONL format above
- The same method for other languages and scripts, and for expert domains such as medicine, law, science and engineering
Request a pilot