Test sets
Native-written test sets
Prompts, reference answers and rubrics written in the language by native experts.
Multilingual & low-resource language evaluation
Evaluation in the language your users actually speak.
Native speakers with subject expertise write, rate and red-team in Arabic (MSA and dialects), Urdu, Spanish, Portuguese, English and other languages per brief. Prompts are written in the language, never translated from English.
72-hour paid pilot · Evidence for every expert · Free replacement
What you receive
- Native-written test sets
- Dialect coverage
- Script & OCR ground truth
- Cultural & safety review
Delivered as
- Native-written prompts
- Ratings
- Transcriptions
- Error logs
The problem
Translated benchmarks measure translation.
Most multilingual evals are English prompts run through machine translation. They miss dialect, script, register and culture, which is where models fail in front of real users.
We built a human-verified Urdu Nastaliq evaluation from a 1908 dictionary. A frontier model made a mistake in about 70% of lines, at about 90% real accuracy against the 98% it claimed.
- Prompts translated from English
- Dialects and scripts left untested
- No native speaker in the review loop
What we deliver
What you receive.
Native speakers with subject expertise in Arabic, Urdu, Spanish, Portuguese, English and other languages per brief. Delivered in your format, with a provenance record for every item.
Dialects
Dialect coverage
Arabic MSA alongside dialects such as Gulf, Levantine and Egyptian, staffed per brief.
Scripts
Script & OCR ground truth
Nastaliq, Naskh and other scripts typed blind by native experts and adjudicated.
Culture
Cultural & safety review
Answers checked for register, cultural fit and local safety norms.
Localization
Translation QA
Native review of translated and localized output, with errors labelled by type.
Corpus
Regional-language book corpus
Held-out pages from a regional-language book corpus, held with publisher and writer permissions.
Typical outputs
- Native-written prompts
- Ratings
- Transcriptions
- Error logs
How vetting works
Every expert comes with the evidence.
Rather than a profile and a score, you get the record behind each person on your project, and you can review it before they start.
- Recorded AI video interview with the full transcript and a scored rubric
- ID and liveness check, with duplicate-account detection
- Credentials checked with the issuing body where the role needs them
- Integrity flags from the interview, shown to you in plain words
- Paid calibration against gold answers before any client work
How an engagement runs
From first call to scale, pilot first.
Scope
Tell us the task, the domain, the languages and your quality bar. We write the rubric with you.
NDA on requestMatch
We staff vetted experts whose field and language fit the task, and share their evidence packs.
Evidence per expertPilot
A 72-hour paid pilot on your real task, with gold items, dual review and agreement scores.
72 hours, paidScale
Scale on the model that suits you: hourly, per task or a managed pod. Free replacement throughout.
Free replacement
Related solutions
Often paired with this.
- Evaluate LLM evaluation Held-out test sets, rubric scoring and side-by-side comparisons by experts in your field.
- Safety Red-teaming & AI safety Adversarial probes and safety evaluations by domain experts and native speakers.
- Speech Audio & speech data Recording, transcription and speech-model evaluation by verified native speakers.
Get started
Tell us the gap. We'll scope the pilot.
Every engagement starts with a 72-hour paid pilot. We reply to every request within two business days.
Talk it through
A 30-minute call with the founding team. Bring the task, the domain and your quality bar.
Get a quote
Share the scope in two minutes: service, team size, languages, start date and budget band.