Test sets
Held-out test sets
New, unpublished prompts and reference answers written by practitioners and never posted online.
LLM evaluation
Know what your model gets wrong before your users do.
Held-out test sets, rubric scoring and side-by-side comparisons, written and graded by vetted experts in the field and language you ship in. Every score traces back to a verified rater.
72-hour paid pilot · Evidence for every expert · Free replacement
What you receive
- Held-out test sets
- Rubric scoring
- Side-by-side comparisons
- Error taxonomies
Delivered as
- JSONL or CSV
- Your eval harness
- Scorecards
- Error logs
- Adjudication notes
The problem
Benchmarks saturate. Your users don't.
Public benchmarks leak into training data and stop telling you anything new. Model-graded evals share the blind spots of the models they grade. And a generalist rater can't tell a plausible answer from a correct one in cardiology, tax law or Urdu poetry.
In our published Urdu study, a frontier model claimed about 98% accuracy. Vetted human review measured about 90%, with a mistake in about 70% of lines. Only people who know the field catch that gap.
- Public benchmarks that leaked into training data
- Model-graded evals that miss confident errors
- Raters without the knowledge to judge the answer
What we deliver
What you receive.
Physicians, lawyers, financial analysts, engineers, STEM PhDs and native speakers, matched to the task. Delivered in your format, with a provenance record for every item.
Scoring
Rubric scoring
Item-level scores against a rubric you approve, each with a short written rationale.
Comparison
Side-by-side comparisons
Pairwise judgments between checkpoints or competing models, blind to which is which.
Diagnosis
Error taxonomies
Every failure labelled by type and severity, so your engineers know what to fix first.
Quality
Agreement you can see
Dual review on a sample, inter-rater agreement and adjudication notes in every report.
Tracking
Regression runs
The same vetted panel re-scores new checkpoints, so results stay comparable over time.
Typical outputs
- JSONL or CSV
- Your eval harness
- Scorecards
- Error logs
- Adjudication notes
How vetting works
Every expert comes with the evidence.
Rather than a profile and a score, you get the record behind each person on your project, and you can review it before they start.
- Recorded AI video interview with the full transcript and a scored rubric
- ID and liveness check, with duplicate-account detection
- Credentials checked with the issuing body where the role needs them
- Integrity flags from the interview, shown to you in plain words
- Paid calibration against gold answers before any client work
How an engagement runs
From first call to scale, pilot first.
Scope
Tell us the task, the domain, the languages and your quality bar. We write the rubric with you.
NDA on requestMatch
We staff vetted experts whose field and language fit the task, and share their evidence packs.
Evidence per expertPilot
A 72-hour paid pilot on your real task, with gold items, dual review and agreement scores.
72 hours, paidScale
Scale on the model that suits you: hourly, per task or a managed pod. Free replacement throughout.
Free replacement
Related solutions
Often paired with this.
- Audit Hallucination & fact-check audits Claim-by-claim checks against primary sources, with every error logged and graded.
- Experts Domain expert evaluation Physicians, lawyers, finance professionals and STEM PhDs, credentials checked at the source.
- Languages Multilingual & low-resource language evaluation Arabic (MSA and dialects), Urdu, Spanish, Portuguese and more, by native speakers.
Get started
Tell us the gap. We'll scope the pilot.
Every engagement starts with a 72-hour paid pilot. We reply to every request within two business days.
Talk it through
A 30-minute call with the founding team. Bring the task, the domain and your quality bar.
Get a quote
Share the scope in two minutes: service, team size, languages, start date and budget band.