Preferences
Pairwise & ranked preferences
Side-by-side choices and full rankings, each with a written rationale.
RLHF & preference data
Preference data from people who know the right answer.
Pairwise rankings, rubric ratings, ideal-response rewrites and reasoning traces from credentialed professionals and native speakers. Each item traces back to the vetted expert who made it.
72-hour paid pilot · Evidence for every expert · Free replacement
What you receive
- Pairwise & ranked preferences
- Rubric ratings
- Expert rewrites
- Reasoning traces
Delivered as
- Pairwise JSONL
- Ranked lists
- SFT pairs
- Rubric scores
- Rationales
The problem
A preference is only as good as the person choosing.
Post-training learns whatever your raters reward. When raters prefer the longer, more confident answer over the correct one, the model learns to sound right rather than be right.
Expert preference data means fewer, cleaner signals, each with a rationale your researchers can audit and a verified person behind it.
- Raters rewarding length and confidence
- Rankings with no rationale behind them
- Domain errors that slip into the reward signal
What we deliver
What you receive.
Credentialed professionals, STEM PhDs, senior engineers and native speakers, matched by field and language. Delivered in your format, with a provenance record for every item.
Ratings
Rubric ratings
Multi-axis scores such as accuracy, safety, helpfulness and style, on your scale.
SFT
Expert rewrites
Ideal responses written from scratch or corrected from model drafts.
Reasoning
Reasoning traces
Step-by-step solutions in maths, science, code and law, checked by a second expert.
Dialogue
Multi-turn conversations
Realistic conversations, written and rated by people who do the work every day.
Quality
Gold items in every batch
Calibration tasks seeded throughout; raters who fall below the bar come off the project.
Typical outputs
- Pairwise JSONL
- Ranked lists
- SFT pairs
- Rubric scores
- Rationales
How vetting works
Every expert comes with the evidence.
Rather than a profile and a score, you get the record behind each person on your project, and you can review it before they start.
- Recorded AI video interview with the full transcript and a scored rubric
- ID and liveness check, with duplicate-account detection
- Credentials checked with the issuing body where the role needs them
- Integrity flags from the interview, shown to you in plain words
- Paid calibration against gold answers before any client work
How an engagement runs
From first call to scale, pilot first.
Scope
Tell us the task, the domain, the languages and your quality bar. We write the rubric with you.
NDA on requestMatch
We staff vetted experts whose field and language fit the task, and share their evidence packs.
Evidence per expertPilot
A 72-hour paid pilot on your real task, with gold items, dual review and agreement scores.
72 hours, paidScale
Scale on the model that suits you: hourly, per task or a managed pod. Free replacement throughout.
Free replacement
Related solutions
Often paired with this.
- Evaluate LLM evaluation Held-out test sets, rubric scoring and side-by-side comparisons by experts in your field.
- Experts Domain expert evaluation Physicians, lawyers, finance professionals and STEM PhDs, credentials checked at the source.
- Code Agentic code review & coding evaluation Senior engineers grade AI-written code and agent runs, with tests and written verdicts.
Get started
Tell us the gap. We'll scope the pilot.
Every engagement starts with a 72-hour paid pilot. We reply to every request within two business days.
Talk it through
A 30-minute call with the founding team. Bring the task, the domain and your quality bar.
Get a quote
Share the scope in two minutes: service, team size, languages, start date and budget band.