RLHF Preference Rater (Generalist)
Paid in USD
Data & AI operations
New opportunity
Talent pool SpecialistContractRemote
Evaluation Rubric Writer
Write clear scoring rubrics that let reviewers grade AI responses consistently, whatever the subject.
At a glance
-
Paid in USDHourly or per task, in USD. Your rate is agreed per project.
-
Remote, any timezoneWork from anywhere with a stable connection.
-
Flexible, part-timeChoose your own hours.
-
Ongoing, project-based Starts when a matching project opens · Rolling applications
-
Contract · SpecialistMultiple openings
About the work
An evaluation is only as good as its rubric. Vague criteria give noisy scores, and noisy scores tell a model builder nothing.
What you will do
- Turn a task description into specific, testable scoring criteria
- Write worked examples of good, borderline and failing answers
- Pilot rubrics with reviewers and fix points of disagreement
- Keep rubrics versioned and documented
What we are looking for
- Excellent written English and structured thinking
- Experience writing marking schemes, style guides or test specifications
- Ability to explain judgement calls in plain words
- Attention to detail
Nice to have
- Teaching, examining or assessment design background
- Prior AI evaluation work
Hiring process
- Step 1Profile and CV review
- Step 2AI interview (about 20 minutes)
- Step 3Identity and credential check
- Step 4Short paid trial task
Free to apply. Vettedlance never charges candidates, and will never ask you to pay for training, tests or equipment.
Related roles
AI Red-Teamer (Generalist)
Paid in USD
Data & AI operations
New opportunity
Multilingual Search Relevance Rater
Paid in USD
Data & AI operations
New opportunity
Speech Transcription QA Reviewer
Paid in USD
Data & AI operations
New opportunity
Dialect Audio Transcriber (Urdu, Punjabi, Saraiki)
Paid in USD
Data & AI operations
New opportunity
Subtitle & Caption Specialist
Paid in USD
Data & AI operations
New opportunity