RemoteWork from anywhere with a stable connection.
Flexible, part-time (10–40 hrs/week)Choose your own hours.
Ongoing, project-based
Starts when a matching project opens
· Rolling applications
Role overview
Experienced QA leads run quality for AI evaluation projects: rubric design, reviewer calibration and agreement tracking for agentic and RAG systems.
Key responsibilities
Own QA for agentic-workflow and RAG/conversational evaluation projects.
Write and refine rubrics and gold sets.
Calibrate reviewers and track inter-annotator agreement (kappa, alpha).
Audit samples and report quality trends.
Ideal qualifications
4+ years owning QA on annotation, evaluation or software testing.
Hands-on experience evaluating LLM, agent or RAG systems.
Comfortable with agreement metrics and spreadsheets.
Fluent written English.
Nice to have
Python for QA scripts.
Finance-domain or audit-trail testing.
Team lead experience, including Urdu or Arabic QA teams.
More about the opportunity
This role does not require or use security clearance.
Compensation and contract terms
$60 – $90 / hour. Paid in USD, weekly. Rate set after a short paid assessment.
Hourly contract. You are engaged as an independent contractor.
Remote. Flexible, part-time (10–40 hrs/week), on your own schedule.
Paid weekly in USD for approved work.
Projects can be extended, shortened or ended early depending on project needs and quality.
Application process
ApplyCreate a free profile and upload your CV.
InterviewAbout 20 minutes. You need a laptop or desktop with a camera and microphone.
ID and credential checkWe verify your identity and the qualifications the role needs.
Matched to projectsOnce verified, you are invited to paid projects that fit your skills.
Vetting for this role: Profile and CV review, AI interview in English, Arabic or Urdu, proctored (about 20 minutes), Identity check: government ID and liveness, Employment verification, Short paid QA assessment and Expert Score.
Free to apply. Vettedlance never charges candidates, and will never ask you to pay for training, tests or equipment.