Hiring for an AI lab or data vendor? See our vetted talent bench

Hiring now

Evaluation QA Lead (Agentic and RAG Evaluation)

$60 – $90 / hour

Hourly contractRemoteFlexible, part-time (10–40 hrs/week)Paid weekly in USD

At a glance

  • $60 – $90 / hourPaid weekly in USD
  • Hourly contractIndependent contractor · Senior review
  • RemoteWork from anywhere with a stable connection.
  • Flexible, part-time (10–40 hrs/week)Choose your own hours.
  • Ongoing, project-based Starts when a matching project opens · Rolling applications

Role overview

Experienced QA leads run quality for AI evaluation projects: rubric design, reviewer calibration and agreement tracking for agentic and RAG systems.

Key responsibilities

  • Own QA for agentic-workflow and RAG/conversational evaluation projects.
  • Write and refine rubrics and gold sets.
  • Calibrate reviewers and track inter-annotator agreement (kappa, alpha).
  • Audit samples and report quality trends.

Ideal qualifications

  • 4+ years owning QA on annotation, evaluation or software testing.
  • Hands-on experience evaluating LLM, agent or RAG systems.
  • Comfortable with agreement metrics and spreadsheets.
  • Fluent written English.

Nice to have

  • Python for QA scripts.
  • Finance-domain or audit-trail testing.
  • Team lead experience, including Urdu or Arabic QA teams.

More about the opportunity

This role does not require or use security clearance.

Compensation and contract terms

  • $60 – $90 / hour. Paid in USD, weekly. Rate set after a short paid assessment.
  • Hourly contract. You are engaged as an independent contractor.
  • Remote. Flexible, part-time (10–40 hrs/week), on your own schedule.
  • Paid weekly in USD for approved work.
  • Projects can be extended, shortened or ended early depending on project needs and quality.

Application process

  1. ApplyCreate a free profile and upload your CV.
  2. InterviewAbout 20 minutes. You need a laptop or desktop with a camera and microphone.
  3. ID and credential checkWe verify your identity and the qualifications the role needs.
  4. Matched to projectsOnce verified, you are invited to paid projects that fit your skills.

Vetting for this role: Profile and CV review, AI interview in English, Arabic or Urdu, proctored (about 20 minutes), Identity check: government ID and liveness, Employment verification, Short paid QA assessment and Expert Score.

Free to apply. Vettedlance never charges candidates, and will never ask you to pay for training, tests or equipment.

Related roles

Video and Sensor Data Annotator for Robotics

Hiring now

Paid in USD

Data & AI operations

Contributor · Open now

Dialect Audio Transcriber (Urdu, Punjabi, Saraiki)

Hiring now

Paid in USD

Data & AI operations

Transcription · Open now

Annotation Team Lead

Hiring now

Paid in USD

Data & AI operations

Senior review · Open now

RL Environment Task Author

Hiring now

$70 – $100 / hour

Data & AI operations

Expert · Open now

GenAI Evaluator, Generalist (US and Canada)

Hiring now

$30 – $50 / hour

Data & AI operations

Contributor · Open now

AI Personalization Evaluator (Multilingual, Remote)

Hiring now

$15 / hour

Data & AI operations

Contributor · Open now

All data & ai operations roles