Evaluation QA Lead (Agentic and RAG Evaluation)

At a glance

• $60 – $90 / hourPaid in USD, weekly. Rate set after a short paid assessment.
• Remote, any timezoneWork from anywhere with a stable connection.
• Hourly contract · paid weekly in USD · remoteChoose your own hours.
• Ongoing, project-based Starts when a matching project opens · Rolling applications
• Contract · Senior reviewMultiple openings · Client: AI labs and data vendors

What you will do

• Own QA for agentic-workflow and RAG/conversational evaluation projects.
• Write and refine rubrics and gold sets.
• Calibrate reviewers and track inter-annotator agreement (kappa, alpha).
• Audit samples and report quality trends.

What we are looking for

• 4+ years owning QA on annotation, evaluation or software testing.
• Hands-on experience evaluating LLM, agent or RAG systems.
• Comfortable with agreement metrics and spreadsheets.
• Fluent written English.

Nice to have

• Python for QA scripts.
• Finance-domain or audit-trail testing.
• Team lead experience, including Urdu or Arabic QA teams.

Hiring process

1. Step 1Profile and CV review
2. Step 2AI interview in English, Arabic or Urdu, proctored (about 20 minutes)
3. Step 3Identity check: government ID and liveness
4. Step 4Employment verification
5. Step 5Short paid QA assessment and Expert Score

Free to apply. Vettedlance never charges candidates, and will never ask you to pay for training, tests or equipment.