QA / AI Evaluation Engineer role in a forward-leaning team.
Run evaluations at scale from small human-UAT batches to millions of automated evals.
Measure factual grounding and accuracy lift; build metrics framework.
Design load and quality tests as corpus scales; automate regression and evaluation suites
Report quality metrics to technical and non-technical stakeholders; collaborate with engineering to

🎯 Requirements

Bachelor’s Degree in a technical field or equivalent experience.
4+ years in QA / test engineering with exposure to data/ML systems.
Strong Python and data-science techniques for measuring factual grounding and answer quality.
Experience with LLM evaluation frameworks and statistical analysis.
Ability to build automated, large-scale eval harnesses plus lighter human-in-the-loop A/B tests.
Comfort working across multiple LLM providers’ outputs; test automation frameworks and scripting.

🎁 Benefits

Apply on employer's website

This employer gathers applications via their own applicant tracking system.

You will be redirected to an external application form.

Share job

Meet JobCopilot: Your Personal AI Job Hunter

Automatically Apply to Engineering Jobs. Just set your preferences and Job Copilot will do the rest — finding, filtering, and applying while you focus on what matters.

Activate JobCopilot