AI Evaluation Infrastructure Engineer

📋 Description

• Build an execution engine that can score candidate versions against task sets in minutes, not hours.
• Create task set tooling that samples from production logs and validates tasks before they are
• Build grader infrastructure across ground truth checks, rubrics, and LLM-as-judge approaches.
• Develop tooling that helps human reviewers calibrate judges, measure judge-to-human agreement, and
• Build leaderboards and reporting systems that include sample size, confidence intervals, and
• Support in-product side-by-side serving, feedback capture, and implicit signal extraction from real

🎯 Requirements

• Experience building production platforms or infrastructure, including distributed batch execution
• Strong statistical literacy, including comfort with confidence intervals, variance, power, and
• The judgment to identify when a result is meaningful and when it is noise.
• Experience evaluating LLM or ML systems, or deep systems engineering experience with a strong
• Product instinct for internal tools. You understand that leaderboards, annotation tools, and
• A bias toward building reliable, observable systems that other engineers can trust.

🎁 Benefits

• Remote work
• Medical insurance
• Flexible time off
• Retirement savings plans
• Modern family planning