Lead Engineer, AI Platform

📋 Description

• Build and own AI evaluation infrastructure, CI/CD pipelines, scorers, and datasets.
• Diagnose quality failures across the agent pipeline and identify fixes.
• Grow evaluation datasets with annotation workflows and simulated conversations.
• Run structured experiments on prompts, models, and the agent harness.
• Lead and grow the AI Quality engineering team, setting technical direction.
• Partner with AI Core engineering to improve product quality.

🎯 Requirements

• 7+ years building production software, including LLM-powered agents.
• Proficiency in Ruby on Rails and Python.
• Experience building evaluation or observability infrastructure for ML/AI systems.
• Experience designing datasets, annotation workflows, or labeling pipelines.
• Fast learner, comfortable with ambiguity and experimentation.
• Strong alignment with company values. Proficient in English (C2).

🎁 Benefits

• Offers equity with ongoing refresh grants
• 35 days of paid time off per year
• Remote-only team with twice-yearly company retreats
• Flexible and autonomous work environment
• Benefits package supporting health, wellbeing, and professional growth