AI Engineer, Evals & Agent Quality

📋 Description

• Build a generalized eval system that measures assistant quality across every surface it touches —
• Stand up golden datasets and the labeling loop that keeps them up to date and constantly checking
• Build model routing and online evaluation tooling to help us learn and route to the best models.
• Make every prompt and system change measurable, so the team can move fast without breaking what
• Partner with engineers across the product to instrument quality and close the loop from signal to

🎯 Requirements

• Have built or owned LLM eval systems, or offline/online quality measurement at scale.
• Think rigorously about measurement. Maybe that came from an MLE or applied-ML background, maybe
• Know the eval landscape hands-on, off-the-shelf tooling and eval frameworks, and have opinions on
• Are comfortable reasoning about model routing and the tradeoffs between models.
• Ship the fixes, not just the dashboards and metrics.
• Are a senior or staff engineer comfortable in greenfield, where the system doesn't exist yet.

🎁 Benefits

• Competitive salary
• Equity
• Health, dental, and vision insurance
• 401(k) plan
• Flexible PTO