Senior Machine Learning Engineer, Agent Eval Platform

📋 Description

• Judge design and calibration: A shared base judge with per-item rubrics expressed as configuration
• Splitting the problem correctly: deterministic validators for checkable world state (e.g., was the
• Scoring that reports its own confidence, so uncertain judgements route to a human instead of
• A standing calibration loop against human-labeled trajectories, run in partnership with our
• Guarding against correlated blind spots: our user simulator and our judge are both LLMs, and they
• Offline↔online divergence: when simulation and production disagree, being the person who can say

🎯 Requirements

• 2+ years in applied ML, data science, or ML-adjacent engineering, with a track record of work that
• Experience turning subjective human judgement into a measurement that holds up - one that other
• Strong applied ML fundamentals, and comfort treating LLMs as a component you evaluate, prompt, and
• Strong Python, and the discipline to ship production-grade code rather than notebooks
• Ability to think and communicate clearly about complex problems
• A high degree of ownership and a bias toward shipping at startup pace

🎁 Benefits

• Base pay of $161,300 - $274,200
• Equity (when applicable)
• Variable/incentive compensation
• Health plans, including flexible spending accounts
• 401(k) Plan with company match
• ESPP