AI Research Engineer

📋 Description

• Design rubrics and test whether they measure what they claim: validity, reliability, and
• Measure how reliable human raters are, and when disagreement is noise versus real signal.
• Calibrate AI judges against real human judgment, and study when they drift away from it.
• Build quality signals that put evidence behind every data point we ship.
• Design Dscout's first benchmark: real users, in real context, over long horizons. Multi-week tasks
• Define the protocol, task design, scoring, and release. Work with our delivery team on fieldwork.

🎯 Requirements

• Published research, or strong preprints and workshop papers, in evals, preference learning, reward
• A PhD, a PhD in progress, or an MS with two or more strong research internships at labs or top
• Strong applied statistics: inter-rater reliability, confidence intervals, power, and a nose for
• Hands-on experience with LLM-as-judge, rubric-based evaluation, and preference data.
• Strong Python, and the ability to build research prototypes and evaluation pipelines other people
• Experience running experiments on messy human data, not only curated datasets.

🎁 Benefits

• $250-400K OTE + Equity (includes built-in bonus and equity program)
• Publication freedom
• Conference budget
• Flexible PTO
• 15 holidays
• 12 weeks of paid parental leave