Senior Data Scientist AI Evaluation

📋 Description

• Design AI evaluations: Define ground truth, metrics, and scoring methods for models and agents.
• Build repeatable eval loops: Track quality over time and catch regressions before release.
• Partner on infrastructure: Work with engineering and analytics engineering to operationalize eval
• Drive iteration: Translate eval results into actionable recommendations for system improvements.
• Establish quality standards: Set evaluation guidelines, documentation, and review practices.
• Mentor and align: Foster evaluation best practices and build a culture of measurable AI quality

🎯 Requirements

• Track record of quantitative measurement rigor (e.g., LLM/model evaluation, metric validation, or
• Strong statistical and ML foundation—you treat evaluations as experiments (sample sizing
• Proficiency in Python and SQL, with experience evaluating models in production environments.
• Strong judgment in defining quality metrics and ground truth for ambiguous outputs.
• Excellent communication and cross-functional collaboration skills to align technical teams and
• Strong problem-solving ability in fast-paced, greenfield environments.

🎁 Benefits

• Competitive Salary & Stock Options
• Health Benefits
• New Hire Home-Office Setup: One-time USD $500
• Monthly Stipend: USD $150 per month via a Brex Card