Senior Data Scientist AI Evaluation

📋 Description

• Define ground truth, quality metrics, scoring methodologies, and evaluation frameworks to assess
• Build repeatable evaluation loops that monitor model quality over time, identify performance
• Apply rigorous statistical methods to evaluation design, including sample sizing, confidence
• Compare automated evaluation methods with human assessments to establish reliability, identify
• Partner with Engineering and Analytics Engineering to operationalize evaluation frameworks
• Interpret evaluation results, identify weaknesses and quality gaps, and provide actionable

🎯 Requirements

• Approximately 6–10 years of experience in quantitative data science, machine learning, or a related
• Strong quantitative and statistical foundations, including experience designing rigorous
• Demonstrated experience defining meaningful evaluation metrics, establishing ground truth, and
• Proficiency in Python and SQL, with experience applying these skills to data analysis, model
• Experience evaluating machine learning models in production environments and translating findings
• Ability to validate automated graders against human judgments and account for variability

🎁 Benefits

• Competitive compensation: Salary package designed to reflect experience and expertise.
• Stock options: Opportunity to participate in the company's long-term growth.
• Health benefits: Benefits designed to support employee health and well-being.
• Home-office setup allowance: One-time allowance of USD $500 to support your remote workspace.
• Monthly stipend: USD $150 per month provided through a Brex card.
• Remote work environment: Opportunity to work remotely within the Americas, with Canada as the