Evaluations Team Lead

📋 Description

• Build a shared evaluation platform for engineering, research, and Applied AI. Continuously add and
• Define evaluation standards for metrics, data splits, leakage prevention, calibration, uncertainty
• Build reproducible pipelines with versioned inputs and artifacts, and integrate regression checks
• Benchmark NEXUS against competing approaches using fair tuning budgets, data access, compute, and
• Support Applied AI’s customer POC evaluations with tooling, methodological guidance, and analysis.
• Measure predictive quality, latency, and cost across deployment configurations, task types, and

🎯 Requirements

• Experience owning evaluation for tabular ML systems used in production or consequential customer
• Strong statistical judgment: choosing metrics and validation schemes, estimating uncertainty
• Practical experience finding leakage in preprocessing, feature construction, joins, temporal
• Strong Python and SQL skills, familiarity with scikit-learn and gradient-boosted trees, and
• Experience designing fair model comparisons, including hyperparameter search, resource budgets, and
• Prior people management experience, including hiring, technical coaching, and performance feedback

🎁 Benefits

• Competitive compensation with salary and equity
• Comprehensive health coverage for you and your dependents
• Paid parental leave for all new parents, inclusive of adoptive and surrogate journeys
• Relocation support for employees moving to join the team in one of our office locations
• A mission-driven, low-ego culture that values diversity of thought, ownership, and bias toward