Senior Product Operations Manager, Evaluation Quality

📋 Description

• Own the quality bar for Harvey’s human evaluations and ensure outputs are decision-grade for
• Author and maintain evaluation guidelines, benchmarks, and gold references used by contract
• Standardize rubric and evaluation design into repeatable templates with documented methodology
• Own onboarding, calibration training, and inter-rater reliability for a distributed pool of raters
• Define and maintain failure-mode taxonomy to translate eval results into actionable signals
• Run QA on vendor and attorney deliverables before launch decisions

🎯 Requirements

• 6+ years in product/ research/ evaluation operations or quality program management
• Experience owning quality inputs (guidelines, benchmarks, QA procedures) for expert-driven work
• Experience onboarding and calibrating distributed expert raters and running feedback loops
• Strong grounding in calibration, inter-rater reliability, sampling, rubric design
• Comfort interpreting evaluation data with or without AI support
• Proven ability to scale quality processes under shipping pressure and maintain documentation

🎁 Benefits

• Compensation: $155,400 - $233,200 USD
• Equity and bonus opportunities
• Applicant privacy notices provided by company