Product Manager II, AI Evaluations

📋 Description

• Run rigorous frontier-model evaluations and mature the evaluation stack and process.
• Define high-quality evaluation datasets and grading criteria aligned to user needs.
• Drive synthetic data generation and pipelines for quality, reliability, and scalability.
• Own online evaluations and derive production-use insights.
• Lead model launches end-to-end, managing process, stakeholders, and timelines.
• Build relationships with model providers and AI companies for early access and collaborations.

🎯 Requirements

• 2+ years of product management experience.
• Deep AI landscape knowledge; able to converse technically with model providers.
• Technical and hands-on: Python, SQL, notebooks; build evals and data pipelines.
• Cross-functional launches across Product Marketing, Communications, Developer Relations, and Engineering.
• Excellent written and verbal communication for senior stakeholders.
• High-agency, able to drive ambiguous problems to resolution on tight timelines.

🎁 Benefits

• Equity and comprehensive benefits.
• Work from an assigned Box office at least 3 days/week.
• Cross-functional collaboration across Product, Marketing, Partnerships, and Engineering.
• Competitive compensation with pay transparency.