AI Product Manager, AI Platform - Quality and Evaluation

📋 Description

• Define and execute product strategy for Ridgeline's AI Quality Platform
• Build scalable LLM evaluation frameworks and automated harnesses
• Establish offline/online evaluation methodologies and benchmarks
• Roadmap AI observability for monitoring quality, latency, cost, reliability
• Partner with engineering/researchers for regression testing and experimentation
• Drive decisions using AI quality metrics and feedback

🎯 Requirements

• 5+ years PM experience on technical platforms or developer products
• Proven track record shipping AI products powered by LLMs
• Experience with LLM eval frameworks, quality metrics, continuous eval
• Deep knowledge of offline/online eval, benchmarking, regression tests
• Experience with LLM observability tools (Datadog LLM Observability, Langfuse/Smith, Arize, etc.)
• Familiar with prompt engineering, eval datasets, LLM-as-a-judge

🎁 Benefits

• Company Stock Plan
• Unlimited vacation
• Educational and wellness reimbursements
• $0 cost employee insurance plans
• Careers page for more perks