Lead Engineer, AI Platform

📋 Description

• Design, build, and own evaluation infrastructure, including CI/CD pipelines, scorers, datasets, and
• Develop observability and diagnostic capabilities to identify exactly where quality issues occur
• Investigate failures across sophisticated AI workflows and turn findings into technical prototypes
• Build and expand datasets through human annotation, AI-generated examples, and simulated
• Develop structured experimentation frameworks for prompts, models, and agent harnesses, including
• Identify opportunities to improve AI system cost and latency through model selection, caching

🎯 Requirements

• 7+ years of experience building and shipping production software, ideally including LLM-powered
• Experience working with complex, tool-using AI systems involving multiple tools, planning
• Strong ability to demonstrate shipped software and explain how its effectiveness and reliability
• Experience with Ruby on Rails and/or Python, with the ability to become productive quickly in
• Experience building evaluation or observability infrastructure for ML/AI systems, including
• Familiarity with evaluation frameworks such as Braintrust, LangSmith, or similar tools.

🎁 Benefits

• Annual cash compensation of $170,000 USD, benchmarked to U.S. compensation levels regardless of
• Equity in the company, including ongoing refresh grants.
• 35 days of paid time off per year.
• Fully remote work environment.
• Significant flexibility and autonomy in how you organize your work.
• Twice-yearly company retreats in international destinations.