Principal Applied Scientist

📋 Description

• Design and lead an end-to-end evaluation framework for planner-executor agents, covering scenario
• Define scenario schemas, grading specifications, and validation processes in collaboration with
• Develop an LLM-powered user simulator capable of realistic multi-turn conversations, using hidden
• Build isolated, parallel experiment execution systems with configurable runs and adapters that
• Develop and implement an initial selection of optimization plug-ins, such as failure-based model
• Instrument model calls and integrate tracing systems to attribute token consumption, cache

🎯 Requirements

• Demonstrated experience with LLM and AI agent evaluation methodologies, including state-based
• Hands-on experience building or evaluating production-grade planner-executor systems or other
• Strong expertise in experimental design and statistical inference, including paired testing
• Experience with multi-objective optimization or cost-aware machine learning, including Pareto
• Familiarity with LLM routing, model cascades, model selection, and inference optimization
• Understanding of LLM inference internals and the ability to evaluate the impact of optimization

🎁 Benefits

• Annual salary range of $200,000–$300,000.
• Remote work opportunity within the United States.
• Customer-facing, research-driven work with substantial ownership of technical strategy
• Opportunity to shape enterprise AI evaluation practices and deliver tools that support real-world
• Close collaboration with engineering teams and business stakeholders.
• Significant technical leadership and mentoring responsibilities.