Senior Software Engineer, AI Evaluation Infrastructure

📋 Description

• Design and ship features end to end: data model, API, background workers, and the UI when the
• Build and improve how agents and models are evaluated: running trials in sandboxed environments
• Improve how benchmark tasks are quality-checked: verifiers, rubric-based and LLM-as-a-judge
• Integrate new models, agents, and benchmarks in a way that generalizes, without a special case for
• Keep the system fast and reliable as usage grows: efficient queries, safe migrations, and
• Review teammates' pull requests against our engineering guidelines and help keep the codebase

🎯 Requirements

• Typically 5+ years building and operating production backend systems, with ownership of at least
• Strong Python. Experience with Django (or a similar framework), PostgreSQL, and a task queue such
• Hands-on experience in LLM evaluation, beyond calling a model API.
• Judgment about task quality. You can spot ambiguous instructions, weak or gameable verifiers
• Good tradeoffs. You reach for the smallest design that solves the problem, extend what already
• Ownership. You take a fuzzy request, find the relevant code, ask questions only when the answer

🎁 Benefits

• Work at the frontier of AI, helping the world's leading AI labs improve their most advanced models
• Contribute to leading-edge AI research and showcase your work at top conferences such as ICLR
• Bring frontier AI innovation to the enterprise, applying lessons learned from leading AI labs to
• Collaborate with and learn from exceptional colleagues with deep AI experience from Google, Meta
• Move at the pace of AI innovation, with the speed, ownership, and impact of a startup.