Research Engineer, Intern – Benchmarking

📋 Description

• Design, implement, and maintain benchmarks and metrics for tool use, agentic behavior, and
• Build and operate LLM evaluation systems end-to-end runs, scoring, dashboards, and reporting
• Run systematic failure analysis on model outputs (e.g., wrong tool use, reasoning errors
• Create and refine rubrics, automated evaluators, and scoring frameworks that drive training and
• Quantify data usability, quality, and impact on key benchmarks
• Work with AI researchers, applied AI teams, and data producers to align evals with training

🎯 Requirements

• Strong applied research background, with focus on model evaluation, benchmarking, and/or failure
• Strong coding skills and hands-on experience with ML models and evaluation code
• Solid grasp of data structures, algorithms, and backend systems
• Comfort with APIs, SQL/NoSQL, and cloud platforms for running and storing eval results
• Ability to reason about model behavior, experimental results, and data quality from evals and
• Excitement to work in person in San Francisco five days a week in a high-intensity, high-ownership

🎁 Benefits

• $1.5K monthly stipend for meals
• Free Equinox membership
• $200 monthly laundry reimbursement
• $200 monthly personal wellness reimbursement