Research Engineer, Benchmarks & Evaluation

📋 Description

• Build benchmarks from real workflows: turn agent traces, customer edge cases and expert labels into
• Own the benchmarking effort: decide what we measure, run benchmark suites across models and agent
• Design graders: rubrics, programmatic checks and llm-as-judge, calibrated against human agreement
• Close the loop with post-training: turn evals into reward functions and RL environments, and catch
• Publish external benchmarks and write-ups on how models perform at real finance work

🎯 Requirements

• Hands-on experience building LLM benchmarks or evals that people used to make real decisions
• Strong Python and software engineering; your eval code is reproducible and fast
• Statistical judgment: variance, confidence intervals, sample size, inter-rater agreement, and
• Working knowledge of post-training (SFT, RL with verifiable rewards) and how evals feed into it
• Comfort with SQL and large datasets; you'll work directly from production traces
• Taste for what is worth measuring

🎁 Benefits

• Flexible PTO
• Centralized home-office equipment ordering
• Health and wellness stipend
• Budget for intra-office travel
• Weekly coffee stipend
• 100% medical, dental & vision insurance coverage for you, with partial coverage for dependents