Applied Research Engineer, Evaluation

📋 Description

• Building and maintaining the evaluation stack that every model checkpoint has to pass through and
• Designing the breadth of evaluation across the taxonomy, from prompt adherence, editing fidelity
• Evaluating the evaluation stack itself: checking that human and automated evals agree, that results
• Benchmarking FLUX against the frontier models continuously, while obtaining a meaningful
• Finding the ways in which the quality of the model's output falls short that are not captured by
• Closing the loop, ideally, with the RL and finetuning researchers, by defining target metrics for

🎯 Requirements

• Advanced degree (MS/PhD) in a relevant field or demonstrated equivalent practical experience
• Practical experience with evaluating generative models, in a hands-on way. You may have worked with
• Very strong Python and software engineering skills, with pipelines that run reliably and that other
• Working knowledge of PyTorch and modern ML infrastructure and practices; you are not a pure Python
• Strong grasp of statistics and its shortcomings in practice, and how to work with those; sample
• Practical experience with building and operating LLM-as-judge systems, and understanding their

🎁 Benefits

• Base compensation of €150K-€220K plus equity depending on experience
• Reasonable travel costs covered for in-person collaboration
• Distributed team with real offices, this role based in Freiburg with at least 2 days a week in