AI Evaluation Engineer

📋 Description

• Design, build, and maintain automated CI/CD testing pipelines to validate AI agent behavior before
• Develop and maintain golden datasets containing regression prompts, edge cases, and synthetic
• Implement fast, cost-effective, deterministic evaluation checks before introducing LLM-as-a-judge
• Create comprehensive test suites for multi-turn agent conversations, evaluating context retention
• Design and implement adversarial testing scenarios to assess resilience against prompt injection
• Build statistical evaluation frameworks to measure non-deterministic AI behavior using pass rates

🎯 Requirements

• Strong proficiency in Python or TypeScript, particularly for API testing, headless execution
• Hands-on experience building LLM evaluation pipelines, AI testing frameworks, or statistical
• Experience integrating automated tests into CI/CD environments, particularly GitHub Actions or
• Strong expertise in validating structured data and outputs, including JSON schema, OpenAPI
• Familiarity with observability and tracing tools such as OpenTelemetry, LangSmith, Arize Phoenix
• Ability to independently design, implement, and maintain an evaluation architecture from scratch

🎁 Benefits

• Fully remote work: Work from anywhere with a reliable internet connection, with the flexibility to
• Flexible schedule: Organize your working day around your productivity while maintaining the
• Professional development: Access paid training and other opportunities to expand your technical
• Modern technology: Work with contemporary AI tools, frameworks, and engineering practices in an
• Consistent project opportunities: Benefit from a steady flow of engaging and challenging product
• Performance recognition: Strong performance and dedication may be rewarded through annual pay