AI SDET (Software Development Engineer in Test)

📋 Description

• Move beyond traditional QA by owning the Evaluation Harness for AI agents.
• Ensure safety, compliance, and accuracy via automated testing of non-deterministic outputs.
• Own Langfuse evaluation harness; create Golden Paths and adversarial test scenarios.
• Validate regulatory quality gates (SR 26-2, NYDFS Reg 187) and perform shadow mode validation.
• Test data quality, population segmentation, and HITL checkpoint reliability.
• Verify output standards of the Designer across evaluation scenarios.

🎯 Requirements

• Advanced proficiency in Python 3.11+, Pytest, and automated testing frameworks.
• Direct experience with AI evaluation tools (Langfuse, RAGAS, or similar) and LLM-as-a-judge
• Knowledge of model risk management and regulatory compliance testing.
• Experience with Shadow Mode deployment strategies and statistical validation.

🎁 Benefits

• Health and dental insurance
• Meal and food allowance
• Childcare assistance
• Extended paternity leave
• Wellness program and related perks
• Continuous learning platform (CI&T University)