AI Agent Quality Engineer

📋 Description

• Build and scale automated evaluation tests for agents before production.
• Design adversarial tests (prompt injection, sycophancy); automate across changes.
• Own the 'break it on purpose' pass: test data exposure and tool boundaries.
• Partner with Data Steward on data sensitivity for test scenarios.
• Define pass criteria per agent capability and automate thresholds for hundreds.
• Maintain guardrails and a negative-test catalog across the platform.

🎯 Requirements

• 4+ years in software quality or security testing; 1–2 years red-teaming LLM-based systems.
• Strong Python skills; production-quality code, not just glue scripts.
• REST APIs and webhook/event-driven patterns to build test harnesses.
• Experience building automated test infrastructure and integrating with CI/CD.
• Hands-on familiarity with an adversarial testing or LLM eval tool and related risk frameworks.
• Prompt injection, jailbreaking, and sycophancy test design.
• Comfort blocking a release when an agent doesn't meet its bar.

🎁 Benefits

• Understanding of data sensitivity and compliance to design real-risk tests.
• Experience in regulated environments with external audits.
• Familiarity with multi-attempt or persistent-attack testing methodology.
• Exposure to how agents are built (prompting, tool schemas, orchestration).