Senior Applied AI Engineer, Agent Quality & Evaluations

📋 Description

• Manage member-facing AI behavior across prompt-to-create, agentic editing and future jobs such as
• Create, test, version and document prompts and agent instructions, with a clear record of what
• Define how the system should use context, member data, tools and product state. Build and prototype
• Own Flodesk's AI evaluation practice, including representative datasets, behavioral scenarios
• Build and operate eval harnesses, monitoring and quality dashboards so the team can measure quality
• Diagnose failures across prompts, context, orchestration, models, tools, data and product code

🎯 Requirements

• A track record of building or meaningfully improving production LLM or agentic products, not just
• Strong software engineering skills in Python, TypeScript or a similar language that let you build
• The interaction between prompts, context, tools, agent loops, orchestration and product state is
• LLM evaluation, observability and experimental design are areas where you have direct experience
• A strong product sense that helps you translate ambiguous member feedback into testable quality
• Systems thinking and reusable patterns guide your work, so today's email experience gets better

🎁 Benefits

• Fully paid health insurance for individual coverage.
• 16 weeks paid parental leave for non-birthing parents; 22 weeks paid maternity leave for birthing
• Unlimited flexible time off.
• 401(k) match (US employees only).
• $1,000 annual stipend for learning and development.