AI Agent Evaluation Engineer

Added
10 days ago
Type
Full time
Salary
Salary not provided

Related skills

python ml evaluation llm regression testing

πŸ“‹ Description

  • Define and maintain the evaluation rubric for the agent across three axes: adherence (did it follow
  • Write annotation guidelines and failure taxonomy; run calibration sessions and measure inter-rater
  • Classify and prioritize failure modes by severity and frequency
  • Review production conversations by hand every week; leverage QA partners; open-code failures
  • Trace failures to likely causes (prompt, knowledge gap, tool/flow bug, speech recognition) and
  • Evaluate across languages and report quality by segment (language, accent, audio quality, intent)

🎯 Requirements

  • 3+ years professional experience in software engineering, applied ML, or data-intensive research
  • Bachelor's or Master's in Computer Science, Computational Linguistics, Cognitive Science
  • Built and operated end-to-end evaluation system for production LLM product (rubric, human labels
  • Deep understanding of evaluation methodology: golden sets, judges, regression suites, calibration
  • Fluent in how LLM agents work: prompts, tool calls, context handling, multi-turn state
  • Strong programming in Python and experience with LLM APIs

🎁 Benefits

  • Comprehensive medical, dental, and vision coverage
  • Flexible PTO
  • Paid parental leave
  • Retirement savings plan
  • Remote work setup budget
  • Monthly wellness and communication stipend
Get alerts for new remote engineering jobs
A weekly email with new matches. Unsubscribe anytime.
Share job

Meet JobCopilot: Your Personal AI Job Hunter

Automatically Apply to Engineering Jobs. Just set your preferences and Job Copilot will do the rest β€” finding, filtering, and applying while you focus on what matters.

Related Engineering Jobs

See more Engineering jobs β†’