Staff Data Scientist, Agent Improvement Loop

Added
7 days ago
Type
Full time
Salary
Salary not provided

Related skills

sql python data science typescript evaluation

๐Ÿ“‹ Description

  • Own and develop the evaluation strategy for AI agents, including pre-release behavioral testing
  • Conduct continuous analysis of agent traces and conversations at scale to identify recurring
  • Drive the complete agent improvement cycle by translating analytical findings into actionable
  • Improve agent skills, orchestration frameworks, supporting infrastructure, and developer tools
  • Partner with Delivery and Customer Success teams to improve underperforming customer conversational
  • Design, calibrate, and validate LLM-as-a-judge evaluation systems against human assessments

๐ŸŽฏ Requirements

  • Bachelor's or Master's degree in a technical discipline such as Computer Science, Statistics
  • At least 8 years of experience delivering data science solutions with measurable business impact in
  • Demonstrated hands-on experience evaluating LLM-based systems or AI agents, including evaluation
  • Advanced proficiency in Python and SQL, with working knowledge of TypeScript or the willingness to
  • Strong software engineering practices, including version control, code reviews, automated testing
  • Practical experience working with production GenAI systems, including prompt engineering, tool

๐ŸŽ Benefits

  • Competitive compensation aligned with market standards.
  • Unlimited paid time off (PTO).
  • Fully remote working opportunities, with working hours offering approximately three hours of
  • Flexible working arrangements focused on results, autonomy, and impact.
  • Opportunity to work in a dynamic, fast-growing startup environment.
  • International and collaborative team with opportunities to work alongside professionals from
Get alerts for new remote data jobs
A weekly email with new matches. Unsubscribe anytime.
Share job

Meet JobCopilot: Your Personal AI Job Hunter

Automatically Apply to Data Jobs. Just set your preferences and Job Copilot will do the rest โ€” finding, filtering, and applying while you focus on what matters.

Related Data Jobs

See more Data jobs โ†’