Added
3 minutes ago
Type
Full time
Salary
Upgrade to Premium to se...

Related skills

python machine learning agent statistical inference llm evaluation

πŸ“‹ Description

  • Design the end-to-end evaluation workflow for a planner-executor agent: scenarios, user simulation
  • Define the scenario schema, grading spec and validators with the client's team, who own scenario
  • Build an LLM-based user simulator with a hidden goal and a stop rule that holds realistic
  • Build isolated, parallel experiment execution: config files, one run per config and scenario, and a
  • Package optimization levers as plug-ins and implement a shortlist of 3 to 4 for v1, such as
  • Tag every model call at the gateway and send traces to a trace store, so tokens, cache usage

🎯 Requirements

  • Experience with LLM and agent evaluation methods, such as programmatic state-based verifiers
  • Experience building or evaluating planner-executor or other multi-step, tool-using agent
  • Strong background in experimental design and statistical inference, including paired designs
  • Experience with multi-objective optimization or cost-aware ML, including Pareto frontier analysis
  • Familiarity with LLM routing, model cascades or model selection, and understanding of LLM inference
  • Experience with observability and tracing for LLM systems, including attributing tokens, cache

🎁 Benefits

  • Salary range $200,000 - $300,000
Get alerts for new remote engineering jobs
A weekly email with new matches. Unsubscribe anytime.
Share job

Meet JobCopilot: Your Personal AI Job Hunter

Automatically Apply to Engineering Jobs. Just set your preferences and Job Copilot will do the rest β€” finding, filtering, and applying while you focus on what matters.

Related Engineering Jobs

See more Engineering jobs β†’