Product Designer, Evals & Prompts

📋 Description

• Write and revise prompts behind Claude's tools, features, and behaviors on a product surface; test
• Build graders that prove a prompt fix and rerun on the next model; turn designers' hand-run rubrics
• Build visual, low-code eval tools designers can use without an engineer: assemble a comparison set
• Watch designers use those tools and make them simpler
• Support model releases: test each surface against the new model, write prompt fixes and migrations
• Stand up and scale the eval harness: build the test environment that exercises our 50 to 100 tools

🎯 Requirements

• Production-quality Python
• Experience building and maintaining evaluation pipelines for LLM products: graders, rubrics
• Experience building internal tools with a real interface for people who do not write code
• Experience standing up test harnesses, sandboxing tool calls, and pinning the settings that make
• Experience shipping prompts, or working closely with people who do, and understanding why a prompt
• Reads transcripts, not only scores

🎁 Benefits

• Competitive compensation and benefits
• Optional equity donation matching
• Generous vacation and parental leave
• Flexible working hours
• Lovely office space in which to collaborate with colleagues