Principal Data Scientist - Agent Builder

📋 Description

• Define evaluation strategy for conversational/agentic search.
• Lead quality metrics for RAG, agents, tools, routing.
• Improve retrieval and re-ranking: dense/sparse, vector search.
• Turn experiments into product decisions on models, routing, tools.
• Collaborate with engineering to productionize pipelines and dashboards.
• Influence roadmap by closing high-impact quality gaps.

🎯 Requirements

• 8+ years applied DS/ML in IR, NLP, ranking, semantic search, RAG or LLM-powered products.
• Strong record in production AI/ML evaluation: offline & online metrics, LLM-as-judge.
• Experience shaping product/tech strategy through data in ambiguous domains.
• Hands-on Python, PyTorch/Transformers, Pandas, notebooks, versioned data, clean code.
• Deep knowledge of retrieval: dense/sparse, vector search, re-ranking, query understanding; metrics like nDCG, MRR, Recall@k; latency/cost trade-offs.
• Collaborating with engineering to productionize: telemetry, dashboards, CI guardrails, regression tracking.
• Practical Elasticsearch or similar search/distributed data systems; ES|QL a plus.
• Excellent written and verbal communication; ability to explain trade-offs to cross-functional teams.

🎁 Benefits

• Competitive pay based on the work you do.
• Health coverage for you and family in many locations.
• Flexible locations and schedules for many roles.
• Generous vacation days each year.
• Company matches up to $2000 for donations and service.
• Up to 40 hours per year for volunteer projects you love.
• Embracing parental leave with minimum of 16 weeks.