Principal Data Scientist - Agent Builder

📋 Description

• Define evaluation strategy for conversational/agentic search (offline/online) and A/B testing.
• Lead quality metrics and decision frameworks for RAG, agents, and tooling.
• Build and compare dense/sparse retrieval and vector search, query understanding.
• Turn experiments into product decisions on models, routing, and tools.
• Partner with engineering to productionize evaluation pipelines and dashboards.
• Influence roadmap by identifying high-leverage quality gaps and trade-offs.

🎯 Requirements

• 8+ years of applied DS/ML with deep expertise in IR, NLP, ranking, semantic search, RAG, or LLM-powered experiences.
• Strong track record defining and leading evaluation for production AI/ML systems, including offline metrics, online experimentation, LLM-as-judge approaches, groundedness, citation quality, and model comparison.
• Hands-on ability with Python, PyTorch/Transformers, Pandas, notebooks, reproducible experiments, versioned datasets, and clean, reviewable code.
• Strong understanding of retrieval systems (dense/sparse retrieval, re-ranking, vector search, query understanding) and metrics like ndCG, MRR, Recall@k, precision, and latency/cost trade-offs.
• Experience collaborating with engineering to move from prototype to production, including telemetry design, dashboards, CI guardrails, and regression tracking.
• Practical Elasticsearch experience, or similar search and distributed data systems; ES|QL familiarity is a plus.

🎁 Benefits

• Competitive pay based on the work you do here and not your previous salary
• Ability to craft your calendar with flexible locations and schedules for many roles
• Generous number of vacation days each year
• Increase your impact - We match up to $2000 (or local currency equivalent) for financial donations and service
• Up to 40 hours each year to use toward volunteer projects you love
• Embracing parenthood with minimum of 16 weeks of parental leave