Principal Data Scientist - Agent Builder

📋 Description

• Define evaluation strategy for conversational/agentic search (offline/online).
• Lead quality metrics for RAG, agents, tools, routing, prompts, cost.
• Build and compare retrieval, dense/sparse, vector search, and re-ranking.
• Turn experiments into product decisions on models, routing, and tooling.
• Partner with инженеринг to productionize eval pipelines, telemetry, dashboards.
• Mentor others in experiment design and evaluation of LL powered systems.

🎯 Requirements

• 8+ years in applied DS/ML with expertise in IR, NLP, ranking, semantic search, RAG, or LLM-powered products.
• Proven track record defining/leading evaluation for production AI/ML systems (offline metrics, online experiments, LLM-as-judge, groundedness, citations, model comparison).
• Experience influencing product/technical strategy through data in ambiguous domains.
• Hands-on with Python, PyTorch/Transformers, Pandas, notebooks, reproducible experiments, and clean code.
• Strong understanding of retrieval systems: dense/sparse, vector search, re-ranking, query understanding, metrics (nDCG, MRR, Recall@k, precision) and latency/cost trade-offs.
• Experience collaborating with engineering to move from prototype to production, including telemetry, dashboards, CI guardrails, and quality regression tracking.

🎁 Benefits

• Competitive pay based on the work you do, not your prior salary.
• Health coverage for you and family in many locations.
• Flexible locations and schedules for many roles.
• Generous vacation days each year.
• We match up to €2000 (or local equivalent) for donations and service.
• Up to 40 hours each year to use toward volunteer projects.