Member of Engineering (Pre-training / Data Acquisition)

📋 Description

• Design, build, and operate a large-scale web crawler for open data
• Develop deep crawlers to high-value sources to improve recall
• Own a long-term data acquisition roadmap with researchers
• Build observability, monitoring, and debugging tooling for crawl infrastructure
• Collaborate with pre-training, post-training, and evaluations teams on data priorities
• Build high-throughput ingestion pipelines for partner data

🎯 Requirements

• Strong distributed systems background with large-scale data pipelines and crawlers
• Proficiency in Python; optimize performance and debug production systems
• Hands-on web crawling or data extraction; HTTP, queues, and parsing at scale
• Familiarity with AWS and Kubernetes/Docker for high-throughput workloads
• Awareness of data privacy and responsible crawling practices
• Nice to have: pre-training LLMs; trillion-scale datasets; translating research to production

🎁 Benefits

• Fully remote work and flexible hours
• 37 days/year vacation and holidays
• 16 weeks of flexible, full-pay parental leave
• Health insurance allowance for you and dependents
• Company-provided equipment
• Well-being, learning, and home office allowances