Staff Software Engineer, Inference

📋 Description

• Lead architecture, performance, and reliability for Kubernetes-native inference at massive scale.
• Define cross-cutting design initiatives: routing, adaptive scheduling, GPU resource mgmt
• Implement advanced inference optimizations: speculative decoding, KV-cache reuse, benchmarking
• Aim for strict P99 SLAs with metrics-driven engineering and observability.
• Collaborate across infrastructure boundaries and mentor senior/mid engineers.

🎯 Requirements

• 8–12+ years in large-scale distributed systems or cloud platforms.
• Proven track record leading cross-team technical initiatives at scale.
• Strong coding in Go, Python, or C++.
• Deep Kubernetes production-scale experience (orchestration, scheduling, service design).
• Strong knowledge of networked systems, performance optimization, distributed design.
• Hands-on inference systems experience: batching, caching, memory opt, mixed precision (BF16/FP8)

🎁 Benefits

• Competitive salary with discretionary bonus and equity.
• Comprehensive benefits package including health, dental, vision.
• 401(k) with employer match and flexible PTO.
• Casual work environment and growth opportunities at a fast-growing AI/cloud company.