Staff Software Engineer, Machine Learning Inference Platform

📋 Description

• Design multi-tenant inference platform architecture across serving, orchestration, and APIs.
• Develop robust API layers (gRPC, WebSockets, REST) and SDKs for orchestration.
• Build multi-tenant control plane for metering, rate limits, quotas, and isolation.
• Optimize inference performance across the stack, including the model engine.
• Implement observability and SLOs to monitor GPU use, cache hits, and cost.
• Collaborate with product and infra teams on model onboarding and capacity planning.

🎯 Requirements

• Education: Bachelor's or Master's in CS, Engineering, or related field.
• 7+ years building and operating backend distributed systems end-to-end.
• Cross-team technical leadership in backend distributed systems, ML infra, or inference serving.
• Strong data & ML fundamentals: data-intensive distributed systems, concurrency, networking, perf profiling.
• Hands-on experience running large-scale GPU inference services (KV caches, prefill/decode).
• Direct experience with inference engines (TensorRT, vLLM) or serving frameworks (Dynamo, Triton).
• C++, Go, Rust or Python; PyTorch; CUDA, NCCL, NVLink; InfiniBand/RoCE.
• Excellent verbal and written communication; AV experience bonus.

🎁 Benefits

• Equal opportunity workplace committed to inclusion and diversity.
• Privacy policy available.