ML Platform Engineer, Distributed Training

📋 Description

• Work with the client's infrastructure team on the GKE training platform (GPU node pools, job
• Diagnose and fix platform failures at the root (for example a controller that keeps running out of
• Tune multi-node GPU networking (RDMA/GPUDirect, NCCL) for training throughput.
• Run, scale and debug multi-node distributed training with PyTorch (FSDP, DeepSpeed, Megatron or
• Profile jobs and remove bottlenecks across compute, memory, input pipelines and interconnect.
• Design and run performance benchmarks for the cluster and training stack: throughput, scaling

🎯 Requirements

• 5+ years of engineering experience, with recent hands-on work on GPU training infrastructure and
• Strong Python software engineering: production-quality code, tests and code review.
• GKE and GPU scheduling: Kubernetes on Google Cloud with GPU node pools, job queueing (Kueue or
• Distributed PyTorch and Ray Train: has personally run and debugged multi-node training jobs.
• RDMA and GPU networking: GPUDirect/RDMA, NCCL tuning, multi-node GB200 (or H200) topology.
• Benchmarking and evals: has designed performance benchmarks and model evaluations, and used them to

🎁 Benefits

• Comprehensive Health Insurance
• Paid Leave (Vacation/PTO)
• Paid Holidays
• Sick Leave
• Parental Leave
• Bereavement Leave