Sr. Software Engineer, Training/Inference (C++)

📋 Description

• Architect and implement scalable distributed infrastructure for model serving (load balancing
• Optimize latency and throughput of model inference under real production workloads
• Build and operate high-concurrency serving systems with excellent tail latency and high reliability
• Benchmark, fine-tune, and accelerate inference engines (including low-level GPU kernel work and
• Develop custom tools to trace, replay, and fix issues across the full stack — from orchestration
• Create robust CI/CD infrastructure for seamless endpoint deployment, image publishing, and

🎯 Requirements

• Bachelor’s degree in computer science, computer engineering, or science discipline
• 5+ years of experience with low-level system programming experience in C/C++ or Rust
• Experience with large-scale, high-concurrent production serving
• Experience with GPU inference engines (e.g., vLLM, SGLang, Triton, TensorRT-LLM)
• Strong background in system optimizations: batching, caching, load balancing, parallelism
• Low-level inference optimizations: GPU kernels, code generation

🎁 Benefits

• Equity
• Comprehensive medical, vision, and dental coverage
• Access to a 401(k) retirement plan
• Short & long-term disability insurance
• Life insurance
• Various other discounts and perks