Inference Performance Engineer

📋 Description

• Build and improve the inference runtime
• Design scheduling, batching, KV cache, and prefill
• Implement low-precision kernels and speculative decoding
• Drive throughput, latency, and cost per token
• Collaborate with hardware teams on kernels and graph optimizations
• Own the OpenAI-compatible API surface and serving protocol
• Build benchmarking and profiling infrastructure

🎯 Requirements

• BS in CS, EE, or related field or equivalent experience
• Software: Rust, Go, Python, or C++
• Understanding of concurrency, memory, and tail latency
• Inference tech: transformers, attention, KV cache, batching, quantization
• Experience with model serving frameworks: vLLM, TGI, SGLang, TensorRT-LLM, llama.cpp
• GPU/ASIC programming: CUDA, ROCm, Triton
• Experience with low-precision inference: FP8, FP4, INT4
• Profiling and benchmarking with Nsight or perf

🎁 Benefits

• Top-tier compensation structured to recognize and retain the best talent
• Meaningful equity
• Comprehensive medical, dental, vision, life, and disability insurance
• Parental leave for all new parents, including adoptive and surrogate journeys
• Flexible PTO
• Paid Holidays
• Relocation support