LLM Inference Frameworks and Optimization Engineer

📋 Description

• Design fault-tolerant, high-concurrency inference engine for text, image, and multimodal models.
• Implement distributed inference strategies: MoE parallelism, tensor parallelism, and pipeline parallelism.
• Apply CUDA graph, TensorRT graph optimizations, and torch.compile for speed.
• Collaborate with hardware teams to identify bottlenecks and co-optimize GPU inference.
• Work with AI researchers and infra engineers to optimize end-to-end model serving.

🎯 Requirements

• 3+ years in deep learning inference, distributed systems, or HPC.
• Familiar with at least one LLM inference framework (TensorRT-LLM, vLLM, SGLang, TGI).
• Background in GPU programming (CUDA/Triton/TensorRT), compilers, quantization, and GPU cluster scheduling.
• Deep understanding of KV cache systems like Mooncake, PagedAttention, or in-house variants.
• Proficient in Python and C++/CUDA for high-performance DL inference.
• Transformer/LLM/VLM/Diffusion model optimization; workload scheduling, CUDA graph, compiled kernels.

🎁 Benefits

• Experience with RDMA/RoCE in large-scale data center networks.
• Familiar with distributed file systems (3FS, HDFS, Ceph).
• Familiar with Kubernetes and open-source distributed scheduling/orchestration.
• Contributions to open-source deep learning inference projects.