Agentic AI Systems Evaluation Engineer

📋 Description

• Translate user stories and intended customer experiences into measurable success criteria
• Design repeatable benchmarking frameworks that assess AI workload effectiveness and system behavior
• Compare model formats, model runners, runtime implementations, and hardware configurations using
• Analyze inference behavior, including prompt processing, time to first token, token generation
• Evaluate system responsiveness and resource utilization under individual and concurrent AI
• Interpret benchmark and telemetry data to distinguish meaningful improvements, regressions

🎯 Requirements

• 7+ years of experience in systems evaluation, performance engineering, software engineering
• Strong understanding of AI systems and inference workloads, including model execution, prompt
• Experience designing benchmarking methodologies, selecting meaningful metrics, and interpreting
• Familiarity with model formats such as ONNX and GGUF, model runners, and the implications of
• Understanding of system-level performance, including compute utilization, memory constraints
• Strong programming and automation skills in Python, C++, or another relevant language, with

🎁 Benefits

• Health insurance
• Dental insurance
• Vision insurance
• Long term/short term disability insurance
• Employee assistance program
• Flexible spending account