Quality Assurance Engineer

📋 Description

• Build, maintain, and expand automated testing for AI-powered voice and chat agents, covering
• Design and execute tests for LLM output quality, including correctness, consistency, structured
• Develop evaluation approaches using LLM-as-a-judge techniques, golden datasets, tolerance-based
• Validate RAG pipelines, with particular attention to retrieval relevance, grounding, faithfulness
• Test voice pipelines, including speech-to-text and text-to-speech accuracy, multilingual behavior
• Automate testing across UI/E2E, API, database, AI-output, desktop, and mobile layers.

🎯 Requirements

• 5+ years of experience as a QA Automation Engineer, including proven hands-on experience testing
• Demonstrated ability to build testing frameworks, QA standards, and CI/CD-integrated automation
• Practical experience testing conversational AI, including voice and/or chat agents, in real-world
• Strong understanding of LLMs, prompt engineering, RAG, and approaches for testing and evaluating
• Knowledge of voice technologies and pipelines, including speech-to-text and text-to-speech, and
• Hands-on experience with LLM evaluation or observability tools such as Langfuse, LangSmith

🎁 Benefits

• Fully remote working environment with the opportunity to work from India within a distributed
• Full-time position within an engineering organization working on AI-powered products.
• High level of ownership and autonomy over QA strategy, automation, evaluation, and release quality.
• Opportunity to work hands-on with advanced AI technologies, including LLMs, RAG, conversational
• Direct impact on the quality, reliability, and customer readiness of AI products.
• Opportunity to build and evolve testing frameworks, evaluation systems, and CI/CD quality