Lead Data Engineer

📋 Description

• Build and maintain scalable ETL pipelines using AWS Glue, Spark, and Athena to process modem
• Develop and manage daily data quality checks, including SQL validations, threshold-based alerts
• Investigate data anomalies such as nulls, duplicates, missing intervals, and unexpected data
• Integrate data from multiple upstream systems, including AOM, IPDR, Scope, CARS, IRIS, and Gisual
• Support data product certification by validating outputs and ensuring they meet established quality
• Coordinate versioned data product releases, including test case development, release notes

🎯 Requirements

• Strong proficiency in SQL, including window functions (LAG, PARTITION BY), grouping sets, null
• Experience with Python for scripting, automation, and data validation.
• Hands-on experience with AWS data services, particularly Glue, Athena, S3, EMR, and Step Functions.
• Experience using Spark for large-scale data processing, preferably on EMR.
• Strong understanding of data quality and the ability to investigate anomalies, identify root
• Experience validating data pipelines and ensuring data accuracy and consistency before downstream

🎁 Benefits

• Certifications in AWS (we are AWS Partners), Databricks, and Snowflake
• Access to AI learning paths to stay up to date with the latest technologies
• Study plans, courses, and additional certifications tailored to your role
• Access to Udemy Business, offering thousands of courses to boost your technical and soft skills
• English lessons to support your professional communication
• Travel opportunities to attend industry conferences and meet clients