Data Engineer

📋 Description

• Creating and supporting batch, incremental, and real-time ETL pipelines
• Standardizing ingestion, validation, and cleaning processes across clients
• Automating data validation to increase data quality
• Managing and evolving schemas in all parts of our pipeline
• Monitoring, tuning, and optimizing Spark jobs
• Becoming a domain-expert in connected vehicle data

🎯 Requirements

• 3+ years as a data engineer
• Experience as a tech lead or mentor
• Proficiency in Python/Scala/Java/C++ and SQL
• 3+ years of experience with Spark or equivalent technologies
• 2+ years of experience with a workflow scheduler (Airflow, Prefect, Argo, etc)
• 2+ years of experience with distributed file-systems (HDFS, S3, etc)

🎁 Benefits

• Experience with Kubernetes
• Experience working with ML teams
• Contributor to open source projects
• Experience in the Automotive industry or a love of cars
• Prior work in small, agile teams