Lead Data Engineer (Databricks, PySpark & GCP)

📋 Description

• Design, develop, test, and maintain scalable ETL data pipelines using Python, PySpark, Databricks
• Architect the enterprise solutions with various technologies like GCP, Azure, Databricks, PySpark
• Work extensively on Google Cloud Platform (GCP) services such as: Dataflow for real-time and batch
• Develop production-grade Databricks notebooks and workflows.
• Build data transformation pipelines using PySpark and Spark SQL.
• Implement Delta Lake architecture.

🎯 Requirements

• 10+ years of hands-on experience in Python for backend or data engineering projects.
• Strong understanding of GCP services (especially Dataflow, BigQuery, Cloud Functions, Cloud
• Working experience with Azure Data Factory, Azure Databricks, Azure Data Lake Storage Gen2.
• Solid understanding of data pipeline architecture, data integration, and transformation techniques.
• Experience with version control (GitHub) and knowledge of CI/CD practices.
• Experience with Apache Spark, Kafka, Redis, Fast APIs, Airflow, GCP Composer DAGs.