Senior Data Engineer - Databricks

📋 Description

• Own the Databricks pipelines for Sugar Predict, ensuring reliability and scalable delivery.
• Migrate legacy ETL/ELT pipelines to Databricks; build automation to minimize manual steps.
• Design and build high-performance Databricks pipelines for ERP/CRM data across Azure and AWS.
• Enforce data security across Databricks, including RBAC and secrets management.
• Own the Delta Lake architecture: schema design, partitioning, data quality, incremental processing.
• Improve pipeline cost efficiency and throughput through optimization.

🎯 Requirements

• 4+ years of data engineering experience.
• 2+ years on Databricks or Apache Spark (Azure/AWS).
• Proficiency in PySpark, SQL, and Python with production-grade pipelines under SLA.
• Strong Delta Lake expertise: ACID, schema design, incremental/streaming.
• PostgreSQL experience in production data pipelines.
• Data governance, security, and multi-tenant patterns experience.
• Ability to collaborate across data science, product, and infra teams.
• Cost governance and cross-cloud data access experience preferred.

🎁 Benefits

• Excellent healthcare for you and family.
• 401(k) match.
• Unlimited Paid Time Off.
• Paid Parental Leave.
• Online Legal Services (Rocket Lawyer).
• Corporate Benefit Program with travel/entertainment discounts.