Databricks Solution Architect

📋 Description

• Architect and lead the enterprise lakehouse on Databricks across AWS, Azure, or GCP.
• Design scalable batch and streaming pipelines with PySpark, Spark SQL, Structured Streaming.
• Define platform standards for data modeling (medallion), CI/CD, testing, and observability.
• Lead governance with Unity Catalog: access control, data lineage, audit, and PII handling.
• Optimize Spark workloads: cluster sizing, Photon, autoscaling, caching, and query tuning.
• Partner with ML engineers to operationalize models using MLflow and model serving.

🎯 Requirements

• 8+ years data engineering, with 4+ years building production Databricks workloads.
• Deep expertise in Apache Spark (PySpark and Spark SQL) — performance tuning and partitioning.
• Hands-on experience with Delta Lake, Unity Catalog, Databricks Workflows, and Delta Live Tables.
• Production cloud experience (AWS, Azure, or GCP) including networking, IAM, S3/ADLS/GCS.
• Proficiency in Python and SQL; Scala is a plus.
• Experience designing medallion architectures and dimensional models for analytics.