Responsibilities of the Candidate:

Design, build, and maintain scalable ETL/ELT pipelines using Databricks (Spark, Delta Lake, Lakeflow/DLT, Unity Catalog)
Develop and productionize feature pipelines for ML use cases, ensuring reliability, freshness, and reproducibility
Collaborate with Data Scientists/ML Engineers to deploy models (batch and/or real-time), including via Databricks Model Serving or ML flow
Optimize Spark jobs, SQL warehouses, and cluster configurations for cost and performance
Build and maintain system-level observability for pipelines and ML jobs (usage, cost, quality, drift)
Implement data quality checks, testing, and monitoring across the medallion architecture (bronze/silver/gold)
Own Unity Catalog governance for datasets and features access controls, lineage, and PII masking
Partner with platform/infra teams on job orchestration, CI/CD for data & ML pipelines, and cost optimization
Contribute to architecture decisions around lakehouse design, streaming vs. batch tradeoffs, and tool selection (build vs. buy)
Perform analysis on top of the data you build, answer ad-hoc business questions, validate metrics, and spot data quality issues before they reach stakeholders
Evaluate and apply LLMs/agentic frameworks responsibly within the team, balancing accuracy, cost, and governance (e.g., row/column-level access control on what an Al agent can query

Requirements:

1-3 years of data engineering experience working on Databricks in production
Strong proficiency in PySpark/Spark SQL and Python
Solid understanding of Delta Lake, Unity Catalog, Lakeflow/DLT, and Databricks system tables (billing, compute, query history) Experience building and maintaining feature pipelines or ML data infrastructure (feature stores, training/serving data parity
Familiarity with MLflow (experiment tracking, model registry) and/or Databricks Model Serving
Strong SQL skills and experience with warehouse performance tuning (query optimization, materialization strategies, cluster sizing)
Understanding of ML fundamentals - enough to have real conversations with Data Scientists about features, drift, and model lifecycle (you don't need to be building models yourself, but you should understand what "good" looks like).