Lead Databricks Data Engineer

EXL Talent Acquisition Team

  • New York, NY
  • Hybrid
  • Director+ level
  • 12+ yrs experience
  • Data Engineering
  • Visa: not specified

The role in 30 seconds

We are looking for a highly experienced Lead Databricks Data Engineer with 10-12 years of experience in Data Engineering, Big Data, and Cloud Data Platforms. The ideal candidate will lead the design, development, modernization, and optimization of enterprise-scale data platforms using Databricks Lakehouse architecture. This role requires strong technical leadership, solution architecture expertise, stakeholder management skills, and hands-on experience building scalable cloud-native data ecosystems.

Is this role a fit for you?

The backgrounds this role is most likely to hire from, as RoleStack reads the posting.

Strong match

  • B2B infrastructure / API / developer tools
  • B2B enterprise platform / workflow software
  • Internal tooling / back-office systems
  • Growth-stage
  • Enterprise-scale

Workable stretch

  • Creator / builder / self-serve tool
  • Data platform / ML pipeline
  • Startup / greenfield

Harder transition

  • B2C consumer-facing application
  • Marketplace / multi-sided platform
  • Embedded systems / hardware-adjacent

Industry: software_saas_ai_platform. See your personal match score

What they're really screening for

Must-haves

  • PLUS 12-15 years of overall IT experience.
  • CORE Minimum 6+ years of hands-on Databricks experience.
  • CORE 7+ years of experience in Spark/PySpark-based data engineering.
  • CORE Experience designing enterprise-scale Data Lake and Lakehouse solutions.
  • CORE Strong experience leading large data transformation and modernization programs.
  • PLUS Experience handling structured, semi-structured, and unstructured data processing.
  • PLUS Experience supporting analytics, AI/ML, and reporting ecosystems.
  • CORE Proven experience managing technical teams and driving architectural decisions.
  • CORE Databricks Lakehouse Platform
  • PLUS Databricks Workflows

Nice-to-haves

  • PLUS Bachelor's or Master's Degree in Computer Science, Engineering, Information Technology, or related field.
  • PLUS Databricks Certified Data Engineer Professional.
  • PLUS Azure Data Engineer Associate.
  • PLUS AWS Data Analytics Specialty.
  • PLUS Google Professional Data Engineer Certification.
  • PLUS TOGAF or Cloud Architecture certifications are a plus.
  • PLUS MLOps and AI Engineering exposure.
  • PLUS Snowflake and Databricks integration experience.
  • PLUS Microsoft Fabric exposure.
  • PLUS Data Mesh implementation experience.

Résumé keywords for this role

Terms the posting stresses. Include the ones you can honestly back up.

Tools & technologies

  • Databricks
  • Lakehouse
  • ETL
  • ELT
  • Data Pipelines
  • Batch Processing
  • Real-time Processing
  • Delta Lake
  • Unity Catalog
  • Data Governance
  • Spark
  • PySpark
  • Spark SQL
  • Kafka
  • Real-Time Streaming

Concepts & phrases

  • Lead Databricks Data Engineer
  • Enterprise-scale data platforms
  • Databricks Lakehouse architecture
  • Scalable ETL/ELT data pipelines
  • Real-time processing
  • Enterprise data models
  • Ingestion frameworks
  • Transformation strategies
  • Delta Lake
  • Unity Catalog
  • Data governance frameworks
  • Highly scalable Spark
  • Large-volume datasets
  • Cloud data platform modernization
  • Databricks and Data Engineering programs

Words that count as the same thing: "Big Data" ≈ Large-volume datasets, Scalable data platforms · "Cloud-native data solutions" ≈ Data solutions on Azure, AWS, or GCP · "real-time processing" ≈ Real-Time Streaming Architectures · "Infrastructure-as-Code" ≈ CI/CD implementation for data workloads

Check my résumé against these keywords

Hiring at EXL Talent Acquisition Team

Open senior roles RoleStack tracks right now, posted in the last 30 days.

More open roles at EXL Talent Acquisition Team

Full job description from EXL Talent Acquisition Team

From EXL Talent Acquisition Team's posting. View the original

Job Description: Lead Databricks Data Engineer

Job Title: Lead Databricks Data Engineer
Experience: 10-12 Years
Employment Type: Full-Time

Job Summary

We are looking for a highly experienced Lead Databricks Data Engineer with 10-12 years of experience in Data Engineering, Big Data, and Cloud Data Platforms. The ideal candidate will lead the design, development, modernization, and optimization of enterprise-scale data platforms using Databricks Lakehouse architecture. This role requires strong technical leadership, solution architecture expertise, stakeholder management skills, and hands-on experience building scalable cloud-native data ecosystems.

Show full description Show less

The candidate will be responsible for defining data engineering standards, mentoring teams, driving modernization initiatives, and delivering high-performance data solutions that support analytics, AI/ML, and business intelligence initiatives.

Key Responsibilities Data Engineering & Architecture

• Design and implement enterprise-scale data platforms using Databricks Lakehouse architecture.
• Architect and develop scalable ETL/ELT data pipelines for batch and real-time processing.
• Define enterprise data models, ingestion frameworks, and transformation strategies.
• Lead the implementation of Delta Lake, Unity Catalog, and data governance frameworks.
• Build and optimize highly scalable Spark and PySpark solutions handling large-volume datasets.
• Drive cloud data platform modernization and migration initiatives.
Technical Leadership

• Serve as the technical lead for Databricks and Data Engineering programs.
• Establish engineering best practices, coding standards, and architecture guidelines.
• Conduct architecture reviews, code reviews, and solution assessments.
• Mentor and guide Data Engineers, Technical Leads, and junior team members.
• Collaborate with Enterprise Architects, Product Owners, Data Scientists, and Business Stakeholders.
Cloud & Platform Engineering

• Design cloud-native data solutions on Azure, AWS, or GCP.
• Implement secure, scalable, and resilient data architectures.
• Optimize costs, performance, reliability, and availability of data platforms.
• Lead CI/CD and Infrastructure-as-Code implementation for data workloads.
• Define and implement monitoring, observability, and operational excellence practices.
Governance & Data Quality

• Establish data governance, lineage, cataloging, and security controls.
• Ensure compliance with organizational and regulatory requirements.
• Implement data quality frameworks and monitoring solutions.
• Collaborate with security and compliance teams to enforce data protection standards.
Stakeholder & Delivery Management

• Partner with business leaders to understand strategic data requirements.
• Translate business requirements into scalable technical solutions.
• Drive delivery planning, risk management, and resource coordination.
• Present architecture decisions and technical recommendations to leadership teams.

Required Technical Skills Databricks

• Databricks Lakehouse Platform
• Databricks Workflows
• Delta Lake
• Unity Catalog
• Structured Streaming
• Databricks SQL
• Delta Live Tables (DLT)
• Databricks Asset Bundles
Big Data Technologies

• Apache Spark
• PySpark
• Spark SQL
• Kafka
• Real-Time Streaming Architectures
• Data Lake & Lakehouse Architectures
Cloud Platforms (Strong Experience in One or More) Azure

• Azure Databricks
• Azure Data Factory (ADF)
• ADLS Gen2
• Azure Synapse Analytics
• Azure Key Vault
• Event Hub
AWS

• S3
• EMR
• Glue
• Redshift
• Lambda
• IAM
GCP

• BigQuery
• Cloud Storage
• Dataflow
• Pub/Sub
• Composer
Programming

• Python
• SQL
• PySpark
• Scala (Preferred)
• Shell Scripting
DevOps & Automation

• Git
• Azure DevOps
• GitHub Actions
• Jenkins
• Terraform
• Docker
• Kubernetes
Data Warehousing

• Dimensional Modeling
• Star & Snowflake Schema
• Data Vault
• Master Data Management
• Data Governance Frameworks

Required Experience

• 12-15 years of overall IT experience.
• Minimum 6+ years of hands-on Databricks experience.
• 7+ years of experience in Spark/PySpark-based data engineering.
• Experience designing enterprise-scale Data Lake and Lakehouse solutions.
• Strong experience leading large data transformation and modernization programs.
• Experience handling structured, semi-structured, and unstructured data processing.
• Experience supporting analytics, AI/ML, and reporting ecosystems.
• Proven experience managing technical teams and driving architectural decisions.

Preferred Qualifications

• Bachelor's or Master's Degree in Computer Science, Engineering, Information Technology, or related field.
• Databricks Certified Data Engineer Professional.
• Azure Data Engineer Associate.
• AWS Data Analytics Specialty.
• Google Professional Data Engineer Certification.
• TOGAF or Cloud Architecture certifications are a plus.

Nice to Have

• MLOps and AI Engineering exposure.
• Snowflake and Databricks integration experience.
• Microsoft Fabric exposure.
• Data Mesh implementation experience.
• Data Governance tools such as Collibra, Alation, or Informatica.
• Expertise in Data Quality frameworks such as Great Expectations or Deequ.

Key Competencies

• Solution Architecture
• Technical Leadership
• Stakeholder Management
• Program Delivery
• Data Strategy & Governance
• Team Mentoring & Coaching
• Problem Solving & Decision Making
• Communication & Executive Presentation Skills
• Performance Optimization
• Innovation & Continuous Improvement
Technology Classification

Technology Tower: Data & AI
Practice Area: Data Engineering
Primary Skill: Databricks Lakehouse Engineering
Secondary Skills: Apache Spark, PySpark, Cloud Data Platforms (Azure/AWS/GCP), Data Architecture, Data Governance, MLOps.

Management of DB Systems, Performance Tuning, Security, Compliance Standards,

Graduate with 4 years BS or Professional Degree. 15 - 18 Years

FAQ

How much experience do you need?

The posting asks for 12+ years in data engineering. It is a director+-level role.

Does EXL Talent Acquisition Team sponsor visas for this role?

The posting does not say whether visa sponsorship is available. Ask the recruiter before you apply.

Is this role remote?

It is a hybrid role: part remote, part in the office.

What tools and technologies does the posting mention?

Databricks, Lakehouse, ETL, ELT, Data Pipelines, Batch Processing, Real-time Processing, Delta Lake, Unity Catalog, Data Governance, Spark, PySpark, Spark SQL, Kafka, Real-Time Streaming.