Lead Data Platform Engineer (Databricks & Data Governance)

AG Coders

  • Remote - US
  • Remote
  • Senior level
  • 8+ yrs experience
  • Data Engineering
  • Visa: not specified
  • $104k–$128k

The role in 30 seconds

AG Coders is seeking a Lead Data Platform Engineer to spearhead the design, implementation, and rollout of Test Data Management and Data Governance frameworks on Databricks. The role involves building scalable pipelines for synthetic data generation and data masking, establishing enterprise-wide security controls, and enabling self-service access to compliant datasets across engineering and QA teams.

Is this role a fit for you?

The backgrounds this role is most likely to hire from, as RoleStack reads the posting.

Strong match

  • Data platform / ML pipeline
  • B2B infrastructure / API / developer tools
  • Enterprise-scale
  • Growth-stage

Workable stretch

  • B2B enterprise platform / workflow software
  • Internal tooling / back-office systems
  • Startup / greenfield

Harder transition

  • B2C consumer-facing application
  • Embedded systems / hardware-adjacent
  • Regulated environment

Industry: Horizontal / industry-agnostic data platform. See your personal match score

What they're really screening for

Must-haves

  • CORE 8+ years of progressive experience in Data Engineering, Data Architecture, or Data Platform roles
  • CORE 3+ years of hands-on experience with Databricks (Lakehouse, Delta Lake, Unity Catalog, and PySpark)
  • CORE Proven experience implementing Medallion Architecture (Bronze/Silver/Gold) and rolling out Unity Catalog or enterprise data governance solutions
  • CORE Strong software engineering skills in Python and PySpark
  • CORE Hands-on experience with Azure (ADF, Azure DevOps, Synapse) and/or AWS (S3, Lambda, Glue, Redshift, MWAA)
  • PLUS Experience building for engineering and QA teams (internal data platform users)
  • CORE Experience in data platform / data governance domain

Nice-to-haves

  • CORE Hands-on experience with Test Data Management (TDM) techniques, including data masking, de-identification, and synthetic data generation
  • PLUS Familiarity with AI-powered data discovery/classification and Generative AI tools (such as Databricks Mosaic AI)
  • PLUS Experience implementing data observability and monitoring frameworks
  • PLUS Track record of rolling out cross-team frameworks and driving technical enablement
  • PLUS Proficiency in CI/CD pipelines and Infrastructure as Code (Terraform, Azure DevOps, GitHub Actions)
  • PLUS Working knowledge of regulatory compliance standards regarding sensitive data (PII, GDPR, CCPA, HIPAA)

Résumé keywords for this role

Terms the posting stresses. Include the ones you can honestly back up.

Tools & technologies

  • Databricks
  • Lakehouse
  • Delta Lake
  • Unity Catalog
  • PySpark
  • Python
  • Azure
  • ADF
  • Azure DevOps
  • Synapse
  • AWS
  • S3
  • Lambda
  • Glue
  • Redshift

Concepts & phrases

  • Lead Data Platform Engineer
  • Test Data Management
  • Data Governance
  • Medallion Architecture
  • Bronze/Silver/Gold
  • Unity Catalog
  • synthetic data generation
  • data masking
  • Role-Based Access Control
  • metadata management
  • data classification
  • data lineage
  • self-service access
  • compliant datasets
  • environment refreshes

Words that count as the same thing: "TDM" ≈ Test Data Management, Test Data Management frameworks · "RBAC" ≈ Role-Based Access Control, Role Based Access Control · "Medallion Architecture" ≈ Bronze/Silver/Gold architecture, Multi-hop data architecture · "Unity Catalog" ≈ Databricks Unity Catalog, Enterprise data catalog · "PII" ≈ Personally Identifiable Information, Personal Data

Check my résumé against these keywords

Hiring at AG Coders

Open senior roles RoleStack tracks right now, posted in the last 30 days.

Full job description from AG Coders

From AG Coders's posting. View the original

**Job Title:** Lead Data Platform Engineer (Databricks & Data Governance)
**Location:** United States (Remote / Hybrid — adjust as needed)
**Employment Type:** Full-time

**About the Role**
We are looking for an experienced Lead Data Platform Engineer to spearhead the design, implementation, and rollout of our Test Data Management (TDM) and Data Governance frameworks on Databricks. In this role, you will build scalable pipelines for synthetic data generation and data masking, establish enterprise-wide security controls, and enable self-service access to compliant datasets across engineering and QA teams.
Key Responsibilities
• TDM Framework Rollout: Lead the adoption and implementation of Test Data Management (TDM) frameworks across multiple projects on Databricks.
• Data Governance & Security: Establish strict governance controls for sensitive data in non-production environments, implementing Role-Based Access Control (RBAC), metadata management, data classification, and lineage.
• Pipeline Engineering: Design and build scalable, automated pipelines for ingestion, transformation, test data generation, data masking, and environment refreshes.
• Synthetic Data & Self-Service: Architect synthetic data strategies for lower environments and develop reusable data provisioning frameworks that allow teams to access compliant test datasets via self-service.
• Cross-Functional Collaboration: Partner closely with QA, application development, data governance, and information security teams to drive platform enablement and best practices.

Show full description Show less

**Must-Have Qualifications**
• Experience: 8+ years of progressive experience in Data Engineering, Data Architecture, or Data Platform roles.
• Databricks Ecosystem: 3+ years of hands-on experience with Databricks (Lakehouse, Delta Lake, Unity Catalog, and PySpark).
• Architecture & Governance: Proven experience implementing Medallion Architecture (Bronze/Silver/Gold) and rolling out Unity Catalog or enterprise data governance solutions.
• Programming: Strong software engineering skills in Python and PySpark.
• Cloud Platforms: Hands-on experience with Azure (ADF, Azure DevOps, Synapse) and/or AWS (S3, Lambda, Glue, Redshift, MWAA).

***Preferred Qualifications (Highly Desired)***
• Hands-on experience with Test Data Management (TDM) techniques, including data masking, de-identification, and synthetic data generation.
• Familiarity with AI-powered data discovery/classification and Generative AI tools (such as Databricks Mosaic AI).
• Experience implementing data observability and monitoring frameworks.
• Track record of rolling out cross-team frameworks and driving technical enablement.

**Nice to Have**
• Proficiency in CI/CD pipelines and Infrastructure as Code (Terraform, Azure DevOps, GitHub Actions).
• Working knowledge of regulatory compliance standards regarding sensitive data (PII, GDPR, CCPA, HIPAA).

**What We Offer**
• Engagement Model: Independent Contractor (1099 / Corp-to-Corp).
• Contract Duration: 1-year initial contract with strong potential for extension based on performance and project needs.
• Competitive Rate: Up to $60 USD/hour, determined by technical assessment and level of experience.
• High-Impact Environment: Opportunity to architect enterprise-level data governance and AI-ready frameworks from the ground up.

FAQ

How much does a Lead Data Platform Engineer (Databricks & Data Governance) at AG Coders make?

The posted pay for this role is $104k–$128k a year.

How much experience do you need?

The posting asks for 8+ years in data engineering, data architecture, or data platform. It is a senior-level role.

Does AG Coders sponsor visas for this role?

The posting does not say whether visa sponsorship is available. Ask the recruiter before you apply.

Is this role remote?

It is a remote role.

What tools and technologies does the posting mention?

Databricks, Lakehouse, Delta Lake, Unity Catalog, PySpark, Python, Azure, ADF, Azure DevOps, Synapse, AWS, S3, Lambda, Glue, Redshift.