Senior Staff Data Engineer, Databricks (R6508)

Shield AI

  • Remote - US
  • On-site
  • Staff/Principal level
  • 12+ yrs experience
  • Data Engineering
  • Visa: not specified
  • $145k–$220k

The role in 30 seconds

Shield AI is a defense-tech company focused on building intelligent systems for protection. This Senior Staff Data Engineer role is responsible for designing and building robust data pipelines and platforms, particularly within the Databricks lakehouse environment, to support data ingestion, transformation, quality, and governance. The engineer will implement and maintain critical data platform constructs, CI/CD pipelines, and ensure compliance with data handling requirements in regulated environments, shaping standards for a fast-moving platform build.

Is this role a fit for you?

The backgrounds this role is most likely to hire from, as RoleStack reads the posting.

Strong match

  • B2B infrastructure / API / developer tools
  • B2B enterprise platform / workflow software
  • Data platform / ML pipeline
  • Enterprise-scale
  • Regulated environment (healthcare, finance, government)

Workable stretch

  • Internal tooling / back-office systems
  • Creator / builder / self-serve tool
  • Growth-stage (scaling existing systems, reliability concerns)

Harder transition

  • B2C consumer-facing application
  • Marketplace / multi-sided platform
  • Embedded systems / hardware-adjacent
  • Startup / greenfield (build from scratch, speed over polish)

Industry: Defense & Aerospace. See your personal match score

What they're really screening for

Must-haves

  • CORE 12+ years of data engineering experience, including hands-on ownership of production data pipelines.
  • CORE Strong Databricks experience, including Delta Lake, Databricks Workflows or Jobs, and Spark with PySpark and/or Spark SQL.
  • CORE Working knowledge of Unity Catalog, including catalogs, schemas, tables, lineage, and access control concepts.
  • CORE Experience with batch, CDC, and/or streaming ingestion patterns and the operational trade-offs associated with each.
  • CORE Experience with CI/CD and deployment automation for data pipelines and platform assets, including version control, testing, and controlled promotion across environments.
  • CORE Strong SQL skills and solid grounding in data modeling fundamentals, even if dimensional modeling is not the primary responsibility of this role.
  • CORE Demonstrated ability to understand the business and regulatory context of the data being processed, not just the mechanics of pipeline development.
  • CORE Experience applying data classification, segregation, security, retention, or compliance requirements in data engineering workflows within a regulated or security-sensitive environment.
  • CORE Ability to design pipelines with awareness of the actual data domains involved, including sensitivity, ownership, permitted use, and downstream impact.
  • CORE Comfort operating in a fast-moving platform build where patterns are still being established and engineers are expected to shape standards, not just follow them.

Nice-to-haves

  • CORE Define and evolve reusable ingestion and transformation patterns, templates, and engineering standards that other domains can adopt as they onboard to the platform.
  • CORE Implement and maintain Databricks platform constructs needed for secure delivery, including catalogs, schemas, service principals, job orchestration, and environment-aware deployment patterns.
  • CORE Build and maintain CI/CD pipelines for data platform assets, including ingestion code, transformation logic, workflow definitions, tests, and environment promotion across dev, test, and prod.
  • CORE Apply data classification, segregation, and handling requirements within the pipeline design, ensuring sensitive and regulated data is processed in accordance with enterprise controls and access policies.
  • CORE Build data quality controls that do more than detect technical failures, including checks that reflect actual business meaning, record integrity, completeness, and expected domain behavior.
  • CORE Quarantine, flag, and route problematic records or datasets according to defined quality and compliance rules rather than silently dropping or obscuring issues.
  • PLUS Maintain documentation for source objects, ingestion logic, applied transformations, data quality rules, and known limitations so downstream teams can trust and use the data correctly.
  • PLUS Partner with the Analytics Engineer and domain teams to ensure Silver-layer data is reliable, well-governed, and suitable for trusted Gold-layer modeling.
  • PLUS Collaborate with domain engineering teams to align on ownership boundaries, onboarding patterns, data contracts, and support expectations as new domains are enabled onto the platform.

Résumé keywords for this role

Terms the posting stresses. Include the ones you can honestly back up.

Tools & technologies

  • Databricks
  • Delta Lake
  • Spark
  • PySpark
  • Spark SQL
  • Unity Catalog
  • CI/CD
  • Infrastructure-as-code
  • CDC
  • SQL

Concepts & phrases

  • Senior Staff Data Engineer
  • Shield AI
  • defense-tech company
  • intelligent systems
  • Hivemind autonomy software
  • V-BAT and X-BAT aircraft
  • Aechelon simulation and synthetic reality technologies
  • design and build ingestion pipelines
  • enterprise source systems
  • Databricks lakehouse
  • Delta Lake
  • Bronze-layer ingestion
  • raw landing patterns
  • metadata capture
  • load traceability

Words that count as the same thing: "Databricks lakehouse" ≈ Databricks Unified Data Analytics Platform, Lakehouse Architecture · "Bronze-layer ingestion" ≈ Raw Data Layer, Landing Zone · "Silver-layer pipelines" ≈ Cleansed Data Layer, Standardized Data Layer · "Gold-layer modeling" ≈ Curated Data Layer, Aggregated Data Models · "Unity Catalog" ≈ Databricks Data Governance, Data Cataloging and Access Control

Check my résumé against these keywords

Hiring at Shield AI

Open senior roles RoleStack tracks right now, posted in the last 30 days.

More open roles at Shield AI

Full job description from Shield AI

From Shield AI's posting. View the original

Shield AI is a venture-backed defense-tech company with the mission of protecting service members and civilians with intelligent systems. Its products include Hivemind autonomy software, V-BAT and X-BAT aircraft, and Aechelon simulation and synthetic reality technologies. With offices and facilities across the U.S., Europe, the Middle East, and Asia-Pacific, Shield AI’s technology actively supports operations worldwide. For more information, visit www.shield.ai. Follow Shield AI on LinkedIn, X, Instagram, and YouTube.

Show full description Show less

• Design and build ingestion pipelines, batch and streaming where appropriate, from enterprise source systems into the Databricks lakehouse using Delta Lake.

• Own Bronze-layer ingestion, including raw landing patterns, metadata capture, load traceability, and recoverable ingestion design.

• Build Silver-layer pipelines for cleansing, standardization, deduplication, conformance, and quality enforcement without embedding unauthorized or undocumented KPI logic.

• Define and evolve reusable ingestion and transformation patterns, templates, and engineering standards that other domains can adopt as they onboard to the platform.

• Implement and maintain Databricks platform constructs needed for secure delivery, including catalogs, schemas, service principals, job orchestration, and environment-aware deployment patterns.

• Build and maintain CI/CD pipelines for data platform assets, including ingestion code, transformation logic, workflow definitions, tests, and environment promotion across dev, test, and prod.

• Apply data classification, segregation, and handling requirements within the pipeline design, ensuring sensitive and regulated data is processed in accordance with enterprise controls and access policies.

• Build data quality controls that do more than detect technical failures, including checks that reflect actual business meaning, record integrity, completeness, and expected domain behavior.

• Quarantine, flag, and route problematic records or datasets according to defined quality and compliance rules rather than silently dropping or obscuring issues.

• Maintain documentation for source objects, ingestion logic, applied transformations, data quality rules, and known limitations so downstream teams can trust and use the data correctly.

• Partner with the Analytics Engineer and domain teams to ensure Silver-layer data is reliable, well-governed, and suitable for trusted Gold-layer modeling.

• Collaborate with domain engineering teams to align on ownership boundaries, onboarding patterns, data contracts, and support expectations as new domains are enabled onto the platform.

• Databricks certification such as Data Engineer Associate or Professional.

• Experience standing up or maturing Unity Catalog structures and access models in an enterprise setting.

• Experience with CI/CD, infrastructure-as-code, and deployment automation for data platforms.

• Experience in defense, aerospace, financial services, healthcare, or another highly regulated environment.

• Experience enabling multiple business domains on a shared data platform while maintaining strong governance and ownership boundaries.

• 12+ years of data engineering experience, including hands-on ownership of production data pipelines.

• Strong Databricks experience, including Delta Lake, Databricks Workflows or Jobs, and Spark with PySpark and/or Spark SQL.

• Working knowledge of Unity Catalog, including catalogs, schemas, tables, lineage, and access control concepts.

• Experience with batch, CDC, and/or streaming ingestion patterns and the operational trade-offs associated with each.

• Experience with CI/CD and deployment automation for data pipelines and platform assets, including version control, testing, and controlled promotion across environments.

• Strong SQL skills and solid grounding in data modeling fundamentals, even if dimensional modeling is not the primary responsibility of this role.

• Demonstrated ability to understand the business and regulatory context of the data being processed, not just the mechanics of pipeline development.

• Experience applying data classification, segregation, security, retention, or compliance requirements in data engineering workflows within a regulated or security-sensitive environment.

• Ability to design pipelines with awareness of the actual data domains involved, including sensitivity, ownership, permitted use, and downstream impact.

• Comfort operating in a fast-moving platform build where patterns are still being established and engineers are expected to shape standards, not just follow them.

FAQ

How much does a Senior Staff Data Engineer, Databricks (R6508) at Shield AI make?

The posted pay for this role is $145k–$220k a year.

How much experience do you need?

The posting asks for 12+ years in data engineering. It is a staff/principal-level role.

Does Shield AI sponsor visas for this role?

The posting does not say whether visa sponsorship is available. Ask the recruiter before you apply.

Is this role remote?

It is an on-site role.

What tools and technologies does the posting mention?

Databricks, Delta Lake, Spark, PySpark, Spark SQL, Unity Catalog, CI/CD, Infrastructure-as-code, CDC, SQL.