Lead Data Engineer – Data Engineering, Analytics & AI

carlyle

  • New York, NY
  • Hybrid
  • Senior level
  • 8+ yrs experience
  • Data Engineering
  • Visa: not specified

The role in 30 seconds

We are seeking a highly experienced, hands-on Lead Data Engineer to design, develop, and own end-to-end enterprise data solutions supporting business applications, analytics, reporting, and AI initiatives. This is a hands-on technical leadership role requiring active involvement in solution design, coding, troubleshooting, engineering standards, and delivery, working closely with business, analytics, AI, and technology teams to build reliable, reusable, and future-ready data products.

Is this role a fit for you?

The backgrounds this role is most likely to hire from, as RoleStack reads the posting.

Strong match

  • B2B enterprise platform / workflow software
  • B2B infrastructure / API / developer tools
  • Internal tooling / back-office systems
  • Enterprise-scale (high availability, compliance, large distributed systems)
  • Growth-stage (scaling existing systems, reliability concerns)

Workable stretch

  • B2C consumer-facing application
  • Marketplace / multi-sided platform
  • Data platform / ML pipeline
  • Startup / greenfield (build from scratch, speed over polish)

Harder transition

  • Creator / builder / self-serve tool
  • Embedded systems / hardware-adjacent
  • Regulated environment (healthcare, finance, government)

Industry: Financial Services & FinTech. See your personal match score

What they're really screening for

Must-haves

  • PLUS Bachelor’s Degree, required
  • CORE Minimum 8+ years of professional experience in data engineering, data warehousing, or related software engineering roles, with significant hands-on technical leadership experience.
  • CORE Proven experience independently designing, building, deploying, and owning end-to-end data engineering solutions.
  • CORE Hands-on experience developing and maintaining ETL/ELT pipelines, including batch ingestion, incremental processing, source-to-target mapping, and data reconciliation.
  • CORE Experience with Apache Airflow, including managed Airflow environments, or equivalent production orchestration frameworks.
  • CORE Strong experience integrating multiple data sources through databases, REST APIs, enterprise applications, and file-based ingestion.
  • CORE Experience designing solutions for large datasets, complex transformations, schema changes, historical tracking, and efficient data processing.
  • CORE Demonstrated ability to lead engineers technically while remaining hands-on with development, code reviews, debugging, and problem resolution.
  • PLUS Experience with Fivetran, SnapLogic, or similar enterprise data integration platforms.
  • CORE Experience designing near-real-time or streaming ingestion solutions, including CDC, event-driven architectures, Kafka, Snowpipe Streaming, or equivalent technologies.

Nice-to-haves

  • PLUS Degree in Computer Science, Data Science, Finance, Business, or a related field, preferred
  • CORE Experience with Snowflake Cortex, Snowpark, Dynamic Tables, Streams, Tasks, or Openflow.
  • PLUS Experience with data observability, cataloging, lineage, and governance platforms such as Atlan, Monte Carlo, or equivalent solutions.
  • PLUS Experience integrating Salesforce, CRM systems, financial applications, or other enterprise SaaS platforms.

Résumé keywords for this role

Terms the posting stresses. Include the ones you can honestly back up.

Tools & technologies

  • Snowflake
  • dbt
  • Python
  • SQL
  • Apache Airflow
  • ETL
  • ELT
  • CDC
  • REST APIs
  • Snowflake Cortex
  • Snowpark
  • Dynamic Tables
  • Streams
  • Tasks
  • Openflow

Concepts & phrases

  • Lead Data Engineer
  • Data Engineering, Analytics & AI
  • enterprise data solutions
  • modern data engineering
  • hands-on technical leadership
  • end-to-end enterprise data solutions
  • design, develop, and own
  • scalable production solutions
  • technical leadership role
  • active involvement in solution design
  • coding, troubleshooting, engineering standards
  • deliver scalable production solutions
  • complex business problem
  • independently design
  • end-to-end data solution

Words that count as the same thing: "AI/LLM applications" ≈ Generative AI, Large Language Models · "RAG" ≈ Retrieval-Augmented Generation · "AI agents" ≈ Autonomous agents, Intelligent agents · "dbt" ≈ data build tool · "ETL/ELT" ≈ Extract Transform Load, Extract Load Transform

Check my résumé against these keywords

Hiring at carlyle

Open senior roles RoleStack tracks right now, posted in the last 30 days.

More open roles at carlyle

Full job description from carlyle

From carlyle's posting. View the original

Position Summary

We are seeking a highly experienced, hands-on Data Engineering Lead to design, develop, and own end-to-end enterprise data solutions supporting business applications, analytics, reporting, and AI initiatives.

The ideal candidate brings strong technical expertise in modern data engineering, particularly Snowflake, dbt, Python, SQL, and orchestration frameworks, along with the ability to independently architect and deliver scalable production solutions.

Show full description Show less

This is a hands-on technical leadership role requiring active involvement in solution design, coding, troubleshooting, engineering standards, and delivery. The individual will work closely with business, analytics, AI, and technology teams to build reliable, reusable, and future-ready data products.

A technically strong, delivery-oriented engineering lead who can take a complex business problem, independently design the end-to-end data solution, develop critical components, guide implementation, and own the solution through production.

The candidate should be equally comfortable writing advanced SQL and Python, designing enterprise data models, troubleshooting production pipelines, collaborating with analytics teams, and enabling emerging AI use cases.

This is not a purely managerial or architecture-only role. Continued hands-on engineering involvement is essential.

In-Office Requirement: 4 days a week

Primary Responsibilities

• End-to-End Solution Ownership: Lead the architecture, design, development, deployment, and ongoing support of enterprise data engineering solutions, from source-system integration through data consumption.
• Data Ingestion & Integration: Design and implement scalable ingestion frameworks supporting databases, enterprise applications, APIs, SaaS platforms, files, and third-party data providers.
• Modern Data Pipelines: Build and optimize batch and incremental ETL/ELT pipelines, including change data capture (CDC), with the ability to evaluate and implement near-real-time and event-driven ingestion patterns as business needs evolve.
• Data Transformation & Modeling: Develop reusable dbt transformations, dimensional models, fact and dimension tables, bridge tables, and curated data marts supporting enterprise reporting and analytics.
• Snowflake Engineering: Design and optimize Snowflake data solutions, including complex SQL transformations, query tuning, workload performance, data sharing, and cost optimization.
• Workflow Orchestration: Develop, manage, and troubleshoot data workflows using managed Apache Airflow or equivalent orchestration frameworks, including dependencies, scheduling, retries, failure recovery, and operational monitoring.
• Data Quality & Reliability: Establish automated data validation, reconciliation, freshness monitoring, alerting, observability, lineage, and production support standards.
• Analytics Enablement: Partner with BI and analytics teams to deliver performant, governed datasets and reusable business metrics for dashboards, reporting, and self-service analytics.
• AI Enablement: Collaborate with AI engineering teams to build trusted, secure, AI-ready data products, including structured and unstructured data integration, reusable data-access interfaces, and data services for AI applications.
• Engineering Excellence: Implement Git-based development, CI/CD, automated testing, code reviews, documentation, and consistent engineering standards across data pipelines and models.
• Technical Leadership: Lead technical design discussions, mentor engineers, resolve complex engineering challenges, and take direct accountability for successful production delivery while remaining actively involved in coding and implementation.

Requirements

Education & Certificates

• Bachelor’s Degree, required
• Degree in Computer Science, Data Science, Finance, Business, or a related field, preferred

Professional Experience

• Minimum 8+ years of professional experience in data engineering, data warehousing, or related software engineering roles, with significant hands-on technical leadership experience.
• Proven experience independently designing, building, deploying, and owning end-to-end data engineering solutions.
• Hands-on experience developing and maintaining ETL/ELT pipelines, including batch ingestion, incremental processing, source-to-target mapping, and data reconciliation.
• Experience with Apache Airflow, including managed Airflow environments, or equivalent production orchestration frameworks.
• Strong experience integrating multiple data sources through databases, REST APIs, enterprise applications, and file-based ingestion.
• Experience designing solutions for large datasets, complex transformations, schema changes, historical tracking, and efficient data processing.
• Demonstrated ability to lead engineers technically while remaining hands-on with development, code reviews, debugging, and problem resolution.
• Experience with Fivetran, SnapLogic, or similar enterprise data integration platforms.
• Experience designing near-real-time or streaming ingestion solutions, including CDC, event-driven architectures, Kafka, Snowpipe Streaming, or equivalent technologies.
• Experience with Power BI, including semantic models, DAX, Import versus DirectQuery, refresh strategies, and performance optimization.
• Experience using AI-assisted development tools, such as Cursor, GitHub Copilot, or equivalent tools, to improve engineering productivity.
• Experience with Snowflake Cortex, Snowpark, Dynamic Tables, Streams, Tasks, or Openflow.
• Experience with data observability, cataloging, lineage, and governance platforms such as Atlan, Monte Carlo, or equivalent solutions.
• Experience integrating Salesforce, CRM systems, financial applications, or other enterprise SaaS platforms.
• Financial services or investment management industry experience.

Competencies & Attributes

• Strong hands-on expertise in Snowflake, advanced SQL, Python, and dbt, including production development, performance optimization, and troubleshooting.
• Strong understanding of enterprise data architecture, dimensional modeling, data warehousing, data marts, and reusable data product design.
• Strong understanding of production-grade engineering practices, including Git, CI/CD, automated testing, data quality, monitoring, alerting, and incident resolution.
• Ability to translate business requirements into technical architecture, data models, implementation plans, and production deliverables.
• Strong communication, problem-solving, and cross-functional collaboration skills.
• Familiarity with AI/LLM applications, including RAG, vector search, Model Context Protocol (MCP), AI agents, and governed data access for AI solutions.
• Familiarity with unstructured data ingestion, document processing, and data preparation for enterprise AI applications.
• Experience with modern data architectures, including medallion architecture, data contracts, and reusable data products.

Benefits/Compensation

The compensation range for this role is specific to New York and takes into account a wide range of factors including but not limited to the skill sets required/preferred; prior experience and training; licenses and/or certifications.

The anticipated base salary range for this role is $180,000 to $200,000.

In addition to the base salary, the hired professional will enjoy a comprehensive benefits package spanning retirement benefits, health insurance, life insurance and disability, paid time off, paid holidays, family planning benefits and various wellness programs. Additionally, the hired professional may also be eligible to participate in an annual discretionary incentive program, the award of which will be dependent on various factors, including, without limitation, individual and organizational performance.

Due to the high volume of candidates, please be advised that only candidates selected to interview will be contacted by Carlyle.

​

​

​

Who We Are

When people, ideas, and capital come together, opportunity expands across private markets. At Carlyle, this belief has shaped how we invest for decades, fueling growth for companies and delivering performance for investors.

Visit our website to learn more about our firm and the ways our platform is shaping private markets.

FAQ

How much experience do you need?

The posting asks for 8+ years in data engineering. It is a senior-level role.

Does carlyle sponsor visas for this role?

The posting does not say whether visa sponsorship is available. Ask the recruiter before you apply.

Is this role remote?

It is a hybrid role: part remote, part in the office.

What tools and technologies does the posting mention?

Snowflake, dbt, Python, SQL, Apache Airflow, ETL, ELT, CDC, REST APIs, Snowflake Cortex, Snowpark, Dynamic Tables, Streams, Tasks, Openflow.