Member of Technical Staff, Distributed Training Performance
Hark
- San Jose, CA
- On-site
- Senior level
- AI/ML Engineering
- Visa: not specified
- $180k–$450k
The role in 30 seconds
Hark is an AI company building advanced, personalized intelligence that is proactive, multimodal, and capable of interacting with the world. The Omni team at Hark is building the next generation of AI experiences beyond text, enabling models to understand and generate content across multiple modalities. As part of this team, you will improve the performance of Hark's Megatron-based pretraining systems across thousands of GPUs, focusing on MFU, training throughput, and kernel performance.
Is this role a fit for you?
The backgrounds this role is most likely to hire from, as RoleStack reads the posting.
Strong match
- Data platform / ML pipeline
- B2B infrastructure / API / developer tools
- Software, SaaS & AI Platform
- Growth-stage
- Enterprise-scale
Workable stretch
- Internal tooling / back-office systems
- B2B enterprise platform / workflow software
- Regulated environment (healthcare, finance, government)
Harder transition
- B2C consumer-facing application
- Creator / builder / self-serve tool
- Marketplace / multi-sided platform
- Embedded systems / hardware-adjacent
Industry: Software, SaaS & AI Platform. See your personal match score
What they're really screening for
Must-haves
- CORE Experience developing distributed training systems and working with the internals of Megatron-LM, NeMo, or a comparable framework, including its parallelism strategies.
- CORE Strong understanding of data, tensor, pipeline, sequence, context, and expert parallelism, including communication patterns, memory tradeoffs, and failure modes.
- CORE Experience using GPU profilers to analyze kernel execution, memory bandwidth, communication overhead, and data-loading stalls.
- CORE Working knowledge of mixed-precision training, including BF16 and FP8, and methods for diagnosing numerical instability.
- CORE Strong Python and C++/CUDA engineering skills, with experience building reliable tools for researchers.
- CORE Experience developing CUDA or Triton kernels, or using CUTLASS, for attention, normalization, or mixture-of-experts models.
- CORE A record of measurably improving MFU or throughput on large-scale training jobs.
Nice-to-haves
- PLUS Experience with mixture-of-experts training at scale, including expert parallelism, load balancing, and all-to-all communication optimization.
- PLUS Experience with GPU communication libraries such as NCCL or network topology-aware scheduling.
- PLUS Experience with fault-tolerant training, asynchronous checkpointing, in-memory recovery, or elastic training.
- PLUS Experience balancing workloads across different encoders in multimodal training.
- PLUS Familiarity with cluster schedulers (Slurm, Kubernetes) and GPU health monitoring at fleet scale.
Résumé keywords for this role
Terms the posting stresses. Include the ones you can honestly back up.
Tools & technologies
- Megatron-LM
- NeMo
- CUDA
- Triton
- CUTLASS
- BF16
- FP8
- NCCL
- Slurm
- Kubernetes
- Python
- C++
- MFU
- GPU
- kernel
Concepts & phrases
- Member of Technical Staff
- Distributed Training Performance
- artificial intelligence company
- advanced, personalized intelligence
- proactive, multimodal
- interacting with the world
- persistent memory
- next-generation hardware
- universal interface
- human and machines
- today's AI
- chat boxes
- decade-old devices
- what comes next
- agentic systems
Words that count as the same thing: "MFU" ≈ Model FLOPs Utilization · "step time" ≈ training step latency · "mixture-of-experts" ≈ MoE · "parallelism strategies" ≈ parallelization techniques, distributed training parallelism · "mixed-precision training" ≈ BF16 training, FP8 training
Check my résumé against these keywordsHiring at Hark
Open senior roles RoleStack tracks right now, posted in the last 30 days.
More open roles at Hark
- Member of Technical Staff, Pretraining Evaluations $180k–$450k
- Member of Technical Staff, Mid-training $180k–$450k
- Member of Technical Staff, Pretraining Architecture and Scaling $180k–$450k
Full job description from Hark
From Hark's posting. View the original
About Hark
Hark is an artificial intelligence company building advanced, personalized intelligence. One that is proactive, multimodal, and capable of interacting with the world through speech, text, vision, and persistent memory.
We're pairing that intelligence with next-generation hardware to create a universal interface between humans and machines. While today's AI largely operates through chat boxes and decade-old devices, Hark is focused on what comes next: agentic systems that interact naturally with people and the real world.
Show full description Show less
To get there, we're developing multimodal models and next-generation AI hardware together - designed from the ground up as a single, unified interface for a new era of intelligent systems.
About the Role
The Omni team at Hark is building the next generation of AI experiences beyond text, enabling models to understand and generate content across multiple modalities, including text, audio, and beyond. Our goal is to create seamless, real-time multimodal intelligence that powers intuitive and immersive user experiences.
As part of the Omni team, you'll improve the performance of Hark's Megatron-based pretraining systems across thousands of GPUs. The role focuses on model FLOPs utilization (MFU), training throughput, and the performance of kernels, communication, and parallelism strategies. You'll work with researchers to support new architectures and with infrastructure engineers to resolve bottlenecks, maintain numerical stability, and keep large runs reliable.
Responsibilities
• Own MFU and step time for pretraining runs across dense and mixture-of-experts models, setting targets per model size and tracking them across runs.
• Write or integrate fused and custom kernels (CUDA, Triton, CUTLASS) for attention, normalization, and mixture-of-experts routing.
• Overlap communication with compute, reduce pipeline bubbles, and tune activation recomputation and memory use to raise throughput per GPU.
• Implement model and optimizer sharding, and optimize data, tensor, pipeline, sequence, context, and expert parallelism for dense and mixture-of-experts architectures.
• Profile GPU kernels, collective communication, data loading, and mixed-precision training to identify performance and numerical stability bottlenecks.
• Develop reusable training components that support new architectures and multimodal encoders for speech, vision, and text.
• Investigate performance regressions and training failures involving communication, numerical changes, hardware faults, or straggling nodes.
• Work with architecture researchers and compute infrastructure engineers to align model designs, training software, and cluster capabilities.
• Improve checkpointing, restart time, failure detection, and observability to reduce lost training work. Participate in on-call support during large runs.
Requirements
• Experience developing distributed training systems and working with the internals of Megatron-LM, NeMo, or a comparable framework, including its parallelism strategies.
• Strong understanding of data, tensor, pipeline, sequence, context, and expert parallelism, including communication patterns, memory tradeoffs, and failure modes.
• Experience using GPU profilers to analyze kernel execution, memory bandwidth, communication overhead, and data-loading stalls.
• Working knowledge of mixed-precision training, including BF16 and FP8, and methods for diagnosing numerical instability.
• Strong Python and C++/CUDA engineering skills, with experience building reliable tools for researchers.
• Experience developing CUDA or Triton kernels, or using CUTLASS, for attention, normalization, or mixture-of-experts models.
• A record of measurably improving MFU or throughput on large-scale training jobs.
Bonus Qualifications
• Experience with mixture-of-experts training at scale, including expert parallelism, load balancing, and all-to-all communication optimization.
• Experience with GPU communication libraries such as NCCL or network topology-aware scheduling.
• Experience with fault-tolerant training, asynchronous checkpointing, in-memory recovery, or elastic training.
• Experience balancing workloads across different encoders in multimodal training.
• Familiarity with cluster schedulers (Slurm, Kubernetes) and GPU health monitoring at fleet scale.
Compensation
The US base salary range for this full-time position is between $180,000 - $450,000 annually.
The pay offered for this position may vary based on several individual factors, including job-related knowledge, skills, and experience. The total compensation package may also include additional components/benefits depending on the specific role. This information will be shared if an employment offer is extended.
FAQ
How much does a Member of Technical Staff, Distributed Training Performance at Hark make?
The posted pay for this role is $180k–$450k a year.
Does Hark sponsor visas for this role?
The posting does not say whether visa sponsorship is available. Ask the recruiter before you apply.
Is this role remote?
It is an on-site role.
What tools and technologies does the posting mention?
Megatron-LM, NeMo, CUDA, Triton, CUTLASS, BF16, FP8, NCCL, Slurm, Kubernetes, Python, C++, MFU, GPU, kernel.