Senior Software Engineer, AI Compute Scheduling
- Santa Clara, CA ; Westford, MA ; Austin, TX ; Durham, NC
- On-site
- Senior level
- 6+ yrs experience
- Platform, Infrastructure & SRE
- Visa: not specified
- $184k–$288k
The role in 30 seconds
The MARS Scheduling team decides how NVIDIA's AI research compute gets used across large GPU clusters. The team builds and scales the infrastructure, platforms, and tools that enable researchers and engineers to develop the next generation of AI/ML systems, supporting workloads from frontier model pretraining to reinforcement learning. You will own real parts of the scheduling stack, shipping code that impacts GPU-hours returned to researchers within days.
Is this role a fit for you?
The backgrounds this role is most likely to hire from, as RoleStack reads the posting.
Strong match
- B2B infrastructure / API / developer tools
- AI/ML platform
- Platform, Infrastructure & SRE
- Enterprise-scale (high availability, compliance, large distributed systems)
- Growth-stage (scaling existing systems, reliability concerns)
Workable stretch
- Internal tooling / back-office systems
- B2B enterprise platform / workflow software
- Startup / greenfield (build from scratch, speed over polish)
Harder transition
- B2C consumer-facing application
- Creator / builder / self-serve tool
- Regulated environment (healthcare, finance, government)
Industry: Software, SaaS & AI Platform. See your personal match score
What they're really screening for
Must-haves
- PLUS BS or MS in Computer Science, Computer Engineering, Electrical Engineering or a related field, or equivalent experience.
- CORE 6+ years building production systems software.
- CORE Deep experience with at least one batch or cluster scheduler, such as Slurm, Kueue, Volcano, KAI, YuniKorn or a scheduler you built. You should be able to explain its internals, not only its configuration.
- CORE Strong programming in Go, C or C++, plus Python for tooling and analysis.
- CORE Hands-on Kubernetes experience: controllers or operators, CRDs, scheduling and resource management.
- CORE Solid Linux systems knowledge: containers (Docker, Podman, Enroot or Apptainer), cgroups, networking basics and debugging under load.
- CORE A track record of owning a service in production and making it measurably better.
- CORE Clear written and spoken communication with engineers and researchers.
- CORE Experience scheduling large-scale AI training: thousands of GPUs, NVLink and InfiniBand or RoCE topology, multi-node jobs that must start together.
- PLUS Contributions to Slurm, Kubernetes SIG Scheduling, Kueue, Volcano, KAI or similar open-source schedulers.
Résumé keywords for this role
Terms the posting stresses. Include the ones you can honestly back up.
Tools & technologies
- GPU
- Slurm
- Kubernetes
- NVLink
- InfiniBand
- RoCE
- Go
- C
- C++
- Python
- Docker
- Podman
- Enroot
- Apptainer
- cgroups
Concepts & phrases
- Senior Software Engineer, AI Compute Scheduling
- Managed AI Research Superclusters
- production systems software
- batch or cluster scheduler
- Kubernetes controllers or operators
- Solid Linux systems knowledge
- owned a service in production
- large-scale AI training
- thousands of GPUs
- NVLink and InfiniBand or RoCE topology
- multi-node jobs
- Slurm, Kubernetes SIG Scheduling
- Kueue, Volcano, KAI
- Slurm on Kubernetes
- Kubernetes operators for batch or HPC workloads
Words that count as the same thing: "AI compute scheduling" ≈ GPU cluster scheduling, AI/ML workload scheduling · "fleet" ≈ cluster, compute infrastructure · "fair sharing" ≈ resource allocation, quotas · "agentic workloads" ≈ agent-based computing, autonomous systems · "goodput" ≈ system efficiency, throughput
Check my résumé against these keywordsHiring at NVIDIA
All NVIDIA hiring dataOpen senior roles RoleStack tracks right now, posted in the last 30 days.
More open roles at NVIDIA
- Senior Product Manager - NVIDIA Sentry and DOCA Security $208k–$380k
- Senior System Software Engineer, NPI Data Center $152k–$242k
- Senior Software Engineer - Quantum Simulations $184k–$357k
- Senior Manager, Technical Program Management — Planning $200k–$322k
- Senior Staff Software Engineer $200k–$322k
Full job description from NVIDIA
From NVIDIA's posting. View the original
NVIDIA is a pioneer in accelerated computing, known for inventing the GPU and driving breakthroughs in gaming, computer graphics, high-performance computing, and artificial intelligence. Our technology powers everything from generative AI to autonomous systems, and we continue to shape the future of computing through innovation and collaboration. Within this mission, our team, Managed AI Research Superclusters (MARS), builds and scales the infrastructure, platforms, and tools that enable researchers and engineers to develop the next generation of AI/ML systems. By joining us, you'll help design solutions that power some of the world's most advanced computing workloads.
Show full description Show less
The MARS Scheduling team decides how NVIDIA's AI research compute gets used. That fleet spans many of the largest GPU clusters in the world, across multiple clouds and partners, running everything from frontier-model pretraining to reinforcement learning and agentic workloads. We run Slurm and Kubernetes at large scale. Our scheduling is coordinated and topology-sensitive on rack-scale systems. We manage fair sharing and quotas for hundreds of research teams. We support preemption and reservations. Elastic jobs recover from failure without losing progress. You will own real parts of the scheduling stack, and the opportunity to ship code into our open-source and internal platform code, and see the result in GPU-hours returned to researchers within days. The researchers training NVIDIA's own models are your users, and they will tell you quickly what works.
What you'll be doing:
• Design and build scheduling for large GPU fleets on both Slurm and Kubernetes. That covers queueing, gang and topology-aware placement on rack-scale NVLink systems, preemption, reservations and backfill that reaches the end of very deep queues.
• Build the policy layer: fair share, quotas and allocations across many research teams, driven from a central allocation system and enforced the same way on every scheduler.
• Run Slurm as a cloud-native service on Kubernetes, and extend Kubernetes batch scheduling (Kueue and its ecosystem) for training, reinforcement learning, sandboxes and inference.
• Build services around the scheduler: reservation automation, node-health-aware placement, workflow orchestration across clusters, and APIs that let other platforms submit and track work.
• Measure and raise goodput: GPU occupancy, waste, fairness and time to start for large jobs. Use the data to choose what to build next.
• Own reliability for the components you build: on-call, incident analysis and fixes that remove whole classes of failure, so problems are found before researchers hit them.
• Work upstream and across teams: contribute to Slurm, Kubernetes and related open-source projects, and partner with platform, storage, networking and research teams.
What we need to see:
• BS or MS in Computer Science, Computer Engineering, Electrical Engineering or a related field, or equivalent experience.
• 6+ years building production systems software.
• Deep experience with at least one batch or cluster scheduler, such as Slurm, Kueue, Volcano, KAI, YuniKorn or a scheduler you built. You should be able to explain its internals, not only its configuration.
• Strong programming in Go, C or C++, plus Python for tooling and analysis.
• Hands-on Kubernetes experience: controllers or operators, CRDs, scheduling and resource management.
• Solid Linux systems knowledge: containers (Docker, Podman, Enroot or Apptainer), cgroups, networking basics and debugging under load.
• A track record of owning a service in production and making it measurably better.
• Clear written and spoken communication with engineers and researchers.
Ways to stand out from the crowd:
• Experience scheduling large-scale AI training: thousands of GPUs, NVLink and InfiniBand or RoCE topology, multi-node jobs that must start together.
• Contributions to Slurm, Kubernetes SIG Scheduling, Kueue, Volcano, KAI or similar open-source schedulers.
• Experience running Slurm on Kubernetes (Slinky or similar), or building Kubernetes operators for batch or HPC workloads.
• Familiarity with RL, agentic or inference workloads and what they need from a scheduler: heterogeneous jobs, sandboxes, elastic scale and fast restarts.
• Experience with workflow orchestrators such as Flyte, Airflow, Ray or Kubeflow at scale.
Your base salary will be determined based on your location, experience, and other objective job-related factors. The expected base salary ranges for this position are:
184,000 USD – 287,500 USD for Level 4; and 224,000 USD – 356,500 USD for Level 5.
You will also be eligible for equity and benefits.
Applications for this job will be accepted at least until November 14, 2026.
This posting is for an existing vacancy.
NVIDIA uses AI tools in its recruiting processes.
NVIDIA is committed to fostering an inclusive work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.
FAQ
How much does a Senior Software Engineer, AI Compute Scheduling at NVIDIA make?
The posted pay for this role is $184k–$288k a year.
How much experience do you need?
The posting asks for 6+ years in production systems software. It is a senior-level role.
Does NVIDIA sponsor visas for this role?
The posting does not say whether visa sponsorship is available. Ask the recruiter before you apply.
Is this role remote?
It is an on-site role.
What tools and technologies does the posting mention?
GPU, Slurm, Kubernetes, NVLink, InfiniBand, RoCE, Go, C, C++, Python, Docker, Podman, Enroot, Apptainer, cgroups.