Principal Site Reliability Engineer (Vehicle Engineering)

SpaceX

  • Bastrop, TX
  • Staff/Principal level
  • 8+ yrs experience
  • Platform, Infrastructure & SRE
  • Visa: not specified

The role in 30 seconds

SpaceX HPC is a shared compute platform used across the company for vehicle and structures simulation, machine learning, and AI inference. This role supports every program at SpaceX to design and operate advanced rockets and satellites by implementing a Site Reliability Engineer operating model to reduce toil, improve automation, enhance observability, and establish a sustainable incident process.

Is this role a fit for you?

The backgrounds this role is most likely to hire from, as RoleStack reads the posting.

Strong match

  • B2B infrastructure / API / developer tools
  • Platform, Infrastructure & SRE
  • Growth-stage (scaling existing systems, reliability concerns)
  • Enterprise-scale (high availability, compliance, large distributed systems)

Workable stretch

  • Internal tooling / back-office systems
  • Startup / greenfield (build from scratch, speed over polish)

Harder transition

  • B2C consumer-facing application
  • Creator / builder / self-serve tool
  • Marketplace / multi-sided platform

Industry: Defense & Aerospace. See your personal match score

What they're really screening for

Must-haves

  • PLUS Bachelor's degree in computer science, engineering, math, or a scientific discipline; OR 2+ years of professional experience operating production infrastructure in lieu of a degree
  • CORE 8+ years of experience with Linux operating systems in production
  • CORE 8+ years of experience operating production infrastructure (servers, services, or networks), including monitoring, debugging, and repairing what you own
  • CORE Linux machines
  • CORE Infrastructure as Code
  • CORE Storage
  • PLUS User facing applications
  • CORE Production instincts
  • CORE Code to delete toil
  • CORE Operable, maintainable infrastructure

Nice-to-haves

  • CORE 8+ years of professional experience in SRE, DevOps, or production infrastructure engineering
  • CORE monitoring and alerting (Prometheus, Grafana, Nagios, or similar)
  • CORE deploying and maintaining configuration management or infrastructure as code (Ansible, Puppet, Terraform, or similar)
  • CORE writing scripts/code (eg. Python or similar languages) to automate common tasks
  • PLUS containers (Docker, Podman, Singularity/Apptainer)
  • PLUS Kubernetes administration for on-premise deployment
  • CORE distributed or high-performance storage (VAST or similar), including capacity, performance, and lifecycle management
  • CORE HPC clusters, schedulers (Slurm, PBS, LSF), or GPU compute
  • PLUS scientific computing (CFD, FEA) and/or ML training workloads (PyTorch, TensorFlow, CUDA) and/or AI inference workloads
  • PLUS version control, testing, continuous integration, build, deployment and monitoring

Résumé keywords for this role

Terms the posting stresses. Include the ones you can honestly back up.

Tools & technologies

  • Linux
  • OS images
  • firmware
  • kernel
  • driver stack
  • Prometheus
  • Grafana
  • Nagios
  • Ansible
  • Puppet
  • Terraform
  • Python
  • Docker
  • Podman
  • Singularity

Concepts & phrases

  • Principal Site Reliability Engineer
  • Vehicle Engineering
  • SpaceX HPC
  • infrastructure as code
  • production instincts
  • delete toil
  • operable, maintainable infrastructure
  • on-call rotation
  • sustainable incident response
  • blameless postmortems
  • node lifecycle
  • configuration management
  • kernel and driver stack
  • build observability
  • HPC administrators

Words that count as the same thing: "SRE" ≈ Site Reliability Engineering, production infrastructure engineering · "Infrastructure as Code" ≈ IaC, configuration management · "toil reduction" ≈ automation of manual tasks, automating operational tasks · "observability" ≈ monitoring, logging, tracing · "HPC" ≈ High-Performance Computing

Check my résumé against these keywords

Hiring at SpaceX

Open senior roles RoleStack tracks right now, posted in the last 30 days.

More open roles at SpaceX

Full job description from SpaceX

From SpaceX's posting. View the original

SpaceX was founded under the belief that a future where humanity is out exploring the stars is fundamentally more exciting than one where we are not. Today SpaceX is actively developing the technologies to make this possible, with the ultimate goal of enabling human life on Mars.

PRINCIPAL SITE RELIABILITY ENGINEER (VEHICLE ENGINEERING)

SpaceX HPC is a shared compute platform used across the company — vehicle and structures simulation, machine learning, AI inference, and more. We support every program at SpaceX to design and operate the worlds most advanced rockets and satellites. This role exists to put a real Site Reliability Engineer operating model on these capabilities and accelerating the world class engineering at SpaceX: toil reduction, automation, observability, and a sustainable incident process.

Show full description Show less

We are looking for a Principal Site Reliability Engineer who wants to own everything from Linux machines and our Infrastructure as Code, storage, and user facing applications – the whole ecosystem as a product, not as a ticket queue. You do not need a prior HPC title. You do need production instincts — you have operated real infrastructure, you write code to delete toil, and you care about whether users can actually get work done, not just whether nodes ping. You’ll work alongside HPC systems engineers who design and commission clusters to help make them more reliable and provide world class services for world class engineers.

Aerospace experience is not required. We value engineers who treat teammates with fairness and respect, who are self-critical, and who will take ownership of hard production problems.

RESPONSIBILITIES:

• Participate in the team's on-call rotation; practice sustainable incident response and blameless postmortems

• Manage node lifecycle with infrastructure as code: OS images, firmware, configuration management, kernel and driver stack

• Build observability for both HPC administrators and end users — cluster, node, and storage health for operators, and job/workflow-level signal for the people running work on the platform

• Reduce toil with automation; split time between operating production systems and writing the software that makes that work smaller

• Sustainably manage resources, including compute and storage

• Lead capacity planning with users across the company: understand what they will need next, and turn that into a concrete picture of tomorrow's compute and storage

• Collaborate with HPC systems engineers and with engineers across all disciplines across the company on operable, maintainable infrastructure

BASIC QUALIFICATIONS:

• Bachelor's degree in computer science, engineering, math, or a scientific discipline; OR 2+ years of professional experience operating production infrastructure in lieu of a degree

• 8+ years of experience with Linux operating systems in production

• 8+ years of experience operating production infrastructure (servers, services, or networks), including monitoring, debugging, and repairing what you own

PREFERRED SKILLS AND EXPERIENCE:

• 8+ years of professional experience in SRE, DevOps, or production infrastructure engineering

• Experience with monitoring and alerting (Prometheus, Grafana, Nagios, or similar)

• Experience deploying and maintaining configuration management or infrastructure as code (Ansible, Puppet, Terraform, or similar)

• Experience writing scripts/code (eg. Python or similar languages) to automate common tasks

• Experience with containers (Docker, Podman, Singularity/Apptainer)

• Experience with Kubernetes administration for on-premise deployment

• Experience with distributed or high-performance storage (VAST or similar), including capacity, performance, and lifecycle management

• Familiarity with HPC clusters, schedulers (Slurm, PBS, LSF), or GPU compute — not required; we will teach this

• Familiarity with scientific computing (CFD, FEA) and/or ML training workloads (PyTorch, TensorFlow, CUDA) and/or AI inference workloads

• Good understanding of version control, testing, continuous integration, build, deployment and monitoring

• Ability to communicate clearly with users, peers, and vendors in both incident and design settings

• Comfortable working with mission-critical and sensitive systems, with a sense of urgency appropriate to the responsibilities

• Eligibility for access to classified material up to TS/SCI with polygraph

ADDITIONAL REQUIREMENTS:

• Must be able to participate in an on-call rotation

• Must be willing to work extended hours and weekends as needed for incidents, cluster bring-up, and time-critical failures

ITAR REQUIREMENTS:

• To conform to U.S. Government export regulations, applicant must be a (i) U.S. citizen or national, (ii) U.S. lawful, permanent resident (aka green card holder), (iii) Refugee under 8 U.S.C. § 1157, or (iv) Asylee under 8 U.S.C. § 1158, or be eligible to obtain the required authorizations from the U.S. Department of State. Learn more about the ITAR here.

SpaceX is an Equal Opportunity Employer; employment with SpaceX is governed on the basis of merit, competence and qualifications and will not be influenced in any manner by race, color, religion, gender, national origin/ethnicity, veteran status, disability status, age, sexual orientation, gender identity, marital status, mental or physical disability or any other legally protected status.

Applicants wishing to view a copy of SpaceX’s Affirmative Action Plan for veterans and individuals with disabilities, or applicants requiring reasonable accommodation to the application/interview process should reach out to EEOCompliance@spacex.com.

FAQ

How much experience do you need?

The posting asks for 8+ years in production infrastructure. It is a staff/principal-level role.

Does SpaceX sponsor visas for this role?

The posting does not say whether visa sponsorship is available. Ask the recruiter before you apply.

What tools and technologies does the posting mention?

Linux, OS images, firmware, kernel, driver stack, Prometheus, Grafana, Nagios, Ansible, Puppet, Terraform, Python, Docker, Podman, Singularity.