Senior Site Reliability Engineer
Andromeda Cluster
- San Francisco, CA ; Global
- Remote
- Senior level
- Platform, Infrastructure & SRE
- Visa: not specified
The role in 30 seconds
Andromeda is building a market and infrastructure platform to buy, sell, and operate compute, specifically focusing on enabling fragmented GPU compute for AI. This Senior Site Reliability Engineer role will design, operate, and debug large-scale GPU infrastructure used for distributed training and inference, directly partnering with customers pushing the limits of modern AI systems.
Is this role a fit for you?
The backgrounds this role is most likely to hire from, as RoleStack reads the posting.
Strong match
- B2B infrastructure / API / developer tools
- Software, SaaS & AI Platform
- Internal tooling / back-office systems
- Enterprise-scale (high availability, compliance, large distributed systems)
- Growth-stage (scaling existing systems, reliability concerns)
Workable stretch
- B2B enterprise platform / workflow software
- Data platform / ML pipeline
- Startup / greenfield (build from scratch, speed over polish)
Harder transition
- B2C consumer-facing application
- Creator / builder / self-serve tool
- Marketplace / multi-sided platform
- Embedded systems / hardware-adjacent
Industry: Software, SaaS & AI Platform. See your personal match score
What they're really screening for
Must-haves
- CORE Deep, hands-on experience operating large-scale GPU clusters (NVIDIA A100/H100/B200 or equivalent). You understand GPU memory hierarchies, ECC behavior, thermal throttling, and hardware failure modes from direct experience not documentation.
- CORE Production experience with InfiniBand, RoCE, or NVLink fabrics in the context of distributed training. You can diagnose why an all-reduce is slow, identify a degraded link in a fat-tree topology, and reason about congestion control at scale.
- CORE Working knowledge of how large training jobs actually run — NCCL, CUDA, PyTorch distributed, DeepSpeed, Megatron, FSDP, or similar. You don't need to write the models, but you need to understand what's happening at the systems level when a 1,000-GPU training run stalls.
- CORE Expert-level Linux knowledge: kernel tuning, driver management (NVIDIA drivers, CUDA toolkit), cgroup/namespace internals, performance profiling at the syscall and hardware level.
- CORE Strong experience running Kubernetes in production with GPU workloads, including device plugins, topology-aware scheduling, multi-cluster federation, and custom operators. Experience with Slurm or other HPC schedulers is equally valued.
- PLUS Strong engineering skills in Python, Go, or Bash. You build production-grade tools and services, not just scripts. Infrastructure-as-Code proficiency (Terraform, Helm, Ansible, or equivalent).
- CORE Hands-on experience building monitoring and alerting for GPU infrastructure, not just Prometheus/Grafana basics, but GPU-specific telemetry (DCGM, nvidia-smi, fabric manager metrics) integrated into actionable dashboards.
- CORE Proven track record leading incident response for complex distributed systems where the failure could be in hardware, firmware, networking, drivers, orchestration, or application code and you need to narrow it down fast.
- CORE Experience building for AI infrastructure
- CORE Experience building for AI factories or similar compute platforms
Nice-to-haves
- PLUS Distributed Storage: Experience with high-performance parallel file systems (VAST, Weka, Lustre, GPFS) and the checkpoint I/O and data-loading bottlenecks that come with large training runs.
- PLUS Training Optimization: Experience profiling and optimizing distributed training performance: identifying stragglers, tuning collective communication strategies, improving MFU (Model FLOPs Utilization), and reducing idle GPU time across large runs.
- PLUS Cluster Buildout & Hardware: Experience involved in physical cluster design - rack layout, power/cooling constraints, network topology design, and hardware validation/burn-in at scale.
- PLUS Team Leadership: Experience leading or mentoring a team of infrastructure engineers. We're growing and need people who raise the bar for everyone around them.
Résumé keywords for this role
Terms the posting stresses. Include the ones you can honestly back up.
Tools & technologies
- GPU
- NVIDIA A100/H100/B200
- ECC
- NVLink
- NCCL
- InfiniBand
- RoCE
- CUDA
- PyTorch distributed
- DeepSpeed
- Megatron
- FSDP
- Linux
- cgroup
- namespace
Concepts & phrases
- Senior Site Reliability Engineer
- market and infrastructure platform
- GPU clusters in production
- distributed training and inference
- modern AI systems
- GPU Cluster Architecture
- multi-provider, multi-region GPU compute clusters
- large-scale training
- topology-aware scheduling
- networking, and storage decisions
- training throughput and cost efficiency
- Customer Technical Partnership
- primary technical point of contact
- large-scale training workloads
- Reliability & Performance Engineering
Words that count as the same thing: "AI factories" ≈ GPU compute clusters, AI infrastructure · "compute" ≈ GPU capacity, Computational resources · "distributed training" ≈ Large-scale model training, Parallel model training
Check my résumé against these keywordsHiring at Andromeda Cluster
Open senior roles RoleStack tracks right now, posted in the last 30 days.
More open roles at Andromeda Cluster
Full job description from Andromeda Cluster
From Andromeda Cluster's posting. View the original
Senior Site Reliability Engineer
Location: Global Remote / San Francisco · Full-Time
ABOUT ANDROMEDA
Andromeda is a market and infrastructure platform to buy, sell, and operate compute.
We believe demand for compute will grow exponentially. So fast that a handful of vertically integrated providers won't be able to scale across operations, capital, supply chains, and politics to serve it. The result is a massive wave of fragmentation, with AI factories of every shape and size coming to market to fill this demand. Our job is to enable all of that fragmented compute to flow through one platform, delivering reliable capacity to model builders, research labs, and inference providers when they need it. We believe every spare electron should be made productive for AI and we're building the platform that makes that possible.
Show full description Show less
We sit at the center of three forces:
- Companies that need reliable, high-performance compute fast
- A fragmented global supply of GPUs across hyperscalers, neoclouds, and independent data centers
- Capital, risk, and operational complexity that most teams are not equipped to manage
When we succeed, trillions of dollars of compute will flow through Andromeda. Builders get capacity when they need it. Providers get a reliable way to monetize, operate, and finance infrastructure at scale. Capital gets an easy way to deploy, hedge, and underwrite.
In five years, Andromeda won't just participate in the AI infrastructure market. We will shape it.
The Role
This is not a generalist SRE role.
You will design, operate, and debug large-scale GPU infrastructure used for distributed training and inference, working directly with customers pushing the limits of modern AI systems.
We’re looking for engineers who have personally run GPU clusters in production, understand the failure modes of distributed training, and can reason about performance from network fabric → kernel → framework.
What You’ll Own
- GPU Cluster Architecture: Design and evolve multi-provider, multi-region GPU compute clusters optimized for large-scale training. Make topology-aware scheduling, networking, and storage decisions that directly impact training throughput and cost efficiency.
- Customer Technical Partnership: Serve as the primary technical point of contact for customers running large-scale training workloads. Onboard, troubleshoot, and optimize, often in real time.
- Reliability & Performance Engineering: Define SLOs and error budgets that account for the unique failure modes of GPU infrastructure (ECC errors, NVLink degradation, NCCL timeouts). Own capacity planning across heterogeneous GPU fleets optimized for training throughput.
- Networking & Fabric Health: Ensure the health and performance of high-speed interconnects (InfiniBand, RoCE, NVLink) that underpin distributed training. Diagnose and resolve fabric-level issues that degrade collective operations.
- Observability: Build deep visibility into GPU utilization, memory pressure, interconnect throughput, training job performance, and hardware health. Go well beyond standard infrastructure metrics.
- Automation & Tooling: Build production-grade automation for cluster provisioning, GPU health checks, job scheduling, self-healing, and firmware/driver lifecycle management.
- Incident Leadership: Lead incident response for complex, multi-layer failures spanning hardware, networking, orchestration, and ML frameworks. Drive blameless postmortems and systemic fixes.
What We’re Looking For
- GPU Systems Expertise: Deep, hands-on experience operating large-scale GPU clusters (NVIDIA A100/H100/B200 or equivalent). You understand GPU memory hierarchies, ECC behavior, thermal throttling, and hardware failure modes from direct experience not documentation.
- High-Performance Networking: Production experience with InfiniBand, RoCE, or NVLink fabrics in the context of distributed training. You can diagnose why an all-reduce is slow, identify a degraded link in a fat-tree topology, and reason about congestion control at scale.
- Distributed Training & ML Frameworks: Working knowledge of how large training jobs actually run — NCCL, CUDA, PyTorch distributed, DeepSpeed, Megatron, FSDP, or similar. You don't need to write the models, but you need to understand what's happening at the systems level when a 1,000-GPU training run stalls.
- Linux & Systems Internals: Expert-level Linux knowledge: kernel tuning, driver management (NVIDIA drivers, CUDA toolkit), cgroup/namespace internals, performance profiling at the syscall and hardware level.
- Kubernetes & Orchestration: Strong experience running Kubernetes in production with GPU workloads, including device plugins, topology-aware scheduling, multi-cluster federation, and custom operators. Experience with Slurm or other HPC schedulers is equally valued.
- Automation & Software Engineering: Strong engineering skills in Python, Go, or Bash. You build production-grade tools and services, not just scripts. Infrastructure-as-Code proficiency (Terraform, Helm, Ansible, or equivalent).
- Observability & Monitoring: Hands-on experience building monitoring and alerting for GPU infrastructure, not just Prometheus/Grafana basics, but GPU-specific telemetry (DCGM, nvidia-smi, fabric manager metrics) integrated into actionable dashboards.
- Incident Management: Proven track record leading incident response for complex distributed systems where the failure could be in hardware, firmware, networking, drivers, orchestration, or application code and you need to narrow it down fast.
Strong Candidates May Have
- Distributed Storage: Experience with high-performance parallel file systems (VAST, Weka, Lustre, GPFS) and the checkpoint I/O and data-loading bottlenecks that come with large training runs.
- Training Optimization: Experience profiling and optimizing distributed training performance: identifying stragglers, tuning collective communication strategies, improving MFU (Model FLOPs Utilization), and reducing idle GPU time across large runs.
- Cluster Buildout & Hardware: Experience involved in physical cluster design - rack layout, power/cooling constraints, network topology design, and hardware validation/burn-in at scale.
- Team Leadership: Experience leading or mentoring a team of infrastructure engineers. We're growing and need people who raise the bar for everyone around them.
Why You’ll Love It Here
This is a high-impact, senior builder’s role. You’ll have significant ownership and autonomy to shape how our systems run at a foundational level, working directly with customers and providers while architecting the infrastructure backbone for reliable, scalable AI compute. You’ll influence technical direction and help define what world-class AI infrastructure operations look like.
Andromeda Cluster is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.
FAQ
Does Andromeda Cluster sponsor visas for this role?
The posting does not say whether visa sponsorship is available. Ask the recruiter before you apply.
Is this role remote?
It is a remote role.
What tools and technologies does the posting mention?
GPU, NVIDIA A100/H100/B200, ECC, NVLink, NCCL, InfiniBand, RoCE, CUDA, PyTorch distributed, DeepSpeed, Megatron, FSDP, Linux, cgroup, namespace.