Staff Network Production Engineer, Operations

Crusoe

  • San Francisco, CA
  • On-site
  • Staff/Principal level
  • 8+ yrs experience
  • Network Production Operations Engineering
  • Visa: not specified
  • $215k–$260k

The role in 30 seconds

Crusoe Cloud is hiring a Staff Network Production Operations Engineer to own production reliability across a global network spanning edge, backbone, data center fabric, and GPU cluster interconnects. The role is hands-on production ownership focused on incident response, root cause analysis, observability automation, and operational excellence that keeps hyperscale AI infrastructure running at scale. The engineer serves as a key escalation point during high-severity network events and builds Python tooling to reduce toil and accelerate resolution.

Is this role a fit for you?

The backgrounds this role is most likely to hire from, as RoleStack reads the posting.

Strong match

  • B2B infrastructure / API / developer tools
  • Data platform / ML pipeline
  • Enterprise-scale
  • Growth-stage

Workable stretch

  • B2B enterprise platform / workflow software
  • Startup / greenfield

Harder transition

  • B2C consumer-facing application
  • Internal tooling / back-office systems
  • Regulated environment

Industry: AI infrastructure / cloud compute. See your personal match score

What they're really screening for

Must-haves

  • CORE 8+ years of production network engineering experience with a focus on operations, incident response, and reliability in large-scale or internet-scale environments
  • CORE Strong Python and scripting proficiency: writing diagnostic tooling, auto-remediation scripts, and automation pipelines from scratch
  • CORE Hands-on experience with observability and monitoring tools including streaming telemetry, SNMP, NetFlow/sFlow, Grafana, Prometheus, and ThousandEyes, and ability to script against their APIs
  • CORE Experience operating RDMA/RoCE lossless fabrics for GPU or HPC workloads, including familiarity with PFC, ECN, and DCQCN tuning
  • CORE Expert hands-on knowledge of BGP, EVPN-VXLAN, IS-IS, OSPF, MPLS, QoS, and TCP/IP in production data center environments
  • CORE Proficiency with Arista (EOS) and Juniper (Junos) platforms in leaf-spine CLOS architectures across multi-vendor environments
  • CORE Comfort operating large device fleets across multi-region environments with on-call responsibility, including experience as an escalation point during critical events
  • PLUS Bachelor's degree in Computer Science, Electrical Engineering, or a related field, or equivalent practical experience
  • CORE Experience building for AI/GPU compute infrastructure or adjacent high-performance compute environments
  • PLUS Experience with incident response, root cause analysis, and operational excellence in high-severity production environments

Nice-to-haves

  • PLUS Experience with NVIDIA/Mellanox networking platforms in GPU cluster environments
  • PLUS Familiarity with Kentik or Arbor for traffic analysis and DDoS visibility
  • PLUS Experience defining or contributing to SLIs and SLOs in partnership with SRE or product teams
  • PLUS Exposure to operating 10K+ device fleets across hyperscale or cloud environments
  • PLUS Background contributing to post-incident learning programs or operational excellence initiatives org-wide

Résumé keywords for this role

Terms the posting stresses. Include the ones you can honestly back up.

Tools & technologies

  • Python
  • RDMA
  • RoCE
  • PFC
  • ECN
  • DCQCN
  • BGP
  • EVPN-VXLAN
  • IS-IS
  • OSPF
  • MPLS
  • QoS
  • TCP/IP
  • Arista
  • EOS

Concepts & phrases

  • Staff Network Production Engineer
  • production network engineering
  • incident response
  • root cause analysis
  • operational excellence
  • lossless fabrics
  • leaf-spine CLOS architectures
  • multi-vendor environments
  • large device fleets
  • multi-region environments
  • escalation point
  • high-severity network events
  • GPU cluster interconnects
  • data center fabric
  • streaming telemetry

Words that count as the same thing: "RDMA/RoCE" ≈ Remote Direct Memory Access, RoCEv2, Converged Ethernet lossless fabric · "EVPN-VXLAN" ≈ Ethernet VPN, VXLAN overlay, EVPN · "SLI/SLO" ≈ Service Level Indicators, Service Level Objectives, reliability metrics · "SRE" ≈ Site Reliability Engineering, reliability engineering · "leaf-spine CLOS" ≈ CLOS fabric, spine-leaf architecture, non-blocking fabric

Check my résumé against these keywords

Hiring at Crusoe

All Crusoe hiring data

Open senior roles RoleStack tracks right now, posted in the last 30 days.

More open roles at Crusoe

Full job description from Crusoe

From Crusoe's posting. View the original

Crusoe is on a mission to accelerate the abundance of energy and intelligence. As the only vertically integrated AI infrastructure company built from the ground up, we own and operate each layer of the stack — from electrons to tokens — to power the world's most ambitious AI workloads. When you join Crusoe, you join a team that is building the future, faster.

We're in the midst of the greatest industrial revolution of our time. The demand for AI compute is boundless, and power is a bottleneck. We're solving that — with an energy-first approach that makes AI infrastructure better for the world and faster for the people innovating with AI.

Show full description Show less

We're looking for problem-solving, opportunity-finding teammates with a sense of urgency, who believe in the scale of our ambition and thrive on a path not fully paved — people who want to grow their careers alongside a team of experts across energy, manufacturing, data center construction, and cloud services.

If you want to do the most meaningful work of your career, help our customers and partners advance their AI strategies, and be part of a high-performing team that believes in each other, come build with us at Crusoe.

About This Role:

Crusoe Cloud is seeking a Staff Network Production Operations Engineer to help own production reliability across our global network infrastructure, including edge, backbone, data center fabric, and GPU cluster interconnects. This is a hands-on production ownership role focused on incident response, root cause analysis, and operational excellence initiatives that keep our hyperscale AI infrastructure running at scale. Your work will directly affect the availability of AI workloads running across thousands of GPUs worldwide.

The ideal candidate is a seasoned network engineer with deep operational experience in large-scale environments who thrives in high-pressure situations and takes pride in keeping systems healthy. You'll contribute to defining SLIs and SLOs, improving observability tooling, building automation to reduce toil, and mentoring peers — all while serving as a key escalation point during high-severity network events.

What You'll Be Working On:

- Production Reliability: Help own uptime across Crusoe's global edge, backbone, data center, and GPU cluster network, directly supporting AI workloads at scale.

- Operational Automation: Build and maintain Python-based tooling to reduce toil, automate common remediation workflows, and accelerate mean time to resolution across the operations org.

- Automate Incident Response: Develop scripts and tooling that speed up detection, triage, and mitigation during high-severity network events, and contribute directly to the response itself, including stakeholder communication and postmortem documentation.

- Automate Root Cause Analysis: Build tooling to surface systemic issues faster during RCAs, and drive remediation plans tracked through to closure.

- Observability Automation: Write automation on top of Crusoe's monitoring stack (streaming telemetry, SNMP, NetFlow, Kentik, Grafana, Prometheus, ThousandEyes) to reduce manual triage and surface actionable signal faster.

- Operational Standards: Author and maintain runbooks, escalation playbooks, and SOPs, with an eye toward what can be scripted or automated rather than performed manually.

- SLI/SLO Contribution: Partner with Architecture and SRE teams to define and track network reliability metrics, and build the automated dashboards and alerting that back them.

- Mentorship: Provide technical guidance to Senior engineers, particularly around automation practices and reducing operational toil, and contribute to a culture of operational excellence.

What You'll Bring to the Team:

- 8+ years of production network engineering experience with a focus on operations, incident response, and reliability in large-scale or internet-scale environments.

- Strong Python and scripting proficiency: comfortable writing diagnostic tooling, auto-remediation scripts, and automation pipelines from scratch, not just modifying existing scripts.

- Hands-on experience with observability and monitoring tools including streaming telemetry, SNMP, NetFlow/sFlow, Grafana, Prometheus, and ThousandEyes, and the ability to script against their APIs.

- Experience operating RDMA/RoCE lossless fabrics for GPU or HPC workloads, including familiarity with PFC, ECN, and DCQCN tuning.

- Expert hands-on knowledge of BGP, EVPN-VXLAN, IS-IS, OSPF, MPLS, QoS, and TCP/IP in production data center environments.

- Proficiency with Arista (EOS) and Juniper (Junos) platforms in leaf-spine CLOS architectures across multi-vendor environments.

- Comfort operating large device fleets across multi-region environments with on-call responsibility, including experience as an escalation point during critical events.

- Bachelor's degree in Computer Science, Electrical Engineering, or a related field, or equivalent practical experience.

Bonus Points:

- Experience with NVIDIA/Mellanox networking platforms in GPU cluster environments.

- Familiarity with Kentik or Arbor for traffic analysis and DDoS visibility.

- Experience defining or contributing to SLIs and SLOs in partnership with SRE or product teams.

- Exposure to operating 10K+ device fleets across hyperscale or cloud environments.

- Background contributing to post-incident learning programs or operational excellence initiatives org-wide.

Benefits:

- Competitive compensation and equity packages

- Restricted Stock Units

- Paid time off, paid holidays & leave of absence programs

- Comprehensive health, dental & vision insurance

- Employer contributions to HSA account

- Paid parental leave

- Paid life insurance, short-term and long-term disability

- Professional development & tuition reimbursement

- Mental health & wellness support

- Commuter benefits (parking & transit)

- Cell phone stipend

- 401(k) Retirement plan with company match up to 4% of salary

- Volunteer time off

- Global travel insurance & emergency assistance

- Daily meals allowance

- Additional perks & programs specific to location

Compensation Range

Compensation will be paid in the range of up to $215,000 - $260,000 + Bonus. Restricted Stock Units are included in all offers. Compensation to be determined by the applicant's knowledge, education, and abilities, as well as internal equity and alignment with market data.

Crusoe is an Equal Opportunity Employer. Employment decisions are made without regard to race, color, religion, disability, genetic information, pregnancy, citizenship, marital status, sex/gender, sexual preference/ orientation, gender identity, age, veteran status, national origin, or any other status protected by law or regulation.

FAQ

How much does a Staff Network Production Engineer, Operations at Crusoe make?

The posted pay for this role is $215k–$260k a year.

How much experience do you need?

The posting asks for 8+ years in production network engineering (operations, incident response, reliability). It is a staff/principal-level role.

Does Crusoe sponsor visas for this role?

The posting does not say whether visa sponsorship is available. Ask the recruiter before you apply.

Is this role remote?

It is an on-site role.

What tools and technologies does the posting mention?

Python, RDMA, RoCE, PFC, ECN, DCQCN, BGP, EVPN-VXLAN, IS-IS, OSPF, MPLS, QoS, TCP/IP, Arista, EOS.