Senior Validation Technical Program Manager, Logical Deployment

Nscale

  • San Francisco, CA ; Seattle, WA
  • On-site
  • Senior level
  • 7+ yrs experience
  • Technical Program Management
  • Visa: not specified

The role in 30 seconds

Nscale builds and operates large-scale AI infrastructure, delivering GPU clusters for leading hyperscalers, AI labs and enterprises. The Deployment PMO turns signed contracts into live AI infrastructure, coordinating many teams and partners. The Validation TPM owns the proof that a cluster is ready for the customer by translating acceptance criteria into test plans, running them at scale, and delivering verifiable results.

Is this role a fit for you?

The backgrounds this role is most likely to hire from, as RoleStack reads the posting.

Strong match

  • B2B infrastructure / API / developer tools
  • Software, SaaS & AI Platform
  • Large-scale compute infrastructure
  • Enterprise-scale (high availability, compliance, large distributed systems)
  • Growth-stage (scaling existing systems, reliability concerns)

Workable stretch

  • B2B enterprise platform / workflow software
  • Internal tooling / back-office systems
  • Startup / greenfield (build from scratch, speed over polish)

Harder transition

  • B2C consumer-facing application
  • Creator / builder / self-serve tool
  • Marketplace / multi-sided platform
  • Embedded systems / hardware-adjacent
  • Regulated environment (healthcare, finance, government)

Industry: Software, SaaS & AI Platform. See your personal match score

What they're really screening for

Must-haves

  • CORE 7-10+ years in software validation, test engineering or QA for infrastructure or distributed systems, including program or project ownership.
  • CORE Hands-on experience building and running test frameworks, and reading and debugging their output.
  • CORE Strong scripting in Python and Bash, and comfort modifying test harnesses and data pipelines.
  • CORE Working knowledge of GPU cluster performance and validation, such as NCCL collectives, HPL, DCGM diagnostics and burn-in.
  • CORE Familiarity with cluster networking and storage testing, including InfiniBand or RoCE bandwidth and latency, and storage throughput.
  • CORE Experience turning requirements or acceptance criteria into test plans with clear pass/fail evidence.
  • CORE Clear, concise communication with engineers and customers, in writing and in the room.
  • PLUS Ability to travel up to 50% of the time, primarily to data center sites.
  • CORE Translate each customer’s acceptance criteria into a clear test plan: what is run, at what scale, and with what pass thresholds.
  • CORE Agree the plan with the customer and engineering before testing starts, so there are no surprises at handoff.

Nice-to-haves

  • PLUS Experience validating NVIDIA GB200/GB300 or Vera Rubin NVL72-class clusters, including NVIDIA cluster validation tooling.
  • PLUS Experience with MLPerf or customer-defined training and inference benchmarks at scale.
  • PLUS Familiarity with Slurm or Kubernetes for scheduling large test jobs.
  • PLUS Experience at a hyperscaler, neocloud or AI lab accepting 10K+ GPU clusters.
  • PLUS Background in CI/CD and test automation for infrastructure.

Résumé keywords for this role

Terms the posting stresses. Include the ones you can honestly back up.

Tools & technologies

  • AI
  • GPU
  • clusters
  • hyperscalers
  • AI labs
  • PMO
  • TPM
  • validation
  • test engineering
  • QA
  • infrastructure
  • distributed systems
  • program management
  • test frameworks
  • scripting

Concepts & phrases

  • Senior Validation Technical Program Manager
  • Logical Deployment
  • large-scale AI infrastructure
  • GPU clusters
  • leading hyperscalers
  • AI labs
  • Deployment PMO
  • signed contracts
  • live AI infrastructure
  • large GPU clusters
  • software validation
  • test frameworks
  • program management
  • cluster scale
  • test plan

Words that count as the same thing: "AI infrastructure" ≈ GPU clusters, Large-scale AI compute · "hyperscalers" ≈ Major cloud providers, Large cloud infrastructure providers · "AI labs" ≈ Machine learning research organizations, Advanced AI development teams · "acceptance criteria" ≈ Customer requirements, Definition of done, Pass/fail thresholds · "handoff date" ≈ Delivery deadline, Customer acceptance deadline

Check my résumé against these keywords

Hiring at Nscale

Open senior roles RoleStack tracks right now, posted in the last 30 days.

More open roles at Nscale

Full job description from Nscale

From Nscale's posting. View the original

About Nscale and the Deployment PMO

Nscale builds and operates large-scale AI infrastructure, delivering GPU clusters for leading hyperscalers, AI labs and enterprises across North America, EMEA and APAC.

The Deployment PMO turns signed contracts into live AI infrastructure, coordinating the many teams and partners involved in bringing large GPU clusters online.

About the role

The Validation TPM owns the proof that a cluster is ready for the customer. You take the customer’s acceptance criteria and turn them into a test plan, run it at cluster scale, and deliver results that stand up to scrutiny. You track the state of validation for every cluster, from first test run to signed acceptance.

Show full description Show less

You come from software validation and test frameworks, not only program management. You go deep into any test, question outputs that look too good or too bad, and write the script yourself when that is the fastest way to unblock testing. You sit between engineering, OEM partners and the customer, and your results decide when a cluster is handed off.

This is an individual contributor role reporting to the Director, Logical Deployment. Expect to travel up to 50% of the time, primarily to data center sites.

What you’ll do

Turn acceptance criteria into tests

• Translate each customer’s acceptance criteria into a clear test plan: what is run, at what scale, and with what pass thresholds.

• Agree the plan with the customer and engineering before testing starts, so there are no surprises at handoff.

• Map every acceptance criterion to the test result that proves it, and keep that traceability current.

Run and own cluster validation

• Drive acceptance validation end to end: cluster-wide performance, interconnect, storage, stability and burn-in.

• Track validation status for every cluster, with a clear view of what has passed, what has failed and what is blocked.

• Coordinate test windows with fabric, provisioning and OEM teams so validation runs without idle time.

Go deep and unblock

• Read raw test output, not just summaries. Question results that don’t make sense and find the root cause.

• Write or modify scripts when a test, harness or data collection step is missing or broken.

• Triage failures to the right owner (hardware, fabric, firmware, software or configuration) with evidence, and drive them to closure.

• Re-run and verify fixes before marking anything as passed.

Close out with the customer

• Produce acceptance reports that tie results to criteria, explain exceptions plainly and support sign-off.

• Brief the customer and leadership on validation status, with an honest view of risk to the handoff date.

• Feed what you learn back into the validation playbook, test suites and automation, so each cluster validates faster than the last.

What success looks like in your first 6–12 months

• Every active cluster has a test plan mapped to its customer’s acceptance criteria, agreed before testing starts.

• Validation status is visible for every cluster without manual chasing.

• Failures are triaged to an owner with evidence within a day, and no cluster reaches handoff with an unexplained result.

• Customers sign off on acceptance reports without rework.

• A reusable acceptance test suite and reporting automation are in use on every new cluster.

What you bring

• 7-10+ years in software validation, test engineering or QA for infrastructure or distributed systems, including program or project ownership.

• Hands-on experience building and running test frameworks, and reading and debugging their output.

• Strong scripting in Python and Bash, and comfort modifying test harnesses and data pipelines.

• Working knowledge of GPU cluster performance and validation, such as NCCL collectives, HPL, DCGM diagnostics and burn-in.

• Familiarity with cluster networking and storage testing, including InfiniBand or RoCE bandwidth and latency, and storage throughput.

• Experience turning requirements or acceptance criteria into test plans with clear pass/fail evidence.

• Clear, concise communication with engineers and customers, in writing and in the room.

• Ability to travel up to 50% of the time, primarily to data center sites.

How you work

• Rigorous: You trust data, not dashboards. A pass needs evidence, and a strange number gets investigated, not explained away.

• Bold: You call a cluster not ready when the results say so, even close to a handoff date, and you put a path to ready next to it.

• Assertive with partners: You hold engineering, OEM partners and vendors to fixing what the tests find. You escalate early, directly and with evidence.

• Accountable: You own validation outcomes. Rather than “waiting on engineering,” you explain what you are doing to unblock it.

• Steady under pressure: You keep testing moving when failures stack up and the handoff date is close.

• Technically deep: You can read a test log, a performance trace or a script and know where the problem is.

Preferred qualifications

• Experience validating NVIDIA GB200/GB300 or Vera Rubin NVL72-class clusters, including NVIDIA cluster validation tooling.

• Experience with MLPerf or customer-defined training and inference benchmarks at scale.

• Familiarity with Slurm or Kubernetes for scheduling large test jobs.

• Experience at a hyperscaler, neocloud or AI lab accepting 10K+ GPU clusters.

• Background in CI/CD and test automation for infrastructure.

For information on how Nscale handles candidate personal data, please see our Employee & Candidate Privacy Notice: Here.

Nscale does not accept unsolicited candidate submissions from recruitment agencies.

FAQ

How much experience do you need?

The posting asks for 7+ years in software validation, test engineering or qa for infrastructure or distributed systems. It is a senior-level role.

Does Nscale sponsor visas for this role?

The posting does not say whether visa sponsorship is available. Ask the recruiter before you apply.

Is this role remote?

It is an on-site role.

What tools and technologies does the posting mention?

AI, GPU, clusters, hyperscalers, AI labs, PMO, TPM, validation, test engineering, QA, infrastructure, distributed systems, program management, test frameworks, scripting.