Senior Software Engineer – Network Observability

Clockwork.io

  • Palo Alto, CA
  • On-site
  • Senior level
  • Backend Engineering
  • Visa: not specified
  • $140k–$210k

The role in 30 seconds

Clockwork.io is pioneering a software-driven approach to AI fabrics by delivering cross-stack observability to catch and quickly resolve problems, workload fault tolerance to keep jobs running through failures, and performance acceleration that dynamically routes and paces traffic to avoid congestion. We are seeking an experienced Senior Software Engineer to contribute to the architecture, development, and scaling of a high-performance network monitoring and observability platform, focusing on systems that provide deep visibility into RDMA, RoCE, InfiniBand, and TCP/IP networks.

Is this role a fit for you?

The backgrounds this role is most likely to hire from, as RoleStack reads the posting.

Strong match

  • B2B infrastructure / API / developer tools
  • B2B enterprise platform / workflow software
  • Internal tooling / back-office systems
  • Developers / technical users
  • Internal users (employees, ops teams)

Workable stretch

  • Creator / builder / self-serve tool
  • Marketplace / multi-sided platform
  • Data platform / ML pipeline
  • Enterprise users (complex workflows, admin panels, role-based access)
  • SMB / small business (self-serve, simplicity-focused)

Harder transition

  • B2C consumer-facing application
  • Embedded systems / hardware-adjacent
  • Consumer / general public (high polish, conversion-focused)
  • Regulated environment (healthcare, finance, government)

Industry: Software, SaaS & AI Platform. See your personal match score

What they're really screening for

Must-haves

  • PLUS Bachelor's or Master's degree in Computer Science, Computer Engineering, Electrical Engineering, or a related technical field.
  • CORE Strong hands-on programming experience in C++, Go, Python, Rust, or similar systems programming languages.
  • CORE Proven experience delivering complex infrastructure projects and contributing to the design and implementation of distributed systems.
  • CORE Experience building distributed systems, backend services, telemetry pipelines, or observability platforms.
  • CORE Hands-on experience with RDMA, RoCE, InfiniBand, or other high-performance network fabrics.
  • CORE Familiarity with libibverbs, RDMA CM, queue pairs, completion queues, memory registration, and related RDMA concepts.
  • CORE Strong knowledge of Linux networking, TCP/IP, DNS, HTTP, routing, MTU, congestion control, packet loss, latency, and performance tuning.
  • PLUS Experience with traceroute-style diagnostics, path discovery, network reachability checks, synthetic probes, or active network measurements.
  • CORE Experience with monitoring and visualization platforms such as Prometheus, Grafana, Datadog, Splunk, OpenTelemetry, or similar tools.
  • CORE Strong debugging skills across software, operating system, server, and network layers.

Nice-to-haves

  • PLUS Experience supporting AI/ML, HPC, storage, or GPU cluster infrastructure workloads.
  • CORE Experience with large-scale RoCE or InfiniBand deployments.
  • PLUS Experience with NCCL, distributed training infrastructure, or AI cluster diagnostics.
  • PLUS Experience with eBPF, XDP, DPDK, perf, tcpdump, Wireshark, ethtool, iproute2, rdma-core, or Linux kernel networking tools.
  • PLUS Experience with cloud infrastructure on AWS, GCP, or Azure.
  • PLUS Experience with Kubernetes, service discovery, configuration management, and infrastructure automation.
  • PLUS Knowledge of security, compliance, and infrastructure best practices.
  • PLUS Experience designing time-series data systems, alerting pipelines, or high-cardinality telemetry platforms.

Résumé keywords for this role

Terms the posting stresses. Include the ones you can honestly back up.

Tools & technologies

  • C++
  • Go
  • Python
  • Rust
  • RDMA
  • RoCE
  • InfiniBand
  • TCP/IP
  • libibverbs
  • RDMA CM
  • queue pairs
  • completion queues
  • memory registration
  • DNS
  • HTTP

Concepts & phrases

  • Senior Software Engineer
  • Network Observability
  • Software Driven Fabrics
  • GPU cluster utilization
  • distributed computing
  • AI workloads
  • network monitoring
  • observability platform
  • RDMA, RoCE, InfiniBand
  • TCP/IP networks
  • distributed systems
  • Linux networking
  • modern observability stacks
  • Grafana/Prometheus
  • backend telemetry services
Check my résumé against these keywords

Hiring at Clockwork.io

Open senior roles RoleStack tracks right now, posted in the last 30 days.

Full job description from Clockwork.io

From Clockwork.io's posting. View the original

About Clockwork Systems

Clockwork.io – Software Driven Fabrics to increase GPU cluster utilization

Clockwork Systems was founded by Stanford researchers and veteran systems engineers who share a vision for redefining the foundations of distributed computing. As AI workloads grow increasingly complex, traditional infrastructure struggles to meet the demands of performance, reliability, and precise coordination. Clockwork is pioneering a software-driven approach to AI fabrics by delivering cross-stack observability to catch and quickly resolve problems, workload fault tolerance to keep jobs running through failures, and performance acceleration that dynamically routes and paces traffic to avoid congestion.

Show full description Show less

To learn more, visit www.clockwork.io.

About the Role

We are seeking an experienced Senior Software Engineer to contribute to the architecture, development, and scaling of a high-performance network monitoring and observability platform. This role will focus on building systems that provide deep visibility into RDMA, RoCE, InfiniBand, and TCP/IP networks. The ideal candidate has strong experience in distributed systems, Linux networking, and modern observability stacks (e.g., Grafana/Prometheus).

What You Will Do

• Design, develop, and scale high-performance network monitoring platforms for RDMA, RoCE, InfiniBand, and TCP/IP infrastructure.

• Build backend telemetry services, observability dashboards, alerts, diagnostics, anomaly detection, SLA monitoring, and traffic analysis workflows.

• Troubleshoot complex production issues across application, OS, server, RDMA, and network layers while optimizing low-latency collection, aggregation, and alerting.

• Collaborate with cross-functional teams to deliver scalable solutions, improve engineering practices, automate operational workflows, and contribute to the technical direction of the platform.

What We’re Looking For

• Bachelor's or Master's degree in Computer Science, Computer Engineering, Electrical Engineering, or a related technical field.

• Strong hands-on programming experience in C++, Go, Python, Rust, or similar systems programming languages.

• Proven experience delivering complex infrastructure projects and contributing to the design and implementation of distributed systems.

• Experience building distributed systems, backend services, telemetry pipelines, or observability platforms.

• Hands-on experience with RDMA, RoCE, InfiniBand, or other high-performance network fabrics.

• Familiarity with libibverbs, RDMA verbs, RDMA CM, queue pairs, completion queues, memory registration, and related RDMA concepts.

• Strong knowledge of Linux networking, TCP/IP, DNS, HTTP, routing, MTU, congestion control, packet loss, latency, and performance tuning.

• Experience with traceroute-style diagnostics, path discovery, network reachability checks, synthetic probes, or active network measurements.

• Experience with monitoring and visualization platforms such as Prometheus, Grafana, Datadog, Splunk, OpenTelemetry, or similar tools.

• Strong debugging skills across software, operating system, server, and network layers.

• Experience operating production systems in Linux-based environments.

• Strong technical judgment and ability to build reliable, scalable, and maintainable systems.

Nice to Have

• Experience supporting AI/ML, HPC, storage, or GPU cluster infrastructure workloads.

• Experience with large-scale RoCE or InfiniBand deployments.

• Experience with NCCL, distributed training infrastructure, or AI cluster diagnostics.

• Experience with eBPF, XDP, DPDK, perf, tcpdump, Wireshark, ethtool, iproute2, rdma-core, or Linux kernel networking tools.

• Experience with cloud infrastructure on AWS, GCP, or Azure.

• Experience with Kubernetes, service discovery, configuration management, and infrastructure automation.

• Knowledge of security, compliance, and infrastructure best practices.

• Experience designing time-series data systems, alerting pipelines, or high-cardinality telemetry platforms.

Enjoy

• Challenging projects.

• A friendly and inclusive workplace culture.

• Competitive compensation.

• A great benefits package.

• Catered lunch.

Compensation for this position will vary based on the skills and experience you bring, as well as internal equity considerations. For candidates hired at the posted level, the expected base salary range is $140,000 - $210,000. The offered compensation package may also include stock options or other equity awards, subject to Clockwork’s equity program and applicable approvals.
In addition to cash compensation, this role is eligible to participate in the company’s equity program, which may include stock options granted in accordance with the company’s equity plan and subject to approval and applicable vesting schedules.

Clockwork Systems is an equal opportunity employer. We are committed to building world-class teams by welcoming bright, passionate individuals from all backgrounds. All qualified applicants will receive consideration for employment without regard to race, color, ancestry, religion, age, sex, sexual orientation, gender identity or expression, national origin, disability, or protected veteran status. We believe diversity drives innovation, and we grow stronger together.

FAQ

How much does a Senior Software Engineer – Network Observability at Clockwork.io make?

The posted pay for this role is $140k–$210k a year.

Does Clockwork.io sponsor visas for this role?

The posting does not say whether visa sponsorship is available. Ask the recruiter before you apply.

Is this role remote?

It is an on-site role.

What tools and technologies does the posting mention?

C++, Go, Python, Rust, RDMA, RoCE, InfiniBand, TCP/IP, libibverbs, RDMA CM, queue pairs, completion queues, memory registration, DNS, HTTP.