NVIDIA
NVIDIA
NVIDIA develops accelerated computing and artificial intelligence technologies used across gaming, data centers, cloud computing, healthcare, manufacturing, automotive, and robotics. Its work spans graphics processing units, AI platforms, simulation, and industry-focused solutions, including NVIDIA Omniverse for collaborative 3D workflows, NVIDIA DRIVE for autonomous vehicle development, and NVIDIA Clara for healthcare applications. The company brings together large technical teams working on the infrastructure and software that support advanced computing, simulation, and AI-driven analytics.

Senior Cloud Infrastructure and Reliability Solutions Architect

Lead Day 2 reliability for NVIDIA Cloud Partner GPU fleets across EMEA. Automate fault recovery, observability, maintenance, and Kubernetes operations at scale.

Description

  • Lead production stability and reliability across NVIDIA Cloud Partner fleets in EMEA.
  • Set and improve reliability and workload-throughput measures for large GPU clusters, and manage platform error budgets.
  • Build automated detection and recovery workflows for GPU, node, network fabric, and storage failures.
  • Automate fault triage, node draining and cordoning, resets, and return-to-service procedures.
  • Plan and deliver preventive maintenance and firmware or field-notice rollouts, from risk reviews and staged deployments through verification.
  • Integrate fleet observability and reliability tools—including Prometheus, Grafana, Loki, DCGM, XID diagnostics, GPUd, and NVSentinel—with partner systems.
  • Coordinate incident response and post-incident reviews with NVEX, technical account managers, and partner operations teams.
  • Lead root-cause investigations, close corrective actions, and turn recurring incidents into automation, health checks, or runbooks.
  • Manage Kubernetes node lifecycles and reliability across Kubernetes, KubeVirt, Slurm, and GPU-aware schedulers.
  • Set node health policies and coordinate upgrades, reboots, and safe workload draining on GPU-accelerated nodes.

Requirements

  • Bachelor’s, master’s, or doctoral degree in computer science, electrical or computer engineering, physics, mathematics, or a related field, or equivalent experience.
  • At least 8 years in site reliability engineering, production engineering, or large-scale infrastructure operations.
  • Demonstrated responsibility for production reliability at scale, including SLOs and error budgets, on-call and incident command, blameless reviews, and measurable gains in availability or mean time between interruptions.
  • Hands-on experience operating HPC or AI clusters and NVIDIA GPU infrastructure, including driver and CUDA toolkit lifecycles, firmware, workload profiling, and troubleshooting CPUs, GPUs, and high-speed interconnects.
  • Extensive experience operating Kubernetes in GPU and HPC environments, including node lifecycles, upgrades and draining, scheduler behavior, Slurm, and mixed bare-metal and virtualized multi-tenant systems.
  • Proficiency in Python and Bash, infrastructure as code and configuration management such as Ansible and Terraform, GitOps-based fleet lifecycle and upgrades, and Prometheus, Grafana, and Loki.
  • Strong Linux expertise, including Red Hat and Ubuntu, operating-system tuning and security, data-center networking fundamentals, and parallel or enterprise storage such as Lustre, GPFS, ZFS, and XFS.
  • Experience planning and executing proactive firmware, microcode, or field-notice rollouts across a large production fleet.
  • Familiarity with DCGM, XID diagnostics, GPUd, NVSentinel, node health agents, and fleet-wide reliability intelligence.
  • Experience with NVIDIA GPU and Network Operators and NVIDIA Base Command Manager.
  • Practical experience with AI scheduling and inference frameworks on Kubernetes, such as KAI, Grove, Dynamo, and NVIDIA Cloud Functions.
  • Expertise in RDMA fabrics such as InfiniBand or RoCE; experience with Cumulus Linux, SONiC, Spectrum-X, DPU/DOCA, and NVLink/NVSwitch partition operations is a plus.

Related Jobs