Cheap Failures: Checkpointing and Deterministic Resumption for GPU Clusters

A long job on GPU clusters is cheap to fail only when two layers work together. The scheduler puts the failed job back in the queue, and the application resumes from its latest checkpoint. Slurm does the first half without extra configuration. After a node failure it marks the node DOWN, ends the job in the NODE_FAIL state and, under the default JobRequeue setting, restarts the batch script from the beginning under the same job ID. Everything since submission is lost unless the job writes checkpoints and loads the latest one.

TechnoLynx starts by inventorying the current job scripts, the scheduler policies and the recompute that past failures actually cost. TechnoLynx then chooses the checkpoint interval and the Slurm mechanisms, builds the checkpoint-aware wrapper, and fault-injects node failures to measure what the setup saves against what it adds, as part of its GPU performance engineering work.

Why `--requeue` alone still restarts a week-long run from zero

Requeue restarts a job; it does not recover its progress. CoreWeave’s Slurm documentation is blunt about this: without a checkpoint to resume from, a requeued job loses all progress made before the failure. The cluster-wide JobRequeue parameter defaults to 1, which makes batch jobs requeueable unless the user opts out, and sbatch --requeue or --no-requeue overrides that per job. Setting --requeue explicitly is still worth doing, because it makes the job’s behaviour independent of whatever the administrator chose.

Node-failure requeue also has a gap that catches people out. A job that dies because the application exited with an error, such as an out-of-memory error or an NCCL timeout, counts as a normal failure and is not requeued even with --requeue set. To cover those cases, RequeueExit in the Slurm configuration lists recoverable exit codes, and Slurm sets SLURM_RESTART_COUNT on every restarted run. Slurm matches the exit code of the batch script, not of each task, and a job that exits with a listed code every time requeues every time. The documented guard is to check SLURM_RESTART_COUNT at the top of the script and exit with an unlisted code once it passes a limit.

The job itself has to do the resuming. Four details decide whether it works:

A stable checkpoint path. Key it on $SLURM_JOB_ID or a fixed “latest” location, never on wall-clock time: a path built from the date points somewhere new after midnight, and a job restarting across the day boundary cannot find its own checkpoint. The job ID is safe because requeue keeps it.
Storage every node can reach. A requeued job can land on different nodes, so checkpoints on node-local disk are gone.
Logs that survive. Slurm’s default open mode truncates output files, so without --open-mode=append a requeue erases the logs from before the failure.
No hung ranks. In a distributed job, one rank can exit while the others wait on a collective that never completes, holding GPUs until a timeout fires; srun --kill-on-bad-exit=1 ends the whole step as soon as any task exits non-zero.

A minimal GROMACS wrapper looks like this:

#!/bin/bash
#SBATCH --requeue
#SBATCH --open-mode=append
#SBATCH --signal=TERM@300

MAX_RESTARTS=5
if [ "${SLURM_RESTART_COUNT:-0}" -gt "$MAX_RESTARTS" ]; then
  echo "restart limit reached" >&2
  exit 3          # deliberately not in RequeueExit
fi

RUNDIR=/shared/runs/${SLURM_JOB_ID}
mkdir -p "$RUNDIR" && cd "$RUNDIR"

if [ -f state.cpt ]; then
  srun --kill-on-bad-exit=1 gmx_mpi mdrun -s topol.tpr -cpi state.cpt
else
  srun --kill-on-bad-exit=1 gmx_mpi mdrun -s topol.tpr
fi

Because srun is the last command, the script exits with the srun exit code, which is what RequeueExit sees. The --signal line has no B: prefix, so the warning goes to the job steps (the mdrun processes) rather than only to the batch shell. Afterwards, sacct -j <id> --duplicates shows every run of the job, including the one that ended in NODE_FAIL.

Which Slurm features support recovery from a partial failure

The table sets out what each one does and when it earns its place.

Mechanism	What it does	What it does not do	Use it when
`JobRequeue=1` / `--requeue`	Returns a batch job to the queue after node failure, admin requeue or preemption, restarting the script under the same job ID	Resume from a checkpoint; requeue on application errors	Always, for any job longer than you are willing to lose
`RequeueExit`	Requeues batch jobs that exit with listed codes	Tell a recoverable failure from a bug: you choose the codes	Recoverable application errors, such as an NCCL timeout, exit with codes you can tell apart
`PreemptMode=REQUEUE`	Preempted jobs are requeued if `--requeue` or `JobRequeue=1` applies, otherwise cancelled	Give any warning unless GraceTime is set	Shared partitions where low-priority work can restart elsewhere
`PreemptMode=SUSPEND,GANG`	Suspends preempted jobs and the Gang scheduler resumes them later	Release the memory suspended jobs still hold on the node	Slurm tracks memory and GANG is set at cluster level
`GraceTime`, `KillWait`, `--signal`	Delay between SIGTERM and SIGKILL, plus an optional advance signal before the time limit	Save anything: your handler must	Any job that writes a checkpoint on demand
`--dependency` chaining	Starts the next segment only after the previous one ends	Resume the program: the script must load the last checkpoint	Runs longer than the job’s time limit
Job arrays	Many tasks under one ArrayJobId, each with its own JobId; `scontrol requeue` works on single elements	Keep one JobId across tasks	Repetitive jobs that follow a common pattern
`PreemptExemptTime`	Minimum run time before a job can be preempted	Protect a job after that window	Jobs that must run a minimum time before they can be preempted

Preemption gives you seconds, not minutes, by default

When Slurm stops a job it sends SIGCONT, then SIGTERM, waits KillWait seconds, then sends SIGKILL, which cannot be caught. For preemption, the partition or QOS GraceTime comes first, and it defaults to zero. On CoreWeave’s SUNK, KillWait defaults to 30 seconds, roughly a handler’s budget to save and exit on cancellation or a time limit. If one checkpoint write takes longer than that, --signal=B:SIGTERM@120 asks for the signal 120 seconds before the job’s time limit, and it can arrive up to 60 seconds earlier, so treat the lead as a minimum.

The handler itself should not write the checkpoint mid-operation. It should set a flag that the training or simulation loop checks between steps, so the state is saved at a consistent point. The periodic interval is the second line of defence: it should be short enough that a SIGKILL landing before the handler finishes never costs more work than you accept. The warning window has to fit one write; the periodic interval bounds the loss when a SIGKILL lands first. We would size those two separately, because they protect against different events.

Chaining segments with dependencies and arrays

A run longer than any allocation can be split into segments that each resume from the last checkpoint. The GWDG HPC documentation shows the pattern: the script traps an advance signal sent five minutes before the time limit (--signal=B:10@300), saves a checkpoint, exits with an error code, and a resubmitted job with --dependency picks up from there until a finished marker file breaks the loop. The same documentation restarts GROMACS with gmx_mpi mdrun -s input.tpr -cpi checkpoint.cpt.

Job arrays need one adjustment. Every task shares an ArrayJobId but receives its own JobId when it starts. A checkpoint path keyed on $SLURM_JOB_ID therefore differs between array tasks, so chained tasks should key on SLURM_ARRAY_JOB_ID instead. Limiting the array to one running task with the % separator (--array=0-9%1) runs the segments one at a time.

Multi-node MPI jobs need more than requeue

A consistent checkpoint across MPI ranks has rules of its own. The MANA work on transparent MPI checkpointing states the core invariant: no checkpoint may be taken while a rank is inside a collective communication call. If a collective has started when the request arrives, the checkpoint must wait until every member of that communicator can complete it. Point-to-point traffic has its own problem: DMTCP’s proxy approach drains all in-flight messages and compares per-rank send and receive counts before it writes anything.

That is why we would choose an application’s own checkpoint first when it exists. The application knows where its safe points are, and a requeue wrapper only has to find the file. MANA’s worst case for GROMACS on 16 MPI ranks was 2.1%, cut to 0.6% with a patched Linux kernel, and its highest observed overhead was 4.5% for 512 ranks over 16 nodes. In those tests, total checkpoint time was dominated by writing to storage, not by coordinating the ranks.

Storage is where multi-node checkpointing strains a shared cluster. Asynchronous flushes compete with the application for CPU, memory and network bandwidth. File-per-process strategies can overwhelm a parallel file system with metadata load when many files are opened at once. On restore, CoreWeave recommends having rank 0 read the checkpoint and broadcast it, because every rank reading the same object at once collapses read throughput.

Checkpoint/restart in GROMACS and RELION under Slurm

GROMACS already has checkpointing built in. When gmx mdrun is halted it writes a checkpoint that restarts the simulation exactly as if there had been no interruption, holding full-precision positions and velocities plus the state that algorithms such as thermostat coupling need. It also writes checkpoints periodically during the run, and before each write it renames the previous file with a _prev suffix, so a failure mid-write loses only recent progress. Restarting is gmx mdrun -cpi state.cpt. The GWDG documentation points to GROMACS and OpenFOAM as programs whose built-in checkpoint options only need switching on and pairing with a resubmitting script.

For RELION, before assuming a refinement or classification job can resume mid-run, check RELION’s own documentation for which job types can continue from their last iteration, and test it under the wrapper above.

Sizing the checkpoint interval from your own failure data

Every interval trades two costs: the time and I/O each write takes, and the progress lost when a node fails between writes. Checkpoint often and the run slows; checkpoint rarely and each failure costs more recompute. At scale the second cost is not abstract: during LLaMA3-405B training on 16K GPUs, checkpointing every 30 minutes meant a single failure repeated 4,096 GPU-hours of work on average, across 419 reported failures.

Worked procedure for choosing an interval

Time one real checkpoint write for the actual job, on the actual storage. For large models trained with Adam in mixed precision, expect the checkpoint to be roughly 14 times the parameter count in bytes.
Measure your failure rate from your own cluster. sacct --duplicates lists every NODE_FAIL run of a requeued job, and those records across jobs are your cluster’s own failure history. Published data, such as Meta’s analysis of 11 months, 4 million jobs and over 150 million A100 GPU hours across two research clusters, fits a failure model projecting mean time to failure across GPU scales, but the rate is not yours.
Decide how much recompute you can tolerate per failure. With interval T, a failure lands on average halfway through it, so expect to lose about T/2 of work plus restart time; Checkmate’s per-iteration scheme correspondingly incurs half an iteration of repeated work per failure.
Check the write cost fraction. Write time divided by T is the steady overhead. FastPersist’s authors cite 10% of training time as a widely accepted overhead; on a shared file system, whose aggregate bandwidth is limited, keep the write cost a small fraction of the work it protects.
Set the interval and validate it. Set the interval so the write cost stays a small fraction of the work it protects, then validate with induced failures.

If the write cost forces an uncomfortably long interval, we would make the write cheaper rather than accept the loss. PyTorch’s distributed asynchronous checkpointing cut the visible downtime for a 7B model from an average of 148.8 seconds to 6.3 seconds, by copying state from GPU to CPU and persisting it to disk on background threads. IBM and PyTorch avoided a collective hang between training and checkpoint threads by giving asynchronous checkpointing its own process group.

Making a GPU training job resumable so preemption does not cost an epoch

A complete checkpoint holds model parameters, optimizer state, the data-loading iterator position and the learning-rate schedule. torchrun’s snapshot pattern follows the same logic: on failure it restarts all processes, each loads the last snapshot (which can include epochs run and optimizer state), and only the progress since that snapshot is lost.

The data-loading iterator state, not just the epoch counter, therefore belongs in the checkpoint. Pair it with the SIGTERM flag from the preemption section and a periodic interval sized as above, so a forced kill never costs more than you accept. A non-preemptible window is the other option: PreemptExemptTime guarantees a minimum run time before preemption, and a partition can set PreemptMode=OFF. We would reserve that for the stretch where a resume genuinely cannot work, because it pushes the cost of preemption onto other users of the cluster.

Can checkpoint/resume be added to CUDA Monte Carlo pricing kernels?

A CUDA Monte Carlo pricing job can be made resumable after node loss by saving its generator state between kernel launches. The application-level route is the simplest. cuRAND’s device API generates the same sequence from the same state, and the same seed always produces the same initial state. For experiments spanning several kernel launches, NVIDIA recommends giving threads the same seed and monotonically increasing ids, and preserving generator state in global memory between launches when the thread configuration stays the same. Saving and restoring that state is much faster than recomputing it.

Kernel launch boundaries give a natural checkpoint point. Split the simulation into batches of paths, one or a few launches each, and after each batch copy the per-thread curandState array, the running payoff accumulators and the batch counter to host memory and persist them. On resume, the job reloads the state with the same launch configuration and continues with the next batch. A single long kernel has no safe point, so batch size is effectively your checkpoint interval.

Transparent tools exist where changing the kernels is not an option. NVIDIA’s cuda-checkpoint locks CUDA driver APIs, completes submitted work, copies device memory to the host and releases the GPU, after which CRIU can checkpoint the process like a CPU workload. Its limits matter for pricing engines: x64 only, a single process rather than a process tree, no UVM or IPC memory, no GPU migration, and it waits for already-submitted work before completing. The research system CRAC reported generally about 1% runtime overhead on its benchmarks, and between 1% and 14% on short-running ones, but its restart relies on using the same CUDA and GPU platform.

Each checkpoint write costs the job time and I/O. The extra I/O and kernel complexity have to fit within the batch window, or the resilience measure becomes the reason the report is late.

Failover for a Slurm and Kubernetes hybrid GPU cluster with hard deadlines

The pattern we would start from gives each job exactly one owner of retries. Slurm requeue retries Slurm batch jobs, and the Job controller retries Kubernetes Jobs. Two layers retrying the same failure independently is how a hybrid cluster ends up with duplicate runs or a job nobody restarts. CoreWeave’s SUNK shows a layered version that works: when a fault is confirmed, Kubernetes evicts the node’s Pods, including the slurmd Pod, the Slurm job fails, and Slurm requeue brings it back on healthy nodes. When severity is uncertain, the node is cordoned and drained and the running job is allowed to finish.

Kubernetes retries add back-off delay of their own. A Job’s backoffLimit defaults to 6 retries, and failed Pods are recreated with exponential back-off (10s, 20s, 40s and so on) capped at ten minutes. Kubernetes can also distinguish retriable from non-retriable Pod failures, so causes such as exceeding memory or ephemeral-storage limits, or a graceful node shutdown, can be handled differently. Map those causes deliberately: a node shutdown should retry from the checkpoint, an out-of-memory loop should fail fast.

For overnight risk runs, the testing discipline matters: kill a node at a known point and measure recovery, as the SWIFT authors did when they killed a machine at iteration 150 after a checkpoint at iteration 100. Do it under full overnight load, not on an idle cluster, because that is the load failover must survive. Whether the repriced batch still fits its window after a failure is a capacity and scheduling question as much as a resilience one, and it needs sizing on its own terms.

Checking that a resumed job gives the same answer as one that never stopped

A resumed run is bitwise identical to an uninterrupted one only under narrow conditions, and on GPUs the default is that it is not. NVIDIA defines a computation as deterministic when repeated runs with the same input produce the same bitwise result, which floating-point non-associativity makes hard in parallel code. GROMACS states the boundary precisely: continuation from a checkpoint gives binary-identical results with the same code, compiler, architecture and processor count without GPUs, while on GPUs the non-bonded force reduction has a non-deterministic summation order by design. Its -reprod flag removes the sources of non-reproducibility it can, given the same executable, hardware, shared libraries, input and command line.

Deep learning stacks have the same shape. cuDNN convolution algorithms and fused attention backward passes are non-deterministic by default because they use atomic accumulation or auto-tuned algorithm selection, and NVIDIA’s TAO documentation states that reproducibility holds only on the same GPU architecture, library versions and GPU count. NVIDIA CCCL offers a run_to_run level for identical results on the same GPU and a gpu_to_gpu level that holds across GPUs, at 20% to 30% extra execution time on an H200 for large problems. That trade-off matters after a requeue onto a different GPU model.

Equivalence is testable, and it works as a regression test. Regression checks of this kind belong to production reliability engineering, which is distinct from model accuracy. Checkmate’s authors ran it in its simplest form: the loss curves of an interrupted and an uninterrupted ResNet152 run overlapped completely.

Resume-equivalence checklist

Pin the environment first: GPU architecture, library versions and GPU count, since identical results are expected only when these match. Pinning the compute stack for reproducible GPU environments covers how to hold those fixed across requeues.
Constrain the requeue to the same GPU model and count, or accept that bitwise equality is off the table.
Enable deterministic paths for the test run (-reprod for GROMACS, deterministic kernel flags or CCCL determinism levels elsewhere), and budget for them being slower.
Run once uninterrupted and once with a forced kill and resume, then compare the two runs at matched steps, as Checkmate’s authors did with loss curves.
For Monte Carlo, confirm that the restored generator state continues the sequence: same seed, same thread layout, same batch boundary.
Note that replay-based recovery schemes depend on this property, since logged replay assumes the same input produces the same output.

For a pricing run or a long MD trajectory on GPUs, equivalence within a stated bound is usually the realistic target, because fast GPU reductions are non-reproducible by design.

FAQ

What is a checkpoint used for in HPC?

In HPC, checkpoints let long batch jobs outlive a single allocation, recover from faults, fit backfill scheduling and migrate between machines. They also allow saved computational state to be analysed or cloned.