To rerun a GPU cluster job months later, we would pin and record these things. Pin the container image by its sha256 digest. Record the host driver, firmware and library versions next to it. Store that record with every run’s results. Both the image and the host matter because the driver lives on the host, while the CUDA runtime and libraries usually live in the image. So a tag refresh or a host driver update can change what a job runs on even when nobody touches the code.
A TechnoLynx engagement sets the reproducibility and audit requirements, chooses the Apptainer and digest-pinning scheme, adds pre-flight checks and metadata capture to the Slurm job scripts, and tests that captured digests re-execute identically across nodes. It starts by defining the reproducibility and audit requirements for the cluster’s jobs.
The image pins CUDA, but the host still owns the driver
A container image bundles the user-space environment, with one exception. The user-mode driver, libcuda.so, is not in the image. The NVIDIA Container Toolkit makes the host’s copy available when the container starts. Apptainer’s --nv flag does the same job: it finds the basic CUDA libraries on the host and binds them into the container so they match the host’s kernel driver. It finds them through the list in nvliblist.conf and the ldconfig cache.
This has a practical consequence that catches people out. When nvidia-smi reports a “CUDA Version” inside a container, that number is the newest CUDA version the host driver supports. It is not the version of any toolkit you installed.
A 2019 paper on HPC container portability (CANOPIE-HPC workshop at SC19) describes pulling host libraries into containers as brittle and a threat to long-term reproducibility. It also notes that accelerator libraries may be tied to driver and firmware versions specific to one facility. libc matters too. Apptainer’s documentation warns that host and container libc versions must not be too far apart. Errors appear when the container’s libc is older than the host’s, and sometimes when it is much newer. An unchanged image can therefore stop working after a host OS upgrade.
So keep two inventories and join them. An image scan cannot see the host layers, and a host package query cannot see what is inside the image.
Pinning a CUDA or ROCm image by digest instead of a tag
A tag is a moving pointer. NVIDIA updates its supported CUDA image tags to the latest CUDA, cuDNN and TensorRT versions, and refreshes them from time to time to fix CVEs. The same tag can point to different content on different days. NVIDIA’s CUDA container support policy also deletes tags. A tag goes six months after the last supported Tesla Recommended Driver reaches end of life, or once a newer update release ships for the same CUDA version. The fix is to reference the base image as nvidia/cuda:<tag>@sha256:<digest>, using a digest you have checked yourself.
Digest pinning has a cost. You no longer get NVIDIA’s CVE refreshes automatically, so you need a job that looks for a newer digest behind your tag, rebuilds, tests and opens a change. That is the right trade for reproducible work, because you decide when content changes.
Keeping the recipe instead of the built image does not reproduce the image exactly. A 2024 survey of reproducibility in HPC notes that two people building from the same Dockerfile can end up with slightly different images. NVIDIA’s HPC SDK Container Guide (release 26.3, March 2026) makes a related point about content fetched from repositories such as GitHub: name a tag or commit so the image build is reproducible. Keep the built artefact, addressed by digest, and keep its recipe with commits pinned. You need both, because the recipe tells an auditor what the image contains and the digest proves it is the same image.
A signed SIF image can be verified byte for byte against what the signer produced. A SIF file is a single portable image. The signature is carried inside the SIF file, and apptainer verify checks the image on disk against it. Administrators can also use Apptainer’s Execution Control List to allow only containers with approved signatures to run. For each run, record the SIF file’s sha256 hash and the signing fingerprint.
AMD’s ROCm stack has the same split between host and image. On the host side, Apptainer’s --rocm flag binds the host’s ROCm libraries so they match the kernel GPU driver. AMD’s compatibility window is different from NVIDIA’s. From ROCm 6.4.0, AMD supports forward and backward compatibility between the amdgpu driver and its user-space software up to a year apart. Earlier releases are supported only within two releases either way.
Will Apptainer containers break Slurm GPU scheduling?
Slurm makes its GPU allocation decisions outside the container, and the container runs inside them. Slurm’s GRES plugin sets CUDA_VISIBLE_DEVICES for each job step so that jobs sharing a node get separate GPUs. Apptainer passes the host environment into the container by default, so the containerised process sees the same allocation. If Slurm uses Linux cgroups to constrain device files, the job sees only the devices visible inside its cgroup.
Three details keep this reliable:
- Device numbering. NVML numbers GPUs by PCI bus ID, so set
CUDA_DEVICE_ORDERto make CUDA’s numbering match. The mapping between device files and NVML numbers can change between boots after hardware or OS changes, so a check after boot is needed. - Driver initialisation. Apptainer blocks privilege escalation, so the setuid binary that initialises parts of the NVIDIA stack cannot run inside the container. If you see
CUDA_ERROR_UNKNOWN, initialise the stack on the host first (for example withmodprobe nvidia_uvmas root) and usenvidia-persistencedso the driver does not unload. - Native OCI as an alternative. In Slurm 21.08 and later,
srunandsbatchaccept--container.oci.confmust be set up on the execution node or Slurm ignores the request. Slurm will not copy the bundle to that node, so it must already be there.
Apptainer or Docker on a shared HPC cluster
For a shared cluster, we consider Apptainer the better fit. Apptainer runs with user privileges, needs no daemon, supports GPUs, MPI and InfiniBand, and packages the environment as a single SIF file. The 2024 survey cites Hu et al. (2019), who found no overhead for Singularity in MPI and GPU parallel applications. From that, the survey concludes that container technologies other than Docker suit HPC and reproducible research. A 2026 set of quick tips for AI-driven HPC workflows says much the same: Apptainer and SingularityCE are common on HPC systems because they work with schedulers and do not normally need elevated privileges at runtime.
Docker still has a place in the setup we recommend. We would build images with Docker or any OCI tool in CI, push them to a registry, refer to them by digest, and turn them into signed SIF files for the cluster. Apptainer can build a SIF directly from an image in an OCI registry. There is one gap to cover. Docker and OCI images can declare runtime requirements in NVIDIA_REQUIRE_* variables, but Apptainer does not yet act on them. The requirement check therefore has to happen elsewhere, and the prolog is the natural place for it.
| Situation | Better fit | Reason |
|---|---|---|
| Shared cluster, users without root | Apptainer | Runs with user privileges, no daemon |
| Image must be checked byte for byte later | Signed SIF (Apptainer) | Signature confirms a bit-for-bit copy |
| CI builds and the registry of record | Docker or any OCI builder | Digest-addressed images, converted to SIF for the cluster |
| Cluster already configured for native OCI | Slurm --container |
Needs oci.conf and bundles already on each node |
Catching a driver mismatch before a week-long run starts
We would catch a mismatch with a Slurm prolog that compares the host’s driver with what the job’s image needs, and exits non-zero if they do not fit. When a prolog fails, Slurm drains the node and requeues the job. The job is held unless nohold_on_prolog_fail is set in SchedulerParameters.
The core rule comes from NVIDIA’s minor version compatibility guide. From CUDA 11 onwards, an application built with a toolkit from a major family can run, with a limited feature set, on a driver from the same family that meets the minimum. The minimums are 580 or later for CUDA 13.x, 525 or later for CUDA 12.x, and 450 or later for CUDA 11.x. Across the whole 11.x family, the threshold is 450.80.02 on Linux. Three caveats turn this from a simple number comparison into real checks:
- Applications that compile device code to PTX will not work on older drivers. If an image depends on PTX JIT, the driver has to be upgraded.
- Minor version compatibility requires passing the target architecture to the compiler, for example
nvcc -arch=sm_xx. - Some features need both a newer toolkit and a newer driver. On older drivers they return
cudaErrorCallRequiresNewerDriver.
Application builds add their own failure mode. RELION’s documentation warns that a newer CUDA toolkit needs a newer GPU driver. It also warns that if the build targets the wrong compute capability, compilation may succeed but the binary can fail at runtime.
The prolog still needs to know what each image requires. The 2019 CANOPIE paper suggested recording requirements such as a CUDA driver version in OCI image labels, so the runtime can read them. We would put the minimum driver and the compiled sm_ targets into labels at build time and have the prolog read them. Users would pass the image reference with the job, for example in a submission flag the prolog parses. NVIDIA’s DeepOps prologs already read user requests from --comment.
Pre-flight checklist for a GPU job prolog
- Read the driver version from
nvidia-smior/proc/driver/nvidia/version, not the package manager. A package query can report no driver on a machine that has one. - Compare the driver with the image’s CUDA family minimum, treating versions as numeric tuples. As text, “580.65.06” sorts after “580.178.04”.
- If the image ships PTX instead of machine code for this GPU, require a driver from the image toolkit’s minor version or later.
- Confirm that
lspcisees every GPU Slurm thinks the node has, as the DeepOpsprolog-lspciscript does. - Run a quick DCGM health check, which takes a few seconds in the DeepOps example and drains the node on failure.
- If jobs load
cuda-compatforward compatibility libraries, check that they are still valid. Older compatibility packages are not supported on new drivers, so load scripts should query the system first.
Inside the container, a second check at job start closes the loop. One 2026 research-computing deployment runs an automated check every time a workspace starts. It records whether nvidia-smi reports the expected driver, whether the CUDA runtime matches the image tag, and whether the main ML framework imports without error.
Two Slurm details affect where the prolog runs. By default, the prolog runs on a node only when that node sees the first job step from an allocation, so a node that never runs a step never runs the prolog. PrologFlags=Alloc forces it to run at allocation time. Prologs also run outside the job’s cgroup. A GPU that a prolog sees as CUDA_VISIBLE_DEVICES=1 can appear to the job as 0. Finally, keep compatibility and security apart: a driver that passes the compatibility check can still be unpatched.
Recording what each GROMACS or RELION job actually ran on
To record the exact stack, capture the host side in the prolog and the image side from inside the job. Then write both under the Slurm job ID. The host commands below follow an inventory published by Safeguard. The image-side command lists the CUDA libraries the image carries, following the same source’s image checks.
# Host side (prolog)
nvidia-smi -q | grep -E "Driver Version|CUDA Version|VBIOS Version|GSP Firmware Version|Inforom Version"
cat /proc/driver/nvidia/version
nvidia-ctk --version
# Image side (inside the job)
sha256sum "$IMAGE_SIF"
apptainer exec --nv "$IMAGE_SIF" bash -lc 'cat /usr/local/cuda*/version.json; ldconfig -p | grep -E "libcudart|libcublas|libcudnn|libnccl|libcufft"'
For PyTorch-based steps, also ask the framework which CUDA, cuDNN and NCCL versions it loaded. If they differ from the image’s OS packages, those OS copies are dead weight. For GROMACS or RELION, record the application’s version and build output, and the -arch targets it was compiled for. If the container toolkit is installed, record whether it runs in legacy hook mode or CDI mode as well as its version, because the mode changes its security exposure. If nvidia-smi reports a driver but no package exists, note that the driver came from a runfile or a driver container. The upgrade procedure for those is different.
The epilog can add what the job did on the GPU. NVIDIA describes a workflow in which the prolog starts DCGM statistics recording labelled with the Slurm job ID. The epilog then stops recording and writes a per-GPU report covering power, memory use, clocks and health.
How much per-run metadata does an audit need?
The minimum set has to answer two questions: which bytes ran, and on which host stack. Library versions, container image identifiers, code commits and random seeds should be captured automatically during the run, not kept by hand. Here is a set we consider enough for both a rerun and an audit:
| Field | Source | Why it is needed |
|---|---|---|
| Slurm job ID, node, allocated GPUs | Slurm environment | Slurm sets CUDA_VISIBLE_DEVICES per job step |
| Driver version | nvidia-smi / /proc/driver/nvidia/version |
libcuda.so comes from the host, not the image |
| GSP firmware and VBIOS versions | nvidia-smi -q |
GSP firmware changes with each driver; VBIOS changes only when someone flashes the board |
| Container toolkit version and mode | nvidia-ctk |
Mode is configuration that a package list will not show |
| Image digest or SIF sha256 plus signature fingerprint | Registry / sha256sum / apptainer verify |
Tags move; digests do not |
| CUDA runtime and loaded library versions | Inside the container | What the application actually linked |
| Application commit and compile targets | Build record | Wrong targets can fail at runtime |
| Inputs, parameters, seeds | Captured with the run | Same inputs and conditions define reproducibility |
The Slurm job comment is one practical place to store this. The comment field accepts any text, including serialised JSON. With AccountingStoreFlags=job_comment in slurm.conf, Slurm saves it in the SlurmDBD accounting database. It then survives after the job ends and can be retrieved with sacct. If your prologs already read user requests from --comment (as the DeepOps examples do), write the metadata from the epilog or into a separate file so the two uses do not collide. Keeping a copy of the JSON beside the job’s outputs also means the record goes wherever the results go.
Retention periods are set by funders and institutions, and our sources cover them only thinly. A US research-administration guide from COGR states that Circular A-110 requires records, including scientific data, to be kept for three years. If litigation starts within that period, they must be kept until any claim or audit is resolved. The same guide says best practice is to keep data that supports a patent application for much longer. Other funders and other countries set their own rules, so take the retention period and format from your grants office, not from a general figure.
What a pinned stack still will not guarantee
Pinning does not guarantee bitwise identical GPU results. The 2019 CANOPIE paper says plainly that bit-wise reproducibility is unlikely, while presenting containers as the foundation for traceability. The 2024 survey adds that in HPC, repeatability often costs performance, because it means turning off GPU performance optimisations that look unreliable. Whether two runs agree within an acceptable numerical tolerance is a separate validation question with its own acceptance protocol. A pinned environment is what makes that question answerable.
Pre-flight checks have a similar limit: they only catch the mismatch types you wrote them for. In 2019 there was no simple way to detect ABI or glibc mismatches before execution. A prolog that checks the driver family and PTX dependence will still miss a failure nobody has seen yet. For week-long jobs, pair pinning with checkpointing and deterministic resumption for long-running GPU jobs, so a failure the prolog missed costs hours rather than the whole run.
The payoff shows up the first time a job that worked last spring now gives different results. The Safeguard guide recommends regenerating the inventory after every stack change and diffing it against the previous one. With per-run records in place, the same habit turns “what changed?” into a comparison of two JSON records: driver branch, image digest, loaded cuDNN version. Without the records, it is an investigation that may never reach an answer.
