A common confusion with concrete consequences A team installs PyTorch, runs torch.version.cuda, and sees a CUDA version that doesn’t match what nvcc --version reports for the system-installed CUDA toolkit. The instinct is that something is misconfigured. Usually it isn’t — the two numbers describe different things, and the discrepancy is the expected behavior of how PyTorch ships CUDA support. The trouble starts when a benchmark is reported as “PyTorch X.Y on GPU Z” and that under-specifies the AI Executor in ways that prevent the result from reproducing on a different host. Understanding the relationship between torch.version.cuda, the system CUDA toolkit, the installed driver, and the GPU’s compute capability is the prerequisite for reading or producing reproducible PyTorch benchmark results. What does torch.version.cuda actually report? PyTorch wheels ship with pre-built CUDA runtime libraries inside the wheel itself. When PyTorch is installed via pip install torch, the wheel contains compiled CUDA kernels and the CUDA runtime libraries (cuBLAS, cuDNN, NCCL, and others) that PyTorch needs. These vendored libraries are built against a specific CUDA toolkit version, and that toolkit version is what torch.version.cuda reports. This is the toolkit version PyTorch was compiled against — not the system-installed CUDA toolkit. A system that has CUDA 11.8 installed system-wide can run a PyTorch wheel built for CUDA 12.1, and torch.version.cuda will report 12.1 because that is what PyTorch was compiled against. The system toolkit is irrelevant to the operation of PyTorch in the typical configuration; PyTorch uses its own vendored runtime. What PyTorch does require from the host is a sufficiently recent NVIDIA driver — one whose forward-compatibility range covers the toolkit version PyTorch was built with. The driver is what actually talks to the GPU, and the driver version determines which CUDA toolkit versions can be used through it. If the system driver is too old for PyTorch’s vendored toolkit, PyTorch fails at GPU initialization regardless of what is installed system-wide. The three numbers that all matter, separately For a PyTorch workload’s behavior to be specified, three (effectively four) numbers have to be captured: torch.version.cuda — the toolkit version PyTorch was built against. This determines the CUDA APIs PyTorch uses and the cuBLAS/cuDNN versions vendored in the wheel. System driver version (nvidia-smi) — what the kernel-mode driver supports. This sets the upper bound on toolkit versions that can be used via this driver. GPU compute capability — the hardware capability the GPU exposes (see the parent article for what this constrains). This determines which precision regimes and kernels can actually execute. PyTorch wheel source — different builds of the same PyTorch version (PyPI default, NVIDIA-published, conda-forge, distribution-vendored) can vendor different toolkit components and produce different runtime behavior. A workload that runs on one combination of (torch.version.cuda, system driver, GPU compute capability) is not guaranteed to reproduce on a different combination, even when the PyTorch version string is identical. The reasons are concrete: A different torch.version.cuda means different vendored cuDNN/cuBLAS, which can select different kernels for the same operation. A different driver can expose different runtime features or have different bug-fix profiles. A different compute capability can route the workload through a different kernel path or fall back to a less-optimized implementation. A different wheel source can vendor a different cuDNN or NCCL version even at the same torch.version.cuda. The torch.version.cuda number alone tells the reader nothing about whether their host will reproduce the result. A reproducibility checklist for PyTorch CUDA benchmarks A benchmark reporting “PyTorch X.Y on GPU Z” is missing the dimensions the reader needs to reproduce or interpret the result. The minimum disclosure surface for a PyTorch CUDA benchmark to be reproducible includes: PyTorch version string (e.g. 2.3.0) PyTorch wheel source (PyPI / NVIDIA NGC / conda-forge / distribution package) torch.version.cuda (the toolkit version PyTorch was built against) System NVIDIA driver version (nvidia-smi) GPU model and compute capability cuDNN version reported by torch.backends.cudnn.version() Whether a custom CUDA toolkit was prepended to the runtime path (LD_LIBRARY_PATH manipulation that overrides PyTorch’s vendored libraries) OS and kernel version, since some driver behavior is OS-conditional Workload precision (FP32 / TF32 / FP16 / BF16 / FP8) and whether tensor cores were exercised A report that satisfies this list can be reproduced by a different team. A report that satisfies a subset cannot, and the subset that’s missing is the difference between informative and indeterminate. Common patterns that produce non-reproducible PyTorch benchmarks Three patterns in particular are responsible for most of the “I can’t reproduce that PyTorch number” experience: The first is silent toolkit override. A user who has set LD_LIBRARY_PATH to include a system CUDA toolkit can shadow PyTorch’s vendored libraries with the system-installed versions. The benchmark then runs on a different cuDNN/cuBLAS than torch.version.cuda suggests. This is invisible from torch.version.cuda alone. The second is wheel-source variance. PyTorch wheels published by NVIDIA NGC, the PyTorch project, conda-forge, and Linux distribution packages can differ in which cuDNN they vendor and how they were compiled. A benchmark reported with a PyTorch version string but no wheel-source disclosure can be reproduced only if the reader installs from the same source. The third is driver mismatch. Different system drivers can produce different observed behavior on the same PyTorch + same GPU, particularly for workloads that exercise newer driver features. A driver version disclosed only as “current” or “latest” produces non-reproducible results six months later, when “current” means something different. The framing that helps torch.version.cuda reports the toolkit version PyTorch was built against, not the system CUDA toolkit. The relationship between this number, the system driver, and the GPU’s compute capability determines which kernels execute and which performance the workload achieves. Benchmark reproducibility requires all three numbers — plus the wheel source and the cuDNN version — to be disclosed; a report that names only torch.version.cuda is under-specifying the AI Executor in ways that block reproduction on any other host. The strategic argument lives in CUDA, frameworks, and ecosystem lock-in; operationally, the ecosystem’s depth produces real performance, and that performance depends on the specific stack tuple under which the workload was validated. LynxBench AI treats the (PyTorch version, wheel source, torch.version.cuda, driver, compute capability, cuDNN) tuple as part of the AI Executor specification — alongside the GPU model — because that tuple is what determines whether the benchmark result transfers to any other host that nominally has the same PyTorch and GPU.
