On Ubuntu, the route Canonical recommends for NVIDIA Linux driver installation is the distribution’s own packages. Install the kernel headers, choose a driver branch that meets your CUDA toolkit’s minimum, install it with ubuntu-drivers or APT, and then pin it to that branch. NVIDIA’s .run installer is the other route, and on a Secure Boot machine you must enrol its signing certificate yourself. The driver is part of what any performance number from that machine describes, so a stack you cannot name is a result you cannot compare.
That installed stack is also the benchmark environment: every LynxBenchAI result discloses its executor, the device together with the software that drives it, and the driver, framework and runtime versions on the machine, because under its methodology the driver is part of what the score measures.
Which NVIDIA driver branch should I install on Ubuntu for AI work?
Work backwards from the CUDA toolkit your framework build targets. NVIDIA’s CUDA Compatibility guide (Release 615, September 2026) sets the minimum driver at 580 or later for CUDA 13.x, 525 or later for CUDA 12.x and 450 or later for CUDA 11.x. The CUDA 13.4 Update 2 release notes add a detail that trips people up: existing CUDA 13.x applications run on drivers from 580 upwards, but the new features and newly enabled platforms in CUDA 13.4 need an R615 or later driver. So “meets the minimum” and “supports everything the toolkit can do” can be two different driver versions.
The GPU sets a second constraint. NVIDIA notes that CUDA toolkits and drivers deprecate and drop support for GPU architectures over a toolkit’s product life cycle. A newer driver or toolkit can therefore stop supporting an old card. Our page on what CUDA compute capability constrains for AI workloads covers the architecture side.
On Ubuntu, the driver type is a separate choice from the branch. Canonical packages Unified Driver Architecture (UDA) drivers, recommended for general desktop use, and Enterprise Ready Drivers (ERD), recommended for servers and compute and identifiable by the -server suffix. For a machine that exists to run training or inference, we would start from the -server variant of the branch the toolkit requires, and move to UDA only if the same machine is also someone’s daily desktop.
How does the driver version relate to the CUDA version PyTorch needs?
The driver and the toolkit are separate pieces that have to be compatible. Neither has to match the other exactly. NVIDIA states that running a CUDA application needs a driver compatible with the toolkit the application was built with, plus the right versions of any dynamically linked libraries. The driver is backward compatible, which means applications built against an older toolkit keep working on later drivers. In NVIDIA’s words, new drivers “will always work with” applications compiled with an older CUDA toolkit.
Three compatibility modes are worth knowing:
- Backward compatibility: a newer driver runs older CUDA builds. A CUDA 11.0 application works on R450 and on every driver after it.
- Minor version compatibility: from CUDA 11 onwards, an application built with a toolkit from within a major release family runs, with a limited feature set, on a driver that meets that family’s minimum.
- Forward compatibility: the
cuda-compat-<major>-<minor>package lets applications built with a newer toolkit run on an older base driver, subject to platform and GPU support. NVIDIA names the package after the highest toolkit it supports, so a 535 driver that has to run 12.5 applications needs the 12.5 compat package.
NVIDIA notes that CUDA applications typically include the libraries they need. The question that matters is therefore whether the driver meets the minimum for the CUDA version the framework was built against. Starting with CUDA 13.4, NVIDIA no longer bundles the Linux driver with the CUDA Toolkit, so the driver has to be obtained separately. The difference between what each one does is the subject of our explainer on the CUDA toolkit vs the CUDA driver.
NVIDIA Linux driver installation on Ubuntu and Linux Mint from the command line
Ubuntu’s server documentation describes two command-line routes: the ubuntu-drivers tool, which it recommends, and plain APT. ubuntu-drivers uses the same logic as the “Additional Drivers” graphical tool, and on a Secure Boot machine it installs only pre-built, signed drivers by default. That default is why Ubuntu recommends it on Secure Boot machines.
Install procedure (Ubuntu, packaged driver)
- Install headers for the running kernel:
sudo apt install linux-headers-$(uname -r). - List the compute drivers on offer with
sudo ubuntu-drivers list --gpgpu. - Install the automatic match with
sudo ubuntu-drivers install --gpgpu, or name the branch, for examplesudo ubuntu-drivers install --gpgpu nvidia:<version>-server. - Reboot, then verify (next section).
If you install manually through APT, put the correct kernel modules in place first and then the metapackage for the driver series. NVIDIA’s own repository also offers a compute-only install that leaves out GL, EGL, Vulkan and the X drivers, which reduces the footprint on a headless training box. Its “full driver” install includes the desktop components together with the CUDA libraries and tools for computational workloads. On that repository, NVIDIA suggests installing its pinning package before the driver when you want to lock a branch.
Linux Mint is built on Ubuntu’s package base. On Mint we would use the same APT route and first confirm that the package names and the -server variants exist in the configured repositories, because the sources here document Ubuntu, not Mint.
When the .run installer is the right call
The .run file comes from NVIDIA’s driver download page. A community guide to Ubuntu 24.04 and 25.04 with Secure Boot has you install build-essential, DKMS, kernel headers for the running kernel and mokutil first. It also warns that mixing .run and apt packages “can cause headaches”, and recommends purging nvidia-* packages first.
| Situation | Route | What you take on |
|---|---|---|
| Ubuntu, Secure Boot on, branch available in the archive | ubuntu-drivers install --gpgpu with a -server driver |
Little: pre-built signed modules by default |
| Compute-only machine, branch to be locked | NVIDIA APT repository, compute-only install, pinning package | Installing the pinning package before the driver |
Using NVIDIA’s official .run file |
.run installer after purging apt drivers |
Module signing and MOK enrolment |
How do I check the driver loaded and PyTorch can see the GPU?
Run nvidia-smi first. NVIDIA documents that the driver reports, through nvidia-smi, the maximum CUDA version it supports, so it can run applications built with toolkits up to that version. That figure is a ceiling the driver allows. It does not tell you which toolkit is installed. Then ask the framework directly: in Python, torch.cuda.is_available() should return True, torch.version.cuda shows the CUDA version the wheel was built for, and torch.cuda.get_device_name(0) should name your card.
If nvidia-smi fails with “Failed to initialize NVML: Driver/library version mismatch”, Ubuntu’s documentation explains that the userspace driver packages were usually upgraded while the kernel module stayed on the older version. This often happens after a system upgrade, and sudo dmesg shows an NVRM: API mismatch line naming both versions. If the module did not build for the new kernel, missing headers are the first thing to check.
Why did my NVIDIA driver install break after Secure Boot or a kernel update?
Three causes are covered here, and each one shows up differently.
- Secure Boot rejects unsigned modules. With Secure Boot enabled, the kernel loads only signed modules, and the signing certificate has to be enrolled with
sudo mokutil --importfollowed by a password entered at the next boot. The.runinstaller can generate the key pair during installation. Without the enrolment the kernel will not load the modules.ubuntu-driversinstalls pre-built signed drivers by default. - nouveau is loaded. Ubuntu’s documentation notes that the open-source nouveau driver, if pre-installed and loaded, conflicts with the proprietary NVIDIA driver.
- The kernel moved and the module did not. The NVIDIA Linux driver consists of several kernel modules (
nvidia.ko,nvidia-modeset.ko,nvidia-uvm.ko). When the userspace packages are upgraded but the kernel module stays on the older version, often after a system upgrade,nvidia-smireports the version mismatch described above.
A reinstall starts with the headers for the running kernel and uses one install channel, not a mix. If .run and apt installs were mixed, purge the apt packages before choosing one. The community guide also sets nvidia-drm.modeset=1 on the GRUB command line.
Should I update the driver on a working AI machine?
Only when you need something the update provides: a toolkit whose minimum you no longer meet, a GPU the current branch does not support, or a fix you are waiting for. Newer is not automatically faster. A 2026 study of NVIDIA’s command streams by Yan, Karlin and Grant found that CUDA 13.0 issued a single submission cycle for a CUDA graph launch where CUDA 11.8 issued several, which reduced host-side launch cost. One measured improvement in one path does not mean the next CUDA or driver update improves your workload. On the Ubuntu MATE community forum, one user reported Wine regressions they attributed to newer drivers. The same thread reported that drivers on 20.04, 22.04 and 24.04 were being bumped to 580 whatever package was installed.
NVIDIA’s documented way to hold a branch or a version is a pinning package, nvidia-driver-pinning-<branch> or nvidia-driver-pinning-<version>. When upgrading, the command is always apt dist-upgrade, and the pinning configuration decides what it does. With Ubuntu’s own packages, apt-mark hold on the driver and its common library, for example sudo apt-mark hold nvidia-driver-550 libnvidia-common-550, stops the updater from replacing them.
Before you upgrade at all, write down the outputs of nvidia-smi and torch.version.cuda. Without them you do not know what you are rolling back to.
Why the driver belongs in every benchmark result
The driver is middleware. The 2026 command-stream study describes the driver as the layer that translates CUDA API requests into hardware commands. It concludes that application performance depends on how the driver interacts with the GPU and on the driver’s own processing overhead, as well as on the hardware. Wang, Wei and Brooks (2019) measured the rapid performance gains that specialised software stacks delivered on TPU and GPU platforms. Frameworks contribute too: PyTorch exposes switches such as allow_fp16_accumulation that trade numerical results for speed. Software sets real ceilings.
LynxBenchAI treats the AI Executor, the device together with the software stack that drives it, as the unit of measurement, and binds every result to the backend it ran through and the driver, framework and runtime present on the machine. No score is attributed to the silicon alone. The same catalogue runs on NVIDIA (cuda), AMD (cuda via ROCm), Intel (xpu) and CPU under one set of rules. A stack difference therefore shows up as a different number from one instrument, not as a difference in whose benchmark was used. Optimisation effort is held fixed: the models are prepared once, before any target device is known, and identical artefacts ship to every vendor’s hardware. The Intel side has its own driver story, covered in Intel Arc Linux support and the oneAPI/SYCL driver stack.
The published data shows why this matters. In the 26Q3 results, the NVIDIA GeForce RTX 3080 on native Linux scores a GT of 160.5 and the RTX 3080 under WSL scores 158.8. The RTX 5090 on native Linux scores 597.5 and the RTX 5090 under WSL scores 545.7. The release lists native Linux and WSL as separate executors. GT is an ordinal aggregate, so neither gap reads as a percentage.
LynxBenchAI measures software anyone can obtain. The benchmark declares the environment it installs. A submitter may use a different driver or a newer runtime as long as every component is a publicly released version. A build made for the benchmark, or tuned against it and shipped to nobody else, falls outside the methodology. MLCommons applies a comparable rule to MLPerf: Available systems must use an Available software stack, which includes the ML framework and the accelerator library. The LynxBenchAI rule is declared, not technically enforced. The client does not check that a submitted stack is publicly obtainable. The submission record captures the driver, framework and runtime versions, so the rule can be audited after the fact.
A driver update moved my benchmark score: what can one before/after pair tell me?
Less than it seems. A before/after pair across a driver change is not a controlled experiment. In LynxBenchAI, each side is one continuous timed window after a discarded warm-up, on one machine, and anything else that changed on that machine between the runs changed with the driver. The measurement does not capture thermal state or clock behaviour, so it cannot rule them out as causes.
Several sources make the limits concrete. Sinha et al. (SC22) found an average performance variation of 8%, up to 22%, between GPUs with identical architecture and vendor SKU in five large clusters. Run-to-run spread on a single card is a different quantity, but the finding shows that nominally identical hardware does not give identical numbers. PyTorch’s documentation states that completely reproducible results are not guaranteed across PyTorch releases or platforms, and that cuDNN’s convolution benchmarking can choose different algorithms between runs. Hoefler and Belli (SC15) proposed a minimal set of rules for reporting parallel-computing performance so that results can be interpreted.
A pair can still tell you something useful. If the score dropped sharply after an upgrade, the new stack deserves suspicion, and rolling back and running again is the next test. Keep the comparison within one release. The catalogue, precisions, correctness thresholds and scoring formula are fixed per release, so 26Q3 results are read against 26Q3 results.
The test is cheap enough to repeat. The LynxBenchAI Personal Edition installs with pip install lynxbench-ai on Python 3.11 or later, on Linux or Windows via WSL2. It needs about 15 GB of free disk, downloads about 2 GB of model data once, takes 15 to 30 minutes per pass and is free for non-commercial use. Run it once on your pinned driver, change only the driver, reboot, run again, and record both stacks next to both scores. If the gap points to a software ceiling you need to raise on a production stack, performance tuning for AI inference is the engineering work that follows.
FAQ
What happens if you don’t update drivers?
Software built against older CUDA toolkits keeps working, because NVIDIA drivers are backward compatible. The risk lies in the other direction: newer toolkits set higher driver minimums, and CUDA 13.4’s new features need R615 or later.
What does CUDA provide for AI workloads?
CUDA for AI workloads is the main interface applications use to run work on NVIDIA GPUs. Much of what it does is carried out by the driver underneath, which translates API calls into hardware commands.
