Change Control for GPU Stacks Under Annex 11 and FDA CSA

EU GMP Annex 11 says any change to a computerised system, including its configuration, should be made only in a controlled manner under a defined procedure. It also says systems should be evaluated periodically to confirm they remain valid and GMP-compliant. FDA guidance puts it more bluntly: when any change, even a small one, is made to software, its validation status has to be re-established, and that includes regression testing of the unchanged parts that could be affected.

GPU stacks make this harder than it sounds. NVIDIA releases CUDA Toolkit versions and GPU drivers on different cadences. A version bump that looks routine can also change results, since PyTorch does not guarantee reproducibility across releases. The stakes are documented. In the FDA’s analysis of medical device recalls from 1992 to 1998, 79% of the software-related recalls came from defects introduced by changes made after the software was first released.

What does an Annex 11 auditor expect to see for a GPU stack?

Under Annex 11, manufacturers should be able to justify their standards, protocols, acceptance criteria, procedures and records from their own risk assessment, and validation documentation should include change control records.

TechnoLynx builds that validation package for regulated teams running GPU inference pipelines. TechnoLynx starts by recording the intended use and finding the points where a change of precision, driver or engine could alter output. TechnoLynx then defines the numerical-equivalence protocol and its risk-based thresholds, locks the bill of materials with reproducible build scripts, and adds change-control gating to deployment. This is part of TechnoLynx’s AI engineering work for life sciences and healthcare.

Documentation checklist for a pinned or changed GPU stack

Item	What it demonstrates
Intended use and risk-based analysis	Why the assurance effort is proportionate to the risk
Version-pinned bill of materials (GPU model, driver, CUDA Toolkit, cuDNN, TensorRT or other runtime, framework)	The defined operating environment, including required hardware, software configurations and versions
Firmware and operating system versions	That these are qualified for intended use, including version or release
Equivalence test protocol, acceptance criteria, test cases and results	Objective evidence that the configuration meets user requirements
Validation summary report with a conclusion statement	The pass, fail and exceptions record, which is often where an auditor starts
Change control SOP	How changes are proposed, assessed for impact (including regression testing), approved and recorded
Configuration management records	Which versions are currently approved, and who can access them

Why a pinned TensorRT engine is defensible

Pinning a TensorRT engine version fits the FDA expectation that the manufacturer defines its operating environment, including software versions. The FDA is explicit that a manufacturer buying off-the-shelf software has to make sure it performs as intended in its own application. TensorRT’s own portability rules also argue for treating the engine file as a versioned artefact:

Serialised engines don’t move between platforms such as Linux and Windows.
Without hardware compatibility mode, an engine doesn’t run on a different GPU architecture.
Across major versions, only two combinations are supported: TensorRT 10 version-compatible engines on a TensorRT 11 runtime, and TensorRT 8 version-compatible engines on a TensorRT 10 runtime.

Pinning the engine doesn’t remove the need to re-validate when the driver or GPU firmware changes.

Why the driver and toolkit are separate items on the bill of materials

CUDA compatibility helps environments where driver updates follow a different qualification schedule from toolkit updates. The driver is backward compatible: applications built against one toolkit keep working on later drivers. Minor-version compatibility lets applications built with any toolkit in a major family run, with a limited feature set, on at least the minimum required driver. That minimum is driver 580 or later for CUDA 13.x, 525 or later for CUDA 12.x, and 450 or later for CUDA 11.x. From CUDA 13.4, the Linux driver is no longer bundled with the toolkit, so the two are now separate items by default.

Compatibility has caveats that belong in the risk assessment. An application that relies on a feature spanning both toolkit and driver can fail on an older driver with cudaErrorCallRequiresNewerDriver. Libraries also depend on each other: each cuDNN version requires a particular cuBLAS version, for example. cuDNN 8.9.6 for CUDA 12.2 needs Linux driver 525.60.13 or later. NVIDIA tuned its heuristics on CUDA 12.2 Update 1 for newer GPUs.

Is there a public precedent for defending mixed precision to a regulator?

We are not aware of a public, citable regulatory decision showing mixed-precision inference to be acceptable across the board. Treat that absence as normal rather than as a blocker. NVIDIA’s TensorRT Edge-LLM separates release-tested “official” combinations from merely “compatible” ones, and checks at load time that the platform, CUDA, TensorRT and GPU match a listed configuration.

The same evidence may need different framing depending on where it goes. Annex 11 covers computerised systems in GMP-regulated activities. FDA CSA covers software used in medical device production and quality management systems.

Pre-qualifying CUDA, cuDNN and driver releases on a fixed cycle

Yes, you can set up a periodic review cycle. Test candidate CUDA, cuDNN and driver releases against a minimum validation suite before anyone asks for them, add the ones that pass to the approved list, and keep an expedited path that runs the same tests on a shorter timeline. Toolkits and drivers arrive on separate cadences. If qualification starts only when an engineer requests a version, it starts when a project already depends on the answer.

TechnoLynx can set up and run that cycle with your team. TechnoLynx defines the cadence and the minimum test suite, builds the profiling and test harness that checks each candidate release for correctness and performance on your workloads, documents the results for audit, and supports the expedited path when a project needs a version outside the cycle.

Not every release deserves a slot. New Feature Branch drivers arrive at least every three months and are meant for early adopters evaluating features. Production Branches come twice a year and are meant for production use. We would pre-qualify Production and Long Term Support branches and evaluate New Feature Branches only when a specific capability is needed.

Known problems in a release belong in the impact assessment, not background reading. CUDA Toolkit 13.2 Update 1 contained a critical cuBLAS bug: cublasLtMatmul() could ignore tensor-wide scaling for NVFP4 matrix multiplications and return incorrect results, fixed by cuBLAS patch 13.4.1. A cuFFT issue identified in CUDA 13.3 affected the correctness of real-side LTO callback kernels for certain large even-size real-to-complex and complex-to-real transforms; cuFFT 13.4 fixed it.

What minimum validation testing should a new GPU stack version pass?

At minimum, the testing has to show that the change works and that nothing else broke. The FDA expects regression analysis and testing to show that parts of the software not involved in the change were not adversely affected. Change-control guidance for instruments says it is not enough to test only the change: potentially affected functionality has to be tested too. A practical minimum suite for a GPU stack covers the following:

Installation and version verification: evidence that every component installed is the version specified.
Functional regression: reference inputs rerun, with results compared to the previous validated outputs.
Numerical equivalence: outputs compared with the validated baseline within acceptance tolerances set in advance and tied to risk.
Determinism settings: confirmation that settings such as torch.backends.cudnn.benchmark = False still behave as documented, since cuDNN benchmarking can pick different algorithms on later runs, even on the same machine.
Performance regression: measured on the profiled production workload.

How rigorous the suite is should follow risk: scripted, or scripted plus unscripted, testing for high process risk functions, and leaner methods elsewhere. The suite itself needs review as audit expectations change, or the approved list becomes a compliance risk of its own.

We urgently need an unapproved CUDA or driver version mid-project. Now what?

Use an emergency change path that was defined in advance. PIC/S PI 011-3 says the change control procedure should define the circumstances and documentation requirements for emergency changes (“hot-fixes”), with each error and each authorised action fully documented. The draft revised Annex 11 adds that relevant security patches should be deployed promptly following vendor recommendations, and for critical vulnerabilities that might mean immediately. A workable expedited path runs in this order:

Raise a change request that states the trigger (security advisory, blocking defect, required feature) and why waiting for the next cycle carries more risk.
Assess impact against the current bill of materials, including library interdependencies and release-note known issues.
Run the same minimum suite on a compressed timeline.
Get system owner and QA approval before production rollout.
Close the change formally and bring the version into the next periodic review.

For the case where you need a newer toolkit without moving the driver, CUDA forward compatibility lets applications built on a newer toolkit run on an older driver than the minimum it normally requires. It applies only to NVIDIA data-centre GPUs, selected NGC Server Ready RTX SKUs and Jetson boards. Some features that need kernel-mode support won’t work through it. The compatibility package is a new component in its own right, so it goes on the bill of materials and through the same tests.

How often should the approved CUDA/cuDNN/driver list be re-certified, and who owns that review?

Neither Annex 11 nor FDA CSA sets the interval. The draft revised Annex 11 says review frequency should be established and justified by the risk the system poses to product quality, patient safety and data integrity. FDA CSA likewise asks manufacturers to choose how often to perform assurance activities based on their risk-based analysis.

Ownership is easier to settle than cadence. PIC/S expects regulated organisations to define clear responsibilities for managing computerised systems, including monitoring and auditing, and change requests are typically approved by the quality assurance unit and the system owner. Our judgement is that the system owner should own the approved-list review, QA should approve it, and the engineering team should run the qualification work. Engineers should not sign off their own list.

On cadence, we would align the review with NVIDIA’s lifecycle rather than the calendar: Production Branches arrive twice a year, so a six-monthly review fits that rhythm. Add event triggers for branch end-of-life notices and critical advisories. The exact interval still has to be set and justified by your own organisation.

The review’s scope is set out in the texts. Annex 11 already lists upgrade history, deviations and incidents among the inputs to periodic evaluation. The draft revision adds hardware, software, configuration, platform and interface changes since the last review, the combined effect of multiple changes, and finding undocumented changes through configuration auditing.

What happens to the validated baseline when NVIDIA ends support for the approved driver branch?

A Production Branch receives bug fixes and security updates for up to one year, and a Long Term Support Branch for three. The draft revised Annex 11 expects validation on updated platforms to be planned and completed before the vendor’s support expires.

Driver branch	Designation	End of life	Max CUDA major without forward compatibility
R535	Long Term Support Branch	June 2026	CUDA 12
R580	Long Term Support Branch	June 2028	CUDA 13
R595	Production Branch	March 2027	CUDA 13
R610	New Feature Branch	August 2026	CUDA 13

Moving from R535 to R580 also moves the supported CUDA ceiling from 12 to 13, so the change may reach beyond the driver into the toolkit and the libraries above it. NVIDIA AI Enterprise runs its own lifecycle, separate from the driver branch dates above: its Infra 4.x line (LTSB / R535) reached end of life in July 2026. NVIDIA’s migration guidance says to validate the target stack in its lifecycle and compatibility explorer. For regulated systems, picking an LTSB at qualification time gives the longest window before this exercise comes round again.

Does mixed-precision or TensorRT inference need its own validation sign-off?

Yes. Treat moving from FP32 to FP16 or BF16, or adding TensorRT, as a separate validated configuration with its own equivalence evidence, not as an extension of the FP32 sign-off. In a study of LLM inference (arXiv 2506.09501), DeepSeek-R1-Distill-Qwen-7B under BF16 with greedy decoding varied by up to 9% in accuracy when GPU count, GPU type and batch size changed. With FP32, 2.2% of samples diverged across configurations; with BF16, more than 90% did.

TechnoLynx can produce the evidence for that sign-off. TechnoLynx defines the protocol for showing numerical and accuracy equivalence between your validated CUDA, cuDNN and PyTorch stack and the candidate one. TechnoLynx then benchmarks the candidate, including mixed precision and TensorRT, against the validated baseline on your own cluster, and documents kernel-level optimisation choices for the risk assessment.

Evidence from other sources points the same way. PyTorch does not guarantee completely reproducible results across releases or platforms, or between CPU and GPU runs. Shanmugavelu et al. (2024) report that the sensitivity of deep learning pipelines to floating-point non-associativity can be extreme, and can prevent certification for commercial applications. NVIDIA’s floating-point white paper discusses fused multiply-add and has a chapter on comparing CPU and GPU results. The acceptance criterion should therefore be a tolerance justified by clinical or quality risk, not bitwise identity. For the related decision of quantising a model, see quantisation as a change-control decision.

Which CUDA, cuDNN and PyTorch versions are already qualified is something only your own change-control records can answer, because your validated environment is the one you defined and tested. If the version is already in the validated register, use it. The lead time depends on how much re-validation the change triggers and on your organisation’s risk tiers. A library upgrade made for performance can still fail validation if it moves outputs beyond tolerance. For the separate decision of which hardware to buy, LynxBenchAI’s published results show how each device runs the public PyTorch stack. They are broadly indicative for custom PyTorch models, but they don’t measure your workload and aren’t validation evidence.

How do we show a GPU system is locked to its validated configuration and cannot update silently?

You show it with a configuration baseline, controls on every update path, and audits that compare the running system with the baseline. Documents describing a frozen state are not enough on their own. Moving an inspection controller from CPU to an edge GPU should be handled as a quality-system equipment change, supported by IQ, OQ and PQ evidence that the frozen configuration performs equivalently to the previously validated state. IQ, OQ and PQ testing is how validation demonstrates accuracy, reliability and consistent intended performance.

We recommend a lockdown package containing the following:

Baseline record: exact firmware, driver, CUDA, cuDNN, runtime and engine versions for the qualified state, with firmware qualified by version.
Image hashes: cryptographic hashes of the driver, firmware, runtime and engine files (or container image digests) recorded against that baseline.
Update controls: automatic updates disabled, package versions held, and remote firmware push either removed or brought under change control.
Startup verification: for TensorRT, NVIDIA advises deserialising during controlled startup, requiring a non-null engine and inspecting the logger before accepting traffic.
Audit trail: change-control and revision transactions recorded in the audit trail.
Configuration auditing in periodic review: a check for undocumented, unapproved changes.
Supplier terms: where a vendor can push updates, a contract that sets out the release process and your opportunity to test before release.

FDA CSA gives a model for that last case. Its SaaS example has the vendor document all automatic updates, and the manufacturer assess each one, test it according to its impact and keep a record.

Each control has limits. A hash proves the image matched at the moment it was checked. It does not stop a later change unless the update mechanisms are also controlled and audited. If the platform allows any auto-update or remote firmware push, the lockdown claim fails however good the paperwork looks. IQ, OQ and PQ equivalence also holds only for the defect types and conditions that were tested, so a new product variant can reopen the question.

Together, version pinning, the qualification cycle, the emergency path and lockdown evidence keep the system in a state your organisation can defend. The hardest open question here is not technical. It is who owns the approved list, and on what risk-justified interval they review it.

FAQ

What is change control in pharma?

Change control is the defined procedure under which any change to a GMP computerised system, including its configuration, is made in a controlled way. Changes are requested in writing, analysed, and approved by QA and the system owner.

What is cuDNN vs CUDA?

CUDA is NVIDIA’s GPU computing platform and toolkit, and cuDNN is NVIDIA’s deep neural network library built on top of it. Each cuDNN release lists the CUDA and driver versions it needs, and depends on a matching cuBLAS version.