This design ships one GPU inference codebase as a locked, CSV-validated release and as a faster branch assured under computer software assurance (CSA), provided the GPU optimisation code (the ONNX export, the TensorRT build configuration and the engine plans it produces) lives in its own versioned layer behind a stable interface. In this design, when TensorRT, the driver or the GPU changes, you re-validate that layer instead of the whole pipeline. Both branches classify every function with the same risk criteria. In this design, they differ in how the GPU stack moves and in how much documentation and testing surrounds the isolated layer.

TechnoLynx designs that boundary so one pipeline ships on both tracks with re-validation limited to the GPU layer. TechnoLynx defines where the risk line between the CSV and CSA tracks falls, builds the isolated TensorRT layer and its interface, sets up the validation trail both tracks need, and checks correctness and performance on the target hardware for each track. TechnoLynx engagements start with that risk framing and end with production AI reliability: deploying and operating the locked release alongside the faster branch.

Release tiering for GPU-optimised inference code, and why the criteria must match on both branches

Release tiering here means deciding, function by function, how much assurance each part of the inference system needs before it ships, and then applying that decision the same way on every branch. FDA’s CSA approach ties the level of assurance to the risk a software function poses to product quality and patient safety. It also asks for that risk to be assessed at the function level rather than the system level, so one system can contain functions in different risk categories. The guidance focuses on the intended use of each software feature, function or operation, and expects validation effort to match the risk it poses.

That is the reason the criteria have to stay the same across branches. Sakara Digital reads the CSA model as having inspectors check whether an organisation has a documented, rational risk-assessment method and whether it has applied that method consistently, with assurance proportionate to the risks it identified. Suppose the post-processing function that turns detector scores into a pass/fail decision is rated high risk on the locked branch and something lower on the internal branch, with nothing in the code to justify the difference. That is the kind of inconsistent application an inspector checks for. Tiering is about the function, not the branch. The branch only decides how much formal evidence gets wrapped around the same classification.

FDA’s CSA guidance covers computers and automated data-processing systems used in medical device production or the quality management system, under 21 CFR Part 820. A model that is part of the device itself, not of production or the quality system, is not what that scope describes. Settle that question before you design two release tracks.

How do we structure one inference pipeline for a locked CSV release and a faster CSA branch?

Split the codebase in two. The core pipeline holds model specification, pre-processing, post-processing and decision logic. In this design, the GPU layer turns a fixed model artefact into a TensorRT plan and runs it. The core talks to the GPU layer through a declared interface: named input and output tensors with fixed shapes, data types and numerical tolerances. Inside the GPU layer, whatever produces outputs inside those tolerances is acceptable.

Branch mechanics then follow an established pattern. In the git-flow model, the main branch always holds production-ready code, a develop branch holds the latest changes for the next release, and release branches handle final preparation, small fixes and release metadata so development can carry on alongside. Vincent Driessen, who described the model, still recommends it for explicitly versioned software, or for teams supporting several versions at once. A team with a locked validated release and a fast internal line is in exactly that position. In this design, the locked release lives on a release line with a frozen bill of materials (TensorRT version, CUDA, driver, GPU model and the plan files). In this design, the CSA branch tracks develop and picks up new GPU-stack versions as they arrive.

Version control for the model specification matters on both branches. The ML Test Score rubric (Breck et al., Google) asks that every model specification be code-reviewed and checked in, because you need to know exactly which code produced a given model, and that record makes training auditable and reproducible.

Locked CSV release versus CSA branch at a glance

Aspect	Locked CSV-validated release	CSA-based internal branch
Core pipeline code	Shared, from the same commits	Shared, from the same commits
Function risk classification	Same method, same results	Same method, same results
GPU stack	Frozen bill of materials: TensorRT, CUDA, driver, GPU model, plan files	Moves to new versions as needed
Engine builds	Reproduced from a retained timing cache	Rebuilt freely, gated by regression checks
Testing around the GPU layer	Whole system validated to one standard, as in traditional CSV	Rigour scaled to risk: scripted suggested for high risk, unscripted for lower
Documentation	Same documentation standard for every function	Risk-proportionate records

Can re-validation be scoped to the TensorRT layer alone?

Yes, provided the interface is the only route between the core pipeline and the GPU layer, and you are clear about what forces a rebuild. By default, a TensorRT engine works only with the TensorRT version that built it, and only on the type of device it was built for. Build-time options can widen both. NVIDIA’s documentation says a plan also records the compute capability it was built for, and it will not deserialise on a GPU whose compute capability does not match. That check guarantees the kernels picked at build time exist and can run. Triton Inference Server follows the same rule: a TensorRT plan is tied to a GPU’s CUDA compute capability, so a model configuration maps each plan file to its compute capability through cc_model_filenames.

A rebuild of the GPU layer is therefore due on any of three changes: a TensorRT version the build was not configured for, a GPU with a different compute capability, or a device type the engine was not built to support. None of those three changes touches the core pipeline’s logic. That fact is what justifies scoping the re-validation.

Version compatibility softens the first trigger, but only within limits. It has been supported since TensorRT 8.6, and the plan and runtime must both be 8.6 or newer. At runtime the available API is the intersection of the build version’s API and the running version’s API. Across major versions, NVIDIA supports exactly two paths: TensorRT 10 version-compatible engines on a TensorRT 11 runtime, and TensorRT 8 version-compatible engines on a TensorRT 10 runtime. A version-compatible build embeds a copy of the lean runtime in the plan, which then executes inside your process, so NVIDIA advises deserialising only trusted plans this way. If many models each carry their own runtime copy the cost adds up, so you can leave the lean runtime out and manage loading yourself. With trtexec, the --versionCompatible flag builds an engine that runs on newer TensorRT versions on the same host OS when used with the dispatch and lean runtimes.

How to tell that a plan was built for a different TensorRT version

TensorRT writes the major, minor, patch and build version of the library into every plan, and it refuses to deserialise the plan if those values do not match the runtime. For version-compatible plans, the embedded lean runtime does that check instead, and it is guaranteed to pass when the lean runtime is included. Even so, we would not rely on a deserialisation error as the control on the locked branch. Record the TensorRT version and compute capability next to each plan in the bill of materials and check them at load time, so a mismatch becomes a logged, attributable event rather than a startup crash.

How testing depth differs between the CSV release and the CSA branch

FDA’s current guidance is Computer Software Assurance for Production and Quality Management System Software. It replaces the final guidance Computer Software Assurance for Production and Quality System Software issued on 24 September 2025. FDA frames the main split as two categories. If a failure could create a quality problem that foreseeably compromises safety, it is “high process risk” and needs testing to match. If it could not, it is “not high process risk” and lighter validation testing is acceptable. Hogan Lovells’ summary of the final guidance adds that manufacturers can use finer categories, such as moderate or low, where that helps. One common three-tier reading, from Sakara Digital, puts direct automated quality impact with no manual check (scripted testing with traceability to requirements) as high, functions that inform quality decisions with human review (a mix of scripted and unscripted testing) as medium, and administrative functions (unscripted testing and vendor documentation review) as low.

Applied to the GPU layer: if an engine’s output drives an automated accept/reject decision with no human in the loop, that function sits in Sakara Digital’s high tier on both branches. For high-risk functions, the guidance suggests more rigorous, traceable testing such as scripted protocols. For lower risks it accepts scenario testing, error guessing or exploratory testing. It does not require any particular method for any particular risk level. Unscripted testing still needs a record: the scope and objective, the software version and environment, who tested and when, what was observed, and any defects. Logs, screenshots and narrative summaries are all acceptable evidence.

A common misreading is that the CSA branch is simply the less-tested branch. Sakara Digital warns that treating CSA as “less testing” is a misreading that leads to inspection findings, and that total effort on high-risk functions can rise under CSA because the guidance expects targeted testing rather than mechanical protocol execution. In this design, what separates the two branches is the release context. In this design, the locked branch carries a full protocol package. The CSA branch carries risk-proportionate evidence, and that evidence is still substantial wherever the function is high risk.

Every track keeps the same baseline records: documented risk assessments justifying the assurance chosen for each function, records of the assurance actually performed, change-control records, and periodic review records. FDA’s own presentation lists what an assurance record should contain: the intended use, the risk determination, a description of the testing, issues found and how they were resolved, a conclusion on acceptability, and the date and name of the person who did the testing. The documentation only needs to show the software is fit for its intended use, and its depth should match the risk level. Sakara Digital notes many organisations use tiered templates: a full protocol for high-risk functions, a streamlined test plan for medium, and a one-page summary for low. How to turn risk tiers into evidence packages for model validation is covered in depth in scoping validation depth from risk tiering to evidence packs.

Regression checks that gate a TensorRT engine rebuild before it reaches the locked branch

Start from one fact: two builds of the same model are not interchangeable by default. TensorRT’s builder times candidate implementations and keeps the fastest. When timings are close, system noise can decide which one wins, and different implementations can accumulate floating-point values in a different order or even run at different precisions, so separate builds usually do not give bit-identical results. When accuracy differs between two engines built from the same model, NVIDIA’s guidance points to a bad tactic chosen for one layer, and recommends using the editable timing cache to inspect the available tactics and pin a correct one.

That gives the locked branch its reproducibility mechanism. On the first build you set BuilderFlag::kEDITABLE_TIMING_CACHE and keep the logs and cache file. The log lists the available and selected tactics for each layer, and the cache records TensorRT’s decisions. Later builds reuse the cache, updated through ITimingCache::update where you want to force a tactic. The cache only works on matching configurations. NVIDIA advises reusing a timing cache only on the same hardware and software (CUDA, cuDNN and TensorRT versions, device model, clock frequency), because otherwise functional or performance problems can follow. With strongly typed networks, TensorRT infers tensor types and adheres to them instead of autotuning them. The engine can be slower, but fewer kernel alternatives are evaluated.

A rebuild gate, in the order the checks run

1. Confirm the bill of materials: the TensorRT version and compute capability match what the plan records, and the plan is mapped to the matching compute capability.
2. In this design, the locked branch builds from the retained timing cache and keeps the build log.
3. Scan layer outputs for NaNs and Infs with Polygraphy’s --validate option.
4. Compare engine outputs with golden values from a reference runtime such as ONNX Runtime, using identical inputs.
5. Apply explicit tolerances: Polygraphy accepts absolute tolerances per output and can check element-wise errors, or the maximum, mean, median or quantile absolute/relative error, with maximum being the strictest.
6. If the comparison fails, check layer by layer to find the operation responsible, then pin a tactic or a layer precision and rebuild.
7. Record the result in the evidence format the release track requires.

Which error statistic to use is a risk decision as much as an engineering one. For a high-process-risk function we would gate on the maximum error, because a mean can hide one badly wrong output on exactly the input that matters. The tolerance values must be fixed and agreed before anyone runs the comparison. Setting them afterwards to fit the result defeats the gate.

A worked example: Zoox’s TensorRT conversion pipeline

Zoox’s TensorRT conversion pipeline combines several of these checks in one harness. Zoox inspects output deviation by feeding identical inputs to the TensorRT engine and the original TensorFlow graph to catch accuracy regressions early. Layer-by-layer inspection traced one regression to an upsampling bug, which was then reported and fixed in TensorRT 7.1. A conversion test suite builds graphs, converts them and measures output deviation, which tells engineers which structures fail and which known regressions a newer TensorRT has fixed. An automated pipeline records the conversion options for each model, rebuilds every recorded model as a TensorRT engine when triggered, runs validation jobs on the new engines, and lets Zoox upgrade all its models to a newer TensorRT with one command. For a team working under CSV or CSA, the transferable point is the design. Because conversion options are recorded and validation is automated, a GPU-stack upgrade produces evidence as it runs, and that evidence is what the locked branch needs before it accepts the change. Approving and reviewing GPU-stack versions as a change-control process is covered in change control for GPU stack version approval and review. A model-level evaluation harness that compares candidate models on held-out data is a separate tool, and it belongs in the core pipeline’s validation rather than this gate.

Versioning the interface so a change behind it cannot slip through

Scoped re-validation is only defensible if the interface itself is versioned and tested. Otherwise a behaviour change inside the GPU layer can reach the core pipeline without anyone noticing. Semantic Versioning gives the rules. Software using SemVer must declare a public API, precisely and comprehensively. A patch version covers only backward-compatible bug fixes, and a minor version covers new backward-compatible functionality or deprecations. A breaking change to that API requires a major version increment.

For the GPU layer, the public API is the tensor contract (names, shapes, data types, tolerances) plus the operators the exported model relies on. ONNX already versions the second part. The IR, operator specifications and model each have their own version. Operators are abstract contracts between the model author and the implementations that run the model. An operator set’s version must increase whenever an operator is added or removed or its semantics change, and every model declares the operator sets it needs. Breaking IR changes require a version increment even when the serialised format stays the same.

The practical rule we would apply: any change to the declared opset, the tensor contract or the agreed tolerances is a major version bump of the GPU layer. That bump triggers the full gate on both branches. A rebuild that keeps the contract but changes the TensorRT version is a patch-level change, and the regression gate above has to prove it. Contract tests in the core pipeline should assert the GPU layer version they were written against, so an unannounced change fails a test rather than shipping silently. Triton’s model repository makes deployment match this scheme: each model version sits in its own numerically named subdirectory, and a version policy decides which versions are served.

What the dual-track design costs, and where it can come apart

Keeping the interface boundary takes ongoing engineering. Someone has to own the contract, the contract tests and the bill-of-materials checks. If nobody does, the isolation decays into a convention that people stop following. We have no published measure of that overhead to cite, so treat it as a cost to budget for, not a figure.

Interpretation is the second risk. The current guidance supersedes the September 2025 final, and it does not require particular testing methods for particular risks. An auditor may draw the boundary of “the GPU layer” more widely than your architecture does. The best defence is a documented risk method that you have applied consistently and can explain.

The third risk is structural. A locked bill of materials limits how far the CSA branch can drift. Version-compatible engines only cover the supported paths, and a timing cache is valid only on the configuration that produced it. Sooner or later the locked release has to move to a newer GPU stack. That move is a full gated rebuild with its own validation package. The CSA branch’s evidence makes the move easier, but it cannot stand in for that package. When you set up the two tracks, decide how often the locked branch will catch up. Do not leave the question until the hardware it was validated on stops being available.

FAQ

What is GAMP 5 in pharma?

GAMP 5 is a guide published by ISPE. Its second edition, published in July 2022, builds in risk-based assurance principles consistent with FDA’s CSA approach. It sorts software into five categories by complexity and configurability, which give a starting point for risk assessment, although CSA still expects risk to be judged per function.

Which TensorRT feature selects the best kernel for a given GPU?

The TensorRT builder times candidate implementations (tactics) and chooses those that globally minimise the engine’s execution time. The timing cache records those choices, and the editable timing cache lets you inspect them and override them.