A model that ran cleanly on NVIDIA gets redeployed to AMD, and throughput quietly drops by a third. The version tag says nothing changed. The weights are identical, the accuracy is identical, and the version record is technically correct — which is exactly why nobody can explain the regression. The team has a perfect record of what the model is and no record of how it was built for the hardware it ran on. That gap is the whole problem. In a single-vendor stack you can get away with treating a model version as a weights-and-checkpoints artifact: tag the trained file, store it, redeploy it, and the runtime is a constant you never have to think about. The moment you support more than one GPU vendor, the runtime stops being a constant. A version that captures accuracy but not the per-target compilation and portability characteristics is a version that reproduces your numbers and loses your performance. What ML model versioning actually is once the target can change Most teams inherit their mental model of versioning from source control. A version is a snapshot; snapshots are immutable; if the bytes match, the thing is the same. For model weights that intuition holds — the tensors are what they are. It breaks the instant those weights have to run on hardware you didn’t train on. A trained model is not an executable. It becomes one only after a target-specific build path turns it into kernels, memory layouts, and scheduling decisions that are specific to the silicon underneath. On NVIDIA that path runs through CUDA and cuDNN, often with a TensorRT engine compiled for a specific GPU architecture and batch profile. On AMD it runs through ROCm and the HIP toolchain. On Intel it goes through oneAPI and SYCL. On Apple silicon it’s Metal. Each of these bakes in kernel selection and memory-access assumptions that do not travel with the .pt or .onnx file. So the honest definition is: in a cross-platform GPU stack, a version is the model plus the target-specific build plus the kernel and memory-access assumptions each build encodes. Reaching a different target is not a deployment detail bolted onto a version — it is a first-class dimension of what the version is. Two builds of identical weights, one for CUDA and one for ROCm, are as different in operational behaviour as two different models would be. Treating them as the same version because the weights match is the category error that produces silent regressions. This is the same principle that underlies software porting as a discipline rather than a recompile: moving a workload to new hardware changes its behaviour in ways that only show up under measurement, not in the artifact hash. Why “the weights are identical” hides vendor-specific debt When you version only the model, every migration starts from zero. Nobody recorded which TensorRT precision mode the NVIDIA build used, which fusion passes fired, what the achieved occupancy was, or whether the ONNX Runtime execution provider fell back to a generic kernel on the AMD side. So when performance drops after a port, there is nothing to diff against. The regression is real, measurable, and completely unattributable. That is how hidden vendor-specific debt accumulates. Each deployment that worked “well enough” on its original hardware carries undocumented assumptions — a batch size tuned for a particular HBM capacity, a kernel path that only exists in one vendor’s library, a graph fusion that torch.compile produced on one backend and silently declined on another. The debt is invisible until someone tries to move the model, and by then the person who tuned the original build has often moved on. The knowledge that made the NVIDIA deployment fast was never a version-controlled artifact. It lived in someone’s head. Teams that version the model against its target matrix keep migrations auditable. The version record answers the question that the weights alone cannot: on this hardware, with this build, what did we actually get? That record is what turns a multi-vendor migration from an unbounded rewrite into a diff against a known baseline. What belongs in a version record for NVIDIA, AMD, and Intel The useful boundary is between metadata that describes the model and metadata that describes the model as built for a target. The first is table stakes. The second is what most teams skip and later wish they hadn’t. Model version record fields for a multi-vendor GPU stack Field Scope Why it belongs in the version Weights hash + training config Model Establishes the accuracy-equivalence class; the constant across targets Framework + op-set version Model ONNX op-set or PyTorch version determines which builds are even valid Target vendor + architecture Per-target CUDA sm_90 vs ROCm gfx942 vs Intel Xe are different executables Build toolchain + version Per-target TensorRT 10.x, ROCm 6.x, oneAPI release — kernel selection changes between them Precision mode used Per-target FP16 on one target, BF16 or INT8 on another, changes both speed and accuracy Achieved throughput + latency Per-target The baseline a future migration diffs against; measured, not spec’d Kernel/fusion decisions Per-target Whether fusions fired or a generic fallback was used explains the numbers Memory footprint at batch profile Per-target Ties the build to a capacity assumption that may not hold on other silicon The per-target rows are the ones that make the record worth keeping. Throughput and latency here are operational measurements from the deployed build on named hardware — not vendor peak-FLOP specs, which predict nothing about a real inference path. The distinction matters: a version record full of datasheet numbers is decorative. A record of what the build actually achieved on the GPU it ran on is a baseline. The choices baked into each build are worth understanding on their own terms; if you are moving off a single-vendor path, the mechanics of porting an inference path off CUDA with HIP show why the ROCm build is genuinely a different artifact and not a recompile. How do you diff performance portability regressions between two versions across vendors? This is the question the version record exists to answer, and it only works if you framed the record correctly upfront. A portability regression is a delta between what a build achieved on its target and what an equivalent build achieved on a different one — or what the same target achieved at a previous version. The diff is not a single number. It is a comparison across the per-target dimensions the record captured: Accuracy parity first. Confirm the weights and precision mode produce equivalent task metrics on both targets. If accuracy drifted, the performance comparison is meaningless until you understand why — a precision downcast that helped throughput may have moved the model outside its accuracy envelope. What to actually track here is covered in which model metrics to measure across multi-platform edge targets. Throughput and latency delta at a fixed batch profile. Compare measured sustained numbers, not peak. A build that wins on burst latency can lose on sustained throughput because of memory-bandwidth pressure the other target doesn’t hit. Kernel-path divergence. If the NVIDIA build fused attention into a single kernel and the AMD build fell back to separate GEMM and softmax passes, the version records will show it. That is the mechanism behind the regression, and it is far more actionable than “AMD is slower.” Memory-footprint shift. A build tuned for 80 GB of HBM may not fit the same batch profile on a card with less capacity, forcing a smaller batch that changes throughput independent of raw compute. When you can attribute a regression to a specific divergence — a fusion that didn’t fire, a fallback kernel, a batch size forced down by capacity — you have converted a mystery into an engineering task. Continuous monitoring of ported inference paths is what keeps that diff current in production rather than a one-time audit artifact. Reproducing a past deployment on different silicon The workflow that a disciplined version record enables looks unremarkable, which is the point. To stand up a past model version on a GPU architecture it never ran on before: Pull the weights and training config — the model-scope constant. Pull the framework and op-set version to know which builds are valid. Select the target-specific build path for the new silicon (ROCm/HIP, oneAPI/SYCL, or a fresh CUDA target). Build with the recorded precision mode as the starting hypothesis, not a guess. Measure against the closest existing per-target baseline in the record, and treat any gap as a portability delta to explain rather than a new normal to accept. The value is not that this is fast — building for a new target rarely is. The value is that you never re-derive lost build assumptions. You know what precision the model tolerated, what batch profile it was tuned for, and what throughput a comparable build achieved, so you are tuning against a reference instead of exploring blind. This is the mechanism by which disciplined versioning reduces multi-vendor migration cost: the second and third targets are diffs, not rewrites. FAQ How does ML model versioning work in practice? In practice, a model version is an immutable snapshot of the weights, training configuration, and framework state, tagged and stored so it can be retrieved and redeployed later. That definition holds for single-vendor stacks where the runtime is a constant. Once the target GPU can change, the version must also capture the per-target build — otherwise it reproduces the model’s accuracy but not its performance on the hardware it originally ran on. Why should a model version in a cross-platform GPU stack capture more than just weights and hyperparameters? Because a trained model is not an executable — it becomes one only after a target-specific build path (CUDA, ROCm, oneAPI, Metal) turns it into kernels and memory layouts specific to the silicon. Two builds of identical weights for different vendors behave as differently as two different models would. Versioning only the weights records what the model is but loses how it was built for the hardware, which is where performance actually lives. What per-target build and performance metadata belongs in a version record for NVIDIA, AMD, and Intel deployments? Alongside the model-scope fields (weights hash, training config, framework and op-set version), the record should carry per-target rows: target vendor and architecture, build toolchain and version, precision mode used, achieved throughput and latency as operational measurements, kernel and fusion decisions, and memory footprint at the batch profile. The per-target measurements — not vendor peak-FLOP specs — are what make the record a usable baseline. How do you detect and diff performance portability regressions between two model versions across GPU vendors? Confirm accuracy parity first, then compare measured sustained throughput and latency at a fixed batch profile, kernel-path divergence, and memory-footprint shifts between the two version records. A regression becomes actionable when it can be attributed to a specific divergence — a fusion that didn’t fire, a fallback kernel, or a batch size forced down by capacity — rather than a vague “the other vendor is slower.” What is the practical workflow for reproducing a past deployment on a different GPU architecture from its version record? Pull the model-scope constant (weights, config, framework/op-set), select the target-specific build path for the new silicon, build using the recorded precision mode as the starting hypothesis, and measure against the closest existing per-target baseline. The point is not speed but that you never re-derive lost build assumptions — you tune against a reference instead of exploring blind. How does disciplined versioning reduce the migration cost of supporting multiple GPU vendors? By turning each additional target into a diff against a known baseline rather than an unbounded rewrite. Teams can quantify per-target throughput and latency deltas at version boundaries instead of discovering regressions in production, and reproduce any past deployment on any supported GPU without re-deriving build assumptions someone tuned and forgot. Where this leaves the version record A version record that captures per-target build and performance characteristics is also the input that makes portability regressions measurable at all — it feeds directly into the cross-platform dimension of a GPU performance audit, where a versioned baseline is the only thing that lets you say a migration lost performance rather than just felt slower. If you are running one vendor today and plan to run two tomorrow, the cheapest thing you can do now is start recording the build, not just the weights. The debt you avoid is the debt nobody can see until they try to move.
