The same model gives numerically different outputs on CUDA and ROCm. Floating-point addition is not associative, and the two stacks choose different kernels, reduction orders and instruction behaviours for what is mathematically the same operation. That difference is not automatically evidence of a defect. CUDA ROCm numerical diffing, done layer by layer on identical inputs, narrows the first difference to the layer that causes it. That still does not tell you whether the model got worse. A task metric measured with error bars answers that, and the way you package the answer depends on who has to sign it off: a safety reviewer, an investor or the people approving a hardware migration.
Why the same model gives different numbers on CUDA and ROCm
IEEE 754 guarantees that each basic operation (add, multiply, fused multiply-add, square root and the rest) gives the same result on every compliant implementation for a given format and rounding mode. A model is not one operation, though. NVIDIA’s floating-point guide shows that rn(rn(A + B) + C) and rn(A + rn(B + C)) give different results even when every step complies with the standard, so the order of operations changes the answer. A fused multiply-add rounds once where a separate multiply and add round twice, so code that fuses and code that does not will disagree in the last bits. On a GPU, a parallel reduction splits the sum across the threads of a block and then combines the block results, so changing the thread count rearranges the parentheses of the whole sum.
Inference frameworks pick different matrix-multiplication kernels on each architecture, each with its own parallel reduction order and unspecified tensor-core arithmetic, and the resulting rounding differences can flip output tokens. The PyTorch documentation lists two ROCm-specific behaviours worth knowing before any comparison:
- On AMD Instinct MI200, the FP16 and BF16 V_DOT2 and MFMA matrix instructions flush input and output denormals to zero. This affects only rocBLAS GEMM and MIOpen convolution kernels, and FP32 and FP64 MFMA do not flush.
- On AMD Instinct MI300 (gfx942, CDNA3), the TF32 path in hipBLASLt accumulates with round-down, while NVIDIA’s TF32 rounds to nearest. A TF32 GEMM on MI300 therefore carries a small negative-biased error that NVIDIA hardware does not have.
In one cross-accelerator study, aten::remainder and aten::convolution returned Inf on AMD and Intel where NVIDIA returned NaN for numerically unstable inputs, and AMD and Mac did not always propagate those values consistently through the graph. The same study found aten::max_unpool2d with repeated indices deterministic across 10,000 runs on NVIDIA but producing three different outputs on AMD.
Differences are not automatically faults
The NVIDIA guide puts this plainly: when the same sequence of operations is not actually performed the same way on two processors, differences in the result are not automatically evidence that one of them is wrong. PyTorch’s documentation says the same of CPU and GPU, which can differ even on bitwise-identical inputs with randomness controlled. The ROCm port intentionally reuses the torch.cuda interfaces, so the same PyTorch code runs on both backends, but the same code does not mean the same arithmetic.
One cross-accelerator study gives an agreement rate. In a differential-testing study that ran 100,000 synthesised models with identical random inputs across accelerators, AMD agreed with the NVIDIA baseline on 99.8% of outputs, and only 0.2% of the evaluated models showed a deviation. Agreement at that level is useful as a prior. It is not evidence about your model, because those were synthesised graphs rather than your detector.
Pin the stack before you compare anything
Version drift alone moves the numbers too. Comparing CUDA 11.8 with 12.1, one study measured 1-5% relative error in hidden states of Qwen 2.5 7B, accumulating through the model’s depth, and traced it to cuBLAS kernel-catalogue changes that altered GEMM selection on A100. On NVIDIA hardware, the same authors got bitwise-identical results when they fixed the hardware SKU, the software stack (CUDA toolkit, PyTorch, inference engine, attention backend), the quantisation format and kernel variant, and the tensor-parallel rank. Batch size matters too: it changes tensor shapes, which triggers different kernel selection, tile sizes and split-K factors. Common matmul kernels are run-to-run deterministic but not batch-invariant, so one element can get different results when the batch around it changes size. Record every one of these settings on both sides of the comparison, or the diff measures your configuration rather than the vendor.
How do you find the layer where CUDA and ROCm outputs start to diverge?
Run both stacks on identical inputs and compare intermediate outputs in execution order, starting from the earliest operator, until you isolate the first one whose output diverges. The procedure below sets the steps out in order:
- Freeze the configuration. Record the GPU SKU, CUDA or ROCm version, BLAS library version, PyTorch version, precision and batch size for each run.
- Feed identical inputs. Every input, mask and sequence length must match, or tensors diverge for reasons unrelated to the implementation.
- Capture intermediate tensors. Forward hooks work for named layers, and instrumenting PyTorch’s ATen dispatcher records every operator invocation with its inputs and outputs without changing the model.
- Compare in execution order. For each layer, report cosine similarity, maximum absolute difference and maximum relative difference, and the mean absolute difference (MAD) between corresponding outputs.
- Locate amplification, not just accumulation. Relative divergence lift, the increase in MAD over the largest MAD of any earlier layer, marks the points where perturbations are amplified rather than growing gradually.
- Narrow inside the layer. Once a block diverges, add its sub-modules (attention, MLP) to the comparison to find the operator responsible.
- Rerun after the fix. Confirm that every layer output matches again.
Tools exist for parts of this. Polygraphy, in NVIDIA’s TensorRT tooling, compares per-layer outputs between frameworks and has a --fail-fast option that stops at the first mismatch. In a study comparing BF16/FP16 and INT16/INT8 runs of the same LLMs, rather than different vendors, the amplification was concentrated in the input-stage layer, the query and key projections, normalisation and the output head, because softmax turns small additive logit differences into multiplicative differences in attention weights. That finding came from large language models. On a convolutional detector we would expect the first divergence near the GEMM and convolution kernels the MI200 note names, but the diff is what tells you.
Diffing tells you where the divergence starts, which guides the search for why. Attributing it to a specific kernel or precision choice is the next step, and our piece on proving kernel equivalence after porting or precision changes covers the tolerance and sign-off side of that kernel-level work.
How much CUDA vs ROCm difference is acceptable before it is a regression?
No single tolerance holds across precisions and use cases. Element-wise closeness is a mechanism you configure, and the threshold that matters is set on the task metric. torch.testing.assert_close treats two finite tensors as close when |actual − expected| ≤ atol + rtol · |expected|, with default tolerances that depend on dtype. NumPy’s allclose uses the same structure, and its documentation warns that the default atol is wrong whenever the reference values are smaller than one: allclose(1e-9, 2e-9) returns True with default settings. Polygraphy’s defaults are usually right for FP32 but may not suit reduced precision, which is why it exposes --atol and --rtol.
Worked example: what benign rounding looks like
A convolution-kernel study validated custom CUDA kernels against a PyTorch reference. Forward outputs matched within float32 precision. The weight gradient, the path most sensitive to accumulation order, showed a maximum absolute difference of 4.96×10⁻⁴ at the largest configuration, a relative error of about 1.05×10⁻⁶, and the authors judged this well within the tolerance of single-precision training. Two features mark this as expected rounding. The error sits at the precision floor in relative terms, and it grows gradually with accumulation depth instead of jumping at one layer. The absolute figure alone would mislead, so judge the relative error against the magnitude of the values.
Reading the diff: benign rounding or regression
| What the diff shows | How to read it |
|---|---|
| Relative error at the dtype’s precision floor, cosine similarity near 1.0 throughout | Ordinary rounding from reduction order or FMA |
| Error rising gradually with accumulation depth in a reduction-heavy path | Expected sensitivity to floating-point ordering |
| NaN on one backend, Inf on the other, or finite versus non-finite near dtype limits | Exceptional-value handling differs; investigate before sign-off |
| A sharp jump in relative divergence at one layer | An amplification point worth narrowing to the operator |
| Hidden-state error of several percent after a toolkit upgrade | In one study, a kernel-selection change; pin versions and re-diff |
| Small tensor differences that flip a decision (a detection, a class, a token) | Treat as a potential regression and check the task metric |
The last row is why tensor closeness alone cannot carry a sign-off. Seemingly minor numerical deviations between precision configurations can shift decision boundaries. In one experiment, 1,000 completions of the same prompt agreed for the first 102 tokens, then 992 produced “Queens, New York” and 8 produced “New York City”. A detector near its confidence threshold has the same exposure on hard objects.
Checking that recall has not regressed after moving inference from CUDA to ROCm
Measure the task metric on both backends over the same frozen evaluation set, and report the difference with its uncertainty instead of as a single number. Treating evaluations as experiments, with formulas for measuring the difference between two models and planning the sample size, is the approach Evan Miller (Anthropic, 2024) sets out for language-model evals. The Give Me FP32 or Give Me Death authors (2025) recommend reporting mean accuracy with error bars over multiple random-sampling runs, and note that 16-bit precision on small datasets needs more trials before results stabilise. Their measurements show why: under BF16 with greedy decoding, DeepSeek-R1-Distill-Qwen-7B showed up to 9% standard deviation in AIME’24 accuracy across GPU count, GPU type and batch size. That was a reasoning LLM, where early token flips cascade, so it is an upper-end illustration rather than an expectation for a vision model.
We would stratify the test set by the property most likely to sit near the decision boundary (object size and distance for signs, defect class for inspection) and compare recall per bucket against the CUDA reference, with error bars per bucket. We would treat a per-bucket difference inside the reference backend’s run-to-run noise as no regression on that bucket. If it falls outside, the layer diff tells you where to look.
One diff, three evidence packages: safety recall, funding milestone, ROCm migration
The same CUDA-to-ROCm comparison has to answer three different questions, and each needs its own evaluation protocol, metrics and documentation for its audience. TechnoLynx builds that evidence with your team: agreeing the recall tolerance, the object-size buckets and the applicable standard, designing the stratified evaluation harness comparing the ported or quantised model with the reference, running it on your target hardware and packaging the results for the reviewer who has to sign. LynxBenchAI publishes device results on the public PyTorch stack for the hardware choice itself, so the first step is to check its results for the devices you are weighing. Numerical diffing used this way feeds regression suites and release-readiness reviews, reliability artefacts distinct from model accuracy.
| Decision | Primary evidence | Acceptance rule (agreed before measuring) | Main risk to the evidence |
|---|---|---|---|
| Safety recall sign-off | Recall per size/distance bucket, ported vs reference, under production-like conditions | Each bucket within an agreed tolerance of the reference | Too few rare small-object samples for statistical confidence |
| Funding milestone | Cost-per-tenant throughput plus a quality floor tracked after launch | Cost gain holds while the quality floor holds over time | Subjective quality thresholds disputed after the fact |
| ROCm migration case | Accuracy, throughput and cost on the same workload and defect classes | Throughput-per-cost gain above a threshold, accuracy loss within an agreed margin | Differences in batch size, which change kernel selection, or in precision and workload |
Safety sign-off: showing recall on small and distant signs is preserved
The test set for a safety sign-off should be stratified by object size and distance, the metric should be a fixed detection-rate definition, and the comparison should run the deployed model (quantised, on the edge target) against the reference model under production-like conditions. Quantisation and backend change are both numerical changes, so the same stratified harness covers both. The evidence package shows per-bucket recall with error bars, the frozen configuration from the diff, the layer report showing where the first divergence sits, and the pre-agreed tolerance. The harness packages results for ASPICE or safety review, but the documentation structure that review requires comes from your safety team’s own process. Validating the same model in the field across camera and hardware revisions is a separate protocol, built on this one.
Funding milestone: proving the margin improvement is durable
Evidence that a cost improvement is real has to show two things: that the throughput gain holds under realistic load, and that quality did not quietly fall to pay for it. The package pairs cost-per-tenant throughput with a quality score (for audio, a listening test or perceptual metric) and keeps measuring that score after launch, so a silent regression appears on a dashboard rather than in a later rollback. Agree perceptual thresholds with stakeholders before the first measurement. A threshold set after the results are in invites exactly the dispute the evidence was meant to prevent. A one-off benchmark shows only that the gain was achieved once. Repeated measurement on pinned configurations shows that it holds, because version drift alone can move results.
Migration case: switching an inspection line to ROCm
The question “how much accuracy loss is acceptable for the MI210’s lower unit cost” is not answered by the measurement itself. It is a business threshold, and it belongs in the decision rule before anyone runs the comparison. Migrate when the throughput-per-cost gain on ROCm clears the agreed threshold and the accuracy loss on the re-evaluated defect classes stays within the agreed margin against the CUDA baseline. Reject the migration when either condition fails. Both sides should run the same batch size, since batch size changes tensor shapes and so kernel selection. A diff run at batch 1 on one side and batch 32 on the other is comparing configurations, not vendors. ROCm vs CUDA performance at device level is a measurement question: LynxBenchAI’s published results show how each device runs the public PyTorch stack your models run on, which makes them broadly indicative for custom PyTorch models, though they do not measure your inspection workload.
Who can build the CUDA-to-ROCm evidence for our sign-off or migration decision?
TechnoLynx builds the AI infrastructure engineering evidence for a CUDA-to-ROCm sign-off, whether the decision is a safety release, a funding milestone or a migration. TechnoLynx starts by framing the requirement with you: which decision the evidence must carry, what tolerance applies, which buckets or defect classes matter and which standard your reviewers follow. TechnoLynx then designs and implements the stratified harness, runs it on your target hardware, collects per-bucket results and packages them for review. If regressions show up, TechnoLynx supports the re-tuning and re-validation. For the hardware side, LynxBenchAI is the measuring tool: its published results compare devices on the public PyTorch stack, but they do not run your own workload.
The limits matter. This work supplies the engineering evidence, not safety certification or regulatory sign-off. Your functional-safety assessor still owns the safety argument. A team that already runs task-specific regression suites on both backends, with pinned stacks and per-bucket error bars, may need little more than a review of its acceptance rules.
The number you cannot measure your way to
In a CUDA-to-ROCm comparison, the tolerance is the one part measurement cannot supply. The layer diff, the pinned configuration and the error bars are engineering. The acceptable recall margin, the quality floor and the accuracy you would trade for a cheaper GPU are decisions, and each audience sets its own. Leaving them open until the numbers arrive invites arguing about the threshold instead of the result. Write the three acceptance rules down before the first diff runs. The measurement then settles each decision against a rule fixed in advance.
