Why can’t we just switch to a different accelerator?
The question comes up in nearly every hardware evaluation meeting. A new chip promises better performance per watt, a lower price point, or competitive benchmark scores. The spec sheet looks strong. The economics look attractive. And then someone from the ML platform team explains, with varying degrees of patience, why switching isn’t as simple as plugging in a different card.
The reason is rarely hardware. The reason is software — specifically, the depth and breadth of the software ecosystem that surrounds the hardware and that years of engineering investment have baked into the organization’s workflows.
CUDA is not just an API
When people talk about CUDA lock-in, they often frame it as a programming interface problem: “We’re locked in because our code is in CUDA.” That framing dramatically understates the issue.
CUDA is a programming model, yes. But more importantly, it’s an ecosystem. That ecosystem includes cuDNN for neural network primitives, cuBLAS for linear algebra, NCCL for multi-GPU communication, TensorRT for inference optimization, Nsight for profiling, and a vast collection of community-maintained libraries, tutorials, Stack Overflow answers, and institutional knowledge that has accumulated over more than fifteen years.
When a framework like PyTorch dispatches a matrix multiplication, it doesn’t call a single CUDA function. It routes through a stack of abstractions — torch.mm → ATen → cuBLAS (or a custom CUTLASS kernel, or a FlashAttention implementation) → the CUDA runtime → the GPU driver. Each layer in that stack has been tuned, debugged, and optimized specifically for NVIDIA hardware. The performance you observe is the product of that entire vertical, not just the hardware at the bottom.
This is what makes the ecosystem valuable: it’s not one thing, it’s everything. And “everything” is very hard to replicate.
Switching costs are predominantly software-driven
The economic argument for switching accelerators usually focuses on hardware acquisition cost. But the harder cost to quantify — and the one that actually blocks transitions — is the software migration.
Moving from CUDA to an alternative ecosystem (ROCm for AMD, oneAPI for Intel, or a proprietary accelerator SDK) requires porting or replacing every CUDA-specific dependency in the stack. That includes not just the obvious kernels and library calls, but also the profiling tools the team relies on, the deployment infrastructure built around NVIDIA-specific tooling, the model optimization pipelines tuned for TensorRT, and the collective knowledge the engineering team has built about diagnosing CUDA-specific performance issues.
We have seen organizations budget six months for a migration and spend closer to eighteen (observed across engagements; not a benchmarked rate, and the spread depends heavily on how much custom kernel code exists). The hardware usually worked. What was mis-sized was the ecosystem gap nobody mapped upfront. As discussed in how the software stack functions as a first-class performance component, the software layer isn’t a thin wrapper around hardware capability — it’s a substantial determinant of what capability gets realized.
The practical switching cost isn’t “recompile for a different target.” It’s “rebuild the performance engineering discipline your team has developed around a specific ecosystem.”
Ecosystem dependency layers and switching costs
| Dependency layer | CUDA ecosystem example | Switching cost |
|---|---|---|
| Low-level kernels | Custom CUDA kernels, CUTLASS, cuBLAS calls | Must be rewritten or replaced with equivalents that may not exist or may be less optimized |
| Optimization libraries | TensorRT, cuDNN, NCCL | Ports exist but often lag in optimization maturity; performance may differ substantially |
| Profiling & debugging | Nsight Systems, Nsight Compute, cuda-gdb | Alternative toolchains are less mature; team expertise must be rebuilt |
| Framework integration | PyTorch CUDA backend, torch.compile CUDA paths |
Framework abstraction layers help, but backend-specific code paths still exist |
| Institutional knowledge | Team experience debugging CUDA-specific performance issues | Cannot be transferred; must be rebuilt for the new ecosystem over months or years |
Read the table top to bottom and the shape of the bill becomes clear: the layers that are easiest to port are the ones nearest the hardware, and the layers that are hardest to port are the ones nearest the people.
What does HIP change about the switching cost?
AMD’s HIP is often presented as the answer to the kernel half of the problem. Source-level CUDA compatibility is real: hipify will mechanically translate a large fraction of straightforward CUDA source, and the resulting kernels compile against ROCm. For teams whose GPU code is mostly ordinary kernels with ordinary memory patterns, this genuinely removes work.
What it does not do is relocate the rest of the stack. A translated kernel still needs a library ecosystem underneath it (rocBLAS and MIOpen in place of cuBLAS and cuDNN), a profiler your engineers can read, an inference optimizer in place of TensorRT, and framework paths that exercise the operators your model actually uses. Kernels that lean on NVIDIA-specific hardware behaviour — warp-level primitives, tensor-core intrinsics, particular async-copy patterns — translate syntactically and then need retuning to perform. So HIP moves the boundary of the problem rather than dissolving it: the syntax cost falls, the tuning and tooling cost stays, and the institutional-knowledge cost is untouched.
Ecosystem depth amplifies hardware capability
This cuts both ways. Lock-in is real and creates strategic risk, but the depth of a mature ecosystem also genuinely amplifies what the hardware can deliver.
Consider FlashAttention. It’s a memory-efficient attention algorithm that achieves substantial speedups by fusing operations and minimizing HBM reads. The original implementation was CUDA-specific, hand-tuned for NVIDIA GPU memory hierarchies. Ports to other platforms exist but often lag in optimization maturity, which means the same algorithm, on comparable hardware, can perform differently depending on how much ecosystem investment has gone into optimizing it for that specific target.
Multiply that example across thousands of kernels, operators, and optimization passes. The cumulative effect is that hardware with a deep ecosystem consistently outperforms hardware with a shallow one — not because the silicon is inherently superior, but because the software has had more time, more contributors, and more production feedback to get fast.
This is the core tension: the very depth that creates lock-in is the same depth that creates performance. Organizations can’t easily have one without the other.
Why a cross-ecosystem comparison usually isn’t one
The lock-in argument is normally hard to test, because each ecosystem tends to be measured with its own tooling, its own preferred model set, and its own idea of how much hand-tuning is fair. Two vendor decks can both be honest and still not be comparable.
What makes a cross-ecosystem number worth reading is that a single instrument produced both sides of it. LynxBenchAI runs the same catalogue across NVIDIA (cuda), AMD (cuda via ROCm), Intel (xpu), and CPU under one set of rules, with the same correctness thresholds applied on every backend. Two constraints do most of the work here:
- The optimization budget is capped identically across ecosystems. Models are prepared once, before any target device is known, and where two vendors’ kernels disagree about what is permissible, the stricter constraint applies to all of them. That cap is what stops a mature ecosystem’s depth from being smuggled into the measurement as a hand-tuning advantage. Depth still shows — but as execution, not as effort spent.
- Comparability holds within a release name. The catalogue changes between releases, so a CUDA-versus-ROCm reading taken from 26Q3 results is a statement about 26Q3, not a trend line you can extend across releases. Compare across release names and you are silently comparing two different test sets.
A last caveat that matters for procurement: a bounded-effort result tells you what the far-side ecosystem sustains under a capped budget. It does not predict what a funded migration, with engineers free to hand-tune for months, would eventually reach. Those are different questions, and the benchmark only answers the first one. The published leaderboard is also useful in the negative direction — it shows which devices have accumulated runs and which are simply absent, and absence is its own procurement signal.
How to think about ecosystem risk without oversimplifying
The temptation is to reduce this to a binary: “CUDA good” or “lock-in bad.” Neither framing helps.
A more productive approach treats ecosystem dependency as a risk factor to be managed, not a verdict. Concretely, this means:
Auditing the dependency surface. How deep does the CUDA dependency go in your stack? Is it confined to a framework layer (PyTorch, JAX) that abstracts the backend, or does it penetrate into custom kernels, deployment tooling, and monitoring infrastructure? The deeper the dependency, the higher the switching cost.
Evaluating abstraction layers. Frameworks increasingly support multiple backends. PyTorch’s torch.compile can target different hardware through backend plugins; JAX’s XLA compiler is hardware-agnostic in principle. These layers don’t eliminate switching costs, but they can contain them — if the organization has invested in using them rather than bypassing them with hardware-specific code. Triton shifts the calculus in a more specific way. Writing kernels directly against CUDA ties you to NVIDIA’s runtime, libraries, and profiling tooling; writing them in Triton expresses the kernel against an abstraction that can, in principle, lower to more than one backend. The dependency doesn’t disappear — the surrounding libraries and tooling still lean NVIDIA-ward — but the most portable part of your code moves up a level, so kernel logic survives a backend change even when the optimization and tooling around it have to be rebuilt.
Splitting the migration bill correctly. When a ROCm move gets scoped, framework-integration cost (backend, operator coverage, torch.compile paths) is largely shared infrastructure the ecosystem amortizes across everyone making the same move. Your own custom kernels are cost you carry alone, because nobody else is porting your CUTLASS. That asymmetry is the single most useful thing to establish early, because it tells you whether you are waiting on a community or staffing a project.
Benchmarking at the stack level. Comparing raw hardware specs is insufficient. The comparison must include the software ecosystem’s maturity for your specific workload — kernel coverage, framework support, profiling tooling, community knowledge. A chip that benchmarks well in a vendor-controlled demo environment may underperform in your production stack because critical operators lack optimized implementations.
Planning for ecosystem evolution. Software ecosystems are not static. ROCm’s coverage has expanded substantially in recent years, Intel’s oneAPI is maturing, and alternative hardware vendors are investing in framework compatibility layers. The landscape a year from now will look different from today’s — which argues for periodic reassessment rather than permanent commitment.
Lock-in is a system property, not a moral failing
No organization chose to be locked in. Lock-in emerged from rational decisions: adopt the most mature tools, optimize for the widest ecosystem, hire engineers with the most available expertise. Each individual decision was reasonable. The cumulative effect is a deep dependency that creates real strategic constraints.
Understanding that dependency — mapping its depth, quantifying its switching costs, and managing it as a technical risk rather than ignoring it or treating it as inevitable — is the difference between informed hardware strategy and reactive procurement. As explored in who actually owns performance outcomes, performance decisions don’t live in hardware or software alone; they live in the intersection. And the ecosystem is what fills that intersection.
So the question worth carrying into the next evaluation meeting isn’t whether an alternative ecosystem is good enough. It’s whether the comparison you are being shown was produced by one instrument, under one budget, within one release — and if not, what part of the gap you are actually looking at.
Frequently Asked Questions
Why is CUDA hard to replace even when competing hardware looks attractive on paper?
Because the lock-in isn’t in the API — it’s in the surrounding ecosystem. CUDA’s value comes from cuDNN, cuBLAS, NCCL, TensorRT, Nsight, and more than fifteen years of community-maintained libraries, tuned kernels, and institutional debugging knowledge. A competing chip can match raw specs and still underperform once it has to run through your actual production stack.
How does ecosystem depth — libraries, tooling, documentation — amplify or suppress raw hardware capability?
When PyTorch dispatches a matmul, it routes through a deep vertical — ATen, cuBLAS or CUTLASS, FlashAttention kernels, the CUDA runtime, the driver — each layer tuned for NVIDIA hardware. FlashAttention on CUDA versus a less-mature port on comparable hardware illustrates the point: same algorithm, different realised performance. Hardware with a deep ecosystem consistently outperforms hardware with a shallow one, because the software has had more time and more production feedback to get fast.
Why can comparing CUDA vs ROCm on a single benchmark be misleading as a procurement signal?
A single benchmark captures one operator on one workload in one configuration. It does not capture kernel coverage across your actual model, framework backend maturity, profiling tool availability, or how quickly your team can diagnose regressions. A chip that benchmarks well in a vendor-controlled demo can underperform in production because critical operators lack optimized implementations. Procurement signals need to be read at the stack level, not the kernel level.
When evaluating a CUDA alternative such as ROCm, which parts of the migration cost live in framework integration (e.g. PyTorch backends) versus in your own kernels?
Framework-integration cost — getting the PyTorch backend, operator coverage, and torch.compile paths working — is largely shared infrastructure the ecosystem amortizes across everyone moving to that platform. Your own custom kernels are cost you carry alone, because nobody else is porting your specific CUTLASS or hand-tuned code. Triton reduces the kernel half of that bill; the framework half depends mostly on how mature the alternative’s backend has become.
What does AMD’s HIP translation layer change about switching cost — does source-level CUDA compatibility remove the ecosystem problem or just relocate it?
It relocates it. HIP genuinely removes much of the syntax cost for straightforward kernels, and hipify handles a large fraction of ordinary CUDA source mechanically. What remains is everything around the kernel: rocBLAS and MIOpen in place of cuBLAS and cuDNN, a different profiler, a different inference optimizer, and retuning for kernels that leaned on NVIDIA-specific hardware behaviour. The institutional-knowledge layer is untouched by translation.
Why can a CUDA-versus-ROCm comparison only be read within a single release name, and what breaks if you compare results across releases?
Comparability is a property of the catalogue, and the catalogue changes between releases. Within one release name, both sides ran the same test cases under the same correctness thresholds and the same optimization cap. Across release names you are comparing two different test sets, so any delta you compute mixes real hardware or software change with catalogue change — and you cannot separate the two after the fact.
Why does capping the optimization budget identically across ecosystems matter when measuring a mature ecosystem against a younger one?
Without a cap, the mature ecosystem wins partly on effort: more people know how to hand-tune it, so more tuning gets applied. Capping the budget identically — models prepared once before the target device is known, and the stricter of two vendors’ constraints applied to both — keeps depth visible as execution rather than as effort spent. That is what makes the comparison a statement about the stacks rather than about who had better tuners on the day.
