XLA and CUDA are not competing choices. XLA is a compiler that fuses and lowers a graph of tensor operations; CUDA is the vendor compute API and toolchain those kernels are ultimately emitted against. On NVIDIA hardware, an XLA-compiled model still lowers to PTX and still links against cuBLAS and cuDNN. The vendor coupling sits underneath the compiler, not beside it.
That distinction matters because the naive reading — “adopt XLA, drop the NVIDIA dependency” — leads teams to make a portability decision on the wrong evidence.
Is XLA an alternative to CUDA, or does it sit on top of it?
It sits on top. XLA takes the graph your framework produced, fuses elementwise and reduction chains, picks layouts, and emits device code. On an NVIDIA backend that device code is PTX, with library calls to cuBLAS/cuDNN for the heavy matmul and convolution paths. Nothing about that removes CUDA from the stack; it changes who writes the kernels.
What XLA genuinely changes is the lock-in profile. A graph expressed at the HLO level has real backends beyond NVIDIA — TPU, and increasingly other accelerators. Hand-written CUDA kernels have exactly one. That is a different kind of dependency, and it is the only part of the comparison that is actually a decision.
| XLA | CUDA | |
|---|---|---|
| Layer | Graph compiler (fusion, layout, lowering) | Vendor compute API + toolchain |
| Unit of work | Whole tensor graph / HLO module | Individual kernel |
| On NVIDIA hardware | Emits PTX, calls cuBLAS/cuDNN | Is the target |
| Retargetable | Yes — TPU and other backends | No |
| Custom kernels | Opaque call-outs; survive fusion but not a backend switch | The thing you wrote |
| What it buys you | Fusion coverage, fewer hand-written kernels | Exact control of the inner loop |
When compiler abstraction beats hand-tuned kernels
XLA shines when you need automatic optimization across hardware targets, while CUDA dominates when you control every memory transaction. It is what fraction of your model is expressible at the graph level versus locked in hand-written kernels — because the compiler-addressable fraction is the fraction you do not rewrite when hardware changes. We look at three things when this comes up in practice: how many custom CUDA kernels survive a backend switch (often: none, unmodified), operator-fusion coverage as a percentage of graph nodes, and the resulting change in inference latency and per-inference cost.
Custom kernels are the part teams underestimate. XLA treats them as opaque call-outs — they run, but they are not fused into neighbouring ops, and they do not travel to a different backend. Neither do the memory-layout assumptions baked around them. A compiler can absorb some portability cost; it does not port those for free.
The two failure modes are symmetrical. Over-trust XLA as an exit from NVIDIA and you discover the porting bill mid-migration. Dismiss it entirely and you keep writing kernels a compiler would have generated for you.
Which of the two you are at risk of depends on your workload shape, and that is measurable before you commit. Our broader treatment of GPU performance engineering and where compute-API choices actually bind covers how we scope that inspection — which parts of a workload are compiler-addressable, and which are kernel-locked.
So the question to carry forward is not “XLA or CUDA?” but: if the target hardware changed next quarter, what percentage of this model would a compiler carry across, and who owns the rest?t?
