CUDA and TensorFlow are not alternatives. CUDA is NVIDIA’s GPU compute API and toolchain; TensorFlow is a machine-learning framework that calls down into CUDA — or ROCm, or oneAPI, or XLA-generated kernels — to reach the hardware. Asking which one to pick is a category error, and the reason it matters is that the error hides the decision that actually carries cost.
Where does CUDA sit relative to TensorFlow?
Read the stack from the model downward. TensorFlow expresses the graph. Below it, a backend lowers that graph to kernels — cuDNN and cuBLAS for the common convolution and matmul paths, XLA when fusion is enabled, hand-written kernels where someone wrote them. Below that sits CUDA, and below CUDA the GPU driver.
| Layer | What it decides | Portable? |
|---|---|---|
| TensorFlow / model code | How the model is expressed | Mostly yes |
| Backend libraries (cuDNN, cuBLAS, XLA) | Which kernels run for standard ops | Vendor-mapped, usually swappable |
| CUDA kernels you wrote | Fused ops, custom layers, memory layout | No — rewrite required |
| Driver / hardware | Execution | n/a |
So “we use TensorFlow, therefore we use CUDA” is a defensible default on NVIDIA hardware — but it is a default, not a conclusion. The same model code often runs on AMD via ROCm builds or on Intel accelerators with minimal change. What does not travel is the hand-written layer.
The number worth having
The decision to make traceable is simple: document the framework choice and the compute-API choice separately, then enumerate the custom kernels and vendor-specific code paths in the repository. That enumeration answers the question a hardware-diversification plan actually needs — how much code would need rewriting to run somewhere other than NVIDIA. In our experience the split is lopsided: a small fraction of framework-level code, and the majority of hand-written kernel code. Without that count, lock-in cost stays a feeling rather than a figure.
Teams usually discover the boundary at a specific moment — a custom operator, a fused attention kernel, a memory-layout trick tuned for one architecture. That is where the compute API stops being invisible plumbing and starts constraining where the workload can run. Everything above it stayed portable; the thing someone hand-optimised did not.
Which decision comes first?
Framework first, in nearly every case. The framework governs how quickly the model can be built and changed, and it is the cheaper decision to reverse. The compute API becomes a real decision only when the framework stops giving you the performance or the hardware flexibility you need — and at that point it should be recorded as its own choice, with its own owner and its own migration cost, rather than inherited silently from the framework.
When we look at where inference spend actually leaks on GPU workloads, this separation is usually the first thing missing. Our GPU engineering work starts by mapping which layer a given performance problem lives in, because optimising the wrong layer is how teams spend a quarter for a few percent.
The open question for most teams is not CUDA or TensorFlow — it is how many lines of their codebase they could not move tomorrow, and whether anyone has ever counted them.
