TensorFlow and CUDA are not two options on the same shortlist. TensorFlow is a machine-learning framework that expresses a model and emits work to a device backend; CUDA is one of the compute APIs that backend can target. Asking which one to “choose” is a layer question phrased as an either/or, and the naive answer — pick the more familiar name — quietly bundles two separate commitments into one: how the model is written, and whose silicon executes the kernels.
That bundling is where the cost shows up later.
What does “TensorFlow vs CUDA” actually decide?
Split the path from model code to executed kernel and the two decisions land in different places.
| Layer | Decides | Vendor-bound? |
|---|---|---|
| TensorFlow (framework) | How the model is expressed: graph, ops, autodiff, data pipeline, training loop | No — framework-level code is nominally portable |
| Device backend / runtime | Which compute API the emitted work targets (CUDA, ROCm, oneAPI, XLA targets) | Yes — this is the vendor decision |
| Kernel libraries (cuDNN, cuBLAS) | How a convolution or GEMM is actually implemented | Yes |
| Hand-written custom ops | Memory layout, block/thread structure, intrinsics | Strongly — this is the hardest layer to move |
TensorFlow can run without CUDA. ROCm builds serve AMD hardware and oneAPI-oriented paths serve Intel accelerators, and high-level graphs do execute on them. What does not automatically travel is performance. In our experience the framework layer ports and the bottom two rows do not, so a team that reports “we’re portable, it’s all TensorFlow” is usually describing the top row of that table and nothing below it.
Where the lock-in actually lives
The dependency is not the string cuda in a build file. It is the accumulated set of assumptions a codebase has made because CUDA is the only backend it has ever run on: memory layouts tuned for one cache hierarchy, custom ops written against warp-level primitives, mixed-precision paths that assume a particular Tensor Core code path, input pipelines shaped around a specific host-to-device transfer pattern. Those assumptions do not raise a compile error on another backend. They just run slower, and nobody budgeted for that.
This is the same structural point the GPU engineering practice makes about compute API choice more generally: the lock-in lives in memory and kernel patterns, not in the API name at the top of the file.
The inventory to run before a hardware decision
Before committing to non-NVIDIA hardware, three numbers make the migration priceable in engineer-weeks rather than guesswork:
- Count of custom ops or kernels containing vendor-specific code — including anything calling cuDNN, cuBLAS, or warp intrinsics directly.
- Share of training and inference wall-clock time spent in those paths — a CUDA-bound op that consumes 2% of step time is a footnote; one that consumes 40% is the project.
- Throughput delta when the same graph runs on a non-CUDA backend — measured, on a representative batch and sequence length, not inferred from spec sheets.
Teams that hold this inventory can quote a backend migration. Teams that do not typically discover their CUDA-bound fraction after the purchase order has cleared.
Writing CUDA directly is still the right call sometimes — when a fused operation has no framework-level equivalent and sits on the critical path. That is a deliberate trade of portability for throughput, and it is defensible as long as it is recorded as such rather than accumulated by accident.
So the useful version of the question is not which to choose. It is: what fraction of this workload would we have to rewrite, and do we know that number today?
