PyTorch and CUDA are not competing options. PyTorch is a tensor and autograd framework; CUDA is NVIDIA’s compute API and toolkit that PyTorch’s GPU backend is built on top of. When you run a model on an NVIDIA card in PyTorch, you are using both — the question is never which one to pick, it is which layer a given problem lives in.
That distinction matters because it decides where optimisation effort goes. A model that is slow because of dataloader stalls or graph breaks will not get faster if you rewrite an attention kernel. A model that is bound by occupancy, memory movement, or precision choices will not get faster if you tune batch size and worker counts.
Which layer owns what?
| Concern | Layer | Where you fix it |
|---|---|---|
| Autograd, module composition, optimiser step | PyTorch | Model and training code |
| Dataloader throughput, host-to-device copies, graph breaks | PyTorch / runtime | DataLoader, pinned memory, torch.compile |
| Operator dispatch, kernel selection, memory allocator behaviour | PyTorch ↔ CUDA boundary | Backend flags, cuDNN/cuBLAS settings, allocator config |
| Kernel occupancy, shared-memory use, warp divergence | CUDA | Custom kernels, Triton, TensorRT |
| Numerical precision (FP32 / TF32 / BF16 / FP8) | Both | Autocast policy above, kernel support below |
The practical diagnostic is ordinary profiling, not intuition. If the GPU timeline shows long idle gaps between kernel launches, the bottleneck is above CUDA. If the timeline is dense but individual kernels are slow relative to the memory bandwidth they consume, the bottleneck is below PyTorch. We use that split as the first cut in a GPU performance review because it removes whole classes of speculative rewrite before anyone touches a kernel.
Does PyTorch tie you to NVIDIA?
Partly, and not where most teams assume. Application code written against torch.Tensor is largely portable — ROCm and XPU backends run the same model code. The lock-in accumulates in what sits underneath: custom CUDA kernels, TensorRT engines, NCCL-specific collective assumptions, and any operator whose fast path only exists on one vendor’s stack. Framework choice, not application code, is often what actually pins a project to a vendor, which is why an honest inventory of “which parts of our stack are NVIDIA-dependent” is worth keeping before any hardware-diversity decision is costed.
torch.compile shifts the boundary rather than removing it. Generated kernels mean fewer hand-written CUDA files, but the compiled path still has to be reasoned about at the kernel level when it underperforms — you inspect what was generated instead of what you wrote.
When a custom kernel is actually justified
Write one when the operation is fused-able in a way no existing operator expresses, the profiler shows that operation dominating step time, and the shape regime is stable enough that the kernel will not be invalidated next quarter. Outside those three conditions, composing existing operators and letting the compiler fuse them is the cheaper bet.
We go deeper into the vendor-lock consequences of that choice, and how it interacts with the CUDA versus OpenCL versus SYCL question, in our GPU acceleration engineering work.
The open question for most teams is not which API to standardise on. It is whether anyone has written down which layer their last three performance wins came from.
