“CUDA vs PyTorch” is not a choice between two options. CUDA is a GPU compute API and toolchain; PyTorch is a deep-learning framework that calls into CUDA — and into cuDNN, cuBLAS, and increasingly vendor-neutral backends — on your behalf. The question turns up in procurement documents and hiring briefs as if it were a fork in the road, and that framing quietly costs money later.
The reason it costs money is specific. Teams that read the two names as alternatives tend to conclude that picking PyTorch settles the vendor question. It does not. Your framework choice governs developer ergonomics; the compute API underneath governs which hardware you can run on and how high your optimisation ceiling goes.
Where does PyTorch end and CUDA begin?
The split is cleaner than most discussions suggest.
| Layer | What it owns | Who writes it |
|---|---|---|
| PyTorch (framework) | Model definition, autograd, data loading, distributed orchestration, device placement | Your ML engineers, in Python |
| Framework backend / libraries | Kernel dispatch, cuDNN and cuBLAS calls, graph capture, ONNX or TensorRT export paths | Mostly vendor-supplied, occasionally configured |
| CUDA (compute API + toolchain) | Kernel launches, memory hierarchy, streams, occupancy, synchronisation | You, only when you drop below the framework |
Most PyTorch code never touches the third row. That is the point of a framework. The divergence arrives when a model needs a custom operator, a non-NVIDIA target, or latency below what the default dispatch path delivers — at that moment you are making the compute-API decision you thought you had avoided. We see this surface during hardware migrations far more often than during design reviews.
Does PyTorch protect you from lock-in?
Partially, and the partial part is where the estimate lives. Backend-agnostic PyTorch — layers, autograd, training loops — moves to ROCm or other backends with modest effort. Hand-written CUDA kernels and CUDA-shaped memory management do not port performantly, because the memory access patterns encode assumptions about one vendor’s hierarchy.
So the useful artefact is a ratio, not an opinion. A team that can say “94% of our model code is backend-agnostic PyTorch, 6% is custom CUDA covering three hot operators” can price an AMD or Intel evaluation. A team that cannot state the ratio ends up refusing the evaluation on unquantified risk — which looks like a technical decision and is actually a missing measurement.
Quick answers, if you need them in a meeting:
- Do I need to write CUDA to work in PyTorch? No, until a profiler says a specific operator is the bottleneck and no fused equivalent exists.
- What triggers a custom kernel? An op with no efficient library implementation, a fusion opportunity the compiler misses, or a latency target the default path cannot hit.
- What should I ask someone who “knows CUDA” versus “knows PyTorch”? Ask the first about occupancy and memory coalescing; ask the second about dispatch, graph capture, and export paths. They are different jobs, and conflating them in a job spec produces the wrong hire.
Our GPU engineering practice treats the framework/kernel boundary as the first thing to establish on any performance or portability question, because everything downstream — rewrite cost, achievable latency, which hardware is genuinely on the table — is a consequence of where that line falls.
The open question for most teams is not which side to stand on. It is whether anyone has actually measured the ratio, or whether it is still an assumption everyone is working from.
