CUDA and cuDNN are not two options you choose between. CUDA is the compute platform — language extensions, driver, runtime, memory model. cuDNN is a closed, hand-tuned library of deep-learning primitives that runs on top of it: convolution, pooling, normalisation, RNN kernels. You do not pick one instead of the other; if you are using cuDNN, you are already using CUDA underneath.
That sounds like a pedantic distinction until someone asks how hard it would be to move the stack off NVIDIA. Then it becomes the whole question.
What does CUDA vs cuDNN mean in practice?
CUDA gives you control. You write the kernel, you decide the memory layout, you decide how work is scheduled across streaming multiprocessors. cuDNN gives you someone else’s answer to a narrow set of problems — the layers that dominate wall-clock time in most vision and sequence models — with tuning you almost certainly could not reproduce by hand in a reasonable timeframe.
The consequence is a second layer of dependency that no API translation layer resolves. A team that has only ever reached cuDNN through PyTorch or TensorFlow has not written portable code. It has written code whose performance lives inside a vendor library.
| CUDA | cuDNN | |
|---|---|---|
| What it is | Compute platform: language extensions, driver, runtime, memory model | Library of deep-learning primitives |
| Source | Toolkit; you write kernels against it | Closed, hand-tuned by NVIDIA |
| Scope | Any parallel computation | Convolution, pooling, normalisation, RNN kernels |
| Who owns performance | Your code | NVIDIA’s kernel implementations |
| Installed | CUDA toolkit | Separately versioned; must match toolkit and framework build |
| Rough non-NVIDIA counterpart | OpenCL, SYCL, ROCm/HIP | MIOpen, oneDNN |
Which layer is your performance actually coming from?
This is answerable, and it should be answered before any portability commitment. Kernel-level profiling separates time spent inside cuDNN primitives from time spent in application-written CUDA kernels and from host-side overhead. We treat that split as the first input to any recommendation about changing API or vendor — it is a step in our GPU engineering work, not an afterthought.
The two halves of the split have very different migration costs. Replacing a handful of custom kernels is a weeks-scale task. Replacing cuDNN-tuned convolution paths with MIOpen or oneDNN equivalents is a benchmarking and re-tuning programme, with a throughput delta to measure per layer class — coverage is not equivalent, and equivalence has to be demonstrated layer by layer rather than assumed. Teams that profile first avoid discovering mid-migration that the “CUDA code” they planned to port was mostly library calls.
There is a smaller, more immediate reason to keep the two straight: cuDNN is versioned and installed separately from the CUDA toolkit, and framework binaries are built against specific combinations. A container that works on one host and fails on another usually fails at that seam, not in the model.
Knowing which layer owns your throughput is a prerequisite for judging whether OpenCL, SYCL, or ROCm equivalents are viable at all. Have you measured the split, or are you assuming it?
