A spec sheet gives you two numbers, and neither one tells you how fast your code will run. CUDA cores execute general-purpose scalar and vector SIMT instructions. Tensor cores execute fixed-shape matrix multiply-accumulate operations at reduced or mixed precision. A kernel only reaches the tensor path when its data layout, tile shape, and precision all permit it — otherwise the tensor cores sit idle while the CUDA-core path does all the work.
That is the whole distinction, and it is structural rather than quantitative. Adding tensor cores to a die does nothing for a branch-heavy simulation or a memory-bound preprocessing kernel. A well-tiled GEMM or an attention block, on the other hand, can be almost entirely dominated by them.
Which of my workloads can actually use tensor cores?
The useful question is not “how many of each does this card have” but “which unit does each of my kernels occupy”. Workload shape decides it.
| Workload shape | Unit it actually occupies | Why |
|---|---|---|
| Well-tiled dense GEMM, convolution, attention block | Tensor cores, if precision and tile shape align | Maps directly onto fixed-shape matrix multiply-accumulate |
| Elementwise preprocessing, normalisation, data movement | CUDA cores, bandwidth-limited | Arithmetic intensity too low for matrix units to matter |
| Branch-heavy simulation, irregular control flow | CUDA cores only | No matrix-shaped work to feed the tensor path |
| FP32-required numerical code | CUDA cores | Tensor path depends on reduced or mixed precision |
Reaching the tensor path in practice means going through vendor-specific primitives — WMMA intrinsics, cuBLAS and cuDNN, or CUTLASS templates. Portable compute APIs expose matrix units unevenly or not at all, so a workload whose performance depends on tensor cores has made a lock-in decision, not merely a hardware one. That trade-off deserves to be priced explicitly: how much of your measured speedup is only reachable through vendor-specific paths?s?s?
Why Workload Type Dictates Core Priority
Matrix multiplication workloads see 10–15× speed gains from Tensor Cores, while traditional compute tasks remain bound by CUDA Core throughput. Achieved FLOPs against both the tensor-core and the FP32 peak. Tensor-core utilisation as a percentage. Arithmetic intensity versus available memory bandwidth. Together they tell you whether a mixed-precision rewrite would convert idle matrix capacity into real throughput, or whether the kernel is bandwidth-limited and a bigger card returns nothing.
We see the same pattern regularly in profiling work: teams commit to hardware on core counts before establishing which execution unit their kernels occupy, then discover they are paying for silicon their code cannot address. Our GPU engineering and optimisation work starts from per-kernel profiling for exactly this reason — the CUDA-vs-tensor distinction is only actionable once you know where each kernel actually spends its time.
One caveat worth keeping in view: mixed and reduced precision is not free. Moving a kernel onto the tensor path changes its numerical behaviour, and whether that matters depends on the tolerance of the model or simulation downstream, which is a question profiling alone will not answer.
