A tensor core and a CUDA core are not two speeds of the same thing. They are different execution units with different instruction shapes, and a kernel only reaches the tensor pipe if its arithmetic is expressed as suitably shaped, aligned, and typed matrix-multiply-accumulate work. Core counts on a spec sheet do not tell you whether your code will ever issue an instruction to that silicon.
CUDA cores execute general scalar and vector FP32 and INT operations — the ordinary arithmetic of element-wise kernels, reductions, indexing, and branch-heavy control flow. Tensor cores execute fixed-shape matrix-multiply-accumulate at reduced or mixed precision. That is the whole distinction, and everything practical follows from it.
What decides whether a kernel reaches tensor cores?
Workload composition determines which silicon executes your operations. Tensor-core instructions are emitted through NVIDIA’s own paths: cuBLAS and cuDNN, CUTLASS, the WMMA/MMA intrinsics, or a compiler that lowers to them — for example a PyTorch model whose GEMMs and convolutions dispatch into cuBLAS, or a TensorRT engine built with mixed precision enabled. A hand-written element-wise CUDA kernel does not get there by accident.
The consequence is blunt: teams that buy tensor-core-heavy hardware for element-wise, branch-heavy, or memory-bound kernels pay for silicon their code never issues instructions to. We see this most often in preprocessing and postprocessing stages that dominate wall-clock time in a vision pipeline while the GEMM-heavy part of the model is already fast.
| CUDA core | Tensor core | |
|---|---|---|
| Operation shape | General scalar / vector FP32, INT | Fixed-shape matrix multiply-accumulate |
| Typical precision | FP32, FP64, INT32 | FP16, BF16, FP8, INT8 (mixed accumulate) |
| Reached by | Any CUDA kernel you write | cuBLAS, cuDNN, CUTLASS, WMMA/MMA intrinsics, or a compiler that emits them |
| Ineligible work | — | Element-wise, branch-heavy, memory-bound, unaligned or odd-shaped arithmetic |
| Portable equivalent | Broadly maps to OpenCL / SYCL / Vulkan compute | Vendor-specific matrix units; no portable equivalent that preserves the fast path |
Verify it from the profiler, not the datasheet
Partition each job by operation type, then measure how many FLOPs land on tensor hardware and how many fall back to CUDA cores. That is readable from profiler counters — tensor pipe utilisation, achieved occupancy, memory throughput — rather than from vendor figures. Checking it before a hardware refresh is what stops a team from buying an accelerator tier whose tensor throughput its kernels cannot reach, and from attributing a quoted 5–10x speedup to a workload that is memory-bound (an observed pattern across our GPU performance work, not a benchmarked rate).
It also prices the portability question honestly. Tensor-core paths are exposed through NVIDIA-only libraries and intrinsics, so however much of your measured speedup depends on them is exactly what an OpenCL or SYCL port would have to reconstruct — usually alongside CUDA-specific memory and layout patterns that do not carry over performantly either. Our GPU engineering practice treats that number as part of the API decision rather than a footnote to it; the same trade-off is laid out in choosing between CUDA, OpenCL and SYCL.
So before comparing two GPUs by tensor-core count, ask a narrower question: which of your kernels currently issue tensor-core instructions at all, and what would it take to make the rest eligible?
