A tensor core and a CUDA core are not two speeds of the same thing. They are different execution units with different instruction shapes, and a kernel only reaches the tensor pipe if its arithmetic is expressed as suitably shaped, aligned, and typed matrix-multiply-accumulate work. Core counts on a spec sheet do not tell you whether your code will ever issue an instruction to that silicon.
CUDA cores execute general scalar and vector FP32 and INT operations — the ordinary arithmetic of element-wise kernels, reductions, indexing, and branch-heavy control flow. Tensor cores execute fixed-shape matrix-multiply-accumulate at reduced or mixed precision. That is the whole distinction, and everything practical follows from it.
What decides whether a kernel reaches tensor cores?
Eligibility, not hardware tier. Tensor-core instructions are emitted through NVIDIA’s own paths: cuBLAS and cuDNN, CUTLASS, the WMMA/MMA intrinsics, or a compiler that lowers to them — for example a PyTorch model whose GEMMs and convolutions dispatch into cuBLAS, or a TensorRT engine built with mixed precision enabled. A hand-written element-wise CUDA kernel does not get there by accident.
The consequence is blunt: teams that buy tensor-core-heavy hardware for element-wise, branch-heavy, or memory-bound kernels pay for silicon their code never issues instructions to. We see this most often in preprocessing and postprocessing stages that dominate wall-clock time in a vision pipeline while the GEMM-heavy part of the model is already fast.
| CUDA core | Tensor core | |
|---|---|---|
| Operation shape | General scalar / vector FP32, INT | Fixed-shape matrix multiply-accumulate |
| Typical precision | FP32, FP64, INT32 | FP16, BF16, FP8, INT8 (mixed accumulate) |
| Reached by | Any CUDA kernel you write | cuBLAS, cuDNN, CUTLASS, WMMA/MMA intrinsics, or a compiler that emits them |
| Ineligible work | — | Element-wise, branch-heavy, memory-bound, unaligned or odd-shaped arithmetic |
| Portable equivalent | Broadly maps to OpenCL / SYCL / Vulkan compute | Vendor-specific matrix units; no portable equivalent that preserves the fast path |
Verify it from the profiler, not the datasheet
The measurable question is what share of a workload’s FLOPs actually execute on tensor cores versus CUDA cores. That is readable from profiler counters — tensor pipe utilisation, achieved occupancy, memory throughput — rather than from vendor figures. Checking it before a hardware refresh is what stops a team from buying an accelerator tier whose tensor throughput its kernels cannot reach, and from attributing a quoted 5–10x speedup to a workload that is memory-bound (an observed pattern across our GPU performance work, not a benchmarked rate).
It also prices the portability question honestly. Tensor-core paths are exposed through NVIDIA-only libraries and intrinsics, so however much of your measured speedup depends on them is exactly what an OpenCL or SYCL port would have to reconstruct — usually alongside CUDA-specific memory and layout patterns that do not carry over performantly either. Our GPU engineering practice treats that number as part of the API decision rather than a footnote to it; the same trade-off is laid out in choosing between CUDA, OpenCL and SYCL.
So before comparing two GPUs by tensor-core count, ask a narrower question: which of your kernels currently issue tensor-core instructions at all, and what would it take to make the rest eligible?
