CUDA cores and tensor cores are not two flavours of the same unit, and comparing their counts on a spec sheet answers almost nothing. CUDA cores execute general-purpose SIMT scalar and vector work — the arbitrary arithmetic, control flow, and memory-bound kernels that make up most of a real pipeline. Tensor cores accelerate a narrow set of fixed-shape matrix-multiply-accumulate operations at reduced or mixed precision, and nothing else.
That narrowness is the whole story. The question is never “how many tensor cores does this GPU have” but “which of my kernels can reach them”.
What does tensor cores vs CUDA cores mean in practice?
It means splitting your workload into two buckets before you quote a single FLOPS number. A kernel reaches the matrix path only if its data layout, tile shapes, and precision policy match what the hardware path expects. In practice that means going through vendor libraries — cuBLAS, cuDNN, or a compiler stack like TensorRT that emits matrix-path kernels for you — or writing vendor-specific intrinsics yourself. Elementwise activation, normalisation, custom loss kernels, most preprocessing, and anything genuinely memory-bound stay on the general SIMT units regardless of how many tensor cores the die carries.
| Property | CUDA cores | Tensor cores |
|---|---|---|
| Work executed | General SIMT scalar/vector arithmetic | Fixed-shape matrix-multiply-accumulate only |
| Typical precision | FP32, FP64, INT32 | FP16, BF16, TF32, FP8, INT8 (mixed accumulate) |
| How you reach it | Ordinary CUDA C++ | Vendor libraries or vendor-specific intrinsics |
| Constraint that blocks you | Occupancy, memory bandwidth | Data layout, tile shape, precision policy |
| Portability | Broadly similar model across vendors | NVIDIA, AMD, and Intel matrix units are exposed differently |
Why achieved throughput falls short of advertised peak
The details of Tensor Cores vs CUDA matter at this point. Teams that quote that number in a capacity plan without confirming which kernels can actually reach the path routinely over-budget performance by an order of magnitude. The honest version of the estimate is per-kernel: which kernels run on general SIMT units at achieved FP32 rates, which reach the matrix path at mixed precision, and what fraction of nominal peak each one actually attains.
There is a second cost that shows up later. Reachability is an API decision, not just a kernel-tuning detail. Because the matrix-core paths on NVIDIA, AMD, and Intel hardware are exposed differently, code written against one does not port performantly to another even where an API translation layer exists — a portability-driven API choice can quietly cost you the matrix path entirely. We see this most often in teams that picked a portable abstraction early and only measured on one vendor.
Getting this split right before hardware is purchased is what gives a defensible answer to “why is our GPU at 20% of its advertised FLOPS”. It also changes accelerator sizing and inference cost per request, which is where the money is. Our wider work on GPU performance engineering goes into how that per-kernel picture is built and what to do with the kernels that can never leave the SIMT units.
The open question for most teams is not which GPU to buy — it is what fraction of their own kernel time is even eligible for the matrix path, and nobody knows that until someone measures it.
