A spec sheet lists CUDA cores and tensor cores side by side, which invites the reading that they are two flavours of the same unit. They are not. CUDA cores execute general-purpose scalar and vector work; tensor cores are fixed-function matrix-multiply-accumulate units that engage only when a kernel expresses its work as tiled GEMM at a supported precision. If your kernel never takes that code path, the advertised tensor throughput is simply unavailable to you — and nothing in the output tells you so.
What does tensor vs CUDA cores mean in practice?
The practical difference is conditional availability. A CUDA core will run whatever you give it: control flow, pointer arithmetic, elementwise activations, reductions, the awkward middle of a preprocessing pipeline. A tensor core will run one shape of computation — a matrix multiply with an accumulate — and only at the precisions its generation supports. Reaching it means the compiler or library has emitted the matrix instruction (via cuBLAS, cuDNN, CUTLASS, TensorRT, or a wmma/mma intrinsic in hand-written code), with tiles and alignment that fit the unit.
When that does not happen, the failure is silent. A mixed-precision GEMM that falls back to general CUDA cores produces correct numbers at a fraction of the available matrix throughput. The deficit shows up as unchanged latency, not as an error — which is exactly why teams discover it after the hardware purchase rather than before.
Where the two units diverge
| CUDA cores | Tensor cores | |
|---|---|---|
| Work shape | General scalar/vector instructions | Tiled matrix multiply-accumulate only |
| Precision | FP32/FP64/INT, broadly | Restricted set per generation (e.g. FP16/BF16/FP8/INT8 accumulate paths) |
| How reached | Ordinary CUDA C++ | Vendor libraries (cuBLAS, cuDNN, CUTLASS, TensorRT) or mma intrinsics |
| Failure mode when unreachable | n/a | Silent fallback: correct results, lost throughput |
| What to measure | SM occupancy, warp stalls | Tensor-pipe utilisation, achieved TFLOPs vs peak |
The one number worth extracting is achieved arithmetic throughput as a fraction of specification for the matrix-heavy portion of the kernel. Nsight Compute reports tensor-pipe active cycles per instruction; if that figure is near zero on a workload you believe is GEMM-dominated, the matrix units are idle and your problem is not the GPU. This is one of the measurement points we lean on during a GPU performance audit, because it separates a genuinely compute-bound workload from one that is merely missing the hardware it paid for.
Why core counts are not a portability signal
Tensor-core paths are reached through the same vendor libraries and intrinsics that carry the rest of the CUDA stack’s lock-in profile. Portable APIs expose matrix units unevenly: AMD and Intel accelerators have equivalent hardware, but a kernel written against a portable API does not automatically land on it, and a translation layer that preserves semantics does not preserve which functional unit executes the maths. Reading core counts as a portability signal is the same category error as assuming API translation preserves performance — a point we develop further in our work on GPU engineering and acceleration.
So the useful question before a hardware or API commitment is not how many of each core a device has. It is what fraction of your specific kernel’s arithmetic currently lands on matrix units — and whether anyone has measured it yet.
