“CUDA vs Tensor Cores” is a category error, and it costs money at purchase time. CUDA is a programming model. CUDA cores are general-purpose SIMT lanes that run whatever you compile. Tensor Cores are fixed-function matrix-multiply-accumulate units on the same die, reached through that same CUDA stack — not an alternative to it. The two core counts on a spec sheet are not competing options, and picking the GPU with the larger number answers a question nobody asked.
The question worth asking is narrower: which unit does your kernel actually saturate?
Which parts of a pipeline run on Tensor Cores?
Dense GEMM and convolution layers in reduced precision map onto Tensor Cores. Almost everything else does not. Elementwise ops, activations, normalisation with awkward shapes, sparse gather/scatter, branch-heavy control code, and the whole image- and text-preprocessing stage stay on the general-purpose lanes regardless of how many Tensor Cores the die carries.
That split is why a Tensor-Core-heavy SKU can deliver almost nothing on a real pipeline. If decode, resize, and colour conversion dominate wall-clock time, the matrix units sit idle and you have paid for silicon that never engages.
Tensor Cores only engage when the hardware’s constraints are met, and there are three of them:
| Condition | What it means in practice |
|---|---|
| Precision (dtype) | FP16, BF16, FP8, or TF32 paths. An FP32 kernel with no TF32 or mixed-precision path does not touch the matrix units. |
| Tile shape | GEMM dimensions must decompose into the hardware’s fixed MMA tile sizes. Odd channel counts and unpadded sequence lengths fall back to the SIMT path. |
| Memory alignment | Operand layout must satisfy alignment requirements; misaligned or awkwardly strided tensors force a slower fallback kernel. |
This is the same memory-pattern dependency that makes CUDA-specific code fail to port performantly to other accelerators — the performance lives in layout assumptions, not in the API name.
Measure before you buy
Datasheet TFLOPS describes a precision and a tile shape, not your workload. Three profiler numbers settle the argument:
- Tensor Core utilisation percentage from profiler counters (Nsight Compute exposes this per-kernel).
- Fraction of kernel time in matrix ops versus elementwise and preprocessing kernels.
- Effective TFLOPS against the datasheet figure for the precision you actually run.
In our experience, teams that run this check before purchase land on one of two outcomes: either a mixed-precision path recovers multiples of current throughput on hardware they already own, or the pipeline turns out to be bound by non-matrix work and a cheaper SKU hits the same latency target. Both change the per-inference cost line, not just the benchmark chart. (Observed across TechnoLynx engagements; not a published benchmark.)
The corollary is that CUDA-core throughput, memory bandwidth, and PCIe or NVLink topology should drive the SKU decision whenever matrix time is a minority of the profile. Tensor Core count is the right headline number only for models whose time really is spent in dense reduced-precision GEMM.
AMD’s Matrix Cores and Intel’s XMX units occupy the same architectural role, so the hardware concept ports even where the API does not. What does not port for free is the tile-shape and alignment tuning underneath — which is the part teams discover late.
We go deeper into profiling and kernel-level throughput work in our GPU acceleration and optimisation practice.
If you cannot say what percentage of your kernel time is matrix work, what exactly is the larger Tensor Core count buying you?
