TensorRT does not compete with CUDA. CUDA is the general-purpose GPU compute platform and programming model; TensorRT is an inference optimisation and runtime library that sits on top of it. Framed as an either/or, the question has no answer — framed as two decisions at two layers, it has a clear one.
Is TensorRT an alternative to CUDA, or does it run on top of it?
It runs on top. When TensorRT builds an engine, it is still emitting CUDA work: fused layers, autotuned kernel selections, and precision-calibrated paths that execute through the same driver and runtime any hand-written kernel would use. Adopting TensorRT does not remove CUDA from the stack, and it does not remove CUDA-level work from the project either — custom operators the builder cannot fuse, plus pre- and post-processing around the model, usually remain yours to write.
The failure we see most often is a team that never states which layer it is deciding at. One version of that mistake is hand-writing kernels for a standard convolutional or transformer topology that a runtime would have fused and quantised automatically. The other is adopting the runtime and assuming nothing below it needs attention. Both are the same error with opposite signs.
Which layer holds your bottleneck?
| Layer | What you decide | What it buys | What it costs |
|---|---|---|---|
| Compute platform (CUDA) | Which GPU compute API the system targets | Full control of kernels, memory, and custom ops | Narrows the hardware roadmap to NVIDIA; engineer-days per kernel |
| Inference runtime (TensorRT) | Whether a vendor runtime sits above that API | Layer fusion, FP16/INT8 calibration, kernel autotuning without writing kernels | Engine files tied to NVIDIA hardware and, in practice, to specific architecture and version combinations |
| Custom CUDA above the runtime | Which operators the runtime cannot handle | Coverage for unfused or novel ops | Justified only where the runtime demonstrably leaves throughput on the table |
The divergence point is portability cost
TensorRT vs CUDA stops being abstract in this section. Adopting TensorRT deepens that commitment rather than adding a separate one: a built engine is not a portable artifact, and rebuilds are expected when GPU generation, driver, or runtime version changes. That re-engineering cost belongs in the portability estimate at the point of decision, not in the incident report eighteen months later.
There are two numbers worth putting against this. The first is inference cost per thousand requests before and after runtime optimisation. The second is engineer-days spent on kernel-level work that a runtime would have covered anyway. In our experience, the second number is the one nobody measures, and it is usually the larger surprise. Our wider view of GPU acceleration and inference optimisation engineering covers how we separate those two budgets in practice.
If cross-vendor portability is a hard requirement, the runtime-layer question does not disappear — it moves. Something has to play TensorRT’s role in a non-NVIDIA stack, and the honest version of that comparison is a runtime-versus-runtime one, not a runtime-versus-platform one.
Before committing inference infrastructure spend, the question to settle is not which of the two to pick, but which layer your throughput is actually being lost at.
