The question is not which llama.cpp backend is faster. It is whether you are deploying onto a fleet you control or onto hardware you ship to — because that, not a tokens-per-second screenshot from someone else’s machine, is what decides the answer.
CUDA buys you a mature, heavily tuned kernel path on NVIDIA silicon: quantised matmul and attention kernels that have had years of attention from people optimising them. Vulkan buys you one binary that runs across NVIDIA, AMD, Intel and integrated GPUs, at a lower optimisation ceiling. Both statements are true at the same time, which is why a single number cannot settle the argument.
What does llama.cpp Vulkan vs CUDA actually decide?
It decides your build and support matrix as much as your throughput. On a fixed NVIDIA fleet, the CUDA backend’s tuned kernels are hard to match and the lock-in cost is theoretical — you were never going to run on anything else. On mixed or customer-owned hardware, the Vulkan backend removes a per-vendor build-and-support matrix that, in our experience, usually costs more engineering time than the throughput it gives up.
The trap is treating a published figure as portable. Teams that measure their own workload typically find the Vulkan-versus-CUDA gap varies by an order of magnitude in either direction depending on model size, quantisation format and GPU generation (observed pattern across GPU engagements, not a benchmarked rate). That is not noise around a true value. It means there is no single true value to quote.
Which backend, given what
| Your situation | Backend that usually wins | Why |
|---|---|---|
| Fixed NVIDIA fleet you own | CUDA | Tuned quantised matmul and attention kernels; no portability requirement to pay for |
| Shipping software to customer hardware | Vulkan | One binary, no per-vendor build and support targets |
| AMD, Intel or integrated GPUs in the mix | Vulkan | Broad vendor coverage without a second code path |
| Undecided, procurement pending | Measure first | The gap is workload-specific; a guess here sets hardware spend for years |
What to measure before you commit
Three numbers, on your own prompts, at your intended context length and batch size:
- Tokens per second, sustained — not a first-run burst.
- GPU memory headroom for the quantisation format you actually intend to ship (Q4_K_M behaves differently from Q8_0).
- The count of build and support targets each choice obliges you to maintain.
A one-day measurement on real prompts replaces a guess. That is the same discipline we apply in a GPU performance audit: if the current backend is structurally wrong for the deployment target, the numbers say so plainly.
One caution on switching later. Memory-layout and quantisation assumptions written against one backend do not transfer performantly just because the API translates — the call compiles, the throughput does not follow. Budget for re-tuning, not recompiling.
So the open question for your team is not which is faster, but: do you know your own tokens-per-second figure at your own context length yet, or are you still quoting someone else’s?
