Vulkan and CUDA in llama.cpp look like two build flags on the same binary. They are two different bets. CUDA gives you a mature kernel path — fused attention, tuned quantised matmul — on NVIDIA hardware only. Vulkan gives you one build that runs on AMD, Intel, Apple and integrated GPUs, at a lower optimisation ceiling. Picking whichever compiles first is how teams end up buying NVIDIA cards they did not need, or shipping a portable build that quietly halves throughput on the one GPU class they actually deploy on.
Why does the same GGUF file run at different speeds on the two backends?
Because the API name is not the divergence point — the memory and dispatch model underneath is. The same quantised GGUF weights are consumed by hand-tuned CUDA kernels on one path and by more generic compute shaders on the other. That is a structural difference, not a tuning gap you can close with a flag.
Two consequences worth writing down before you benchmark:
- The gap is workload- and quantisation-specific, not a fixed percentage. Any number you have read as “Vulkan is X% slower” is a measurement of somebody else’s model, quant level and GPU, and does not transfer to yours.
- Prompt processing usually diverges more sharply than token generation. Prefill is compute-bound and leans hardest on the tuned matmul and attention kernels; generation is more memory-bound, so the backends sit closer together there. If you only measure tokens/sec on a short prompt, you will measure the flattering half of the comparison.
The comparison that matters
| Axis | CUDA backend | Vulkan backend |
|---|---|---|
| Hardware reach | NVIDIA only | AMD, Intel, Apple, integrated GPUs, plus NVIDIA |
| Kernel maturity | Fused attention, tuned quantised matmul | More generic compute shaders, lower ceiling |
| Build artifacts | One build per target stack | One binary across vendors |
| Where the gap shows first | — | Prompt-processing throughput |
| Procurement effect | May force NVIDIA purchase | Can serve inference on GPUs you already own |
Benchmark it in an afternoon
Vulkan vs CUDA llama stops being abstract in this section. Record, per backend and per quantisation level, on the exact GPUs in your fleet:
- Tokens/sec at generation, with a realistic prompt length — not a two-token warm-up.
- Prompt-processing throughput separately from generation.
- Time-to-first-token.
- Peak VRAM per configuration, and the number of offloaded layers that produced it.
The output is a documented backend choice with recorded numbers attached, not a default inherited from a README. In our experience this is the artifact that survives the next hardware refresh, because the reasoning is legible to whoever inherits it.
Vulkan is the correct production choice — not a fallback — when your deployment target genuinely spans vendors, or when the measured gap on your model is smaller than the cost of standardising the fleet on NVIDIA. That is a judgement about numbers you own, and it is exactly the kind of question our GPU engineering work starts from.
The wider version of this question is the same one: portability against optimisation ceiling, whether the API pair is Vulkan and CUDA or something else entirely. llama.cpp just makes it cheap to measure. What would your prompt-processing numbers have to look like before you changed the procurement plan?
