When a large language model costs too much to serve or answers too slowly, the fix is often below the model layer, and swapping the model is not the first move. Two kinds of lever recur throughout this article. Algorithmic levers change what is computed: fewer bits per weight, fewer serial forward passes, a smaller KV cache. Kernel-level levers change how the same maths runs on the hardware: fused attention, fewer trips to memory, less launch overhead. Which family pays off depends on whether a step is compute-bound prefill or memory-bound decode, and on batch size. That is why the first job is to profile the serving path you actually run, not to pick techniques off a list.
Algorithmic and kernel-level LLM inference optimization are different jobs
Surveys split the field in slightly different ways. The “LLM Inference Unveiled” survey uses four groups: model compression, fast-decoding algorithms, compiler and system-level optimisation, and hardware-level optimisation. A survey of efficient foundation model design uses three: model design, system design and model-system co-design. It also treats low-level GPU optimisation as its own way to get more out of the hardware. For a team running a model in production, a three-way split is easier to act on:
| Family | What it changes | Typical techniques | Does it change outputs? |
|---|---|---|---|
| Algorithmic | The numbers or the order of the computation | Quantisation, speculative decoding, KV cache compression, sparsity, distillation | Quantisation and compression can; speculative decoding is designed not to |
| Kernel-level | How identical maths moves through GPU memory and compute units | Operator fusion, FlashAttention, Flash-Decoding, CUDA Graphs, torch.compile | No: the same result with fewer bytes moved or fewer launches |
| Serving-level | When requests are scheduled and where their memory lives | Continuous batching, PagedAttention, chunked prefill | No |
In practice the line between these families is blurry. The AWQ authors shipped their method with TinyChat, which uses kernel fusion and platform-aware weight packing, and report more than 3x speedup over the Hugging Face FP16 implementation.
All three families sit below the model. Replacing the model is a fourth lever, and the most expensive one. It can also leave the economics of the workload unchanged.
Prefill is compute-bound and decode is memory-bound
Every request runs in two phases. Prefill processes the whole prompt in parallel and produces the first token. Decode then generates the remaining tokens one at a time. The roofline analysis in the “LLM Inference Unveiled” survey looks at Llama-2-7B on an NVIDIA A6000 at sequence length 2048 and batch size 1. During prefill, the projection layers reach an arithmetic intensity of 1,024 to 1,215 operations per byte and are compute-bound, while the attention matmuls sit at 114 and are memory-bound. In decode, every layer in that analysis is memory-bound, so the GPU runs far below its compute capacity.
The reason is mechanical. Prefill uses GEMM operators over many tokens at once. Decode uses GEMV or lean GEMM operators that do little arithmetic for each byte of weights they load. The KernelSight-LM authors (2026) measured GEMM at about 77% of prefill time, with its share dropping sharply once decode starts and attention taking over at long context. Databricks’ engineering guidance makes the same point from the hardware side: at small batch sizes, achieved memory bandwidth predicts token generation speed better than peak compute does.
This split decides which technique helps:
- In a memory-bound phase, quantisation helps because each step moves fewer bytes. Once a layer is already compute-bound, compressing its weights further brings no speedup; only faster low-bit compute helps there.
- Kernel fusion raises arithmetic intensity, which helps memory-bound operators and does little for compute-bound ones.
- Speculative decoding spends spare compute to cut serial passes, which only works while compute is spare.
The regime is not a fixed property of the model. With large batches, compressing weights from 4 bits down to 2 or 1 bits stops reducing inference time because the network is already compute-bound. Prefill on short sequences can be memory-bound and move towards compute-bound as sequences get longer. The SPEED-Bench authors (2026) describe the same shift: as concurrency rises, systems move from memory-bound to compute-bound, which changes the cost-benefit of any technique that trades compute for fewer memory passes.
Which LLM inference optimization should a team apply first?
Profile first. Choose a technique afterwards, because best practices only make sense in context, and that context comes from profiling. A study of energy use in inference found that the effect of optimisations depends heavily on workload geometry, software stack and hardware. In its case study of NLP tasks and real-world LLM workloads, applying them properly cut total energy use by up to 73%, but they cannot be applied uniformly. The same study measured torch.compile giving a 29.90% improvement on an A100, 13.28% on an RTX 6000 Ada and 1.96% on an A6000.
TechnoLynx audits the production AI stack for cost and latency improvements before recommending a model replacement.
Once the profile shows where time goes, work in this order:
- Measure the serving path end to end. Record time to first token, time per output token and throughput at realistic concurrency, with your real prompt and output lengths. Databricks recommends always measuring end-to-end server performance, because software inefficiencies and hardware differences produce surprises the theory does not predict.
- Fix scheduling and KV cache memory if they are still naive. Moving from static batching in plain PyTorch to an engine with continuous batching and paged KV cache shows large measured multiples against naive baselines.
- Quantise to match the serving mode. Use weight-only 4-bit (W4A16) for synchronous setups and W8A8 for asynchronous continuous batching.
- Look at the attention kernels. FlashAttention-3 reports gains on H100s, Flash-Decoding at long sequences and FlashInfer against the Triton backend.
- Add speculative decoding only if traffic runs at small batch sizes, since its gains fade at large batches.
- Consider a different model last, since replacing it is the most expensive lever.
Decision table: lever, regime, evidence
| Lever | Family | Helps most when | Little or negative effect when | Published evidence |
|---|---|---|---|---|
| Continuous batching | Serving | Many concurrent requests with variable output lengths | Low-variance output lengths, where static batching performs about the same | 8x over naive batching (Anyscale); 36.9x over FasterTransformer at equal latency for Orca (FriendliAI) |
| PagedAttention | Serving / memory | Longer sequences, larger models, complex decoding | : | 2-4x throughput at the same latency for vLLM |
| Chunked prefill | Serving | Prefills stall ongoing decodes under load | : | 2.6x serving capacity over vLLM, Mistral-7B on one A100 (Sarathi-Serve) |
| Weight-only 4-bit (W4A16) | Algorithmic | Synchronous, decode-bound serving | Large batches that are already compute-bound | 2-3x lower cost per query on 8B and 70B models |
| W8A8 (FP8 or INT8) | Algorithmic | Asynchronous continuous batching; prefill-heavy loads | Synchronous setups, where W4A16 is more cost-efficient | FP8 effectively lossless; INT8 loses 1-3% |
| KV cache quantisation | Algorithmic | Very long contexts | Short contexts, where weights dominate memory | KV cache dominates memory beyond about 50k tokens |
| Speculative decoding | Algorithmic | Batch of 16 or less; low-entropy domains such as code and maths | Large batches | 25.65% more energy than autoregressive decoding at batch 128 |
| Fused attention kernels | Kernel | Memory-bound attention; long context decode | Compute-bound prefill | Up to 8x decode speedup at very long sequences (Flash-Decoding) |
| CUDA Graphs, torch.compile | Kernel | Small batches; faster GPUs | Large batches, where the CUDA Graphs launch saving matters less | 29.90% improvement on A100 from compilation |
How quantisation, KV cache management and speculative decoding cut cost
Quantisation formats: FP8, INT8 and W4A16
Quantisation trades a little precision to cut cost in two ways: each decode step moves fewer bytes, and the model fits on fewer or cheaper GPUs. The “Give Me BF16 or Give Me Death” study, which calls itself the most comprehensive to date, ran over 500,000 evaluations across the Llama-3.1 family. It found FP8 weights and activations (W8A8-FP) effectively lossless, well-tuned INT8 (W8A8-INT) at 1-3% degradation, and 4-bit weight-only INT4 (W4A16) competitive with 8-bit. On performance, W4A16 cut cost per query by 2-3x and improved latency by 1.5-2.5x against BF16 for the 8B and 70B models. At 405B it reached 5-7x cost reduction and needed fewer GPUs.
The best format depends on model size, hardware and serving mode. Weight quantisation mainly speeds up decode by reducing memory traffic, while quantising weights and activations together improves compute efficiency in prefill. The authors therefore recommend W4A16 for synchronous setups and W8A8 for asynchronous continuous batching. Their measurements used vLLM 0.6.4.post1 on A6000, A100 and H100 GPUs, so the exact ratios belong to that stack.
GPTQ, a post-training quantisation method, quantised weights to 3 or 4 bits with negligible accuracy loss and gave about 3.25x end-to-end speedup over FP16 on an A100 and 4.5x on an A6000. AWQ showed that protecting only 1% of salient weights, chosen from activation statistics, greatly reduces quantisation error. Databricks reports that moving from 16-bit to 8-bit weights can halve the number of GPUs needed in memory-constrained setups such as Llama2-70B on A100s.
The caveats concern tuning and configuration. INT8 in the BF16 study caused noticeable drops at 70B until SmoothQuant and better calibration data were applied, and the smallest models lost more at INT4. We go deeper on why memory bandwidth decides LLM quantization gains and where accuracy breaks in a separate piece.
KV cache management
The KV cache stores the keys and values of earlier tokens so they are not recomputed at each step, and its memory grows linearly with batch size and sequence length. NVIDIA’s formula is batch size × sequence length × 2 × layers × hidden size × bytes per FP16 value. For Llama 2 7B in 16-bit precision at batch 1 and 4,096 tokens, that comes to about 2 GB. Multi-query and grouped-query attention shrink this footprint without changing accuracy.
On the serving side, KV cache memory management and quantisation both matter. PagedAttention addresses fragmentation and is covered under batching below. KV cache quantisation targets size: the roofline survey’s analysis shows the KV cache taking most of the memory once sequences pass about 50k tokens, which is where quantising it makes a large difference. We compare KV cache quantization and weight quantization in a separate piece.
Speculative decoding
Speculative decoding lets a cheap draft model propose several tokens, which the target model then verifies in one parallel pass. Leviathan et al. showed it can make exact decoding faster without changing the output distribution, with 2-3x acceleration on T5-XXL and identical outputs. It works because decode is memory-bound: verifying several tokens at once loads the weights once, not once per token.
The 2025 energy study found speculative decoding saved energy at batch sizes of 16 or less but used 25.65% more energy than plain autoregressive decoding at batch 128. SPEED-Bench confirms that larger batches push inference towards compute-bound and shrink speculative gains, sometimes into slowdowns. Domain matters too: coding and maths prompts produce longer accepted runs than creative writing or roleplay, and N-gram speculation caused net slowdowns in some settings. So does the engine. SPEED-Bench found TensorRT-LLM reached higher peak throughput with one CUDA graph covering the whole draft-and-verify loop, while vLLM’s multi-engine design had some host overhead but more flexibility in drafting strategy.
What FlashAttention and Flash-Decoding change in LLM inference performance
FlashAttention computes exact attention and uses tiling to cut reads and writes between GPU high-bandwidth memory and on-chip SRAM. It never writes the full attention matrix to HBM. The output is the same as standard attention. What changes is the number of memory reads and writes. The roofline survey shows that in decode, fusing attention cuts inference time by as much as it cuts memory access. In prefill the time saving is smaller, because parts of prefill are compute-bound.
Each version targets a different bottleneck:
- FlashAttention-2 reached only about 35% utilisation on an H100, against 80-90% for optimised GEMM kernels.
- FlashAttention-3 uses Hopper’s asynchronous Tensor Cores and TMA. It is 1.5-2.0x faster than FlashAttention-2 and reaches up to 740 TFLOPs/s in FP16 (75% of the H100’s theoretical peak) and close to 1.2 PFLOPs/s in FP8. Its FP8 path, with block quantisation and incoherent processing, is 2.6x more accurate than standard per-tensor quantisation when features have outliers.
- Flash-Decoding fixes a decode-specific problem. FlashAttention parallelises over batch and query length, and in decode the query length is typically 1, so at batch size 1 it can use less than 1% of the GPU. Flash-Decoding also splits the keys and values across the sequence. The PyTorch team reports up to 8x faster generation at very long sequences, with the attention itself up to 50x faster.
- FlashInfer v0.2 reuses the FlashAttention kernel skeleton in customisable templates, with block-sparse KV cache formats and a load-balancing scheduler. Against the Triton backend it reports 29-69% lower inter-token latency on a serving benchmark and 28-30% lower latency for long-context inference.
FlashAttention already ships inside vLLM, TGI and TensorRT-LLM. The useful question is which kernel the engine actually dispatches on your GPU generation. The KernelSight-LM authors found attention efficiency to be highly device-specific and to depend on which kernel the engine picks, unlike GEMM, which carries across devices predictably.
How continuous batching in engines like vLLM improves GPU throughput
Static batching keeps a batch fixed throughout inference. When output lengths vary, idle compute accumulates after early requests finish. Continuous batching swaps finished requests for new ones at every iteration. The idea comes from Orca’s iteration-level scheduling. Orca’s research page, published by FriendliAI, reports 36.9x throughput over NVIDIA FasterTransformer at the same latency on GPT-3 175B. An Anyscale benchmark measured 8x throughput over naive batching from continuous batching alone, and up to 23x when combined with vLLM’s memory optimisations. Databricks puts the gain over dynamic batching at 10-20x.
These multiples are against different baselines, models and traffic, so they show the size of the effect rather than a number to plan around. Continuous batching improves median latency as well as throughput, because new requests join a running batch straight away. That advantage fades as the server saturates and free slots get scarce.
PagedAttention attacks the memory limit that caps batch size. The vLLM authors found that KV cache memory is large, grows and shrinks dynamically, and is wasted through fragmentation and duplication when managed naively. PagedAttention splits each sequence’s KV cache into fixed-size blocks and maps them through a block table, much as virtual memory works in an operating system, so memory is allocated as needed instead of reserving the maximum context length up front. vLLM reports near-zero KV cache waste and 2-4x throughput at the same latency compared with FasterTransformer and Orca, with larger gains for longer sequences and larger models. TensorRT-LLM also offers in-flight batching, another name for continuous batching, and paged KV caching.
Request latency rises with batch size even as throughput grows. On one A100, Databricks measured that running at batch size 64 raised throughput by 14x and latency by 4x. Chunked prefill eases that trade-off by splitting prefills into chunks so new requests do not stall ongoing decodes. Sarathi-Serve reports 2.6x higher serving capacity than vLLM for Mistral-7B on a single A100 and up to 5.6x for Falcon-180B with pipeline parallelism. How that trade-off plays out in conversational products is covered in our piece on chatbot serving latency.
Sizing GPU hardware for an optimised LLM deployment: worked example
Assumptions: Llama-3-8B, a single NVIDIA L40, FP16 weights, a 4,096-token context per request.
Decode speed from weight bytes over memory bandwidth. Decode is memory-bound, so time per output token ≈ bytes of weights moved ÷ memory bandwidth = (8B parameters × 2 bytes) ÷ 864 GB/s ≈ 18.5 ms.
Prefill from FLOPs over compute. Prefill is treated as compute-bound: (8B × 2) FLOP ÷ 181 TFLOP/s ≈ 0.088 ms per prompt token. A 4,000-token prompt with a 256-token answer therefore takes about 4,000 × 0.088 ms + 256 × 18.5 ms ≈ 5.1 s.
KV cache per sequence. For a grouped-query model like Llama-3-8B, KV cache per token = 2 × 2 bytes × 32 layers × 8 KV heads × 128 head dimension = 128 KB. A 4,096-token sequence therefore needs 128 KB × 4,096 = 512 MiB.
Apply the optimisation. With FP8 or INT8 weights, use 1 byte per parameter in the formulas. The weights drop to about 8 GB, and the Step 1 formula gives 8 GB ÷ 864 GB/s ≈ 9.3 ms per token. The roughly 8 GB freed holds about 14 more 4,096-token sequences at 512 MiB each.
Treat the result as a best case. VMware warns that communication, network delay and software overhead can add 50% or more to these estimates. Batching works because all sequences in a batch share one load of the weights, until the batch grows large enough to become compute-bound.
The formula gives a best-case estimate for shortlisting GPUs. It is not a commitment. Even a dedicated kernel-level simulator predicting from datasheets alone has a 12.1% per-kernel error on an unseen GPU generation, falling to 3.8% after one microbenchmark sweep on the target. Databricks has also observed a 2x latency difference between 8xA100 servers from two cloud providers. So confirm the shortlist by measuring end-to-end server performance on the actual hardware.
What latency, cost and reliability targets should you commit to before promoting a prototype?
Set targets for time to first token, time per output token, latency and throughput only after characterising your real input and output lengths. Databricks recommends against fixing a number like “under 20 ms per token” before that characterisation. For chat, VMware’s guidance suggests a time to first token under 200 ms and at least 30 tokens per second, which is about 33.3 ms between tokens when streaming.
The quantisation study above reports throughput as queries per second and queries per US dollar.
The KernelSight-LM authors note that latency-SLO-oriented deployments typically operate at 60-80% of sustained capacity, as queues build near saturation. We would set latency targets at a percentile, measured at the load you expect at peak, and plan capacity so peak traffic stays inside that band.
Where GenAI prototypes break between the notebook and production traffic
Production traffic adds concurrency, long inputs and varied output lengths. Concurrency moves decode from memory-bound towards compute-bound, so a technique that sped up the demo, speculative decoding above all, can turn into a slowdown. Long inputs limit how large a batch can be, and varied output lengths waste slots in static batches. Benchmarks built from random tokens overestimate speculative-decoding throughput by an average of 23%, and random inputs can push a mixture-of-experts router onto a narrow set of experts, which changes step times even without speculation. Prompts longer than the draft model’s training length degrade its accuracy quickly.
A pruning choice that held up on one domain can degrade others. SPEED-Bench found vocabulary pruning cost little on maths and coding but degraded multilingual, RAG and summarisation inputs significantly. Any change in precision or drafting should rerun the quality evaluation on a sample of real traffic before rollout.
Auditing the serving path before replacing the model
The levers below the model are batching, caching, routing, quantisation, kernel work, serving configuration and runtime choice, each tunable on its own. TechnoLynx’s wider generative AI work is described under Generative & Agentic AI R&D.
The question to settle first is which regime your traffic lives in. A synchronous, low-concurrency product is decode-bound, so weight-only quantisation and speculative decoding pay off. A busy multi-tenant API heads towards compute-bound, where continuous batching, paged memory and W8A8 do the work and speculation can hurt. The same model can sit in either regime depending on how it is served.
