LLM inference optimization divides into two fundamentally different categories: algorithmic changes that reduce the amount of work being done, and kernel-level changes that make the existing work run faster on hardware. Confusing the two leads to wasted effort — a finely tuned kernel for an inefficient algorithm rarely outperforms a mediocre implementation of a better algorithm. This article covers the most impactful techniques in both categories, when to apply each, and what results to reasonably expect. The framing matters because most of the prototype-to-production gap for generative AI systems is really an inference-cost gap, and the order in which you apply these optimizations determines whether you ever close it. Algorithmic Optimizations These changes reduce the computational or memory cost of inference independent of hardware execution efficiency. What does quantization actually buy you? Quantization reduces model weights (and optionally activations) from FP32 or FP16 to lower-precision formats: INT8, INT4, or FP8. The key tradeoffs: INT8 weight-only quantization (W8A16): Weights stored as INT8, dequantized to FP16 for matmuls. Reduces memory footprint by roughly 2x with typically under 1% accuracy degradation on most models (observed pattern across deployments we have profiled, not a model-agnostic guarantee). Common in llama.cpp, GPTQ, AWQ. INT4 weight-only (W4A16): Around 4x memory reduction versus FP16. Accuracy impact varies by model and quantization method; GPTQ and AWQ handle this better than naive rounding. INT8 activation quantization (W8A8): Both weights and activations quantized. Enables INT8 matmuls via CUTLASS or cuBLAS INT8 paths. Faster than W8A16 on Turing+ (SM 7.5+), but requires careful calibration. FP8: Available on Hopper (SM 9.0). Near-FP16 accuracy with roughly 2x throughput on tensor cores. The Transformer Engine in H100s handles FP8 scaling automatically. KV cache management Autoregressive generation caches key and value tensors from previously generated tokens. KV cache size grows linearly with sequence length and batch size, and is frequently the binding memory constraint for long-context inference. The techniques worth knowing: Paged attention (vLLM): Allocates KV cache in fixed-size pages rather than contiguous buffers, eliminating fragmentation and enabling much higher concurrent batch sizes. Multi-Query Attention (MQA) and Grouped Query Attention (GQA): Architecture-level changes (applied at training time) that reduce KV heads, shrinking KV cache by 4–8x with minimal quality impact. Llama 3, Mistral, and Gemma models use GQA. KV cache quantization: Store cached KV tensors in INT8 instead of FP16, halving KV memory at a small accuracy cost. Speculative decoding Speculative decoding uses a small draft model to propose several tokens in parallel, which the large target model verifies in a single forward pass. When draft tokens are accepted, you get multiple output tokens per target-model call. It is effective when: The target model is large (70B+) and memory-bound during decoding. The draft model is small enough that its cost is negligible. Token acceptance rate is high (above roughly 60%), which requires draft and target models to be aligned in distribution. In our experience, speculative decoding delivers a 2–3x decoding throughput improvement for appropriate model pairs and workloads — observed pattern from production GenAI systems we have helped move past prototype, not a universal benchmark. Kernel-Level Optimizations These change how the computation executes on hardware without changing the algorithm’s mathematical output. FlashAttention FlashAttention rewrites the attention computation — softmax over QK^T followed by weighted sum over V — as a single fused kernel that keeps intermediate results in SRAM (shared memory) rather than writing them to HBM. The result: Memory: O(n) instead of O(n²) HBM usage for attention intermediates. Speed: roughly 2–4x faster than standard attention for long sequences on A100 (benchmark figures reported by the FlashAttention authors; sensitive to sequence length and head dimension). Correctness: Mathematically equivalent to standard attention (uses online softmax normalization). FlashAttention v2 and v3 extend this with improved parallelism across sequence length and better utilization on Hopper hardware. FlashAttention is the single most impactful kernel-level optimization for transformer inference and training, which is why “always on” is the right default once your stack supports it. Fused kernels and operator fusion Standard inference implementations launch separate kernels for each operation: LayerNorm, GEMM, activation function, another GEMM. Each kernel reads from and writes to HBM. Fusing operations keeps intermediate data in registers or shared memory: Fused LayerNorm + linear projection Fused activation (gelu/silu) + gate multiplication (for gated MLPs) Fused attention (FlashAttention) Fused residual add + normalization torch.compile with the inductor backend performs some of this fusion automatically. For more aggressive fusion, custom CUDA kernels or Triton kernels are required. Continuous batching Not a kernel optimization per se, but a scheduling change that dramatically improves GPU utilization. Rather than waiting for all sequences in a batch to finish before starting new ones (static batching), continuous batching inserts new requests as soon as a slot becomes free. This keeps the GPU busy and improves throughput at the cost of per-request latency variance — a tradeoff worth committing to explicitly before a system goes to production traffic. Optimization decision framework Optimization Primary Benefit When to Apply INT8/INT4 quantization Reduced memory, higher batch size Memory-bound inference; almost always worth evaluating FP8 (Hopper) ~2x matmul throughput H100 hardware, available via TensorRT-LLM KV cache paging Higher concurrency Long context or high concurrent request count GQA / MQA 4–8x KV cache reduction Model training or selection phase Speculative decoding 2–3x decoding throughput Large target model, latency-sensitive workload FlashAttention 2–4x attention-kernel speed Always, once the runtime supports it Operator fusion Reduced memory bandwidth Complex custom inference pipelines Continuous batching Higher GPU utilization Serving with variable-length requests Interaction with algorithmic restructuring Our companion piece on when algorithmic restructuring beats kernel tuning addresses the broader decision of when to change algorithms versus when to optimize execution. For LLM inference specifically: algorithmic changes (quantization, KV management, speculative decoding) typically deliver larger improvements than kernel tuning, because the bottleneck is usually memory bandwidth from loading large weight matrices rather than arithmetic throughput. Kernel optimization matters most after algorithmic changes have been applied. This sequencing is the same logic that governs the broader prototype-to-production transition for generative AI — the failure modes we discuss in what it takes to move a generative AI prototype into production almost always start with cost and latency assumptions that ignored the algorithmic layer. Wrapping up LLM inference optimization starts with algorithmic changes: quantization reduces weight memory and enables faster matmuls, KV cache management increases concurrent capacity, and speculative decoding improves decoding throughput. Kernel-level optimizations — FlashAttention, fused operations — then reduce the execution cost of the remaining computation. Applying kernel optimizations before algorithmic ones is a common mistake that produces modest gains while leaving larger wins on the table. FAQ Where do GenAI prototypes typically break when promoted from notebook to production traffic? Inference cost and tail latency. A prototype runs single requests through an FP16 model on a developer GPU; production has to sustain concurrent traffic, which is what forces the algorithmic stack — quantization, KV cache paging, continuous batching — into the design. What latency, cost, and reliability targets should I commit to before promoting a prototype? Commit to a target tokens-per-second per concurrent user, a per-request cost ceiling, and a p95 (not mean) latency budget. Without those numbers, you cannot pick between FlashAttention-and-go versus quantization plus speculative decoding plus continuous batching. How do I monitor a production GenAI system for hallucination, drift, and edge cases the prototype never saw? Track per-request quality signals — answer-acceptance rate, fallback rate, retrieval-hit rate for RAG — alongside the usual latency and throughput. Inference optimizations such as quantization can shift output distributions slightly, so quality monitoring has to run continuously after each optimization step, not just at deployment.
