Stable Diffusion VRAM requirements vary dramatically by model. The figure runs from about 4 GB for SD 1.5 to more than 20 GB for the largest models at full precision, and it moves with output resolution, batch size and precision. When the same GPU is also running Unreal Engine 5 with Lumen or the path tracer, you have to add both peaks into one budget, quantise where the asset quality bar allows, and profile the combined load before you write a number into a hardware spec.
LynxBenchAI publishes results showing how each device runs the public PyTorch stack that custom models are built on, so the first step is to check candidate cards there; those results do not measure your scene or pipeline.
TechnoLynx sizes GPU nodes for this mixed workload by measuring both pipelines together on candidate cards.
Stable Diffusion VRAM requirements by model: SD 1.5, SDXL and SD 3.5
There is roughly a tenfold gap between the smallest and largest Stable Diffusion versions. GIGAGPU publishes peak VRAM during generation at each model’s native resolution (512×512 for SD 1.5 and 2.1, 1024×1024 for SDXL and Flux):
| Model | Parameters | FP32 | FP16 | FP8 | NF4 |
|---|---|---|---|---|---|
| SD 1.5 (512×512) | 860M | ~6 GB | ~3.5 GB | ~2.5 GB | ~2 GB |
| SDXL Base (1024×1024) | 3.5B | ~16 GB | ~6.5 GB | ~4.5 GB | ~3.5 GB |
| SDXL Base + Refiner | 6.6B | ~28 GB | ~7.5 GB | ~5 GB | ~4 GB |
| SD 3 Medium | 2B | ~10 GB | ~5.5 GB | ~4 GB | ~3 GB |
| Flux.1 Dev (1024×1024) | 12B | ~48 GB | ~20 GB | ~12 GB | ~8 GB |
The SDXL-plus-refiner row assumes sequential loading, with one model resident at a time, so its peak is close to the base model alone. If you keep the refiner resident, nexgpu puts the extra cost at about 6 GB on top of the roughly 8 GB SDXL needs in fp16.
Other measurements put the same models higher. VirtusLab’s benchmark records SDXL at about 11 GB and SD 1.5 at about 4 GB in fp16. A bestgpuforai snapshot puts SDXL text-to-image at 1024 px at 6.8 GB. We do not read this spread as one source being wrong. Some published figures exclude text encoders and the VAE (Thunder Compute; Stability AI’s SD 3.5 Medium figure), so measure your own pipeline instead of copying a table.
SD 3 and 3.5: the text encoder decides whether 24 GB is enough
SD 3 uses three text encoders, one of them T5-XXL. That encoder has 4.7B parameters and takes about 9.5 GB in fp16. Stability AI quotes SD 3.5 Medium at 9.9 GB of VRAM, but that figure excludes the text encoders. Once you count T5-XXL at fp16, nexgpu says the bill roughly doubles. For SD 3.5 Large, nexgpu reports about 26 GB for the full pipeline at bf16 with fp16 T5-XXL, against 11–12 GB for an fp8 all-in-one checkpoint. Loading T5 at fp8 (about 5 GB) instead of fp16 (about 9.5 GB) is the single choice that decides whether the pipeline fits a 24 GB card.
Hugging Face measured 18.765 GB for SD 3 inference at FP16. In Stability AI’s early, unoptimised tests, the 8B SD 3 model fitted in the 24 GB of an RTX 4090 at 1024×1024.
How do output resolution and batch size change the VRAM a texture pipeline needs?
VRAM grows roughly with the square of resolution, because doubling the edge length quadruples the pixel count. For SDXL at FP16, the figures go from about 6.5 GB at 1024×1024 to about 12 GB at 1536×1536 and about 20 GB at 2048×2048. For high-resolution output, the source recommends tiled upscaling instead of generating at full resolution, to avoid out-of-memory errors.
Batch size adds memory almost linearly. SDXL at 1024×1024 FP16 goes from about 6.5 GB at batch 1 to about 15 GB at batch 4 and about 26 GB at batch 8. On the same 24 GB card, that source fits eight SD 1.5 images at once but only one or two Flux.1 images. A practitioner guide on propelrc gives a rule of thumb: maximum batch size equals available VRAM minus about 2 GB of overhead, divided by the per-image cost. Batch count is different. It runs batches one after another, so it adds time, not memory.
Larger batches stop paying off at some point. Lambda found A100 80GB throughput rose 64% from batch 1 to batch 28 (the largest batch that did not run out of memory), but the gain flattened once the tensor cores were saturated. For a texture team on a shared card, that flattening argues for a small batch at the target resolution over a large one.
How much VRAM do fp16, quantised and distilled variants save, and what do they cost?
Half precision cut VRAM by about 40% against FP32 in Lambda’s test. Lambda measured about 7.7 GB for single-precision inference at batch 1 and about 4.5 GB at half precision. Half precision also cut generation time by about 40% on Ampere GPUs. It is not bit-identical: Lambda saw visible differences from fp32 output, mostly in early steps, though that did not necessarily mean the half-precision output was worse.
Thunder Compute says FP8 checkpoints cut VRAM by roughly 40–50% with minimal quality loss. For SD 3.5 Large, Stability AI reports that its TensorRT build with FP8 quantisation reduces memory from 19 GB to 11 GB on NVIDIA RTX 40 and 50 series GPUs. In Hugging Face’s Quanto test on SD3 Medium, with the transformer quantised to FP8 and all three text encoders quantised too, memory fell to 8.200 GB from 16.403 GB unquantised, and latency rose from 2.118 to 2.858 seconds.
The speed side depends on the architecture. VirtusLab separates weights-only quantisation, which only saves VRAM, from weights-plus-compute formats, which can also speed things up but need newer GPUs. Its fp8 speedups require Ada Lovelace or newer. On the UNet models in the SD family, VirtusLab’s Blackwell run found fp8 roughly breakeven, about 2.5% slower, because Conv2d dominates the compute.
Distillation saves time, not memory. SDXL Turbo, Lightning and LCM-SDXL all sit at about 11 GB in VirtusLab’s table, the same as base SDXL, because they cut denoising steps and keep the backbone. VirtusLab calls fewer steps the most universal optimisation available. On a shared node, fewer steps shortens generation time, but the peak VRAM stays the same.
Dropping T5 entirely is the other lever for SD 3. Stability AI found that removing it left visual aesthetics unchanged (50% win rate) and only slightly reduced text adherence (46%), while typography dropped more (38%). Unless your textures carry lettering, T5 removal is the first thing to test on SD 3-family texture work, checked against your own asset quality bar.
What causes Stable Diffusion out-of-memory errors, and how do teams fix them?
The UNet or transformer uses the most memory during denoising, but VAE decode is a separate peak. Decoding a batch of four images at once raises peak activation memory fourfold.
The Hugging Face Diffusers documentation describes these fixes:
- VAE slicing decodes one image of a batch at a time.
- VAE tiling decodes overlapping tiles. Tone can vary slightly from tile to tile, but there should be no obvious seams.
- Model offloading keeps one main component (text encoder, UNet or VAE) on the GPU at a time. It is faster than fine-grained offload but saves less memory.
- CPU offloading moves submodules back and forth. It saves more memory than model offloading and is extremely slow.
- Sliced attention brought SD down to as little as 3.2 GB in Lambda’s tests, at about 10% slower inference.
Components outside the base model add to the bill. Text encoders, LoRAs and ControlNet push real usage above published minimums, and bestgpuforai says adding ControlNet raises every figure by 2–4 GB. The allocator matters as well. PyTorch’s garbage_collection_threshold setting makes its allocator reclaim unused memory blocks once usage passes a set fraction of capacity. On a shared GPU, that threshold is worth setting deliberately rather than leaving at the default.
Can Stable Diffusion share a GPU with Unreal Engine 5 running Lumen or path tracing?
It can, but only if the combined peak fits with headroom and you accept that the two workloads will slow each other down. The combined figure has to come from summing the two peaks and then profiling the real load.
Epic gives the UE5 side a floor, not a ceiling. Epic recommends 8 GB or more of graphics RAM. Its typical internal development workstation uses an RTX 4080 with 16 GB. Lumen needs DirectX 12 with SM6 and an NVIDIA RTX-2000 series, AMD RX-6000 series or Intel Arc A-series card or newer. Hardware ray tracing gives higher quality than Lumen’s software path but costs more. Its setup cost grows in large scenes with many overlapping meshes, and skinned meshes pay to rebuild acceleration structures every frame. The path tracer caps each dispatch at r.PathTracing.DispatchSize pixels wide and high (default 2048); larger images render in several steps.
Lumen runs on async compute in parallel with other GPU work, and Epic recommends turning that off with r.Lumen.AsyncCompute 0 when you want to measure Lumen on its own. Stat GPU shows the timing of each Lumen pass. TechnoLynx measures peak VRAM and throughput with both workloads running together on candidate cards before sizing anything.
Worked example: summing the budget (assumptions stated)
Assume a workstation previewing a Lumen scene while generating SDXL textures at 1024×1024 in FP16:
- UE5 floor: 8 GB, from Epic’s recommended specification. This is a planning floor, not a measured peak for your scene.
- SDXL at batch 1: about 6.5 GB. At batch 4: about 15 GB.
- Sums: 8 + 6.5 = 14.5 GB at batch 1. At batch 4, 8 + 15 = 23 GB, which leaves about 1 GB on a 24 GB card.
- Add allowances for what the tables leave out: ControlNet at 2–4 GB per ControlNet added, and a 4K desktop at 200–500 MB (both from bestgpuforai).
With those numbers, batch 1 fits on a 24 GB card with room to spare and batch 4 does not. A Lumen scene needing more than Epic’s floor changes the answer, which is why the final number comes from profiling and not from this sum.
When should texture generation run sequentially on shared render nodes instead of on a separate pool?
Running jobs one after another on the same nodes is the safe default when the hardware cannot enforce a memory limit. In Kubernetes, NVIDIA’s GPU Operator time-slicing has no memory or fault isolation between replicas. Each pod can run as many processes as it likes, and the GPU only shares time between them. MIG, by contrast, partitions the GPU into instances with memory and fault isolation in hardware. MPS lets kernels from different processes run at the same time, and on Ampere and newer GPUs it supports static SM partitions for isolation between clients. MPS v3 can also partition device memory. Under that scheme, an allocation that would exceed a client’s hard limit fails with an out-of-memory error, and a client over its soft limit becomes a candidate for eviction under pressure.
MPS and MIG manage CUDA processes, and a UE5 renderer is a DirectX 12 graphics workload. So on nodes that mix the two, these tools mainly constrain the SD side, and the practical control over UE5 is scheduling. That leads to this decision table:
| Situation | Arrangement |
|---|---|
| Combined profiled peak fits with headroom, and render deadlines tolerate slower frames | Run concurrently on one GPU, with SD capped via batch size, VAE tiling and allocator threshold |
| Combined peak is near the card limit, or render latency is contractual | Run sequentially on shared nodes: generation between render jobs |
| Peaks of the two workloads do not fit one SKU, or both run around the clock | Separate node pools by workload type |
| Several SD jobs share data-centre GPUs that support MIG | Partition with MIG; use time-slicing only where losing isolation is acceptable |
Keeping a render queue within its latency targets while other workloads share the pool is a separate scheduling problem from sizing VRAM, and it needs its own admission control.
What VRAM figure should a studio specify per GPU node for both workloads?
For an artist workstation doing concurrent SD texture generation and UE5 preview, 24 GB is the card class to shortlist, on the condition that SD runs at batch 1–2, at around 1024×1024, with fp16 or fp8 weights. That condition rests on the sum above. If the spec instead says SD 3.5 Large with fp16 T5 resident next to a Lumen scene, 24 GB is not enough. That pipeline alone is about 26 GB, so you either quantise the encoder or run the two workloads sequentially.
For a render-node fleet that also runs another AI workload (an in-betweening model, for example), the logic is the same. Profile each workload’s peak, standardise on one SKU if the peaks fit together, and split into pools if they do not. Three caveats apply to any figure you write down. It moves when production raises resolution or batch size. A quantised variant has to pass your asset quality review before it counts toward the budget. And a measurement on one scene or texture set may not hold across your whole content library.
Who can size and validate a GPU spec for our Stable Diffusion and UE5 pipeline?
TechnoLynx works out the VRAM per node or workstation for studios running Stable Diffusion texture generation next to Unreal Engine 5 rendering, and validates it on real hardware. TechnoLynx delivers a VRAM-per-node or per-workstation spec for both workloads, validated by measuring peak VRAM and throughput on candidate cards running them together, with quantised and distilled SD variants checked against your quality bar. For the hardware shortlist, LynxBenchAI publishes device results on the public PyTorch stack. The engineering side is described under GPU Performance Engineering.
This is not the right engagement if you only need to buy one hobby card. If the open question is price-performance across specific cards, start with published, like-for-like device results before you commission any sizing work.
FAQ
Is 8GB of VRAM acceptable?
For SD 1.5, and for SDXL in FP16 at batch 1, 8 GB is enough by GIGAGPU’s figures. Thunder Compute says FP8 and GGUF quantisation bring SD 3.5 within reach of 8–12 GB cards. Running UE5 at the same time needs more, since Epic recommends 8 GB or more for the engine alone.
