For deep learning on GPU, the short answer is yes: a GPU substantially cuts neural network training time compared with a CPU. A GPU runs thousands of threads in parallel and has much higher memory bandwidth than a CPU in a similar price and power envelope. One comparative study measured GPU training speedups of 11x to 246x over a CPU on vision models, depending on the model. The catch is that the speedup is not automatic. A model moved to a GPU runs only as fast as whichever resource saturates first allows: compute throughput, memory bandwidth or launch overhead. A bigger card does nothing for those limits until someone measures which one is in play.
Why GPUs train neural networks faster than CPUs
CPUs and GPUs are built for different goals. NVIDIA’s CUDA Programming Guide describes it this way: a CPU runs a serial sequence of operations as fast as possible and can handle a few tens of threads in parallel, while a GPU gives up single-thread speed to run thousands of threads at once, with more of its transistors spent on data processing and fewer on caching and flow control. Neural network training is dominated by matrix multiplications and tensor operations, which is exactly the data-parallel work that this design rewards.
Memory bandwidth matters alongside arithmetic. NVIDIA’s deep learning performance guide gives the A100 as an example: 108 streaming multiprocessors, a 40 MB L2 cache and up to 2039 GB/s of bandwidth from 80 GB of HBM2 memory. The “Democratizing AI” comparative study attributes its measured gains mainly to the GPU’s Single Instruction Multiple Threads execution model and its high memory bandwidth. Libraries carry the rest of the load: cuDNN, cuBLAS and CUTLASS are tuned for each GPU architecture, and AI frameworks get much of their acceleration by calling them.
The size of the gain depends on the model. In the same study, a lightweight Conv6 network trained 246x faster on GPU, VGG16 and ResNet18 trained 51x to 116x faster, and the CycleGAN generative model trained 11x faster. The authors note that these results are limited to vision models tested on a single GPU and CPU pair. For inference, the InferBench study measured speedups on a V100 from 3.6x to 47.4x across object detection, GAN, text classification and image classification models. The two studies used different hardware and methods, so they cannot be compared directly. Still, both point the same way: the GPU’s lead is real, and it varies a lot with the workload.
Is a GPU needed for machine learning, and when does it start to pay off?
For deep learning, the answer is yes. The scikit-learn maintainers themselves say that deep learning needs GPUs for efficient computing. For small models the answer is less clear. TensorFlow’s profiling guide warns that offloading to a GPU is not always worth it for small models, because moving data from host to device costs time and so does launching kernels from the host. That guide puts kernel launch latency at around 20 to 40 microseconds. If the host cannot enqueue kernels fast enough, the GPU spends its time waiting on the CPU.
Our view is that a GPU pays off when the work arrives in large, parallel chunks: wide layers, large batches, many training iterations, or inference traffic heavy enough to batch. It pays off less, and sometimes not at all, for tiny models, single requests at low arrival rates, or workloads limited by memory capacity rather than compute.
In Facebook’s study of recommendation model training (HPCA 2021), GPU throughput beat the CPU setup in every configuration tested, but GPU power efficiency was not always best, and it was highest for models with more dense features. One production model got 2.25x the throughput on GPUs and was 4.3x more power efficient. Another reached only 0.85x the throughput of its CPU setup, although it was still 2.8x more power efficient. The same paper notes that most of Facebook’s recommendation models were trained on CPU servers, because their embedding tables need more memory capacity and bandwidth than GPU memory provided. If energy per result is your deciding metric, our energy-efficient GPU for machine learning piece looks at it in more detail.
Training, inference or data preparation: which gains most from deep learning on GPU
The three stages behave differently, so it is worth looking at them separately.
| Workload part | Measured GPU gain (source) | What decides the gain | What usually limits it |
|---|---|---|---|
| Neural network training | 11x to 246x on vision models (“Democratizing AI” study) | Model complexity | Data stalls, memory-bound operations |
| Neural network inference | 3.6x to 47.4x on a V100 across four model types (InferBench) | Model type | Transmission time and under-utilisation at low load |
| Classical ML and data preparation | Vendor charts only, cuML 25.02 on H100 vs Xeon 8480CL (NVIDIA) | Whether a GPU implementation of the estimator exists | Falls back to CPU when it does not |
| Recommendation models with large embeddings | 2.25x for one production model, 0.85x for another (Facebook) | Model configuration: dense versus sparse features | Memory capacity and irregular lookups |
Inference performance depends on batch size and request arrival rate. InferBench found GPUs had lower latency than CPUs at batch sizes under eight, but latency grew much longer at large batch sizes on two of the GPU types. Most energy went into batch size one, which the authors attribute to per-request start-up overhead that larger batches spread out. They also found that, at small batch sizes, network transmission took about as long as inference itself. In inference, the GPU is one part of a larger system whose end-to-end pipeline design determines service performance.
Classical machine learning is a separate case. NVIDIA’s cuML runs scikit-learn-style code on the GPU and falls back to standard CPU scikit-learn when it cannot accelerate an estimator. NVIDIA’s own speedup charts, measured with cuML 25.02 on an H100 against scikit-learn, UMAP and HDBSCAN on a Xeon Platinum 8480CL, compare GPU with CPU training. These are vendor claims, not independent tests. scikit-learn’s own GPU support is narrower: since 2023, a limited but growing list of estimators can run on GPUs when given PyTorch or CuPy arrays through the Array API, and some algorithms are not suited to GPUs at all. Our walkthrough of GPU-powered machine learning with NVIDIA cuML covers that library in more depth.
What PyTorch and TensorFlow need to run a model on a GPU
TensorFlow uses the GPU by default. Its documentation says TensorFlow code and tf.keras models run on a single GPU with no code changes. When an operation has both a CPU and a GPU kernel, TensorFlow puts it on the GPU unless told otherwise. By default, TensorFlow maps nearly all the memory of every visible GPU to the process, to reduce fragmentation.
In PyTorch, you place tensors on a device explicitly. You create a device such as torch.device('cuda') and move tensors and models to it with .to(device) or .cuda(). torch.cuda.is_available() returns True when PyTorch was built with GPU support. In both frameworks, the basic requirements are the same: a GPU driver, a framework build compiled for that GPU’s platform, and the vendor libraries (cuDNN and cuBLAS on NVIDIA) that do the actual work. When a model runs but runs slowly, a CPU-only build or a tensor quietly left on the host is worth ruling out before anything more involved.
Does deep learning on GPU only work on NVIDIA hardware?
No. The same PyTorch code can run on AMD and Intel GPUs with little or no change. PyTorch’s ROCm build intentionally reuses the torch.cuda interfaces, so code addresses AMD GPUs through device('cuda'), and torch.version.hip tells you which backend you have. For Intel GPUs, PyTorch’s documentation describes migration as changing cuda to xpu. It supports training and inference, eager mode and torch.compile, and FP32, BF16, FP16 and automatic mixed precision. AMD’s ROCm documentation covers installing PyTorch, JAX and TensorFlow on Instinct and Radeon GPUs and on Ryzen APUs.
We treat CUDA’s dominance as a starting condition rather than a fixed rule, and the real decision is which compute API a team commits to for a given workload and hardware roadmap. The trade-offs between CUDA, OpenCL, SYCL and HIP as GPU compute APIs are covered on their own page.
Why a model on GPU still runs slower than expected
GPU performance is set by whichever hardware resource saturates first (compute throughput, memory bandwidth or launch overhead), and the right fix depends entirely on which one it is. Guessing is how teams spend days optimising the wrong subsystem.
TechnoLynx’s GPU Performance Engineering work profiles GPU training and inference workloads to find the resource that is actually saturated, whether that is memory bandwidth, the data pipeline, launch overhead or batching, and then fixes it in your code and serving stack.
An operation is memory-limited when its arithmetic intensity (operations per byte accessed) is lower than the processor’s ratio of math bandwidth to memory bandwidth. Reduction operations and layers that perform few operations per byte accessed fall into this category. On Google TPUs, the ParaDnn benchmarking study found memory bandwidth to be the bottleneck for many models, and found that even a heavily optimised compute-bound model like ResNet-50 had 13% memory-bound operations. Those operations need more memory bandwidth or less memory traffic to speed up. Why that happens, and which levers work in each regime, is covered in our page on memory-bound versus compute-bound inference.
The data feed is the next suspect. Mohan et al. (VLDB 2021) found that in many cases DNN training time is dominated by data stalls, meaning time spent waiting for data to be fetched and preprocessed, and that their CoorDL loader cut training time by as much as 5x on a single server compared with DALI. Speeding up the device can create this problem. In the ParaDnn study, on a TPU, switching ResNet-50 from float32 to bfloat16 moved its bottleneck from the device to data infeed, and resolving that infeed limit was worth a further 34%.
Batch size also matters. GPUs need larger mini-batches to use their compute capacity fully, and in Facebook’s measurements training throughput rose roughly linearly with batch size, up to a limit. Larger batches raise arithmetic intensity and, in ParaDnn’s fully connected models, turned memory-bound layers into compute-bound ones. They are not free, though: the same Facebook study saw the accuracy gap grow as batch size increased, even after retuning hyper-parameters.
Finding the bottleneck, in order
These steps are ordered so that each one rules out a cause before you spend effort on the next:
- Time it correctly. The “Democratizing AI” study averaged 100 iterations after a 10-iteration warm-up and called
torch.cuda.synchronize(). - Check whether you are input-bound. TensorFlow’s Profiler includes an input-pipeline analyser for this. A quicker test is to train on random synthetic data: if throughput jumps, the input pipeline is the limit.
- Fix the loader before the model. PyTorch’s
DataLoaderdefaults tonum_workers=0, which loads data synchronously in the main process, andpin_memory=Trueenables faster asynchronous copies from host to GPU. - Profile operators. The PyTorch Profiler records CPU operators and on-device CUDA kernels, reports memory per operator, and has a schedule API so long training loops do not produce huge traces.
- Sort hot operations by what limits them. Memory-bound pointwise chains benefit from kernel fusion. Matrix-heavy work benefits from mixed precision, which PyTorch says can give up to 3x overall speedup on Volta and newer GPUs when tensor dimensions meet Tensor Core requirements.
- Look for launch overhead. When the CPU cannot launch kernels fast enough, CUDA graphs keep the work on the GPU without paying launch and synchronisation costs each time.
- Re-check the input pipeline after lowering precision or adding GPUs.
Framework profilers tell you which operators are expensive. To see why a kernel is slow, turn to our page on profiling GPU kernels with Nsight Systems and Nsight Compute.
Who can find out why our GPU training or inference is slower than it should be?
For production inference under cost or latency pressure, TechnoLynx looks at batching, quantisation, kernel work, runtime choice and serving configuration before suggesting a new model, because those levers sit below the model and can be tuned on their own. If the question is which accelerator to buy, LynxBenchAI is the tool for measuring GPU benchmarks for AI: the AI hardware directory of LynxBenchAI results shows how devices compare under one published methodology, which is a better starting point than spec sheets, though it measures the hardware rather than your own model.
When more GPUs are the right answer
Whether more hardware helps depends on which resource is saturated. Scaling across GPUs is not linear either. TensorFlow’s guide notes that going from one GPU to two on the same host will not give an exact 2x speedup, because gradient communication and host-thread work add overhead. It also says to re-check the input pipeline after scaling up, since one host is now feeding more devices. Before signing a hardware order, the question to answer is which resource the profile says is saturated, and whether the new hardware actually has more of that resource.
