The Adam optimizer combines momentum and RMSprop in one update rule. It keeps a running average of each parameter’s gradients (the first moment, which acts as momentum) and a running average of their squares (the second moment, which is how RMSprop scales the step). It then divides one by the square root of the other, so every parameter gets its own effective learning rate. Bias correction fixes the fact that both averages start at zero, and the published defaults are 0.9 for beta1, 0.999 for beta2 and 1e-8 for epsilon. Picking an optimizer has less to do with which one is “best” and more to do with which one fits your architecture, your tuning budget and your memory. AdamW is the safe default for transformers, well-tuned SGD with momentum still competes on convolutional image models, and the newer optimizers (Lion, Muon) earn a trial in some settings rather than a blanket switch.
How the Adam optimizer combines momentum and RMSprop
Kingma and Ba proposed Adam in 2014, and it has since become one of the most widely used optimisation algorithms in deep learning. Its name, adaptive moment estimation, describes the mechanism: the method builds adaptive learning rates from estimates of the lower-order moments of the gradient.
Momentum adds a fraction of the previous update to the current one. That speeds progress along directions where gradients keep pointing the same way and damps oscillation where they flip sign. RMSprop, as Hinton’s lecture notes put it, divides the learning rate for a weight by a running average of the magnitudes of recent gradients for that weight. Hinton’s lecture states the goal as moving quickly in directions with small but consistent gradients and slowly in directions with large but inconsistent ones.
Here is the Adam optimizer formula, step by step, for gradient g at step t:
m_t = β1 · m_(t−1) + (1 − β1) · g_t # first moment: mean of gradients
v_t = β2 · v_(t−1) + (1 − β2) · g_t² # second moment: uncentred variance
m̂_t = m_t / (1 − β1^t) # bias-corrected first moment
v̂_t = v_t / (1 − β2^t) # bias-corrected second moment
θ_(t+1) = θ_t − η · m̂_t / (√v̂_t + ε)
The first two lines are exponentially decaying averages, with m_t estimating the mean of the gradients and v_t their uncentred variance. Both averages are initialised as zero vectors, so they are biased towards zero early in training, especially when beta1 and beta2 are close to 1. The division by (1 − β^t) counteracts that bias. The final line applies the corrected estimates in the same way RMSprop and Adadelta apply theirs.
Worked example: why the first Adam step has a fixed size
Take the default betas and a single parameter whose first gradient is g. After one step, m_1 = 0.1·g and v_1 = 0.001·g², which would shrink the update badly if used raw. Bias correction divides m_1 by 1 − 0.9 = 0.1 and v_1 by 1 − 0.999 = 0.001, recovering g and g². The update becomes η·g / (|g| + ε), which is roughly ±η whatever the gradient’s scale. A 2019 study of warmup for adaptive optimisation confirms that at the first iteration every Adam parameter update is either −α or +α when ε is zero. That fixed first step matters later, when we get to warmup.
What beta1, beta2 and epsilon control
Beta1 and beta2 set the decay rates of the two moving averages. A beta1 of 0.9 matches the value that Ruder’s 2016 overview notes the momentum term is usually set to. A beta2 of 0.999 gives the variance estimate a much longer memory than beta1 gives the mean. Epsilon is a term added to the denominator for numerical stability, defaulting to 1e-8 in PyTorch.
The defaults are a starting point, not a law. The AlgoPerf authors (Dahl et al., 2023) point to evidence that tuning Adam’s beta2 and epsilon can significantly improve performance compared with leaving them at their defaults.
RMSprop vs Adam: what the extra pieces buy
Adam adds two things to RMSprop: momentum and bias correction. A 2022 survey of deep learning optimisers notes that RMSprop, lacking the correction factor, has a high bias in the early stages of training, while Adam is robust to the choice of hyperparameters. On the theory side, a 2025 analysis shows that on locally well-behaved problems Adam converges at the faster rate of momentum gradient descent, while RMSprop only matches plain gradient descent. Kingma and Ba also reported that bias correction helps Adam slightly outperform RMSprop towards the end of optimisation, as gradients become sparser. RMSprop is still a reasonable choice where it is already tuned and working, but for a new project there is little reason to prefer it over the Adam family.
Setting up the Adam optimizer in PyTorch
In PyTorch, Adam and AdamW live in torch.optim, and every optimizer follows the same three-step loop. optimizer.zero_grad() resets the gradients (they accumulate by default), loss.backward() deposits new gradients for each parameter, and optimizer.step() applies the update.
import torch
model = MyModel().cuda()
optimizer = torch.optim.AdamW(
model.parameters(),
lr=1e-3,
betas=(0.9, 0.999),
eps=1e-8,
weight_decay=1e-2,
amsgrad=False,
fused=True, # fused CUDA implementation
)
warmup_steps = 2000 # 2 / (1 - 0.999), see the learning-rate section
scheduler = torch.optim.lr_scheduler.LambdaLR(
optimizer, lambda step: min(1.0, (step + 1) / warmup_steps)
)
for inputs, targets in loader:
optimizer.zero_grad()
loss = loss_fn(model(inputs), targets)
loss.backward()
optimizer.step()
scheduler.step()
A few parameters deserve a sentence each. The weight_decay argument on torch.optim.Adam is an L2 penalty and defaults to 0, while on torch.optim.AdamW it is a decoupled coefficient that defaults to 1e-2. The amsgrad flag switches on the AMSGrad variant from the paper “On the Convergence of Adam and Beyond”, and is off by default. PyTorch’s documentation tabulates the implementations Adam and AdamW support, including foreach, with their stability status. Choosing between these implementations is part of performance engineering for scalable deep learning systems.
Adam’s memory cost is the other setup consideration. Optimizers such as RMSprop and Adam keep first and second moment estimates per parameter, which triples the required memory. Because of that tripling, optimizer state belongs in any exercise of planning GPU memory for deep learning training. Adafactor cuts that auxiliary memory drastically without hurting performance compared with full accumulators.
What learning rate should I start Adam with?
The studies below used Adam at about 1e-3. The 2022 optimiser survey trained its comparison models with initial learning rates of 0.1 for SGD, 0.001 for Adam and 0.01 for AdamW. The 2019 warmup study’s Adam configuration used α = 1e-3 with beta1 0.9, beta2 0.999 and epsilon 1e-8. These are reference points, not guarantees: the AlgoPerf V1 results (Kasimbeg et al., 2026) found learning rate and momentum to be highly sensitive across domains and algorithms. For a sense of the useful range, AlgoPerf’s narrow AdamW search space ran from 2e-4 to 5e-3, and a 2026 tabular MLP benchmark searched AdamW learning rates between 3e-5 and 1e-3. A rate of 1e-4 falls inside the tabular benchmark’s AdamW range and just below AlgoPerf’s narrow one.
Warmup is the second half of the answer. The 2019 warmup study shows by simulation that Adam’s update magnitudes start high and decay only gradually. Unlike SGD, Adam will always encounter some early-stage instability from large updates, even when initialised at a local minimum. The study recommends linear warmup over 2/(1 − β2) iterations, which it found functionally equivalent to RAdam across a wide range of settings. With beta2 at 0.999 that works out to 2/0.001 = 2,000 steps, the figure used in the code above. If you lower beta2, the rule shortens warmup with it.
How the schedule after warmup interacts with the optimizer is a broader topic, and the best practices for training deep learning models cover decay schedules and their trade-offs in more detail.
How is Adam different from gradient descent, and when is SGD with momentum the better choice?
Plain stochastic gradient descent applies one global learning rate to every parameter: θ ← θ − η·∇L, computed on a mini-batch. Adam replaces that single rate with a per-parameter one derived from gradient history. A 2021 large-scale optimisation survey describes that approach as giving more precise control on the step size, because each dimension relates to the loss differently. The memory cost is higher, but the advantages usually credited to Adam are rapid initial progress and relative insensitivity to hyperparameters.
Whether that adaptivity costs you accuracy is a long-running argument, and the evidence pulls both ways.
The case for SGD rests on generalisation. Wilson et al. (2017) built an illustrative, linearly separable toy problem on which gradient descent and SGD reach zero test error while AdaGrad, Adam and RMSprop reach test errors arbitrarily close to one half. They also observed that adaptive methods often generalise worse than SGD on real models even when their training loss is better. Keskar and Socher (2017) found that adaptive methods do well early in training but are overtaken by SGD later, and proposed SWATS, which starts with Adam and switches to SGD once a trigger condition is met. In the 2022 survey’s ResNet20 experiment, SGD reached 93.01% accuracy against 90.73% for Adam and 91.96% for AdamW. AlgoPerf also reports a Wide ResNet run where Nesterov momentum beat AdamW on test error after a simple learning-rate sweep, largely because AdamW overfitted.
The case for Adam rests on tuning and robustness. Choi et al. argued that Wilson et al.’s conclusion holds only under restrictive conditions (Adam’s beta2 and epsilon fixed at defaults, a constant learning rate). They found that tuning all of Adam’s hyperparameters with a decay schedule lets Adam match or slightly beat SGD with momentum. In that same Wide ResNet study, changing the final block’s stride from 2×2 to 1×1 raised Nesterov’s test error by 3.5% while AdamW was largely unaffected. On transformers, Zhang et al. (2019) give evidence that heavy-tailed gradient noise is one cause of SGD’s poor performance on attention models. Kunstner et al. (2024) link Adam’s larger margin on language models to heavy-tailed class imbalance, where gradient descent reduces the loss on rare words slowly.
Our reading is that both sides are right about their own conditions. SGD with momentum is worth the effort on convolutional image models when you have the budget to tune a learning-rate schedule and final test accuracy matters more than time to a decent result. Adam-family optimizers are the better choice for transformers, for language and other heavily imbalanced outputs, for architectures that are still changing, and whenever the tuning budget is small.
Adam vs AdamW: when the weight decay difference matters
AdamW differs from Adam in how it applies weight decay. The weight decay is applied directly to the weights instead of being added to the gradient, so it does not accumulate in the momentum or variance estimates. Loshchilov and Hutter (ICLR 2019) showed that L2 regularisation and weight decay are equivalent for standard SGD but not for adaptive methods such as Adam. Their decoupled version made the best weight decay independent of the learning rate and substantially improved Adam’s generalisation, letting it compete with SGD with momentum on image classification.
With weight decay at zero, Adam and AdamW perform the same update. With a nonzero value, PyTorch’s decoupled_weight_decay=True flag on Adam makes it equivalent to AdamW. The defaults differ too (0 for Adam, 1e-2 for AdamW), so swapping one class for the other silently changes your regularisation.
Is AdamW still the best optimizer, or do Lion and Muon beat it?
AdamW is still the reference point, and newer optimizers have not consistently beaten Adam across tasks. A standardised benchmark of fifteen popular optimizers analysed more than 50,000 runs. They found that performance varies greatly across tasks, that no method clearly dominates, and that Adam remains a strong contender, with newer methods failing to significantly and consistently outperform it. In the AlgoPerf V1 results, Schedule-Free AdamW ranked first of 13 submissions on wall-clock time to target.
Lion
Lion keeps track only of momentum, so it is more memory-efficient than Adam. Its update takes the sign of a momentum term, which gives every parameter the same step magnitude. Its authors report gains on vision, diffusion and vision-language training, and similar or better results than Adam on language modelling. Their own analysis adds the conditions: the gain grows with batch size, Lion needs a smaller learning rate than Adam because the sign update has a larger norm, and there are scenarios where its improvements are small or not statistically significant. Lion is worth a trial when optimizer memory is the binding constraint and batches are large; on small-batch work we would not expect much.
Muon
Muon applies to the 2D weight matrices of hidden layers: it takes the SGD-momentum update and approximately orthogonalises it with a few Newton-Schulz iterations. Embeddings, biases, other scalar and vector parameters, and the input and output layers should still use a standard method such as AdamW, so a Muon run is really a Muon-plus-AdamW run. PyTorch’s Muon optimizer defaults to a momentum of 0.95. Its documentation notes that with a learning-rate adjustment Muon can reuse the learning rate and weight decay tuned for AdamW.
The reported gains depend on who measured them and at what scale. Muon’s author reports a 35% training-speed improvement in the NanoGPT speedrun and acknowledges that Muon has a slower per-step wall-clock time than AdamW. Moonshot AI (2025) reports roughly 2× computational efficiency over AdamW in compute-optimal training once weight decay and per-parameter update scaling are added. Wen and colleagues (2025), comparing against well-tuned baselines, found the speedup of matrix-based optimizers such as Muon shrinking from 1.4× over AdamW at 0.1B parameters to 1.1× at 1.2B. Outside language models, a 2026 benchmark of 15 optimizers on 17 tabular datasets found Muon consistently beat AdamW on MLPs, at the cost of slower training.
The two scaling results disagree. The fair-tuning study’s central point is that hyperparameters tuned for one optimizer may be suboptimal for another, which makes blind transfer unfair. The Moonshot result is still real evidence that Muon can scale with the right changes. Muon’s author also lists open questions, including whether it works for fine-tuning or reinforcement-learning workloads, which is a reason for caution before using it in agents that learn through trial and error. AlgoPerf V1 adds an engineering caveat: matrix-preconditioned methods are sample-efficient but need more aggressive computational optimisation to compete on runtime. So the honest answer to “is Muon better than AdamW” is yes on some measures, at some scales, if you pay for the extra compute per step.
Why Adam sometimes fails to converge
The original Adam paper’s convergence proof contains an error. Reddi et al. found that RMSprop and Adam can diverge even on convex problems, because their effective step size (the learning rate divided by √v) is not guaranteed to shrink, even when the learning rate itself decays. AMSGrad, the amsgrad=True option in PyTorch, is the variant from that paper. More generally, the 2025 convergence analysis notes that on stochastic problems Adam typically converges not to a minimiser or critical point of the objective but to different points, the zeros of its own update field.
In practice we would try warmup and a lower learning rate before reaching for AMSGrad. A 2019 survey of optimisation for deep learning sums up the theory honestly: why adaptive methods work as well as they do in practice is still largely unknown.
Choosing an optimizer: a decision table
| Situation | Start with | Why |
|---|---|---|
| Transformer or language-model training or fine-tuning | AdamW with linear warmup | Heavy-tailed gradient noise and class imbalance favour adaptive methods; linear warmup suits Adam |
| Convolutional image classification with budget to tune a schedule | SGD with momentum or Nesterov, compared against AdamW | SGD beat Adam on ResNet20, and Nesterov beat AdamW on a Wide ResNet |
| New or changing architecture, small tuning budget | Adam or AdamW at defaults | Adam’s advantages are relative insensitivity to hyperparameters and rapid initial progress |
| Regularising an Adam-trained model | AdamW, not Adam with weight_decay | L2 and weight decay are not equivalent under Adam |
| Optimizer memory is the constraint | Lion or Adafactor | Lion keeps only momentum; Adafactor cuts auxiliary memory |
| Speed-sensitive pretraining of hidden-layer matrices | Muon for 2D weights plus AdamW for the rest, as a tuned trial | Speedup shrinks with scale and costs per-step time; other parameters need AdamW |
One finding from the 2021 benchmark is worth keeping in mind alongside this table: trying several optimizers at their default settings worked about as well as tuning the hyperparameters of a single fixed one. When you are unsure which row you are in, a short run with AdamW, SGD with momentum and one newer method at sensible defaults will often settle the choice faster than a long sweep over one of them.
The question that remains open is the comparison itself. AlgoPerf’s authors argue that a tuning protocol, search space included, is part of an optimizer’s definition, because the same algorithm with different tuning can give significantly different results. Any claim that one optimizer beats another, including the claims above, is only as good as the tuning behind both sides, so before you trust a published speedup on your own workload, check how the baseline was tuned.