Reach for Adam, set the learning rate to whatever the tutorial used, and move on. That is how most teams treat the optimizer — as a fixed hyperparameter that “just works.” It is also the quiet reason a lot of training runs stall, diverge, or converge to a worse model than the architecture deserves.

The optimizer is not cosmetic. It is the algorithm that decides how each gradient turns into an actual change in your weights. Get it wrong and no amount of architecture tuning saves the run. Get it right and you converge in fewer epochs, spend less GPU time, and — the part people underestimate — get training curves you can actually reproduce. That reproducibility matters most when the trained model is a component other systems depend on, such as a reinforcement-learning agent feeding a larger coordinated pipeline.

This is the piece to read before you accept a default. We build and debug training pipelines as part of our generative AI work, and optimizer choice is one of the most common under-diagnosed causes of a run that “just won’t train.”

How does a deep learning optimizer actually work?

Strip away the names and every optimizer answers one question: given the gradient of the loss with respect to the weights, how much should each weight move, and in what direction?

Plain gradient descent gives the blunt answer — move every weight a fixed fraction (the learning rate) down its gradient. The problem is that a single global learning rate has to serve a loss surface with wildly different curvature in different directions. Some weights need big steps; some need tiny ones. A ravine-shaped loss surface, common in deep networks, makes plain descent oscillate across the steep walls while crawling along the shallow floor.

Every modern optimizer is a strategy for coping with that mismatch. They differ in two levers: how they use the history of past gradients (momentum-style accumulation) and whether they scale the step per-parameter (adaptive methods). Once you see optimizers through those two levers, the zoo of names collapses into a small, understandable family.

SGD, momentum, RMSProp, Adam: what actually differs

Stochastic gradient descent (SGD) computes the gradient on a mini-batch and steps directly. It is noisy by design — the mini-batch is a sample, not the full dataset — and that noise is not purely a nuisance. It acts as a mild regularizer and helps the run escape sharp, poorly-generalizing minima.

SGD with momentum adds a velocity term: instead of stepping on the current gradient alone, it accumulates an exponentially-decayed average of past gradients. In the ravine case, momentum cancels the oscillating components across the walls and reinforces the consistent component down the floor. This is why momentum is the workhorse for large-scale image classification training.

RMSProp takes the other lever. It keeps a running average of squared gradients per parameter and divides the step by the square root of that. Parameters with large, spiky gradients get damped; parameters with small, steady gradients get amplified. This is per-parameter step scaling, and it is what makes training tolerant of very different gradient magnitudes across layers.

Adam combines both — momentum (a running average of gradients, the “first moment”) and RMSProp-style per-parameter scaling (a running average of squared gradients, the “second moment”), with a bias correction for the early steps. That combination is why it converges quickly out of the box on a wide range of problems, and why it became the default in PyTorch and TensorFlow tutorials. AdamW, which decouples weight decay from the adaptive step, is the variant you almost always want over vanilla Adam when regularization matters — this is the standard for transformer training today.

Optimizer selection matrix

Optimizer	History lever	Per-parameter scaling	Converges fast out of box	Best-generalization tendency	Typical fit
SGD	none	no	no	strong (with tuning)	small models, when you have time to tune
SGD + momentum	yes	no	moderate	strong	large-scale vision (ResNet-style training)
RMSProp	no	yes	yes	moderate	RNNs, non-stationary objectives, some RL
Adam	yes	yes	yes	moderate	fast baselines, transformers, prototyping
AdamW	yes	yes	yes	good	transformer / LLM fine-tuning, default modern choice

The honest summary, and this is an observed-pattern across the training runs we have debugged rather than a single benchmark: Adam and AdamW win on time-to-a-working-model, while well-tuned SGD with momentum often edges them out on final generalization for convolutional vision models. If you are prototyping, the fast baseline is the right call. If you are shipping a vision model where the last point of accuracy is worth a week of tuning, do not assume Adam is the ceiling.

How do the optimizer and learning-rate schedule interact?

This is where the “optimizer as a fixed knob” mindset does the most damage. The optimizer and the learning-rate schedule are not independent choices — they are one coupled decision.

An adaptive optimizer like Adam already scales steps per-parameter, so it is more forgiving of a poorly chosen base learning rate than SGD is. But “more forgiving” is not “immune.” Two interactions bite people repeatedly:

• Warmup. Adam’s second-moment estimate is unreliable in the first few hundred steps because it has almost no history to average over. Starting at full learning rate during that window is a common cause of early divergence in transformer training. A short linear warmup — ramping the learning rate up over the first fraction of steps — is not optional folklore; it directly addresses the cold-start of the adaptive statistics.
• Decay shape. Cosine decay, step decay, and linear decay change what the same optimizer does at the end of training. Momentum SGD paired with a step-decay schedule behaves very differently from momentum SGD with cosine decay, even though the optimizer object is identical.

The practical consequence: when you copy an optimizer configuration from one project to another, you are copying half a decision. If the batch size changed, the effective learning rate changed too, and the schedule that worked before may now be wrong. We treat schedule and optimizer as a single unit in review for exactly this reason. Weight initialization sits in the same coupled system — see our note on why weight initialization governs training stability for the other half of the convergence story.

Why optimizer choice matters for reinforcement-learning agents

For teams training reinforcement-learning agents that feed multi-agent systems, optimizer instability is a common and badly-under-diagnosed source of behavioural drift. The reason is structural: RL objectives are non-stationary. The data distribution shifts as the policy changes, so the gradient statistics an adaptive optimizer accumulates can go stale in a way that never happens in supervised training on a fixed dataset.

An optimizer that quietly amplifies a transient gradient spike does not just slow training — it can nudge the policy into a different behavioural basin. When that agent then feeds a coordinated system, the drift shows up as an emergent behaviour problem that looks like an architecture or reward-design bug but is actually an optimizer-stability bug. If you are building on top of actor-critic methods, the interaction between optimizer choice and value/policy learning rates is worth its own attention — our walkthrough of advantage actor-critic in practice covers where those learning rates diverge.

This is why establishing that a learned component can be trained reliably — not just once, but reproducibly — is part of any honest feasibility assessment. A model that trains to target accuracy on one seed and diverges on the next is not a shippable component.

What are the symptoms of a badly chosen optimizer?

The failure modes are recognizable once you know their shapes. Reading the loss curve is the cheapest diagnostic you have.

Optimizer-instability diagnostic checklist

• Loss explodes to NaN in the first hundreds of steps → learning rate too high for the optimizer, or missing warmup with Adam. Add warmup or drop the base LR.
• Loss decreases then suddenly spikes and recovers repeatedly → step size too large for the local curvature; the run is oscillating across a ravine. Reduce LR or add momentum.
• Loss plateaus early and never moves → learning rate too low, or the schedule decayed it to near-zero prematurely. Check the effective LR at the plateau step, not the initial value.
• Training loss fine, validation loss diverges → this is usually not the optimizer; suspect regularization or data. But vanilla Adam’s coupling of weight decay into the adaptive step can worsen it — switch to AdamW.
• Run trains cleanly on one seed, diverges on another → the configuration is on the edge of stability. It “works” but is not robust. Widen the warmup or lower the LR until multiple seeds converge.

A useful reflex: before blaming the architecture, plot the gradient norm alongside the loss. A gradient norm that spikes right before a loss spike tells you the problem is in how steps are being taken, not in what the model can represent. Gradient clipping is the blunt fix; the right optimizer-and-schedule pairing is the durable one.

How do I diagnose and fix optimizer-related instability?

Work from cheapest to most expensive. Start by logging the effective learning rate per step (not the nominal one) and the gradient norm — most instability is visible in those two traces before it is visible in final metrics. If you see early divergence, add or lengthen warmup before touching anything else, because the cold-start of adaptive statistics is the single most common culprit in transformer training.

If warmup does not settle it, reduce the base learning rate by a factor and re-check across at least two random seeds. Reproducibility across seeds is the real acceptance test, not a single good run. Only after the run is stable should you optimize for final quality — and that is the point to consider whether a well-tuned SGD-with-momentum configuration would generalize better than the fast Adam baseline you prototyped with. This regularization interaction connects to the broader question of what L1 and L2 regularization actually do, which shares the same weight-decay coupling that AdamW was designed to fix.

The ROI here is concrete. A well-matched optimizer and schedule converge in fewer epochs, which is fewer GPU-hours and lower cost per trained model. Just as important, a run that reliably converges across seeds does not waste engineer-days on restarts — and it produces a learned component whose behaviour downstream systems can actually depend on. When you scale that training across devices, the optimizer’s sensitivity to effective batch size interacts with how you shard the work; our comparison of data parallelism versus model parallelism covers why the effective learning rate changes when you scale the batch.

FAQ

What matters most about a deep learning optimizer in practice?

An optimizer decides how much each weight moves and in what direction, given the gradient of the loss. It answers the mismatch between a single learning rate and a loss surface with very different curvature in different directions, using two levers: how it uses past-gradient history (momentum) and whether it scales each step per-parameter (adaptive methods).

What is the difference between SGD, momentum, RMSProp, and Adam — and when should I use each?

SGD steps directly on the mini-batch gradient. Momentum adds an accumulated average of past gradients to smooth oscillations. RMSProp scales steps per-parameter using squared-gradient history. Adam combines both. Use momentum SGD for large-scale vision when final generalization matters and you can tune; use Adam or AdamW for fast baselines, transformers, and prototyping.

How do the optimizer and the learning-rate schedule interact to affect convergence?

They are one coupled decision, not two. Adaptive optimizers like Adam need warmup because their per-parameter statistics are unreliable in the first steps, and decay shape (cosine, step, linear) changes what the same optimizer does at the end of training. Copying an optimizer config without its schedule — especially after a batch-size change — copies half a decision.

Why does optimizer choice matter when training reinforcement-learning agents that feed multi-agent systems?

RL objectives are non-stationary, so the gradient statistics an adaptive optimizer accumulates can go stale as the policy shifts — something that does not happen in supervised training on a fixed dataset. An optimizer that amplifies a transient spike can nudge the policy into a different behavioural basin, which surfaces downstream as behavioural drift that looks like a reward or architecture bug but is actually an optimizer-stability bug.

What are the symptoms of a badly chosen optimizer — divergence, plateaus, or unstable training curves?

Early NaN loss points to too-high LR or missing warmup; repeated loss spikes with recovery indicate steps too large for local curvature; an early flat plateau suggests LR too low or decayed prematurely; and a run that converges on one seed but diverges on another is on the edge of stability. Plotting gradient norm alongside loss usually reveals which of these you have.

How do I diagnose and fix optimizer-related instability in a training run?

Work cheapest-first: log effective learning rate and gradient norm per step, add or lengthen warmup for early divergence, then reduce the base LR and re-check across at least two seeds. Reproducibility across seeds — not a single good run — is the acceptance test before you optimize for final quality.

Optimizer choice is one of the levers that determines whether a learned agent component is trainable reliably enough to meet the monitoring and reproducibility requirements a multi-agent system depends on. If a run only converges on a lucky seed, the honest feasibility answer is that the component is not ready — and knowing that early is cheaper than discovering it downstream.