Optimizers and Learning Rate Schedules: Turning Instantaneous Gradients into a Controllable Training Trajectory
From the state updates of SGD, Momentum, Adam, and AdamW, to warm-up, plateau, decay, batch size, and mixed precision; understand why a training recipe must be tuned as a whole.
- Why don't we always perform a single SGD step right after obtaining the gradient?
- How does Momentum accumulate consistent directions, and how does Adam scale parameters according to historical squared gradients?
- Why are L2 regularization and AdamW's decoupled weight decay not the same implementation?
- How do warmup, peak learning rate, decay, and batch size form a recipe?
- When seeing oscillation, stagnation, NaN, or validation degradation, what runtime evidence should you look at?
1Why a Gradient Is Not Yet a Reliable UpdatePositioning
Backpropagation has already computed the slope for each parameter; what is still missing if you simply subtract it?
The gradient only describes the first-order direction at the current batch and current position. It may contain sampling noise, different parameters may differ in numerical scale by several orders of magnitude, and in narrow valleys it may repeatedly change sign along steep directions. The optimizer's job is not to redefine the objective but to decide how to turn this local, noisy directional evidence into stable updates.
| Component | Question answered | Not responsible for |
|---|---|---|
| Loss Function | What outcome counts as better? | How to efficiently compute derivatives |
| Backpropagation | How sensitive is the current loss to the parameters? | How far the parameters move |
| Optimizer | How to combine gradient history to form updates? | Whether the objective represents actual needs |
| Learning Rate Schedule | What global step size to use at different stages of training? | Fixing data and label errors |
2SGD is the minimal baseline, not “no optimizer”Math
What state does the simplest optimizer actually store?
SGD uses only the current mini-batch gradient gₜ and the learning rate ηₜ; it has no per-parameter historical state, so extra memory is small and behavior is easy to interpret. Gradient noise makes each step jitter, but the long-term average may still move in a useful direction.
SGD’s “simplicity” is an important baseline: if a complex optimizer is faster only on early training loss but does not improve the same number of steps, compute, or validation metrics, it cannot be claimed to be better overall.
3How Momentum Makes Consistent Directions Accumulate and Alternating Directions CancelNumerical Example
In a narrow valley, how do historical gradients turn left-right oscillations into forward speed?
Using a common notation:
Let β=0.5, the horizontal gradients are, in order, +4,−4,+4,−4, the vertical gradient is always −1, initial velocity is 0:
| Step | Gradient g=(horizontal,vertical) | Momentum v=(horizontal,vertical) | Observation |
|---|---|---|---|
| 1 | (4,−1) | (2,−0.5) | Starts responding |
| 2 | (−4,−1) | (−1,−0.75) | Horizontal reversal is weakened by history |
| 3 | (4,−1) | (1.5,−0.875) | Vertical continues to accumulate |
| 4 | (−4,−1) | (−1.25,−0.938) | Vertical approaches stable −1 |
The horizontal component still oscillates, but its amplitude is no longer equal to the original gradient; the vertical component accumulates because its sign remains the same over time.β The higher it is, the longer the smoothing window, and the more likely it is to react slowly to new terrain or overshoot the target due to inertia.
4Why Adam forms different effective step sizes for different parametersMechanism
When averaging directions alone is not enough, what information does the squared gradient history provide?
m smooths the direction,v estimates the squared gradient scale; parameters whose gradient history is always large are shrunk by the denominator, and sparse or smaller-scale parameters may receive relatively larger steps. The hats indicate correction for the early bias caused by initialization to zero,ε prevents division by zero and affects very small-scale parameters.
η, and β₁、β₂、ε, batch size, and gradient clipping all alter the effective update. The defaults are common starting points, not cross-task laws.5AdamW: Why Take Weight Decay Out of the GradientDisambiguation
Adding (λ/2)||θ||² to the loss, and directly shrinking parameters each step—when are they no longer equivalent?
In plain SGD, under this coefficient convention, the L2 term produces the gradient λθ, and is closely related to proportionally shrinking the parameters. But Adam also divides this part of the gradient by v, causing different parameters to receive different strengths of “regularization.” AdamW decouples the decay:
This way λ is closer to a uniform parameter shrinkage rate. Biases and normalization scales are often excluded from decay, but this too is a recipe choice that needs verification, not a syntax rule.
| Mechanism | Where it enters | In adaptive optimizers |
|---|---|---|
| L2 regularization | Added to loss, changes gradient | Is scaled by the per-parameter preconditioner |
| Decoupled weight decay | Shrinks parameters independently during updates | Does not mix with the gradient scale |
| Early stopping / data augmentation | Training process or data | Not an alternative notation for parameter decay |
6Why does the learning rate need to vary with training stage?Schedule
Why is it hard for the same step size to simultaneously suit the early, middle, and final stages of training?
Scroll horizontally to view the full diagram on small screens.
Warmupis the stage in the initial training steps during which the learning rate gradually rises from a lower value to the peak; at this point, parameters, activation statistics, and Adam moment estimates are not yet stable, and limiting early step sizes can avoid unrecoverable spikes.Peak stagehandles the main learning.Decayallows the later stage to move finely within an existing basin and reduces the parameters' continuous wandering in noise. After the total number of steps or batch size changes, you cannot simply copy the absolute step counts from the original curve.
7Why batch size belongs in the optimizer recipescaling
Does increasing the global batch size only change throughput, or also change gradients and usable learning rate?
Larger batch sizes reduce gradient sampling noise, reduce the number of updates per epoch, and change the amount of data covered by one update. Linear learning rate scaling and warmup are common starting points, but they are only effective within a certain range and under certain conditions.Critical batch sizeis the turning-point range where continuing to increase batch size no longer significantly improves statistical efficiency: after fixing a validation target and separately tuning the recipe, increase batch size step by step; if the number of samples or update steps needed to reach the target no longer decreases significantly, you are near this range. Beyond it, continuing to scale up may yield benefits mainly from parallel throughput rather than fewer training samples or update steps.
| Change | Also affected | Fair comparison should fix/report |
|---|---|---|
| batch size increase | noise, update count, memory, throughput | sample count, step count, compute, and wall-clock time |
| gradient accumulation | effective batch size and optimizer update frequency | micro-batch size, accumulation steps, normalization method |
| data parallelism scale | global batch size and communication | number of devices, per-GPU batch size, synchronization strategy |
8Which layer do gradient clipping and mixed precision protectNumerical
When the loss suddenly becomes NaN, are lowering the learning rate, clipping, and loss scaling solving the same problem?
Mixed precisionMixed precision lets forward and backward tensor operations that are suited to low precision use FP16 or BF16, while keeping reductions, optimizer state, and the usual master parameter copies—all more sensitive to numerical range—in FP32. The input is the same batch of data and parameters, and the output is still one gradient for the optimizer; this combination trades smaller memory and higher throughput for extra precision management, and exactly which operators remain FP32 is determined by the framework and hardware.
Global norm clippinglimits the length of the update direction from an anomalous gradient, primarily mitigating explosion;Loss scalingscales the loss up before low-precision backward to prevent small gradients from underflowing, then restores it before the update;Finite-value detectionskips the update and adjusts the scaling when Inf/NaN occurs. They protect different stages and cannot "clip away" vanishing gradients, wrong objectives, or long-term excessively large peak learning rates.
9Worked Example: How to Tell from Logs What the Updates Are Actually DoingDiagnosis
When you only see loss=1.23, why is it almost impossible to judge whether the optimizer is healthy?
| Must record | Question answered | Warning sign |
|---|---|---|
| Training/validation loss and breakdown | Whether the objective decreases and whether it generalizes | Training decreases while validation increases; sub-losses are masked by the total |
| Current learning rate | Where the scheduler actually is | Curves misalign after resuming training; warm-up repeats |
| Global and per-layer gradient norms | Whether the backward signal exists/explodes | Long-term zero, spikes, Inf |
| Clipping rate | How many steps have their direction or scale changed | Almost every step is clipped |
| Update norm / parameter norm | How much relative movement the gradient ultimately produces | Nonzero gradient but almost no update, or a single step jumps too far |
| Adam m/v and finite rate | Whether moment estimates and precision are healthy | Extreme v, frequent skipped updates |
| Sample throughput and global batch | Whether the configuration matches expectations | Gradient accumulation or parallel scaling errors |
- First, use a few dozen samples to confirm overfitting, ruling out problems with the objective or the graph.
- Run short experiments over several learning-rate orders of magnitude to determine the range of "start decreasing—stable—diverging".
- Compare SGD/Momentum/AdamW with fixed data order and budget.
- Then jointly search peak learning rate, decay, weight decay, and batch; do not change five things at once and then credit the optimizer name.
- When scaling up, first run a small-scale trial and observe gradients, update ratio, and numerical overflow.
10Connecting the Entire Causal ChainSynthesis
How does a reproducible training recipe connect gradients all the way to validation performance?
- Loss and data produce the target signal for the current mini-batch.
- Backpropagation computes noisy gradients.
- The state of Momentum or Adam extracts historical direction and scale.
- The global learning rate determines the overall movement magnitude.
- Warm-up protects early, and decay controls fine updates later.
- AdamW and similar mechanisms independently impose parameter preferences.
- Clipping and precision management prevent abnormal values from corrupting updates.
- Training logs prove that updates actually occur, and the validation set judges whether the trajectory generalizes.
- Any change in batch size, step count, or scale requires revalidating the entire combination.
11Common MisconceptionsDisambiguation
| Misconception | More Accurate Statement |
|---|---|
| Adam automatically finds a suitable learning rate | It performs per-parameter scaling, yet still relies on a global learning rate and schedule. |
| AdamW is just Adam plus L2 | AdamW decouples parameter decay from adaptive gradient updates. |
| Warmup is only for drawing a smoother curve | It limits the update magnitude under unstable early statistics, often related to whether training is possible. |
| Gradient clipping solves all gradient problems | It mainly limits explosions, and cannot fix vanishing gradients, broken graphs, or objective misalignment. |
| An optimizer that reduces training loss faster is definitely better | It also requires validation, stability, memory, and final task metrics under the same budget. |
12Supplementary Exercises: Check Whether You Really UnderstandSupplementary
- In the running scenario, why does Momentum weaken lateral oscillation while accumulating longitudinal velocity?
- What do Adam's first moment, squared-gradient second moment, and global learning rate each control?
- Why is the L2 gradient not equivalent to uniform weight decay in Adam?
- After resuming training, the loss jumps suddenly; which scheduler and optimizer states would you check?
- After scaling up the global batch by 8 times, how do you design a fair recipe re-validation?
Reference Answers
- Alternating gradients cancel each other in the exponential average; long-term same-sign components continue to accumulate.
- The first moment smooths the direction, the second moment scales parameters by the historical magnitude, and the global learning rate controls the overall magnitude.
- The L2 term is scaled together with the task gradient by the per-parameter denominator, so the regularization strength is no longer uniform; AdamW decays the parameters independently.
- Check the global step, current learning rate, whether warmup is repeated, whether m/v are restored, loss scaling, and gradient accumulation count.
- Clarify which budget is fixed—samples, compute, or wall-clock—rescan learning rate and warmup, and report update count, throughput, validation, and stability.
13Supplementary Route: Conceptual Dependencies and Further LearningSupplement
| Direction | Read Next | What to Ask |
|---|---|---|
| Where does the local direction come from? | Gradient Descent | What is the mathematical basis of negative gradients and learning rates? |
| How historical gradients are computed | Backpropagation | Do layer-wise gradient anomalies originate from the computation graph? |
| Why weight decay is useful | Regularization | How do parameter preferences affect unseen data? |
| Why deep-layer training is unstable | Vanishing Gradient Problem and Layer Normalization and RMSNorm | How do structure and scale change optimizability? |
| Recipes after scaling up | Distributed Training and Parallelism Strategies | How do global batch, communication, and optimizer state change together? |
- Adam: A Method for Stochastic Optimization: Adaptive moment estimation and bias correction.
- Decoupled Weight Decay Regularization: AdamW and decoupled weight decay.
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour: Large minibatch, linear scaling, and warmup.
- Deep Learning — Optimization for Training Deep Models: Momentum, stochastic optimization, and condition number.
The curves, numerical momentum examples, tables, and diagnostic workflows are all originally organized by this project.