A model's loss surface usually isn't a smooth hill that drops straight into the best solution. It often has flat stretches, narrow valleys, and shallow dips where one direction changes fast while another barely moves. Plain gradient descent only looks at the slope at its current point. That's easy to reason about, but it can make optimization sluggish - or get stuck in a not-so-great pocket.
Momentum adds a little memory to the update. Instead of using the latest gradient by itself, it mixes that gradient with a running "velocity" built from earlier ones. If gradients keep roughly pointing the same way, the updates start to build up speed. If they swing back and forth, part of the signal cancels out. In the right setup, that's exactly what helps optimization move more smoothly across the loss surface.
This guide walks through the mechanics, shows a shallow-minimum example, and compares momentum to a narrow-valley case. It also covers how momentum can overshoot, what changes with Nesterov momentum, and what to check before turning either option on in a PyTorch training loop.
Why can plain gradient descent get stuck?
Gradient descent updates a parameter by stepping opposite the loss slope at where you are:
wt+1 = wt − η∇L(wt)
Here, w is a model parameter, L is the loss, ∇L is its gradient, and η is the learning rate. The gradient points toward the fastest increase in loss, so stepping the other way is meant to reduce it.
The catch is that each update has no memory. If the gradient becomes zero, the update becomes zero too. That can happen at a local minimum, but it can also happen at a saddle point or some other stationary point. Once the optimizer lands there, it has no "sense" of which direction it was moving before.
A shallow dip and a deeper destination
Picture a one-dimensional loss with two dips: one shallow, one deeper (the global minimum). A small bump separates them. If you start on a slope above the shallow dip, gradient descent will slide down toward it. As the slope flattens, the steps get smaller. At the bottom, the gradient is zero, and the method just... stops there.
One worked example uses:
L(w) = 0.12(w − 2.5)² − 0.55 exp(−(w + 1)² / 0.12)
Starting at w = −3.5, plain gradient descent with learning rate 0.1 reaches the shallow dip near w = −0.904 after about 21 steps. It stays there because the local gradient gives it no push out of the dip. The deeper minimum is at w = 2.5, but the update rule has no clue that a better destination exists "over the bump."
This isn't a claim that every training run will get permanently trapped. Real neural-network training is high-dimensional, and gradients are usually stochastic because you estimate them from mini-batches. What this example isolates is a real weakness: an update based only on the current gradient can't carry the optimizer across a flat spot, even if its previous direction would have pushed it onward.
What the example does - and does not - show
A zero gradient doesn't automatically mean "bad." Sometimes you really are at the minimum you want. Momentum also isn't a magic wand that guarantees a better point will be found. The point of the example is to make the difference between "no memory" and "has memory" obvious, and to show that whether it helps depends on the loss surface, the learning rate, and the rest of the optimizer setup.
How does momentum use past gradients?
Momentum adds a velocity term to the update. In one common convention:
vt = βvt−1 + ∇L(wt)
wt+1 = wt − ηvt
The velocity v accumulates gradients over time. The momentum coefficient β controls how much old velocity you keep. With β = 0.9, you carry over 90% of the previous velocity before adding the latest gradient. In practice, each parameter gets its own velocity buffer - so you end up with a velocity tensor aligned with each parameter tensor.
The effective memory of the update
If you expand the recurrence, velocity becomes a weighted sum of past gradients:
vt = ∇Lt + β∇Lt−1 + β²∇Lt−2 + ...
The latest gradient gets full weight, and earlier ones get smaller weights. With β = 0.9, the weights are 1, 0.9, 0.81, and 0.729. The infinite geometric series sums to 1 / (1 − β), or 10 here. That's a helpful way to think about the overall scale of the accumulated signal - but it isn't saying "only the last ten steps matter." Older gradients keep contributing; they just matter less and less.
Two patterns matter:
- If gradients keep pointing in the same direction, they reinforce each other. With a repeated gradient of +1 and β = 0.9, the velocity grows as 1, 1.9, 2.71, and 3.439, approaching 10.
- If gradients alternate between +1 and −1, they mostly cancel. The sequence goes 1, −0.1, 0.91, −0.181. The velocity still swings, but its magnitude stays much smaller than in the reinforcing case.
This is one reason momentum can help in narrow valleys: gradients across the steep direction may alternate, while gradients along the valley floor tend to stay more consistent.
Momentum can carry a parameter through a flat spot
Back to the shallow dip. Using the same learning rate 0.1 and β = 0.9, the parameter still has substantial velocity as it approaches the bottom. The gradient can hit zero right at the dip, but the existing velocity doesn't vanish instantly. So the update continues, carrying the parameter over the bump and toward the deeper minimum.
In that particular numerical run, the parameter passes the bump around step 7, overshoots the deeper minimum, and settles near it by step 89. Those exact numbers depend on the chosen loss and settings, so treat them as a demonstration of the mechanism, not a universal expectation. Change the start point, learning rate, or β, and the path can change a lot.
Some references write the velocity update as vt = βvt−1 + (1 − β)∇L(wt) instead. That version changes how the accumulated velocity is scaled. You can make it equivalent to the unscaled form by adjusting the learning rate - but using the same numeric learning rate in both conventions will not produce identical updates. If you compare formulas or implementations, check the convention instead of assuming the symbols mean the exact same step size.
Why does momentum help in narrow valleys, and when can it hurt?
A narrow valley has very different curvature depending on direction. Across the valley, the loss changes quickly. Along the valley floor, it changes more gradually. If you pick a learning rate that's big enough to move fast along the floor, that same rate can become unstable across the steep direction. If you pick a smaller rate to avoid that instability, progress along the floor can feel painfully slow.
Consider this quadratic loss:
L = ½(w1² + 50w2²)
The factor 50 makes the loss much steeper along w2 than w1. Starting at (−4, 1), the loss is 33. With learning rate 0.015, plain gradient descent takes 298 steps to drive the loss below 0.001 in the worked comparison.
Cranking the learning rate to 0.045 is not a safe shortcut. In the steep direction, each update multiplies the parameter by 1 − 0.045 × 50 = −1.25. The magnitude grows instead of shrinking, so the optimization diverges. So plain gradient descent has to stay at a smaller learning rate - and then it crawls along the shallow direction.
Faster movement, with more oscillation
Momentum can improve that trade-off. Along the valley floor, gradients tend to agree, so the accumulated velocity builds up and speeds things along. Across the valley, gradients often flip sign as you cross from one side to the other. Those opposing signals partially cancel, which reduces the effective push in the steep direction.
In the same worked setup, momentum with β = 0.9 hits the loss threshold in 73 steps instead of 298. Faster, yes - but it also swings side to side across the valley. That oscillation isn't automatically "bad"; it becomes a problem only when it's large enough to destabilize training or prevent settling.
Momentum can also make things worse. A high learning rate paired with a strong accumulated velocity can overshoot repeatedly. Noisy batch gradients can cause the velocity to lag behind the most recent information. And momentum can carry you past a helpful solution just as easily as it carries you past an unhelpful bump. In other words, it changes the dynamics - it doesn't "scan the whole landscape" and choose the best direction.
Read the training behaviour, not just the step count
A lower training loss in fewer optimizer steps is nice, but it doesn't automatically mean better model quality or lower total training cost. You still have to check the validation metric that matters, plus total wall-clock time and stability. If momentum reduces the number of steps but needs much more tuning - or gives worse validation results - then the step count is the wrong target.
Key takeaways:
- Momentum accumulates direction when gradients agree and dampens motion when they alternate.
- A high momentum coefficient increases persistence, but can also increase overshoot.
- Compare validation results and run stability, not just the number of updates.
- Treat numerical examples as demonstrations of a mechanism, not as guaranteed training benchmarks.
When should you use Nesterov momentum in practice?
Nesterov momentum changes where you evaluate the gradient. Instead of computing it at the current parameters, it first "looks ahead" in the direction of the current velocity and then computes the gradient at that estimated position:
vt = βvt−1 + ∇L(wt − ηβvt−1)
The idea is simple. If the velocity is about to carry you uphill, the look-ahead gradient can signal that sooner, acting like an early brake. In some problems, that can reduce overshoot compared to standard momentum.
In the narrow-valley comparison, the step counts were 298 for plain gradient descent, 73 for standard momentum, and 18 for Nesterov momentum. Those numbers are specific to that quadratic loss and those chosen settings. They show a potential advantage, not a guaranteed ratio you should expect everywhere.
Use the optimizer deliberately
In PyTorch, you can enable momentum and Nesterov with stochastic gradient descent:
optimizer = torch.optim.SGD(
model.parameters(),
lr=0.015,
momentum=0.9,
nesterov=True,
)
The optimizer keeps the velocity state for your model parameters. If you're comparing a custom implementation against a framework optimizer, pay attention to details like buffer initialization, dampening, weight decay, and the exact Nesterov convention. Similar-looking equations can differ in the first update or scaling, and that can change behaviour enough to confuse a "why did it diverge?" debugging session.
You can also check the sign/accumulation idea directly with a tiny example:
velocity = 0.0
history = []
for gradient in [1, -1, 1, -1, 1]:
velocity = 0.9 * velocity + gradient
history.append(round(velocity, 2))
print(history) # [1.0, -0.1, 0.91, -0.18, 0.84]
This kind of toy run is useful for sanity-checking how momentum accumulates. It won't replace validating your optimizer settings in an actual training run, though. Real models have many parameters, gradients that change from batch to batch, and interactions between the learning rate and other training choices.
A practical decision sequence
If training is slow or acting weird, don't turn five knobs at once. Do something controlled:
- Establish a baseline. Record the learning rate, optimizer, training and validation metrics, and time per epoch.
- Identify the failure mode. Slow progress in a consistent direction is a reasonable sign that momentum could help. Divergence usually points to an overly large learning rate or an issue in the data/implementation.
- Change one variable. Compare plain SGD, SGD with momentum, and (if relevant) Nesterov, using the same data order and evaluation method.
- Watch for overshoot. Look for unstable loss, big oscillations, or worse validation. If those show up, revisit learning rate and momentum together.
- Keep what improves the actual objective. The best setting is the one that meets quality and operational constraints - not the one with the flashiest update rule.
Momentum is a solid default to try with SGD, especially when gradients stay directionally consistent across updates. It's not a universal fix. If the underlying problem is bad data, a flawed loss, mismatched preprocessing, or a learning rate that's simply wrong, adding memory won't magically fix it.
Conclusion
Momentum gives gradient descent a running memory of recent slopes. That helps when gradients keep pointing in roughly the same direction, and it can reduce the back-and-forth motion that happens in a narrow valley. It also helps the optimizer cross flat spots that would stop plain gradient descent.
The trade-off is persistence: momentum can overshoot, amplify a poor learning-rate choice, or push you past a point you'd rather not go beyond. Start with a clear baseline, change one setting at a time, and judge by validation quality, stability, and wall-clock cost. Use momentum when training behaviour suggests it will help - not because "memory" is automatically better than using the current gradient directly.