Overview of gradient descent

optimization algorithms
HYUNG IL KOO
Based on
http://sebastianruder.com/optimizing-gradient-descent/
Problem Statement
• Machine Learning  Optimization Problem
• Training samples: 𝑥 𝑖 ,𝑦 𝑖

• Cost function: 𝐽(𝜃; 𝑋; 𝑌) = 𝑖 𝑑(𝑓(𝑥 (𝑖) ; 𝜃), 𝑦 (𝑖) )

𝜃 = arg min 𝐽(𝜃; 𝑋; 𝑌)

𝜃
Optimization method
• Gradient Descent
• The most common way to optimize neural networks
• Deep learning library contains implementations of various gradient descent
algorithms
• To minimize an objective function 𝐽(𝜃) parameterized by a model's
parameters 𝜃 ∈ ℜ𝑑 by updating the parameters in the opposite direction of
the gradient of the objective function 𝛻𝜃 𝐽(𝜃) with respect to the
parameters.
CONTENTS
• Gradient descent variants
• Batch gradient descent
• Stochastic gradient descent
• Mini-batch gradient descent

• Challenges
• Gradient descent optimization algorithms
• Momentum
• Adaptive Gradient

• Visualization

• Which optimizer to choose?

• Additional strategies for optimizing SGD

• Shuffling and Curriculum Learning
• Batch normalization
GRADIENT DESCENT
VARIANTS
Batch gradient descent
• Computes the gradient of the cost function w.r.t. 𝜃 for the
entire training dataset:
𝜃 𝑛𝑒𝑤 = 𝜃 𝑜𝑙𝑑 − 𝜂𝛻𝜃 𝐽(𝜃; 𝑋; 𝑌)

• Properties
• Very slow
• Intractable for datasets that don't fit in memory
• No online learning
Stochastic Gradient descent (SGD)
• To perform a parameter update for each training example 𝑥 (𝑖)
and label 𝑦 (𝑖)

𝜃 𝑛𝑒𝑤 = 𝜃 𝑜𝑙𝑑 − 𝜂𝛻𝜃 𝐽(𝜃; 𝑥 𝑖 ; 𝑦 (𝑖) )

• Properties:
• Faster
• Online learning
• Heavy fluctuation
• Capability to jump to new (potentially better local minima)
• Complicated convergence (overshooting)
SGD fluctuation
Batch Gradient vs SGD
Batch gradient Stochastic gradient descent

• It converges to the minimum • It is able to jump to new and

of the basin the parameters potentially better local
are placed in. minima.
• This complicates
convergence to the exact
minimum, as SGD will keep
overshooting
Mini-batch (stochastic) gradient descent
• To perform an update for every mini-batch of 𝑛 training
examples:

𝜃 𝑛𝑒𝑤 = 𝜃 𝑜𝑙𝑑 − 𝜂𝛻𝜃 𝐽(𝜃; 𝑥 𝑖:𝑖+𝑛 ; 𝑦 (𝑖:𝑖+𝑛) )

Properties of mini-batch gradient descent
• Compared with SGD
• It reduces the variance of the parameter updates, which can lead to more
stable convergence;
• It can make use of highly optimized matrix optimizations common to
state-of-the-art deep learning libraries that make computing the gradient
w.r.t. a mini-batch very efficient

• Mini-batch gradient descent is typically the algorithm of

choice when training a neural network and the term SGD
usually is employed also when mini-batches are used.
CHALLENGES
Challenges
• Choosing a proper learning rate can be difficult.
• Small learning rate leads to painfully slow convergence
• Large learning rate can hinder convergence and cause the loss function to
fluctuate around the minimum or even to diverge

• Learning rate schedules and thresholds

• It has to be defined in advance and unable to adapt to a dataset's
characteristics.

• Same learning rate applies to all parameter updates.

• If our data is sparse and our features have very different frequencies, we
might not want to update all of them to the same extent, but perform a
larger update for rarely occurring features.
Challenges
• Avoiding getting trapped in their numerous suboptimal local
minima and saddle points
• Dauphin et al. argue that the difficulty arises in fact not from local
minima but from saddle points.
• These saddle points are usually surrounded by a plateau of the same error,
which makes it notoriously hard for SGD to escape, as the gradient is
close to zero in all dimensions.
MOMENTUM
Momentum
• One of the main limitations of gradient descent) is local minima
• When the gradient descent algorithm reaches a local minimum, the gradient becomes
zero and the weights converge to a sub-optimal solution

• A very popular method to avoid local minima is to compute a temporal average

direction in which the weights have been moving recently
• An easy way to implement this is by using an exponential average

𝑣𝑡 = 𝛾𝑣𝑡−1 + 𝜂𝛻𝜃 𝐽(𝜃)

𝜃 𝑛𝑒𝑤 = 𝜃 𝑜𝑙𝑑 − 𝑣𝑡
• The term 𝛾 is called the momentum
• The momentum has a value between 0 and 1 (typically 0.9)

• Properties
• Fast convergence
• Less oscillation
Momentum
• Essentially, when using momentum, we push a ball down a hill. The ball
accumulates momentum as it rolls downhill, becoming faster and faster on
the way (until it reaches its terminal velocity if there is air resistance).
• The same thing happens to our parameter updates: The momentum term
increases for dimensions whose gradients point in the same directions and
reduces updates for dimensions whose gradients change directions. As a
result, we gain faster convergence and reduced oscillation.
Momentum
• The momentum term is also useful in spaces with long
ravines characterized by sharp curvature across the ravine
and a gently sloping floor
• Sharp curvature tends to cause divergent oscillations across
the ravine
• To avoid this problem, we could decrease the learning rate, but this is too
slow
• The momentum term filters out the high curvature and allows the effective weight
steps to be bigger
• It turns out that ravines are not uncommon in optimization problems, so the use of
momentum can be helpful in many situations
• However, a momentum term can hurt when the search is close to the minima
(think of the error surface as a bowl)
• As the network approaches the bottom of the error surface, it builds enough
momentum to propel the weights in the opposite direction, creating an
undesirable oscillation that results in slower convergence
Smarter Ball?
NGD (Nesterov accelerated gradient)
• Nesterov accelerated gradient improved on the basis of Momentum algorithm
• Approximation of the next position of the parameters.

𝑣𝑡 = 𝛾𝑣𝑡−1 + 𝜂𝛻𝜃 𝐽(𝜃 − 𝛾𝑣𝑡−1 ).

𝜃 𝑛𝑒𝑤 = 𝜃 𝑜𝑙𝑑 − 𝑣𝑡
ADAPTIVE GRADIENTS
Adaptive Gradient Methods
• Let us adapt the learning rate of each parameter, performing
larger updates for infrequent and smaller updates for frequent
parameters.

• Methods
• AdaGrad (Adaptive Gradient Method)
• AdaDelta
• RMSProp (Root Mean Square Propagation)
• Adam (Adaptive Moment Estimation)
Adaptive Gradient Methods
• These methods use a different learning rate for each parameter
𝜃𝑖 ∈ ℜ at every time step 𝑡.
(𝑡)
• For brevity, we set 𝑔𝑖 to be the gradient of the objective function w.r.t.
𝜃𝑖 ∈ ℜ at time step 𝑡:
(𝑡+1) (𝑡) (𝑡)
𝜃𝑖 = 𝜃𝑖 − 𝜼 ∙ 𝑔𝑖

• These methods modify the learning rate 𝜼 at each time step (𝑡) for every
parameter 𝜃𝑖 based on the past gradients that have been computed for 𝜃𝑖 .
Adagrad
• Adagrad modifies the general learning rate 𝜂 at each time step 𝑡
for every parameter 𝜃𝑖 based on the past gradients that have
been computed for 𝜃𝑖 :
(𝑡+1) (𝑡) 𝜼
𝜃𝑖 = 𝜃𝑖 − 𝑔𝑡,𝑖
𝑮𝒕,𝒊 + 𝝐
2
𝑘
• 𝐺𝑡,𝑖 = 𝑘≤𝑡 𝑔𝑖

(𝑘) 𝜕𝐽(𝜃)
𝑔𝑖 =
𝜕𝜃𝑖 𝜃(𝑘)
Adagrad
• Pros
• It eliminates the need to manually tune the learning rate. Most
implementations use a default value of 0.01.

• Cons
• Its accumulation of the squared gradients in the denominator: the
accumulated sum keeps growing during training. This causes the learning
rate to shrink and eventually become infinitesimally small. The following
algorithms aim to resolve this flaw.
RMSprop
• RMSprop has been developed to resolve Adagrad's diminishing
learning rates.

2
𝑡
𝜆𝑡,𝑖 = 𝛾 𝜆𝑡−1,𝑖 + 1 − 𝛾 𝑔𝑖
(𝑡+1) (𝑡) 𝜼
𝜃𝑖 = 𝜃𝑖 − 𝑔𝑡,𝑖
𝝀𝒕,𝒊 + 𝝐

η : Learning rate is suggest to set to 0.001

𝛾 : Fixed momentum term
Adam (Adaptive moment Estimation)
• Adam keeps an exponentially decaying average of past gradients 𝑚𝑡 , similar to
momentum.
𝑡
𝑚𝑡,𝑖 = 𝛽1 𝑚𝑡−1,𝑖 + (1 − 𝛽1 )𝑔𝑖 𝑚𝑡 : Estimate of the first moment(the mean)
𝑣𝑡 : Estimate of the second moment
2 (the un-centered variance)
𝑡 𝛽1 : suggest to set to 0.9
𝑣𝑡,𝑖 = 𝛽2 𝑣𝑡−1,𝑖 + 1 − 𝛽2 𝑔𝑖 𝛽2 : suggest to set to 0.999

• They counteract these biases by computing bias-corrected first and second

moment estimates.
𝑚𝑡,𝑖 𝑣𝑡,𝑖
𝑚𝑡,𝑖 = 𝑣𝑡,𝑖 =
1 − 𝛽1 1 − 𝛽2

• which yields the Adam update rule.

(𝑡+1) (𝑡) 𝜼
𝜃𝑖 = 𝜃𝑖 − 𝑚𝑡,𝑖 𝜖 : suggest to set to 10−8

𝑣𝑡 + 𝜖
COMPARISON
Visualization of algorithms
Which optimizer to choose?
• RMSprop is an extension of Adagrad that deals with its
radically diminishing learning rates.

• Adam slightly outperform RMSprop towards the end of

optimization as gradients become sparser.
CONCLUSION
Conclusion
• Three variants of gradient descent, among which mini-batch
gradient descent is the most popular.