Optimization for ML (2)

CS771: Introduction to Machine Learning

Piyush Rai
 2
The Plan
 Some basic techniques for solving optimization problems
 First-order optimality
 Gradient descent
 Dealing with non-differentiable functions
 Sub-gradients and sub-differential

CS771: Intro to ML
 3
Optimization Problems in ML
 The general form of an optimization problem in ML will usually be
Usually a sum of the
training error +
regularizer However, possible to have
linear/ridge regression
 Here denotes the loss function to be optimized where solution has some
constraints (e.g., non-neg,
sparsity, or even both)
 is the constraint set that the solution must belong to, e.g.,
Linear and ridge regression
 Non-negativity constraint: All entries in must be non-negative that we saw were
 Sparsity constraint: is a sparse vector with atmost non-zeros unconstrained ( was a real-
valued vector)

 If no is specified, it is an unconstrained optimization problem

 Constrained opt. probs can be converted into unconstrained opt. (will see later)
 For now, assume we have an unconstrained optimization problem
CS771: Intro to ML
 4

Methods for Solving

Optimization Problems

CS771: Intro to ML
 5
Method 1: Using First-Order Optimality
 Very simple. Already used this approach for linear and ridge regression
Called “first order” since only gradient
is used and gradient provides the first
order info about the function being
optimized
The approach works only for very
simple problems where the objective
is convex and there are no
constraints on the values can take

 First order optimality: The gradient must be equal to zero at the optima

=0
 Sometimes, setting and solving for gives a closed form solution

 If closed form solution is not available, the gradient vector can still be used in
iterative optimization algos, like gradient descent
CS771: Intro to ML
 6
Method 2: Iterative Optimiz. via Gradient Descent
Can I used this approach For max. problems we Iterative since it requires
to solve maximization several steps/iterations to find
can use gradient ascent
problems? the optimal solution
Fact: Gradient gives the For convex functions, Good
direction of steepest Will move in the
GD will converge to initialization
change in function’s direction of the gradient
the global minima needed for non-
value Gradient Descent convex functions
The learning rate
very imp. Should be
 Initialize as set carefully (fixed
or chosen
adaptively). Will
 For iteration (or until convergence) discuss some
 Calculate the gradient using the current iterates strategies later
 Set the learning rate Will see the Sometimes may be
justification shortly tricky to to assess
 Move in the opposite direction of gradient convergence? Will
(𝑡 +1) (𝑡 ) (𝑡 ) see some methods
𝒘 =𝒘 −𝜂 𝒈
𝑡 later
CS771: Intro to ML
 7
Gradient Descent: An Illustration
Negative gradient here . Let’s move Learning rate is very important
𝐿(𝒘 ) in the positive direction

Positive gradient
here. Let’s move in
the negative direction

𝒘
(0 )
𝒘
𝒘 𝒘
(1) (3 )
∗
𝒘
(2 )
𝒘
(2 )
𝒘(3 )
𝒘∗ 𝒘
(1) 𝒘
(0 )
𝒘
Woohoo! 
Stuck at a local
Global minima
minima 
found!!!
GD thanks you for the Good initialization
good initialization  is very important CS771: Intro to ML
 8
GD: An Example
 Let’s apply GD for least squares linear regression

=
Training
 The gradient: examples on
Prediction error of current model which the current
 Each GD update will be of the form on the training example model’s error is
large contribute
more to the
update

 Exercise: Assume , and show that GD update improves prediction on the

training input (, ), i.e, is closer to than to
 This is sort of a proof that GD updates are “corrective” in nature (and it actually is true
not just for linear regression but can also be shown for various other ML models)CS771: Intro to ML
 9
Dealing with Non-differentiable Functions
 In many ML problems, the objective function will be non-differentiable

 Some examples that we have already seen: Linear regression with absolute
loss, or Huber loss, or -insensitive loss; even norm regularizer is non-diff
¿ 𝑦 𝑛 − 𝑓 ( 𝒙 𝑛 ) ∨−𝜖
Loss ¿ 𝑦 𝑛 − 𝑓 ( 𝒙 𝑛 ) ∨¿ Loss Loss

Not diff. here Not diff. here

) −𝛿 𝛿 ) −𝜖 𝜖 )
Not diff. here Not diff. here Not diff. here

 Basically, any function in which there are points with kink is non-diff
 At such points, the function is non-differentiable and thus gradients not defined
 Reason: Can’t define a unique tangent at such points
CS771: Intro to ML
 10
Sub-gradients
 For convex non-diff fn, can define sub-gradients at point(s) of non-
differentiabilty Convex, thus lies
𝑓 (𝑥 )
above all its tangents
Equation of unique tangent at differentiable
here One extreme tangent at
non-differentiable
here Region containing all sub-gradients
The other extreme tangent at

𝑥1 𝑥2

 For a convex, non-diff function , sub-gradient

⊤
atis any vector s.t.
𝑓 ( 𝑥 ) ≥ 𝑓 ( 𝒙 ∗) + 𝒈 ( 𝒙 − 𝒙 ∗)
CS771: Intro to ML
 11
Sub-gradients, Sub-differential, and Some Rules
 Set of all sub-gradient at a non-diff point is called the sub-differential
𝜕 𝑓 ( 𝒙 ∗ ) ≜ { 𝒈 : 𝑓 ( 𝐱 ) ≥ 𝑓 ( 𝒙 ∗ ) +𝒈 ( 𝒙 − 𝒙∗ ) ∀ 𝐱 }
⊤

The affine transform

 Some basic rules of sub-diff calculus to keep in mind rule is a special case of
 Scaling rule: the more general chain
rule
 Sum rule:
 Affine trans: , where
 Max rule: If then we calculate at as
 If , If
 If

 is a stationary point for a non-diff function if the zero vector belongs to the
sub-differential at , i.e.,
CS771: Intro to ML
 12
Sub-Gradient For Absolute Loss Regression

0 𝑦 𝑛 − 𝒘 ⊤ 𝒙𝑛 0 𝑡
 The loss function for linear reg. with absolute loss:
 Non-differentiable at
 Can use the affine transform rule of sub-diff calculus
 Assume . Then
 if
 if
 where if
CS771: Intro to ML
 13
Sub-Gradient Descent
 Suppose we have a non-differentiable function

 Sub-gradient descent is almost identical to GD except we use subgradients

Sub-Gradient Descent
 Initialize as

 For iteration (or until convergence)

 Calculate the sub-gradient
 Set the learning rate
 Move in the opposite direction of subgradient
(𝑡 +1) (𝑡 ) (𝑡 )
𝒘 =𝒘 − 𝜂𝑡 𝒈
CS771: Intro to ML
 14
Coming up next
 Making GD faster: Stochastic gradient descent
 Constrained optimization
 Co-ordinate descent
 Alternating optimization
 Practical issue in optimization for ML

CS771: Intro to ML