Assignment 1

APM462 – Nonlinear Optimization – Summer 2016

Christopher J. Adkins

Solutions

Question 1 Let f (x, y) = 2x2 + y 2 + xy − y

(a) Find a point satisfying the first order conditions for f

(b) Prove that the point you found in a) is a global minimum for f

Solution Notice we may rewrite f as

1
f (x) = (x, Qx) − (b, x)
2
where (notation def: (x1 , x2 ) := xT1 x2 )
! !
4 1 0
Q= & b=
1 2 1

(a) Now it’s easy to see the first order condition (∇f = 0) gives

∇f (x∗ ) = Qx∗ − b = 0 =⇒ x∗ = Q−1 b

One may now easily compute x∗ :

! !
−1 1 2 −1 −1/7
Q = =⇒ x∗ =
7 −1 4 4/7

(b) We compute the eigenvalues of Q using P (λ) = det(Q − 1λ) = λ2 − 6λ + 7. Finding the roots gives the
two eigenvalues as
√
λ± = 3 ± 2

i.e. λ+ > λ− > 0 which means Q is positive definite. Now if we “complete the square” on f , we see
1 1 1
f (x) = (x − x∗ , Q(x − x∗ )) − (x∗ , Qx∗ ) > − (x∗ , Qx∗ )
2 2 2
Thus x∗ is a global minimum for f .

Question 2 Find all local minimum points for the function

f (x, y, z) = 2x2 + xy + y 2 + yz + z 2 − 6x − 8y − 8z + 9

Prove that your solution really is a global minimum.

1
Assignment 1 – Spring 2016 - CJA APM462

Solution Notice we may rewrite f as

1
f (x) = (x, Qx) − (b, x) + 9
2
where    
4 1 0 6
   
Q = 1
 2 1
 & b = 8


0 1 2 8
−1
One may easily compute x∗ using x∗ = Q b as we saw in the previous question. Plug and Chug
   
3 −2 1 1
1 
Q−1
  
= −2 8 −4 =⇒ x∗ = 2
 
10  
1 −4 7 3
Now we show Q is positive definite by checking the eigenvalues. We see the characteristic equation is given by

P (λ) = det(Q − 1λ) = −λ3 + 8λ2 − 18λ + 10

and the obvious bound of

P (λ) > 10 when λ60
shows that all eigenvalues are positive(since Q is symmetric, the eigenvalues must be real, and we’ve shown
there are no non-positive eigenvalues), hence x∗ is a global minimum by completing the square as we saw in the
previous question. Another method of of checking if Q is positive definite (could use this for question 1 as well)
is through Sylvester’s Criterion. This states that a symmetric matrix M is positive definite if and only if the
following matrices have positive determinate: the upper left 1-by-1 corner of M , the upper left 2-by-2 corner of
M , . . . , M itself. In this case its easy to see that
 
! 4 1 0
4 1  
det(4) =⇒ 4 & det =7 & 1 2  = 10
1
1 2 
0 1 2
Thus Sylvester’s Criterion gives us Q is positive definite.

Question 3 To approximate a function g : [0, 1] → R by an n-th order polynomial, one can minimize the
function f defined by Z 1
f (a) = (g(x) − pa (x))2 dx
0
where, for a = (a0 , . . . , an ) ∈ Rn+1 , we use the notation

pa (x) = a0 + a1 x + . . . + an xn = (x, a) & x = (1, x, . . . , xn )

In this question we will investigate approximating the parabola g(x) = x2 by the linear polynomials pa (x) =
a0 + a1 x.

(a) Show that f (a) can be written in the form

f (a) = aT Qa − 2bT a + c

for a 2 × 2 matrix Q, a vector b ∈ R2 and a number c. Find formulas for Q, b and c. It should be clear
from your formula that Q is symmetric.

(b) Find the first-order necessary condition for a point a∗ ∈ R2 to be a minimum point for f .

2
Assignment 1 – Spring 2016 - CJA APM462

Solution

(a) Expanding out f (a), we see that

Z 1 Z 1
2
f (a) = (g(x) − pa (x)) dx = (g(x) − (x, a))2 dx
0 0
Z 1
(x, a)(x, a) − 2g(x)(x, a) + g(x)2 dx
 
=
0
Z 1
(a, xxT a) − 2(g(x)x, a) + g(x)2 dx
 
=
0

=(a, Qa) − 2(b, a) + c

where Z 1 Z 1
1
Q= xxT dx =⇒ Qij = xi−1 xj−1 dx =
0 0 i+j−1
Z 1 Z 1
b= g(x)xdx =⇒ bi = g(x)xi−1 dx
0 0
Z 1
c= g(x)2 dx
0

Note that the particular form of Q implies that it is positive definite (since (v, xxT v) = (v, x)2 > 0 and
det Q 6= 0). If g(x) = x2 and x = (1, x), we see
!
1 1/2
Q=
1/2 1/3
! !
1
x2 1/3
Z
b= dx =
0 x3 1/4
Z 1
c= x4 dx = 1/5
0

(b) As usual with functions of this form, we know that critical point a∗ must satisfy Qa∗ = b, and since Q is
positive definite we have that a∗ = Q−1 b.

Question 4 Assume that g is a convex function on Rn , that f is a convex function of a single variable, and
in addition that f is a nondecreasing function (which means that f (r) > f (s) whenever r > s).

(a) Show that F (x) := f ◦ g(x) = f (g(x)) is convex by directly verifying the convexity inequality

F (tx1 + (1 − t)x2 ) 6 tF (x1 ) + (1 − t)F (x2 )

explain where each hypothesis (convexity of g, convexity of f , and the fact that f is nondecreasing) is
used in your reasoning.

(b) Now assume that f and g are both C 2 . Express the matrix of second derivatives ∇2 F (x) in terms of f
and g. Prove directly (without using part a)) that ∇2 F (x) is positive semidefinite at every x.

3
Assignment 1 – Spring 2016 - CJA APM462

Solution

(a) By direct computation, we see

F (tx1 + (1 − t)x2 ) =f (g(tx1 + (1 − t)x2 )) definition of F

6f (tg(x1 ) + (1 − t)g(x2 )) g is convex and f nondecreasing
6tf (g(x1 )) + (1 − t)f (g(x2 )) f is convex
=tF (x1 ) + (1 − t)F (x2 ) defintion of F

(b) First compute the gradient, we see that

∇F (x) = f 0 (g(x))∇g

Computing the matrix of second derivatives now shows we have (by product rule)

∇2 F (x) = f 00 (g(x))∇g∇g T + f 0 (g(x))∇2 g

Since f is nondecreasing and convex, we have that f 0 > 0 and f 00 > 0 at every x. Since g is convex, we
have that ∇2 g is positive semidefinite at every x. As we’ve mentioned before, matrices of the form xxT
are positive semidefinite(in this case we have ∇g∇g T ). Thus ∇2 F is positive semidefinite at every x, i.e.

(y, ∇2 F (x) y) = f 00 (g(x))(y, ∇g∇g T (x) y) + f 0 (g(x))(y, ∇2 g(x) y) > 0 ∀y ∈ Rn

Question 5 Prove that if f1 and f2 are two convex functions on Rn , then

g(x) := max{f1 (x), f2 (x)}

is also convex.

Solution This is easy to verify directly:

g(tx1 + (1 − t)x2 ) = max{f1 (tx1 + (1 − t)x2 ), f2 (tx1 + (1 − t)x2 )}

6 max{tf1 (x1 ) + (1 − t)f1 (x2 ), tf2 (x1 ) + (1 − t)f2 (x2 )} f1 and f2 are convex.
6t max{f1 (x1 ), f2 (x1 )} + (1 − t) max{f1 (x2 ), f2 (x2 )} bound by the bigger function at x1 and x2
=tg(x1 ) + (1 − t)g(x2 )