Home Work 1

Due Date: 12th September 2019 (8am, in class).

(Attempt all questions)

Q. 1]
(i) Consider the following subset of R2 : Q = {x = (x1 , x2 ) ∈ R2 : 9x21 + 16x1 x2 +
21x22 ≤ 16}. Say whether Q is closed and bounded. If Q is bounded then find
M > 0 & x̂ ∈ Q such that kxk2 ≤ M for all x ∈ Q and kx̂k2 = M . If Q is not
bounded then find x̂ ∈ Q such that kxk2 > N , where N is an arbitrary big positive
number. . [5 marks]

(ii) Consider the following subset in the space of 2×2 real matrices: S = {X ∈ R2×2 :
X is symmetric & positive semidefinite, trace(X) = 1}. Say whether S is closed and
bounded. If S is bounded then find M > 0 & X̂ ∈ S such that kXkF ≤ M for all
X ∈ S and kX̂kF = M . Here k·kF denotes the Frobenius norm of a matrix. [5 marks]

Ans: (Present your answer in tabular form given below)

Closed? Bounded ? M (if finite) x̂ or X̂

Q. 1(i)

Q. 1(ii)

Q. 2] Let β denote the number consisting of last three digits of your S.R. number and
set  = (0.001β + 1)10−4 . Consider an iterative algorithm for solving the optimization
problem f ∗ = minx f (x). Let xk be the solution produced by the algorithm after
k-th iteration and ek = f (xk ) − f ∗ is the corresponding error. For the following cases
plot ek vs k and calculate the minimum number of iterations k required to reduce ek
below , 0.1, 0.01 and 0.0001: √
• [Sublinear Convergence:] (i) ek = 0.5/ k, (ii) ek = 0.5/k, (iii) ek = 0.5/k 2 ,
• [Linear Convergence:] (iv) e1 = 0.5, ek = 0.975ek−1 , k ≥ 2,
• [Quadratic Convergence:] (v) e1 = 0.5, ek = 1.9e2k−1 , k ≥ 2.
. [10 marks]
Ans: Attach the following plots:

• ek (log scale) vs k (log scale) for (i), (ii) & (iii) together in one plot.

• ek (log scale) vs k (normal scale) for (iv).

 • ek (log scale) vs k (normal scale) for (v).

β =

 =

Fill the blanks in the below table with Nδ (for δ = , 0.1, . . .) which is the minimum
number of iterations needed such that error ek ≤ δ, ∀k ≥ Nδ .

δ=  0.1 0.01 0.0001

Q. 2(i)

Q. 2(ii)

Q. 2(iii)

Q. 2(iv)

Q. 2(v)

Q. 3(i)
Let f : R4 → R be twice continuously differentiable function. The largest absolute
eigen value of the Hessian matrix of f at all points is bounded above by 25. We are
given that x0 = [4, 0, −2, 1]> , f (x0 ) = 6 and ∇f (x0 ) = [8, 4, 4, 2]> . Using 2nd
order Taylor Series find a quadratic function g : R4 → R such that g(x0 ) = f (x0 )
and f (x) ≤ g(x) ∀x ∈ R4 . Can the minimum value of f be more than 4? If the
answer is no then find x̃ such that f (x̃) ≤ 4. If the answer is yes then find x̃ such
that 4 < f (x̃) < 6. [ 4+3 marks]

Ans: As g is quadratic, it can be expressed as: g(x) = x> Ax + b> x + c, where

A ∈ R4×4 , b ∈ R4 and c ∈ R. Therefore, you need to simply specify the values of
A, b, c for your g.
Q. 3(ii)
Consider the function f (x) = (p> x)2 + (q> x)2 , p, q, x ∈ R3 . Assume p 6= 0, q 6= 0,
p + q 6= 0. Two friends Siri and Cortona have opinions about f . Siri claims that
there is an x ∈ R3 with kxk2 = 1 but f (x) = 0. Cortona says f is coercive. What
can you say about f : (a) Both claims are correct, (b) Siri is correct and Cortona’s
claim is false, (c) Siri’s claim is false and Cortona is correct, (d) Both claims are false.
. [ 3 marks]

Q. 4]
Recall that x̄ is a local maxima of f if the function value does not increase (de-
crease for local minima) when you move very small distance from x̄ in any direction.
We call x̄ a saddle point of f if the function value increases when we move from x̄
in some direction and descreases when we move from x̄ in another direction. Let
dθ = [cos(θ), sin(θ)]> donote the direction vector with angle θ ∈ [0, 2π].

(i) Consider the function f (x) = 10x21 +10x1 x2 +x22 +4x1 −10x2 +2 over [−3, 3]×[−3, 3].
Plot the function and see whether it has a local maxima / minima / saddle point at
x̄ = (1.8, −4). For clarity, plot [f (x̄ + αdθ ) − f (x̄)] vs θ for α = 0.01 and see whether
it is always non-negative (if local minima), always non-positive (if local maxima) or
has both positive and negative (if saddle point). Also, calculate ∇f (x̄) and eigen
values of ∇2 f (x̄). [ 5 marks]

(ii) Repeat part (i) with f (x) = 16x21 + 8x1 x2 + 10x22 + 12x1 − 6x2 + 2, x̄ = (−0.5, 0.5).
. [ 5 marks]

Ans: Attach the plots and fill up the following table:

∇f (x̄) eigen values of ∇2 f (x̄) at x̄ local maxima/minima/saddle point?

Q. 4(i)

Q. 4(ii)
Q. 5] Consider the following 1-dimensional minimization problem:

min f (x) . (1)

x∈[a,b]

We are given a, b, a subroutine named “zoof” (Zero Order Oracle for f ). The sub-
routine “zoof” is suppose to return only the value of f for any given x ∈ [a, b] i.e.
“fvalueAtx = zoof(x)”. Let x∗ be the unique minimizer of the problem (1). Given a
tolerance value ε > 0, we are interested in finding x̂ ∈ [a, b] such that |x̂ − x∗ | ≤ ε.
Note that x∗ and the function f are unknown but we are allowed to call the “zoof”
subroutine multiple times. Here is a generic pseudo code to find x̂:

• Initialize: xl = a, xu = b.

• In loop:

• d = (xu − xl ) ∗ ρ. [0 < ρ < 1 is an algorithm specific choice]

• x− = xu − d, x+ = xl + d.
• If zoof(x− ) < zoof(x+ ) then xu = x+ otherwise xl = x− .

• Output: x̂ = 0.5(xl + xu ), tolerance = 0.5 ∗ (xu − xl ), NSC= number of times

“zoof” subroutine was called.

We will now specialize the above generic structure to describe two line search tech-
niques. In√ Golden search (GS) we set ρ to a fixed value λ/(1 + λ), where λ =
0.5 ∗ (1 + 5). Now, implement Golden Search as a function (in Matlab / Python)
with structure:
[x̂, tolerance, NSC ] = GS(a, b, zoof, ε).
Note that the “loop” part of the GS algorithm should continue until tolerance ≤ ε is
satisfied.

Similarly, implement Fibonacci search (FS) as a function:

[x̂, tolerance, NSC ] = FS(a, b, zoof, N ).

Here the “loop” part should be executed only (N − 1) times and at k-th iteration use
ρ = FFNN−k+1
−k
, where F0 = 1, F1 = 1, Fn = Fn−1 +Fn−2 , n ≥ 2 is the Fibonacci sequence.

Now, implement two different “zoof” subroutine such that

(i) zoof(x) returns |2 − x| + |5 − 4x| + |8 − 9x|,
(ii) zoof(x) returns 3(x − 1)2 − ex−1 .

Fix a = 0, b = 3. Now, for each of the above choices of “zoof” first call the FS
subroutine with N = 5, N = 10, and N = 20. Then for each case call the GS
subroutine with ε set to the tolerance value returned by the FS subroutine. Report
x̂, tolerance, NSC for all the runs. Also, plot the functions (i) and (ii) over the inter-
val [a, b]. Note that the above line search subroutines do not need differentiability of
f : (ii) is differentiable where as (i) is not. The crucial property that the above line
search methods need is that f has a unique local minima over [a, b].

Reference: Details on the above line search techniques can be found in section 2.1
of this book.
. [5 + 5 marks]

Ans: You do not need to submit the code. But keep the code with you as we will use
those functions in later assignments as well. Present the output of your line search
subroutines as follows:

FS subroutine GS subroutine

case x̂ tolerance NSC x̂ tolerance NSC

(i)zoof, N = 5

(i)zoof, N = 10

(i)zoof, N = 20

(ii)zoof, N = 5

(ii)zoof, N = 10

(ii)zoof, N = 20