COL726: Numerical Algorithms

Sem 2, 2024-25
Assignment 2

Deadline - 23 April, 2025

Please carefully read the instructions given below before attempting the questions:
1. Originality and Acknowledgment: All solutions must be written in your own words. If you
collaborate with others, acknowledge their names at the start of the respective question. Similarly,
if you refer to online resources or other materials, cite them clearly. However, the solutions must
still be written independently.
2. Consequences for Non-Compliance: Failure to follow these guidelines will result in a zero score
for the assignment, and further disciplinary actions may be taken if necessary.
3. Use single precision for the numerical problems if nothing is specified. All the pro-
gramming questions are to be done in MATLAB. It is mandatory to submit MATLAB
code for all programming problems. You have to paste the code in the submission
document.
4. You can submit your solutions in either handwritten format (ensure clear readability) or LaTeX-
typed format.
5. If you find any problem is incorrect/missing information before a week of the deadline, kindly let us
know. If it is in the last week before the deadline, then make your own assumptions. The deadline
for assignment will not be extended.
6. The assignment consists of 10 questions, each worth 5 marks.
7. Submit a single PDF file named <Entry number>.pdf (Example - 2020CS10111.pdf ), containing
solutions to all the problems as well as the code to programming questions.

1. Let A ∈ Cm×m be a matrix.

(i) Suppose A∗ A = AA∗ . Show that there is a unitary matrix X such that X ∗ AX is a diagonal
matrix.
(ii) Let W (A) denote the following set
u∗ Au
 
m
W (A) = :u∈C \0 .
u∗ u
Show that W (A) contains the convex hull (in the complex plane) of all the eigenvalues of A.
(iii) Suppose the matrix A satisfies the condition in (i) above. Then show that W (A) is equal to
the convex hull of the eigenvalues of A.

2. The matrix exponential of an n × n matrix A is defined as

A2 A3
eA := I + A + + + ...
2! 3!

1
 (a) Write a program to compute eA using the definition above.
(b) Write a program based on the eigenvalue-eigenvector decomposition of A (assume it has distinct
eigenvalues and you are given the eigenvalues and the eigenvectors).
(c) Test the above two methods on the matrices:
   
2 −1 −49 24
A1 = , A2 = .
−1 2 −64 31

Compare your results with the MATLAB function for matrix exponential. Which of the two
methods is more accurate (on the above examples), and why?

3. Write a program implementing the Lanczos method. Test your method on a random symmetric
matrix of order n having eigenvalues 1, 2, . . . , n. To generate such a matrix:

(a) First, generate a random n × n matrix B with random entries uniformly distributed in [0, 1).
(b) Compute the QR factorization of B, i.e., B = QR.
(c) Define A = QDQT , where D is a diagonal matrix with diagonal entries 1, 2, . . . , n.

After applying, the Lanczos iteration you are required to compute the eigenvalues of the tridiagonal
matrix – use a MATLAB library routine to find these eigenvalues. For the purpose of this exercise,
run the Lanczos iteration for a full n iterations.
To visualize how the Ritz values behave, construct a plot with the iteration number on the vertical
axis and the Ritz value at each iteration on the horizontal axis. Plot each pair (γ, k), where γ is a
Ritz value in iteration k, as a discrete point.
Try several values of n = 30, 40, 50.
4. Consider the system of equations    
x p
M = ,
y q
where  
A B
M= ,
B∗ 0
with A ∈ Cm×m being Hermitian positive definite, and B ∈ Cm×n being full rank with m ≥ n.
Correspondingly, p ∈ Cm and q ∈ Cn .

(a) Show that the matrix M is indefinite, i.e., there exist vectors u and v such that u∗ M u > 0
and v ∗ M v < 0.
(b) Give a method to solve this system as efficiently as possible. In particular, your method should
take fewer flops than naively applying LU or QR factorization to M .
Hint: Consider eliminating x and solving for y alone first.

5. Suppose we run the Arnoldi iteration on an m × m matrix A and it terminates with hn+1,n =
0 for some n < m (i.e., the vector qn+1 = 0). Let Kn denote the Krylov space spanned by
⟨b, Ab, . . . , An−1 b⟩.

(i) Show that Kn = Kn+1 = Kn+2 = . . ..

(ii) Show that each eigenvalue of the matrix Hn is an eigenvalue of A.
(iii) Show that if A is invertible, then the solution x to Ax = b lies in Kn .

6. Prove the following properties related to convex functions:

(a) Consider a real symmetric matrix A(x) that depends on x ∈ Rn , with all its entries being
convex functions of x. Show that the largest eigenvalue λmax (A(x)) is a convex function of x.

2
 (b) Prove that the function

f (x) = − log det(A + diag(x))

is convex for a given symmetric positive definite matrix A and vector x satisfying the condi-
tion that (A + diag(x)) is positive definite. It can be shown that the hessian of the function is
positive definite.
−1
You may use ∂(B∂t ) = −B −1 ∂B −1 ∂
ln(det(B)) = Tr B −1 ∂B


∂t B and ∂t ∂t without proving. Every
other result or lemma needs to be proved.
7. Suppose f (x) is a convex quadratic function of the form:
1 T
f (x) = x Ax − bT x
2
where A is a symmetric positive definite matrix.
(a) Suppose we apply gradient descent to minimise this function with a certain constant learning
rate α. Show that the best rate of convergence is achieved when learning rate is chosen as:
2
λmax +λmin where λmax and λmin are the largest and smallest eigenvalues of A, respectively. You
need to minimise the error = ∥xk − x∗ ∥2 at every iteration, where x∗ is the optimal solution.
(b) Show that the application of Newton’s method on this function converges in a single step. Why
doesn’t the same happen for a general non-quadratic function? Argue using the second-order
Taylor expansion.
(c) Suppose now we apply gradient descent to minimise this function with an adaptive learning
rate αk . For each step, find αk in terms of xk , which minimizes f (xk+1 ), where the update
equation is given by xk+1 = xk − αk ∇f (xk )(basically, a line search).
(d) Find the rate of convergence for part (a). Show that it depends on the ratio k = λλmax
min
. Argue
quantitatively or qualitatively why, in part (c), the convergence rate would not depend on k.
8. Find the minimum of Rosenbrock’s function:

f (x, y) = 100(y − x2 )2 + (1 − x)2

using each of the following methods:

(a) Steepest Descent Method (Gradient descent with Line Search).

(b) Newton’s Method.
(c) Damped Newton’s Method (Newton’s Method with a Line Search). Update equation: xk+1 =
xk − γHf (xk )−1 ∇f (xk ), search is carried out on γ.

Each method should be tested from the following three starting points:
     
−1 0 2
, ,
1 1 1
For any line searches and linear system solutions required, you may use MATLAB routines. Plot
the path taken in the plane using each of the methods for each of the starting points.
9. Consider the probability distribution of a random variable X given by X = eµ+σZ , here Z is a
standard normal variable. It can be shown that the logarithm of the random variable X has a
normal distribution with µ and σ as mean and standard deviations, respectively.
In order to generate a log-normal distribution with given mean(µX ) and standard deviation (σX ),
we can use a standard normal distribution directly if we know the values of the parametes µ and
σ. The relation between the parameters is given by:
2
 
2 σX
σ = log 1 + 2
µX
1
µ = log (µX ) − σ 2
2

3
 (a) Generate 10000 samples from a log-normal probability distribution with parameters µX = 1
and σX = 10−7 . You can use MATLAB’s built in function randn to generate the standard
normal distribution. Set the seed value of the random generator to 2 using command rng(2);
before generating the samples. Show the histogram for the sampled numbers. Use MATLAB’s
built-in function histogram to show the plot with ’Normalization’ argument set to ’pdf’. For
example, if you have samples in a variable named samples, then the plot can be generated
using the following command: histogram(samples, ’Normalization’, ’pdf ’);.
(b) There is an issue with using the above formulations to calculate parameters µ and σ for the
given values of µX and σX . Identify the issue and propose a modified formulation free of that
issue. Repeat the previous step using the modified formulation to generate the samples. Use
MATLAB subplots to show the histograms side by side.
(c) Report the empirical mean and variance of the generated samples using the original formulation
and the modified formulation. You can use MATLAB’s built-in functions mean and var. Do
you notice any differences?
CAUTION: Use single precision for generating the samples. You will get zero for using double
precision for this problem. You are not allowed to use any other built-in MATLAB function
for this problem. The allowed functions are already stated in the problem description.
10. A linear system of the form Ax = b can be solved using iterative solvers such as Jacobi and Gauss-
Seidel. Consider the following linear systems:
i.  
3.50104431 0.76049899 0.94900442 0.68591364 0.05987737
0.90828514 3.98803024 0.90664645 0.49033591 0.02432771
 
0.68503541 0.69695709 4.04621174
A= 0.60751771 0.31092343
 0.4930187 0.4933185 0.61623597 3.23993414 0.23298047
0.26899025 0.32922291 0.32813053 0.43759326 2.78896836
 
0.53312846
0.37067101
 
b=0.11539274

0.69388962
0.23550072
ii.  
1.19978173 0.57396235 0.06640492 0.705216 0.42379898
0.50406865 0.76522395 0.175033 0.31193115 0.04410332
 
0.16480431 0.81287099
A= 1.10015456 0.07493729 0.87673584

 0.1370814 0.23830759 0.42274182 0.84025606 0.55507874
0.73728438 0.40809756 0.35715129 0.0538071 1.03679079
 
0.50591368
0.34140362
 
0.47161319
b= 
0.86696864
0.29814066
Implement the Jacobi Method and Gauss-Seidel Method to solve the two linear systems described
above. Compare their performance by analyzing the convergence rate and the number of iterations
required to reach a specified tolerance.

• Implement both the Jacobi Method and Gauss-Seidel Methods.

• Solve both linear systems (i. and ii.) using both methods.
• Plot the convergence of both methods for each scenario, showing the error (L2 norm of the
residual) versus the number of iterations.
• Analyze the results and explain why one method performs better than the other in each
scenario.
• Discuss the characteristics of the matrices that lead to better performance for each method.

4
Note:
• Use a tolerance of 10−6 for the convergence criterion.
• Set a maximum of 1000 iterations for both methods.
Your analysis should include:
(a) MATLAB code
(b) Convergence plots for both methods in each scenario
(c) A table comparing the number of iterations and final error for each method in both scenarios
(d) An explanation of the observed performance differences based on the properties of the matrices