Solutions: Problem Set 1

January 17, 2013

Total points: 20 Solutions provided here are very detailed on purpose. 1. (a) Prove that for rectangular matrices B, D, and invertible matrices A, C A(A + BCD)1 = I B(C1 + DA1 B)1 DA1 . (b) Using the above result, prove that (A1 + BT C1 B)1 BT C1 = ABT (BABT + C)1 . (c) Find an expression for (A + xyT )1 in terms of A1 , x and y for invertible matrix A and (column) vectors x and y. Sol 1(a) Let Y = C1 + DA1 B so that DA1 B = Y C1 . Right-multipling both sides of the expression in part (a) by (A + BCD), the right-hand side can be written as, (A + BCD) BY1 DA1 (A + BCD) = A + BCD BY1 DA1 A BY1 DA1 BCD = A + BCD BY1 D BY1 (DA1 B)CD = A + BCD BY1 D BY1 (Y C1 )CD = A + BCD BY1 D BY1 YCD + BY1 C1 CD = A + BCD BY1 D BCD + BY1 D =A which is equal to the left-hand side. Sol 1(b) This part can be done in a similar way. Set Y = A1 + BT C1 B and mltiply both sides by (BABT + C) on the right. The the left-hand side then becomes Y1 BT C1 (BABT + C) = Y1 BT C1 BABT + Y1 BT C1 C 1

= Y1 BT C1 BABT + Y1 BT = Y1 (Y A1 )ABT + Y1 BT = ABT which is the same as the right-hand side. Sol 1(c) Set B = x, C = 1, and D = yT in part (a) to get A(A + xyT )1 = I x(1 + yT A1 x)1 yT A1 (2) (1)

Left-multiplying both sides by A and noting that 1 + yT A1 x is just a scalar, we get the so-called Sherman-Morrison formula, (A + xyT )1 = A1 2. Let i (A) denote an eigenvalue of A (a) i (AB) = i (BA) where A, B are not necessarily square. (b) If A = AT , nd the following in terms of i (A) i. ii. iii. iv. Tr(Ap ). i (I + cA). i (A cI). i (A1 ). A1 xyT A1 . 1 + yT A1 x (3)

Sol 2(a) The proof is easy if you assume that A and B are square and |A| = 0. In this case the characteristic polynomial can be written as |I BA| = |A1 A(I BA)| = |A|1 |A ABA| = |A|1 |(I AB)A| = |A|1 |(I AB)||A| = |I AB|.

(4)

Since the characteristic polynomials of AB is the same as that of BA, so are their eigenvalues. You get full marks if you prove this case. In the general case, let A Rmn and B Rnm , with m n without loss of generality. First we prove that |I AB| = |I BA| by observing the following matrix decomposition into two square matrices
[

I B I 0 = A I A I 2

][

I B 0 I AB

(5)

Also, the same matrix can be decomposed as

[

I B I B = A I 0 I

][

I BA 0 A I

(6)

Next observe that for any matrices A and B,

[

I B 0 I

I 0 A I

=1

(7)

Thus taking determinant of both expansions, we get |I AB| = |I BA|. This implies that the following two polynomials are equal |I AB/| = |I BA/| (8)

Note that the matrix on the right-hand side is n n while the matrix on the left is m m. Thus if m n, we get |I AB| = mn |I BA| (9)

Thus the larger matrix BA has at least m n zero eigenvalues, and all its other eigenvalues are equal to those of AB. Sol 2(b) If A is symmetric, i (A) are all real and the decomposition A = EET holds with ET E = EET = I and diagonal . (i) Ap = Ep ET , which implies that Tr(Ap ) =

i (Ap ) =

p i.

(ii) Note that I + cA = EET + cEET = E(I + c)ET , which implies that i (I + cA) = 1 + ci . (iii) Similarly, A cI = EET cEET = E( cI)ET , implying that i (A cI) = i c. (iv) A1 = (EET )1 = E1 ET , which implies that i (A1 ) = 1/i . 3. The notation A 0 denotes the fact that A is positive denite. Prove the following results for A 0: (a) A1 0 (b) [A]ii 0 for all i, where [A]ii denotes the i-th diagonal entry of A. (c) For any B, rank(BABT ) = rank(B). (d) If B is full row-rank, then BABT 0. (Optional) If A 0, then for any matrix X, XT AX = 0 AX = 0. 3

Sol 3(a) This was shown in the class. Let i be the eigenvalues of A. Then, A 0 i > 0i 1/i > 0 A 0. Sol 3(b) We use the following denition of positive deniteness. A 0 xT Ax > 0 x = 0 (10)

Since the right-hand side holds for all x = 0, it also holds when x1 = 1 and xi = 0 for all i = 1. In this case, it is easy to see that xT Ax = [A]11 > 0 since x = 0. The proof is similar for other diagonal entries of A. Sol 3(c) Let B Rmn . First we prove that BT x = 0 BABT x = 0 for all A 0. One direction is straightforward, i.e., if there is an x Rm such that BT x = 0, it holds that BABT x = 0. In the other direction, assume that for some y Rm , BABT y = 0 yT BABT y = 0 yT B A ABT y = 0 ABT v = 0 ABT v = 0

BT v = 0 The last implication follows from the fact that A is full-rank. In particular, since A 0, A 0, which means that A is full rank, and invertible, which means that for any u, if Au = 0 u = 0 (11) since the null space of A is empty. Having proved this equivalence, it is clear that the null spaces of BT and BABT are the same. By rank nullity theorem, rank(BT ) + null(BT ) = m = rank(BABT ) + null(BABT ) which implies that rank(BABT ) = rank(BT ) = rank(B). T T T Sol 3(d) Here, for any x = 0 , it holds that x BAB x = x B A ABT x = ABT x > 0. The last inequality follows from the fact that A and B are both full rank. Sol 3(e) Use a similar technique as in 3(c). XT AX = XT A AX = AX = 0 AX = 0 AX = 0. 4 (12)

4. Prove the following for the operator norm (with a = b, and all vector norms are .a ). (a) Ia,a = 1 (b) Aa,a = supx=1 Axa (c) Aa,a = supx=0 (e) A1,1 = maxj

Ax . x T (d) Let AT A 0, then A2 2,2 = maxi i (A A) i |[A]i,j |

(f) Ax Aa,a x (g) ABa,a Aa,a Ba,a Sol 4(a) Ia,a = supx1 Ix = supx1 x = 1. Sol 4(b,c) We can prove both these equivalences at the same time as follows. Let b1 = sup Ax
x1

b2 = sup Ax
x=1

b3 = sup

Ax x=0 x

Then since there are more restrictions on the choice of x in b2 as compared to b1 , it holds that b1 b2 . In other words, since there are more options for choosing x in b1 , I should be able to nd an x that could potentially result in a bigger value of Ax. In the denition of b3 , consider the unit norm vector u = x/ x. Since u = 1, the following inequality holds for all x = 0 Ax = Au sup Ay = b2 x y=1 which implies that b3 b2 . Finally, note that in general for any x 1, it holds that Ax Ax / x b3 . Since b3 Ax for all x 1, it implies that b3 b1 . Combining the three inequalities, b1 b3 b2 b1 , which implies that b1 = b2 = b3 . (13)

Sol 4(d) Let the eigenvalue decomposition of AT A = EE. Then, A2 2,2 = sup
x=0

xT AT Ax xT EET x = sup xT x xT x x=0

(14)

Introduce the projected u = ET x and observe that xT x = uT u. Further, there is a unique u for every x, so that it is possible to write

A2 2,2 Now, observe that

i = sup u=0

i u2 i 2 u i i

(15)

i u2 i max

i i

u2 i , which implies that

i u2 i max 2 u i i A2 2,2 max

(16)

One possible way to attain this maximum is to choose ui = 1 for i corresponding to the largest eigenvalue, and set all other ui to zero. Thus, it follows that A2 2,2 = max . Sol. 4(e) We have already seen the property that in general, will apply it here as follows. Ax1 = =

i i ai

(maxi ai )

i i .

We

Aij xj |

( (
j i j

|Aij xj |

|xj |

|Aij |
) )
j

max
j

|Aij |

|xj |. (17)

= max
j

|Aij | x1

This allows us to have the following bound, which should hold for any value of x
Ax1 max |Aij | j x 1 i

(18)

But this bound can indeed be achieved by setting xj = 1 corresponding to the j that maximizes i Aij , so the result follows. 6

Sol 4(f) Follows from denition in 4(c). Ax x=0 x Ax Aa,a x Aa,a x Ax Aa,a = sup The inequality holds trivially when x = 0. Sol 4(g) We repeatedly use the result in 4(f), ABx = A(Bx) Aa,a Bx Aa,a Ba,a x Aa,a Ba,a Aa,a Ba,a ABx x ABx sup = ABa,a x x=0 x = 0

x = 0 x = 0

5. Consider X = {x Rn |Ax = b} for A Rmn and b Rm with b R(A). Prove that the following two statements are equivalent (i.e., (a) (b) and (b) (a)) (a) cT x = d for all x X (b) There exists v Rm such that c = AT v and d = bT v. Sol. 5 First we prove (b) (a) since it is easier. Suppose there exists v Rm such that c = AT v and d = bT v. Then, cT x = vT Ax = vT b = bT v = d for all x that satisfy Ax = b. Since b R, there must exist at least some x0 such that Ax0 = b (i.e. X is nonempty). Then any x X can be written as x = x x0 + x0 . Now, A(x x0 ) = 0, so in other words, any x X can be written as x0 + y, where y is a vector that satises Ay = 0. Therefore to prove the other direction, suppose that cT x = d for all x such that Ax = b. Starting with this condition, cT x = d cT (x0 + y) = d c y=0
T

xX y : Ay = 0 y : Ay = 0

The vector c is therefore orthogonal to the null space of A, N (A). As we saw in the class, N (A) = R(AT ). This means that c lies in the range space of AT , or in other words, there exists some v Rm such that c = AT v. It follows that d = vT Ax = vT b = bT v. 7

6. Prove that
2 2 T (a) u + v2 2 = u2 + v2 if and only if u v = 0.

(b) 2 < a, b > +2 < x, y >=< a + x, b + y > + < a x, b y > (c) x1 n x2 (d) x1 x2 x
2 2 T Sol. 6(a) u + v2 2 u2 v2 = 2u v

Sol. 6(b) (a + x)T (b + y) + (a x)T (b y) = aT b + aT y + xT b + xT y + aT b aT y xT b + xT y = 2aT b + 2xT y. Sol. 6(c) Observe that
2 n x 2 2 x1 = (n 1)(

x2 i) 2

i=j

|xi xj |

= =

i=j i=j

2 x2 i + xj 2|xi xj |

(xi xj )2 0

(19)

Sol. 6(d) For the rst inequality, x 2 1 =

|xi |2 +

i=j

|xi xj |

2 x2 i = x2

(20)

The second inequality is obvious since maxi {x2 i}

x2 i.

(Optional) Consider A Rmn where m n and rank of A is n. Suppose there exists B such that BA = I and that AB = BT AT . (a) Let X = AB. Show that i. X2 = X ii. Tr(X) = rank(X) (b) Find an expression for B in terms of A. (c) For any two vectors b Rn and c Rm , prove that Ab c2 ABc c2 . Sol. 7(a) (i) ABAB = A(BA)B = AB (ii) Let and v be such that Xv = v. Multiplying both sides by X, we get X2 v = (Xv) 8

 Thus, tr(X) =

Xv = 2 v 2 = {0, 1}

(21)

i = number of non-zero eigenvalues = rank(X).

Sol. 7(b) Since rank of A is n, the matrix AT A is full rank (see 3(c) with A = I). If we set B = (AT A)1 AT , we can see that all properties are indeed satised. Sol. 7(c) Introduce a new vector x := b Bc, and obseve that Ab c = Ax + ABc c. Taking the norm, it follows that
2 Ab c2 2 = Ax + (AB I)c2

= xT AT Ax + cT (X I)T (XT I)c + cT (X I)T Ax + xT AT (X I) = xT AT Ax + cT (I X)c I Xc = A(b Bc)2 + 2

2 2

(22)

where since eigenvalues of X are either 0 or 1, I X 0. Thus, Ab c is minimized when we pick b = Bc. Note that an easier way is to simply take the derivative of Ab c2 2 with respect to b and set it to zero.