INTRODUCTION

TO

LINEAR

ALGEBRA

Fifth Edition

MANUAL FOR INSTRUCTORS

Gilbert Strang
Massachusetts Institute of Technology

math.mit.edu/linearalgebra
web.mit.edu/18.06
video lectures: ocw.mit.edu
math.mit.edu/∼gs
www.wellesleycambridge.com
email: [email protected]

Wellesley - Cambridge Press

Box 812060
Wellesley, Massachusetts 02482
Solutions to Exercises 169

Problem Set 11.1, page 516

1 Without exchange, pivots .001 and 1000; with exchange, 1 and −1. When the pivot is
 
1 1 1
|entry|  
 
larger than the entries below it, all |ℓij | = ≤ 1. A =  0 1 −1 .
|pivot|  
−1 1 1
 
9 −36 30
 
 
2 The exact inverse of hilb(3) is A−1 = −36 192 −180 .
 
30 −180 180
         
1 11/6 1.833 0 1.80
         
         
3 A  1  =  13/12  =  1.083  compares with A  6  =  1.10 . k∆bk < .04 but
         
1 47/60 0.783 −3.6 0.78 k∆xk > 6.
The difference (1, 1, 1) − (0, 6, −3.6) is in a direction ∆x that has A∆x near zero.

4 The largest kxk = kA−1 bk is kA−1 k = 1/λmin since AT = A; largest error 10−16 /λmin .

5 Each row of U has at most w entries. Use w multiplications to substitute components

of x (already known from below) and divide by the pivot. Total for n rows < wn.
1 2
6 The triangular L−1 , U −1 , R−1 need 2
n multiplications. Q needs n2 to multiply the
right side by Q−1 = QT . So QRx = b takes 1.5 times longer than LU x = b.

7 U U −1 = I: Back substitution needs 12 j 2 multiplications on column j, using the j by

j upper left block. Then 12 (12 + 22 + · · · + n2 ) ≈ 12 ( 13 n3 ) = total to find U −1 .
         
1 0 2 2 2 2 0 1 1 0
8   →   →   = U with P =   and L =  ;
2 2 1 0 0 −1 1 0 .5 1
       
2 2 0 2 2 0 2 2 0 2 2 0
       
       
A →  1 0 1  →  0 −1 1  →  0 2 0  →  0 2 0  = U with
       
0 2 0 0 2 0 0 −1 1 0 0 1
   
0 1 0 1 0 0
   
   
P =  0 0 1  and L =  0 1 0 .
   
1 0 0 .5 −.5 1
170 Solutions to Exercises
 
1 1 0 0
 
 
1 1 1 0  has cofactors C13 = C31 = C24 = C42 = 1 and
9 A=



0 1 1 1  C14 = C41 = −1. A−1 is a full matrix!
 
0 0 1 1
10 With 16-digit floating point arithmetic the errors kx − xcomputedk for ε = 10−3 , 10−6 ,

10−9 , 10−12 , 10−15 are of order 10−16 , 10−11 , 10−7 , 10−4 , 10−3 .
    
√ √ 1 3 1 −1 10 14
11 (a) cos θ = 1/ 10, sin θ = −3/ 10, R = √110    = √1 
10
.
−3 1 3 5 0 8
(b) A has eigenvalues
  4 and 2. Put one  eigenvectors in row 1 of Q: either
 of the unit
1 1 −1 2 −4
Q= √   and QAQ−1 =   or
2 1 1 0 4
   
1 1 −3 4 −4
Q= √   and QAQ−1 =  .
10 3 1 0 2
12 When A is multiplied by a plane rotation Qij , this changes the 2n (not n2 ) entries in

rows i and j. Then multiplying on the right by (Qij )−1 = (Qij )T changes the 2n
entries in columns i and j.

13 Qij A uses 4n multiplications (2 for each entry in rows i and j). By factoring out cos θ,

the entries 1 and ± tan θ need only 2n multiplications, which leads to 23 n3 for QR.
1
14 The (2, 1) entry of Q21 A is 3 (− sin θ + 2 cos θ). This is zero if sin θ = 2 cos θ or
√ √ √
tan θ = 2. Then the 2, 1, 5 right triangle has sin θ = 2/ 5 and cos θ = 1/ 5.

Every 3 by 3 rotation with det Q = +1 is the product of 3 plane rotations.

15 This problem shows how elimination is more expensive (the nonzero multipliers in L

and LL are counted by nnz(L) and nnz(LL)) when we spoil the tridiagonal K by a
random permutation.

If on the other hand we start with a poorly ordered matrix K, an improved ordering
is found by the code symamd discussed in this section.
Solutions to Exercises 171

16 The “red-black ordering” puts rows and columns 1 to 10 in the odd-even order 1, 3, 5, 7,

9, 2, 4, 6, 8, 10. When K is the −1, 2, −1 tridiagonal matrix, odd points are connected
only to even points (and 2 stays on the diagonal, connecting every point to itself):

 
2 −1
   
 
 −1 2 −1  2I D
K=

 and P KP T = 

 with
 · · · DT 2I
 
−1 2
 
−1 1 to 2
 
 
 −1 −1  3 to 2, 4
 
 
D =  0 −1 −1  5 to 4, 6
 
 
 −1 −1  7 to 6, 8
 
−1 −1 9 to 8, 10

17 Jeff Stuart’s Shake a Stick activity has long sticks representing the graphs of two linear

equations in the x-y plane. The matrix is nearly singular and Section 9.2 shows how to
compute its condition number c = kAkkA−1 k = σmax /σmin ≈ 80, 000:

   
kA−1 k ≈ 20000
 1 1.0001  −1  −1 1.0001 
A=  kAk ≈ 2 A = 10000  
1 1.0000 1 −1 c ≈ 40000.

Problem Set 11.2, page 522

√
1 kAk = 2, kA−1 k = 2, c = 4; kAk = 3, kA−1 k = 1, c = 3; kAk = 2 + 2=
√ √
λmax for positive definite A, kA−1 k = 1/λmin , comd = (2 + 2)/(2 − 2) = 5.83.
√ √
2 kAk = 2, c = 1; kAk = 2, c = ∞ (singular matrix); AT A = 2I, kAk = 2, c = 1.

3 For the first inequality replace x by Bx in kAxk ≤ kAkkxk; the second inequality is

just kBxk ≤ kBkkxk. Then kABk = max(kABxk/kxk) ≤ kAkkBk.

4 1 = kIk = kAA−1 k ≤ kAkkA−1 k = c(A).

172 Solutions to Exercises

5 If Λmax = Λmin = 1 then all Λi = 1 and A = SIS −1 = I. The only matrices with

kAk = kA−1 k = 1 are orthogonal matrices.

6 All orthogonal matrices have norm 1, so kAk ≤ kQkkRk = kRk and in reverse kRk ≤

kQ−1 kkAk = kAk. Then kAk = kRk. Inequality is usual in kAk < kLkkU k when
AT A 6= AAT . Use norm on a random A.

7 The triangle inequality gives kAx + Bxk ≤ kAxk + kBxk. Divide by kxk and take

the maximum over all nonzero vectors to find kA + Bk ≤ kAk + kBk.

8 If Ax = λx then kAxk/kxk = |λ| for that particular vector x. When we maximize

the ratio kAxk/kxk over all vectors we get kAk ≥ |λ|.

     
0 1 0 0 0 1
9 A+B =  + =  has ρ(A) = 0 and ρ(B) = 0 but ρ(A + B) = 1.
0 0 1 0 1 0
 
1 0
The triangle inequality kA + Bk ≤ kAk + kBk fails for ρ(A). AB =   has
0 0
ρ(AB) > ρ(A) ρ(B); thus ρ(A) = max |λ(A)| = spectral radius is not a norm.

10 (a) The condition number of A−1 is kA−1 kk(A−1 )−1 k which is kA−1 kkAk = c(A).

(b) Since AT A and AAT have the same nonzero eigenvalues, AT has the same norm
as A.

11 Use the quadratic formula for λmax /λmin , which is c = σmax /σmin since this A = AT

is positive definite:
 p   √ 
c(A) = 1.00005 + (1.00005)2 − .0001 / 1.00005 − ≈ 40, 000.

12 det(2A) is not 2 det A; det(A + B) is not always less than det A + det B; taking

| det A| does not help. The only reasonable property is det AB = (det A)(det B). The
condition number should not change when A is multiplied by 10.

13 The residual b − Ay = (10−7 , 0) is much smaller than b − Az = (.0013, .0016). But

z is much closer to the solution than y.

 
659 −563
14 det A = 10−6 so A−1 = 103  : kAk > 1, kA−1 k > 106 , then c > 106 .
−913 780
Solutions to Exercises 173
√
15 x = (1, 1, 1, 1, 1) has kxk = 5, kxk1 = 5, kxk∞ = 1. x = (.1, .7, .3, .4, .5) has
kxk = 1, kxk1 = 2 (sum), kxk∞ = .7 (largest).

16 x21 +· · ·+x2n is not smaller than max(x2i ) and not larger than (|x1 |+· · ·+|xn |)2 = kxk21 .
√
x21 + · · · + x2n ≤ n max(x2i ) so kxk ≤ nkxk∞ . Choose yi = sign xi = ±1 to get
√ √
kxk1 = x · y ≤ kxkkyk = nkxk. The vector x = (1, . . . , 1) has kxk1 = n kxk.

17 For the ℓ∞ norm, the largest component of x plus the largest component of y is not

less than kx + yk∞ = largest component of x + y.

For the ℓ1 norm, each component has |xi + yi | ≤ |xi | + |yi |. Sum on i = 1 to n:
kx + yk1 ≤ kxk1 + kyk1 .

18 |x1 | + 2|x2 | is a norm but min(|x1 |, |x2 |) is not a norm. kxk + kxk∞ is a norm;

kAxk is a norm provided A is invertible (otherwise a nonzero vector has norm zero;
for rectangular A we require independent columns to avoid kAxk = 0).

19 xT y = x1 y1 + x2 y2 + · · · ≤ (max |yi |)(|x1 | + |x2 | + · · · ) = ||x||1 ||y||∞ .

20 With λj = 2 − 2 cos(jπ/n+1), the largest eigenvalue is λn ≈ 2 + 2 = 4. The smallest

 2
π
is λ1 = 2 − 2 cos(π/n+1) ≈ n+1 , using 2 cos θ ≈ 2 − θ2 . So the condition number
is c = λmax /λmin ≈ (4/π 2 ) n2 , growing with n.

Problem Set 11.3, page 531

1 The iteration xk+1 = (I − A)xk + b has S = I and T = I − A and S −1 T = I − A.

2 If Ax = λx then (I −A)x = (1−λ)x. Real eigenvalues of B = I −A have |1−λ| < 1

provided λ is between 0 and 2.

 
−1 1
3 This matrix A has I − A =   which has |λ| = 2. The iteration diverges.
1 −1
4 Always kABk ≤ kAkkBk. Choose A = B to find kB 2 k ≤ kBk2 . Then choose A =

B 2 to find kB 3 k ≤ kB 2 kkBk ≤ kBk3 . Continue (or use induction) to find kB k k ≤

kBkk . Since kBk ≥ max |λ(B)| it is no surprise that kBk < 1 gives convergence.
174 Solutions to Exercises

5 Ax = 0 gives (S − T )x = 0. Then Sx = T x and S −1 T x = x. Then λ = 1 means

that the errors do not approach zero. We can’t expect convergence when A is singular
and Ax = b is unsolvable!
 
0 1
6 Jacobi has S −1 T = 13   with |λ|max = 1 . Small problem, fast convergence.
3
1 0
 
1
0 3  1
7 Gauss-Seidel has S −1 T =   with |λ|max = 9 which is (|λ|max for Jacobi)2 .
1
0 9

 −1    
a 0 −b 0 −b/a
8 Jacobi has S −1 T =    =  with |λ| = |bc/ad|1/2 .
d −c 0 −c/d 0
 −1    
a 0 0 −b 0 −b/a
Gauss-Seidel has S −1 T =    =  with |λ| = |bc/ad|.
c d 0 0 0 −bc/ad
So Gauss-Seidel is twice as fast to converge if |λ| < 1 (or to explode if |bc| > |ad|).
 
π
9 Gauss-Seidel will converge for the −1, 2, −1 matrix. |λ|max = cos2 n+1
is given
on page 527, together with the improvement from successive overrelaxation.

10 If the iteration gives all xnew

i = xold
i then the quantity in parentheses is zero, which

means Ax = b. For Jacobi change xnew on the right side to xold .

11 uk /λk1 = c1 x1 + c2 x2 (λ2 /λ1 )k + · · · + cn xn (λn /λ1 )k → c1 x1 if all ratios |λi /λ1 | <
 
0 1
1. The largest ratio controls the rate of convergence (when k is large). A =  
1 0
has |λ2 | = |λ1 | and no convergence.

12 The eigenvectors of A and also A−1 are x1 = (.75, .25) and x2 = (1, −1). The inverse

power method converges to a multiple of x2 , since |1/λ2 | > |1/λ1 |.

(j−1)π
jπ
13 In the jth component of Ax1 , λ1 sin n+1 jπ
= 2 sin n+1 − sin n+1 − sin (j+1)π
n+1 .
jπ π π
The last two terms combine into −2 sin n+1 cos n+1 . Then λ1 = 2 − 2 cos n+1 .
Solutions to Exercises 175
         
2 −1 1 2 5 14
14 A =   produces u0 =  , u1 =  , u2 =  , u3 =  .
−1 2 0 −1 −4 −13
 
1
This is converging to the eigenvector direction   with largest eigenvalue λ = 3.
−1
Divide uk by kuk k to keep unit vectors.
         
1 2 1 2 5 14 1/2
15 A−1 =   gives u1 = 1  , u2 = 1  , u3 = 1   → u∞ =  .
3 1 2 3 1 9 4 27 13 1/2
   
1 cos θ sin θ cos θ(1 + sin2 θ) − sin3 θ
16 R = QT A =   and A1 = RQ =  .
0 − sin2 θ − sin3 θ − cos θ sin2 θ
17 If A is orthogonal then Q = A and R = I. Therefore A1 = RQ = A again, and the

“QR method” doesn’t move from A. But shift A slightly and the method goes quickly
to Λ.

18 If A − cI = QR then A1 = RQ + cI = Q−1 (QR + cI)Q = Q−1 AQ. No change in

eigenvalues from the shift and shift back, because A1 is similar to A.

19 Multiply Aq j = bj−1 q j−1 + aj q j + bj q j+1 by q T T

j to find q j Aq j = aj (because the

q’s are orthonormal). The matrix form (multiplying by columns) is AQ = QT where

T is tridiagonal. The entries down the diagonals of T are the a’s and b’s.

20 Theoretically the q’s are orthonormal. In reality this important algorithm is not very

stable. We must stop every few steps to reorthogonalize—or find another more stable
way to orthogonalize the sequence q, Aq, A2 q, . . .

21 If A is symmetric then A1 = Q−1 AQ = QT AQ is also symmetric. A1 = RQ =

R(QR)R−1 = RAR−1 has R and R−1 upper triangular, so A1 cannot have nonzeros
on a lower diagonal than A. If A is tridiagonal and symmetric then (by using symmetry
for the upper part of A1 ) the matrix A1 = RAR−1 is also tridiagonal.

22 From the last line of code, q 2 is in the direction of v = Aq 1 − h11 q 1 = Aq 1 −

(q T
1 Aq 1 )q 1 . The dot product with q 1 is zero. This is Gram-Schmidt with Aq 1 as the

second input vector; we subtract from Aq 1 its projection onto the first vector q 1 .
176 Solutions to Exercises

Note The three lines after the short “pseudocodes” describe two key properties of con-
jugate gradients—the residuals r k = b − Axk are orthogonal and the search directions
are A-orthogonal (dT
i Adk = 0). Then each new approximation xk+1 is the closest

vector to x among all combinations of b, Ab . . . , Ak b. Ordinary iteration Sxk+1 =

T xk + b does not find this best possible combination xk+1 .

23 The solution is straightforward and important. Since H = Q−1 AQ = QT AQ is

symmetric if A = AT , and since H has only one lower diagonal by construction, then
H has only one upper diagonal: H is tridiagonal and all the recursions in Arnoldi’s
method have only 3 terms.

24 H = Q−1 AQ is similar to A, so H has the same eigenvalues as A (at the end of

Arnoldi). When Arnoldi is stopped sooner because the matrix size is large, the eigen-
values of Hk (called Ritz values) are close to eigenvalues of A. This is an important
way to compute approximations to λ for large matrices.

25 In principle the conjugate gradient method converges in 100 (or 99) steps to the exact

solution x. But it is slower than elimination and its all-important property is to give
good approximations to x much sooner. (Stopping elimination part way leaves you
nothing.) The problem asks how close x10 and x20 are to x100 , which equals x except
for roundoff errors.
   
1 1 1 q
26 A =   has An =   with q = 1 + 1.1 + · · · + (1.1)n−1 =
0 1.1 0 (1.1)n
 
0 10
(1.1n − 1)/(1.1 − 1) ≈ 10 (1.1)n . So the growing part of An is (1.1)n  
0 1
√
with ||An || ≈ 101 times 1.1n for larger n.