Matrix algorithms (part 1)

Computational Techniques II

Contents
1 System of linear equations 2 Gaussian elimination 3 Gauss-Jordan elimination 4 Pivoting strategies 4.1 Partial pivoting . . . . 4.2 Complete pivoting . . 4.3 Keeping track of swaps 4.3.1 Method 1 . . . 4.3.2 Method 2 . . . 3 4 8 9 9 11 12 12 13 14

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

5 Some theory behind Gauss-Jordan elimination

6 Inversion using Gauss-Jordan elimination 15 6.1 Ecient implementation . . . . . . . . . . . . . . . . . . . . . . . 16 7 Determinant using Gaussian elimination 8 LU 8.1 8.2 8.3 8.4 decompositions Application . . . . . . . Algorithm . . . . . . . . Ecient implementation Some theory . . . . . . . 18 18 19 19 20 20

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

9 QR 9.1 9.2 9.3 9.4

decomposition Householders transformation Using Householder for QR . . Ecient implementation . . . Application to least squares .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

21 22 22 23 27

System of linear equations

Example: Consider the following three equations: 2x 3y + z 3x + y + 2z 2x + 3z = = = 5 15 2

This is an example of a system of linear equations. Here system means a collection, and the equations are called linear because they do not involve any squares or higher powers of the unknowns like x2 , z 3 , or product terms like xy, xy 3 z etc. In the above example we have three unknowns: x, y and z. We can write a system of m linear equations in n unknowns using matrix notation: Ax = b, where Amn is called the coecient matrix, xm1 is called the vector of unknowns, and bn1 is called the rhs vector. Example: The system in the last example can be written as Thus, here 2 3 1 A = 3 1 2 , 2 0 3 x x = y , z 5 b = 15 . 2 2 3 2 3 1 x 5 1 2 y = 15 . 0 3 z 2

If the rhs vector b is 0, then we call the system homogeneous, else we call it non-homogeneous. Any value of x for which Ax = b, is called a solution of the system. A system of linear equations has either exactly one solution, or innitely many solutions or no solution at all. In the last case we call the system inconsistent, otherwise it is called consistent.

If A is a nonsingular matrix then for any b, the system Ax = b is consistent, and has exactly one solution, viz., x = A1 b.

Gaussian elimination

This is the matrix version of a method for solving a system of linear equations that we learn at high school. Example: Consider the following system of three linear equations, which we call 1 , 1 and 1 . 1 : 1 : 1 : x y 2x +5y x +2y +z z 3z = 2 = 9 = 4

In high school we used to solve this by eliminating the unknowns one by one until only one remained. Then we could solve for that one. Once we know one of the unknowns we substitute this value in the equations to get a system with only two unknowns. Then we eliminated one of these, and so on. Example: Let us rst eliminate x from the last two equations by subtracting multiples of the rst equation from them. Here are the resulting 3 equations, which we call 2 , 2 and 2 . 2 = 2 = 2 = 1 : 1 21 : 1 1 : x y 7y 3y +z 3z 4z = = = 2 5 6

Now let us eliminate y from the last equation 3 = 2 : 3 = 2 : 3 2 : 3 = 2 7 x y 7y +z 3z 19 7 z = = = 2 5 57 7

Now the third equation involves only one unknown, z. So we can solve for z. The second equation involves both z and y. So once we have solved for z from the third equation we can use the second equation to solve for y. The rst

equation (which involves x as well as y and z ) can then be used for solving x. This approach is called back substitution. From the third equation: z = 3, then from the second equation y= 1 (5 + 3z ) = 2, 7

and nally, from the rst equation, x = 2 + y z = 1.

This is what is called Gaussian elimination (with back substitution) in matrix theory. While doing Gaussian elimination it is customary to write the system at each step in the augmented matrix form. This is done in the example below. Example: The augmented matrix form of the given system is as follows. 1 2 1 1 1 2 9 5 1 2 3 4

It is obtained by appending the rhs after the matrix. We draw a vertical line to keep the rhs separate. While doing Gaussian elimination we work with only the augmented matrices. To move from one augmented matrix to the next we need certain kind of operation that we call an F -move. Before each F -move we have to choose some nonzero entry in the matrix. This is called the pivot for that step. The position of the pivot may change from step to step. The row (or column) that contains the pivot is called the pivotal row (or column). Forward elimination (F): In this move we add multiples of the pivotal row to the rows below it so that the entries in the pivotal column below the pivot become zero. In the example below, we need to add 5 times the rst row to the second row, and 2 times the rst row to the third row. 1 2 1 -1 5 2 1 2 9 1 3 4
F

1 7 3

-1 0 0

1 2 4 19 1 0

This operation only aects the rows below the pivotal row. The pivotal row itself, as well as the rows above it remain unchanged. 5

An F -move may be computed easily if you remember the following diagram. Here p is the pivot, and we want to know how a will change by a F - or B -move. First locate the entry b in the pivotal row which is in the same column as a. Similarly, let c be the entry in the pivotal column having the same row as a. Then a F -move will change a to a bc/p.
pivotal column

pivotal row

p c

b a

a could also be to the left of the pivotal column. Example: Let us use this method see how the bold entry changes when we apply F -move to the following augmented matrix: 2 2 3 7 4 5 6 15 1 1 1 3 The entry in the pivotal row sharing same column with 15 is 7. The entry in the pivotal column sharing the same row with 15 is 4. The pivot is 2. So 15 will change to 74 = 1. 15 2

In Gaussian elimination of an n n system (i.e., where the augmented matrix is n (n + 1)) we start with the (1, 1)-th entry as the pivot. Then we perform the following operations. F, , , F, .
n1

times

The symbol means shift the pivot diagonally downward by one place. If the pivot turns out to be zero at any step, then stop and output Failure. Otherwise, you will end up with an augmented system b A ,

will be an upper triangular matrix with nonzero diagonal entries. Now where A you have to apply back substitution to solve for xn , ..., x1 in this order as follows. n 1 bk a kj xj , xk = a kk
j =k+1

for k = n, n 1, ..., 2, 1. It might help to write down the back substitution part explicitly for the 3 3 case where the unknowns are x, y and z. = b3 /a 33 y = ( b2 a 23 z )/a 22 x = (b1 a 13 z a 12 y )/a 11 z

Example: Here is the algorithm in action. We have applied it to the system from the last example. 1 1 1 2 2 5 1 9 1 2 3 4 1 1 0 7 0 0 1 3 -19/7 2 5 57/7

1 0 0

2 1 1 7 3 5 3 4 6

Now we have an upper triangular system, i.e., a system where the coecient matrix (the part to the left of the vertical divider in the augmented matrix) is upper triangular. We shall now apply back substitution to solve for z, then y, and then x as follows. 57/7 z= = 3, 19/7 7

and y= and nally,

5+33 1 (5 + 3z ) = = 2, 7 7

x = 2 + y z = 2 + 2 3 = 1.

The following diagram shows the order of the Gaussian elimination operations for a 4 4 matrix. The gray regions hold the nonzero entries. The white regions are all zeroes. The little triangles to the left mark the rows that are going to be aected by the next move. The position of the pivot is shown as a circle.
F F F

Project 1:Implement the Gaussian elimination algorithm discussed above.

Gauss-Jordan elimination

Gauss-Jordan elimination is basically the same as Gaussian elimination except that here we have two new moves: the S -move and the B -move. Scaling (S): This operation involves dividing the entire pivotal row by the pivot. So in place of the pivot we shall have 1. This operation aects only the pivotal row, and leaves the rest of the matrix unaltered. Here is an example. The pivot is shown in a box. 1 2 1 2 1 1 5 1 9 2 3 4
S

2 1 1 1 2/5 1 1/5 9/5 1 2 3 4

Back substitution (B): This is same as a F-move, except that now we work on the rows above the pivotal row. This operation leaves the pivotal row and the ones below it unaltered. 1 1 1 2 4/3 1/3 0 2/3 B 2 5/3 13/3 9 5 1 0 31/3 1 2 -3 4 1 2 -3 4 8

Here we have subtracted 1 3 times the last row from the second row, and added 1 times the last row to the rst row. 3 In Gauss-Jordan elimination of a n n system we start with the pivot at the (1, 1)-th position. Then we perform the following operations. S, F, B, , , S, F, B, , S, B.
n1

times

As before the symbol means shift the pivot diagonally downward by one place.

Example: Let us perform Gauss-Jordan elimination for the same system. We start with the original system in augmented matrix form. 2 1 1 1 1 1 1 2 (S,F ) 2 0 7 3 5 1 9 5 1 2 3 4 0 3 4 6 (S, F, B ) 19/7 1 0 4/7 1 0 0 1 (S,B ) 5/7 3/7 0 1 0 2 0 1 0 0 1 3 0 0 -19/7 57/7

Pivoting strategies

We have already mentioned the three basic steps: B,F, and S. All these steps involve division by the pivot. So the pivot needs to be nonzero. However, it may happen that while performing the elimination the pivot turns out to be zero. Then we need to swap rows and/or columns to bring a nonzero number in the position of the pivot. This is called pivoting. Pivoting is useful even when the pivot is nonzero, but is very small in absolute value. This is because, division by numbers new zero introduces large errors in the computer output. We shall discuss two pivoting strategies: partial pivoting, and complete pivoting.

4.1

Partial pivoting

Partial pivoting involves swapping rows. Suppose that at some step in the Gaussian or Gauss-Jordan algorithm you have the following augmented matrix, 9

where the pivot is shown in a box. 1 2 3 6 2 9 4 3 5 1 2 2 To perform partial pivoting look at the pivot as well as all the entries below the pivot in the pivotal column. These are shown bold. Locate among these the entry with the maximum absolute value. Here it is 5. Swap the pivotal row with with the row containing this entry. In this example we swap row 1 with row 3 to get -5 1 2 2 2 9 4 3 6 1 2 3 Note that the position of the pivot (shown by the box) has remained unchanged, though the pivot itself has changed from 1 to 5. In partial pivoting you look for a candidate pivot from among the current pivot and the entries below it:

Partial pivoting

Example: Here are some more examples of partial pivoting. 2 0 0 10 1 1 2 3 1 2 1 Pp 2 0 5 0 10 1 3 1 1 2 1 5 2 3 5 9

10 3 4 10 1 3 2 13 8

3 10 3 4 Pp 5 10 1 3 9 2 13 8

We have called the partial pivoting step as Pp . So now we have four dierent kinds of steps: S, F, B and Pp .

10

In Gaussian elimination with partial pivoting we perform the following steps: Pp , F, , , Pp , F, ,

times

and then we perform back substitution as before. If the pivot turns out to be zero at any step (i.e., all the entries at and below the pivotal position are zero then the system is singular, and the pivotal variable can be set to any value. For Gauss-Jordan elimination we do Pp , S, F, B, , , Pp , S, F, B, , S, B.
n1

times

4.2

Complete pivoting

In complete pivoting we swap rows and/or columns. Suppose the pivot is at the (i, i)-th entry. Then we look at all the (r, s)-th entries where r, s i.

Complete pivoting

Notice that we do not cross the vertical divider. 1 2 3 6 3 2 1 5 4 1 4 10 Locate the entry with the maximum absolute value among the bold entries. If this is the (r, s)-th entry then swap i-th row of the augmented matrix with r-th row, and the i-column with the s-th column. Note that the order of row and column swaps does not matter. In this example we swap row 2 with row 3, and column 2 with column 3 to get 6 1 2 3 4 -4 1 10 3 1 2 5 Recall that the columns of the matrix correspond to the variables of the equations. So swapping the columns also involves reordering the variables. A simple way to keep track of the order of the variables is to write the variables above the columns. If we call the variables as x, y, z in the last example then we shall write: 11

x 1 3 4

y 2 2 1

z 3 6 1 5 4 10

x 1 4 3

z 2 -4 1

y 3 6 1 10 2 5

Here Pc is our symbol for complete pivoting.

In Gaussian elimination with complete pivoting we perform the following steps: Pc , F, , , Pc , F, ,

times

followed by back substitution as before. If we cannot avoid having a zero pivot at some step (i.e., all the entries at and to the south-east of the pivotal position are zeros) then the system is singular, and the remaining variables (including the pivotal variable) can be set to any arbitrary value if the system is consistent. For Gauss-Jordan elimination we do Pc , S, F, B, , , Pc , S, F, B, , S, B.
n1

times

4.3
4.3.1

Keeping track of swaps

There are two ways to keep track of the column swaps. Both uses an extra integer array. In the rst version we keep an integer array colPerm[0],...,colPerm[n-1], say. Initially colPerm[i] stores i for all i. This means all the columns are in their right places to start with. Every time we swap two columns we also swap the corresponding entries of the colPerm array. Thus, at the end colPerm[i] stores the index of the variable at the i-th position now. We need to nd its inverse permutation, to know where a particular variable is. Remember that if i a permutation, then its inverse permutation is 1 where i 1 ( (i)) = i. So we can dene a new array called, say, invColPerm[0,...,n-1] and ll it up as for(i=0;in;i++) invColPerm[colPerm[i]] = i; Now the solution for the i-th variable can be found in a[invColPerm[i]][n].

12

4.3.2

Method 2

This is a conceptually simpler, though computationally more demanding solution. Here we keep an integer array colIndex[0,...,n-1]. Instead of referring to the (i, j )-th entry of A as a[i][j] we shall use a[i][colIndex[j]]. Initially, colIndex[i] will store i for all i = 0, ..., n 1. So a[i][colIndex[j]] will be the same as a[i][j]. When we need to swap two columns we shall instead only swap the corresponding elements of colIndex. The following example will make things clear. Example: Let n = 3. After we swap columns 0 and 1, we have colIndex[0] = 1, colIndex[1] = 0, colIndex[2] = 2. So the (0, 1)-th entry of the new matrix is now a[0][colIndex[1]], which is a[0][0], as it should be. If we perform the entire Gauss-Jordan algorithm like this, then there is no need to change anything at the end. The last c9olumn of the nal augmented matrix will automatically store the required solution in the correct order!

Exercise 1:Show that the operations F, S and B all correspond to premultiplication by special matrices. These matrices are called elementary matrices. Write down the elementary matrices for performing these three basic moves for an n n matrix assuming that the pivotal position is (i, i). What similar statement can you make about Pp and Pc ? Compute the product of the elementary matrices for S, F and B moves.

Project 2:Implement Gauss-Jordan elimination with complete pivoting.

Exercise 2:Suppose that we want to solve multiple systems with a common coecient
matrix but dierent r.h.s. vectors and unknown vectors. Ax1 = b1 Ax2 = b2 . . . Axk = bk . This is the same as solving the matrix equation: AX = B,

13

where the bi s are the columns of B, and the xi s are the columns of X. Here we can apply Gauss/Gauss-Jordan elimination with the augmented matrix: [A | B ]. Write a program to implement Gauss-Jordan elimination (with complete pivoting) in this set up.

Some theory behind Gauss-Jordan elimination

Notice that we are using four basic actions: dividing a row by a number, subtracting multiple of one row from another, swapping two rows, and swapping two columns. Of these the rst three are row operations and can be done by left multiplication with elementary matrices. The last operation is a column operation, and hence done by right multiplication. If we accumulate all the left and right marices then at the end of Gauss-Jordan elimination we are left with LAR = I. So A = L1 R1 , or A1 = RL. Convince yourself that left multiplying by R is same as applying the column swaps to the rows in the reverse order. From now on we shall assume that R = I for ease of exposition. The results should be easily modiable for general R. Here is another interesting point. After p steps the matrix is of the form Ip O P b1 Q b2 .

When does total pivoting fail? When Q = O. Thus the system is inconsistent if b2 = 0. If b2 = 0 then the general solution is y arbitrary and x = b1 P y. Another interesting point is
1 Q = A22 A21 A 11 A12 .

This is because if all the accumulated left multiplier matrix is L then it is of the form Upp V L= . W I Do you see why the lower right hand block is I ? Now we have LA = Upp W V I A11 A21 14 A12 A22 = Ip O P Q .

So Q = W A12 + A22 . To nd W we note that W A11 + A21 = O, or

So combining everything we have

an easy proof by just performing the block matrix multiplication. This result is useful because we often have to compute the rhs in statistics. It is called the conditional covariance matrix.

Inversion using Gauss-Jordan elimination

Gauss-Jordan elimination (with pivoting) may be used to nd inverse of a given nonsingular square matrix, since nding inverse is the same as solving the system AX = I. 1 1 5 1 . 2 3

1 Example: Suppose that we want to nd the inverse of the matrix 2 1 Then we append an identity matrix of the same to get the augmented matrix 1 1 1 1 0 2 5 1 0 1 1 2 3 0 0

size to the right of this matrix 0 0 . 1

Now go on applying Gauss-Jordan elimination until the left hand matrix is reduced to identity. The right hand at the nal step will give the required inverse. Each step combines the S,F and B-moves. 1 1 1 1 0 0 1 1 1 1 0 0 (S,F ) 2 5 1 0 1 0 0 7 3 2 1 0 1 2 3 0 0 1 0 3 4 1 0 1 After a set of S, F and B steps we arrive at 5/7 4/7 1 0 -3/7 -2/7 0 1 0 0 -19/7 -1/7 15 0 0 , 1

1/7 1/7 -3/7

which after the nal S, B steps yields 1 0 0 1 0 0 Thus, the required inverse is 0 0 1
91/133 -35/133 1/19 7/133 28/133 3/19 4/19 -3/19 -7/19

91/133 -35/133 1/19

7/133 28/133 3/19

4/19 -3/19 -7/19

6.1

Ecient implementation
do not need to perform any row/column swap. 1 0 0 1 0 0 0 0 . 1 0 0 . 1

First consider the case where we Then we start out like this:

Here a denotes an entry that depends sweep we have 1 0 0 Notice that the rst column is now have 1 0 0 1 0 0 After the third step we have 1 0 0

on the matrix being inverted. After one 0 1 0

forced to be (1, 0, 0) . After another step we 0 0 . 1 .

0 0 1 0 0 1

The s in the right hand half now constitute the inverse matrix. Notice that at each step there are exactly as many s as there are entries in the original matrix. Clearly, all that we really need to store are the s, because the othr entries are anyway constants irrespective of the matrix being being inverted. It should not be dicult to check that the following algorithm computes the s:

16

For p = 0 to n 1 you compute aij aip apj app = = = = aip apj i, j = p app aip /app i = p aij apj /app j = p 1/app .

Then in the above algorithm the i-th column of A is needed only to compute the i-column of A1 . So we can overwrite the i-th column of A with the i-th column of A1 to save space. This is an example of an in place algorithm, i.e., where the output gradually overwrites the input. If we need to perform row/column swaps, then the situation is very similar. To see this consider the same example once again. We start out as 1 0 0 0 1 0 . 0 0 1 After one sweep we have 1 0 0 0 1 0 0 0 . 1

Now suppose we swap the second and 1 0 0 The next sweep will produce 1 0 0 0 1 0

third rows: 0 0 0 1 . 1 0 0 0 1 .

Notice that we still have exactly the same more side by side. The third sweep will yield 1 0 0 0 1 0 0 0 1

number of s, but now they are no

Here the right hand half stores the inverse of the row-swapped version of the original matrix, i.e., (LA)1 , where L is the row-swap matrix. Now (LA)1 = A1 L1 . Hence A1 = (LA)1 L. 17

When L is multiplied from the right, it swaps the columns in the reverse order. Thus we need to keep track of the swaps and apply them in the reverse order to the inverse. Remember that a row swap becomes a column swap and vice versa. Project 3:Implement the in-place version of the Gauss-Jordan inversion algorithm using complete pivoting.

Determinant using Gaussian elimination

One may use the Gaussian elimination or Gauss-Jordan elimination algorithm to compute the determinant.

Start by applying the n 1 moves of type (Pc , F ), as in Gaussian elimination. Then multiply the diagonal entries of the resulting matrix. If you have done k row/col swaps, then multiply with (1)k . A row swap is counted separately from a column swap.

Project 4:Implement the above algorithm to compute the determinant of a given square matrix.

LU decompositions
A = LU,

We say that a nonsingular matrix has LU decomposition if it can be written as

where L is a lower triangular and U is an upper triangular matrix. Exercise 3:Show that such a factorization need not be unique even if one exists. If L has 1s on its diagonals then it is called Doolittle decomposition and if U has 1s on its diagonals, it is called Crouts factorization. If L = U then we call it Cholesky decomposition (read Cholesky as Kolesky.) We shall work with Crouts decomposition as a representative LU decomposition. The others are similar.

18

8.1

Application

LU decomposition is mainly used as a substitute for matrix inversion. If A = LU then you can solve Ax = b as follows. First write the system as two triangular systems Ly = b, where y = U x. Being triangular, the systems can be solved by forward or backward substitution. Apply forward substitution to solve for y from the rst eqation, and then apply backward substitution to solve for x from the second equation. Notice that, unlike Gausian/Gauss-Jordan elimination, we do not need to know b while computing the LU decomposition. This is useful when we want to solve many systems of the form Ax = b where A is always the same, but b will change later depending on the situation. Then the LU decomposition needs to be computed once and for all. Only the two substitutions are to be done afresh for each new b.

8.2

Algorithm
n

From denition of matrix multiplication we have aij =

lik ukj .

Now, lik = 0 if k > i, and ukj = 0 if k < j. So the above sum is eectively
min{i,j }

aij =
k=1

lik ukj .

You can compute li1 s for i 1 by considering ai1 = li1 u11 = li1 , since diagonal entries of U are 1. Once l11 has been computed you can compute u1i s for i 2 by considering u1i = a1i /l11 . Next you will compute li2 s and after that u2i s, and so on. The order in which you compute the lij s and uij s is shown in the diagram below.
1 3 5 2 4 6 7 9 11 13 = one 8 10 12 = zero

L=

U=

LU decomposition computation order

19

The general formulas to compute lij and uij are lij uij = = aij
1 lii j 1 k=1 lik ukj i1 k=1 lik ukj

(i j ) (i < j ).

aij

Exercise 4:What should we do if for some i we have lii = 0? Does this necessarily mean that LU decomposition does not exist in this case?

8.3

Ecient implementation

Notice that L and U have nonzero elements at dierent locations. The only place where both has nonzero elements is the diagonal, where U has only 1s. So we do not need to explicitly store the diagonal entries of U. This lets us store L and U in a single n n matrix. Also, observe that aij for i < j is required to compute only uij . Similarly aij for i j is required to compute only lij . Thus, once uij is computed (for i < j ) we can throw away aij . Similarly, for the case i j. This suggests that we overwrite A with L and R. Here is how the algorithm overwrites A :
U L A
Snapshot of A during Crouts decomposition

Project 5:Implement the ecient version of Crouts decomposition discussed above. Your software should also be able to solve a system Ax = b by forward and backward substitution.

8.4

Some theory

Exercise 5:Show that if all the leading principal minors are nonzero then all the lii s will be nonzero. In fact, if i is the smallest number such that the i-th leading principal minor is zero, then i is also the smallest number with lii = 0. [Hint: If A = LU and you partition A as Aii B , A= C D 20

where Aii is i i, then what is the LU decomposition of Aii ? Now apply the formula for determinant of partitioned matrix to show that lii = |Aii |/|Ai1,i1 |.

Exercise 6:Use the above exercises to characterize all square matrices having LU
decomposition.

9
Theorem

QR decomposition

For any n by p matrix A with n p we have an n by n orthogonal matrix Q and an n by p upper triangular matrix R such that A = QR. This decomposition is the matrix version of Gram-Schmidt Orthogonalization (GSO). To see this we rst consider the case where A is full column rank (i.e., the columns are independent.) Call the columns a1 , ..., ap . Apply GSO to get an orthonormal set of vectors q1 , ..., qp given by
i1

qi = unit(ai
j =1

qj ai )

computed in the order i = 1, ..., p. Here unit(v ) denotes the unit vector unit(v ) = v/ v , for any nonzero v. If v = 0, then unit(v ) is not dened. Notice that ai is a linear combination of q1 , ..., qi only and not of qi+1 , ..., qp . So we can write qi = r1i a1 + r2i a2 + + rii ai . Dene the matrix R using the rij s.

21

 If A is not full column rank then some (ai j =1 qj ai ) will be zero, hence we cannot apply unit to it. But then we can take qi equal to any unit norm vector orthogonal to q1 , ..., qi1 and set rii = 0. However, GSO is not the best way to compute QR decomposition of a matrix. This is because in the unit steps you have to divide by the norms which may be too small. The standard way to implement it is using Householders transformation.

i1

9.1

Householders transformation

If A is any orthogonal matrix then Ax = x . In other words an orthogonal matrix does not change shape or size. It can only rotate and reect objects. We want to ask if the reverse is true: if x and y are two vectors of same length then does there exist an orthogonal A that takes x to y and vice versa? That is, we are looking for an orthogonal A (possibly depending on x, y ) such that Ax = y and Ay = x? The answer is Yes. In fact, there may be many. Householders transform is one such: A = I 2uu , where u = unit(x y ).

Exercise 7:Show that this A is orthogonal and it sends x to y and vice versa.

Exercise 8:Show that in 2 dimensions Householders transform is the only such

Exercise 9:In general you need n2 scalar multiplications to multiply an n n matrix

9.2

Using Householder for QR

The idea is to shave the columns of X one by one by multiplying with Householder matrices. For any non zero vector u dene Hu as the Householder matrix 22

that reduces u to

v=

u 0 . . . 0

Exercise 10:Let us partition a vector u as

where both u1 , u2 are vectors (u2 being k 1). Consider the n n matrix I 0 . H= 0 Hu2 Show that H is orthogonal and compute Hu. We shall say that H shaves the last k 1 elements of u.

Let the rst column of X be a. Let H1 shave its lower n 1 entries. Consider the second column b of H1 A. Let H2 shave o its lower n 2 entries. Next let c denote the third column of H2 H1 X, and so on. Proceeding in this way, we get H1 , H2 , ..., Hp all of which are orthogonal Householder matrices. Dene Q = (H1 H2 Hp ) and R = Q X to get a QR decomposition. Exercise 11:Carry this out for the following 5 4 case.
2 6 6 6 6 4 1 1 1 7 2 2 3 2 2 8 3 2 5 1 5 4 4 0 3 4 3 7 7 7 7 5

9.3

Ecient implementation
I 2uu

Notice that though the Householder matrix

is an n n matrix, it is actually determined by only n numbers. Thus, we can eectively store the matrix in linear space. In particular, the matrix H1 needs only n spaces, H2 needs only n 1 spaces and so on. 23

So we shall try to store these in the shaved parts of X. Let H1 = 1 2u1 u 1 and H1 X be partitioned as v . 0 X1 Then we shall try to store u1 in place of the 0s. But u1 is an n 1 vector, while we have only n 1 zeroes. So the standard practice is to store (which is the squared norm of the rst column) in a separate array, and store u1 in place of the rst column of A. The nal output will be a n p matrix and a p-dimensional vector (which is like an extra row in A). The matrix is packed with the us and the strictly upper triangular part of R :
strict upper triangular part of R

u1

u2

u4 u3 u5

u6

diagonal entries of R Output of ecient QR decomposition

The extra row stores the diagonal entries of R. It is possible to unpack Q from the us. However, if we need Q only to multiply some x to get Qx, then even this unpacking is not necessary. Exercise 12:Write a program that performs the above multiplication without explicitly computing Q.

Project 6:Write a program to implement the above ecient version of QR decomposition of a full column rank matrix A. Your program should be able to detect if A is not full column rank, in which case it should stop with an error message. This is not entirely easy to do. So lets proceed step by step. First we chalk out the basic owchart. 24

n, p, A

i = 0, ..., p 1 i A.

The main algo

The easiest part to expand here is the input part. The only subtlety is to remember to allocate an extra row in A.

n, p

An+1p A n p
The input part

Now comes the tricky step: the shaving. There are two parts here. First, we have to compute the u vector, and save it in the A matrix. Second, we have to multiply the later columns by Hu .

25

 i. j = i + 1, ..., p 1 j

How to shave

The owchart below is quite technical in nature. It tells you how to compute the u vector, and save it in the A matrix. It also saves the diagonal element of R in the extra row of A.

normsq 1 = norm =

n1 2 j =i+1 aji

normsq 1 + a2 ii

aii = aii norm normDif f = normsq 1 + a2 ii

aii, ..., an1,i

normDif f ani

norm

Processing the shaved column (i)

Updating the subsequent columns is basically multiplying by a Householder matrix.

26

mult = 2

n1 k =i aki akj

k = i, ..., n 1

akj = akj mult aki

Updating column j

9.4

Application to least squares

An important use is in solving least squares problems. Here we are given a (possibly inconsistent) system Ax = b, where A is full column rank (need not be square.) Then we have the following theorem. Theorem The above system has unique least square solution x given by x = A(A A)1 A b. Note that the full column rankness of A guarantees the existence of the inverse. However, computing the inverse directly and then performing matrix multiplication is not an ecient algorithm. A better way (which is used in standard software packages) is to rst form a QR decomposition of A as A = QR. The given system now looks like QRx = b. The lower part of R is made of zeros: Q or R1 O R1 O x = b,

x = Q b,

Partition Q b appropriately to get R1 O x= c1 c2 ,

where c1 is p 1. This system is made of two systems: R1 x = c1 27

and Ox = c2 . The rst system is always consistent and can be solved by back-substitution. The second system is trivial and inconsistent unless c2 = 0.
1 c1 is the least square solution of the original system. Exercise 13:Show that x = R1

Notice that you use back-substituting to nd this x and not direct inversion of R1 .

Exercise 14:Find a use of c2 to compute the residual sum of squares:

28