Appendix C:

Review of Linear Algebra

and Related MATLAB Commands

C.1 Matrix and Vector Operations

M atrix A is an (m × n) array of elements arranged in rows and columns, as shown
below:

⎡ a11 a12 .... a1n ⎤

⎢a a22 .... a2 n ⎥
A = ⎢ 21 ⎥
⎢ ... ... ... ... ⎥
⎢ ⎥
⎣ am1 am 2 ... amn ⎦

In MATLAB, elements in a row are separated by commas, while rows are separated
by semicolons or by line breaks. Thus,
>> B=[1,3,5;2,0,4;1,5,6]
B =
1 3 5
2 0 4
1 5 6

517
518 APPENDIX C: REVIEW OF LINEAR ALGEBRA

The trace of a matrix is the sum of the elements on the main diagonal of a square
matrix. The MATLAB command for calculating the trace is
>> trace(B)
ans = 7

A matrix of one column of n elements is referred to as an n-dimensional column

vector,
>> x=[1; 3; 5; 2; 0]
x =
1
3
5
2
0

A matrix of only one row of n elements is referred to as an n-dimensional row vector,

>> y=[5, 6, 7 , 8, 9]
y =
5 6 7 8 9

Matrix addition and subtraction can be performed if matrices have the same number
of rows and the same number of columns.
Matrix multiplication between A and B is allowed if A has the same number of
columns as the number of rows of B. When this condition is satisfied, then A and B
are said to be conformable. The multiplication of the matrix A, that has (m × n)
elements, with matrix B, of (n × p) elements, results in a matrix with m rows and p
columns that has an order of (m × p). For example,

⎡ a11 a12 a13 ⎤

⎢a ⎡b b ⎤
a22 a23 ⎥ ⎢ 11 12 ⎥
AB = ⎢ ⎥ b
21
b
⎢ a31 a32 a33 ⎥ ⎢ 21 22 ⎥
⎢ ⎥ ⎢⎣ b31 b32 ⎥⎦
⎢⎣ a41 a42 a43 ⎥⎦
( 4 × 3) (3 × 2)
⎡ ( a11b11 + a12b21 + a13b31 ) ( a11b12 + a12b22 + a13b32 ) ⎤
⎢ ⎥
⎢( a21b11 + a22b21 + a23b31 ) ( a21b12 + a22b22 + a23b32 )⎥
=⎢
( a b + a b + a b ) ( a31b12 + a32b22 + a33b32 ) ⎥⎥
⎢ 31 11 32 21 33 31
⎢⎣( a41b11 + a42b21 + a43b31 ) ( a41b12 + a42b22 + a43b32 )⎥⎦
(4 × 2)
C.1 MATRIX AND VECTOR OPERATIONS 519

Matrix multiplication is not commutative, that is, AB ≠ BA. It should be noted that to
an element-by-element product of two matrices is denoted by the (.*) operator:

⎡a a12 ⎤ ⎡ b11 b12 ⎤ ⎡ a11b11 a12b12 ⎤

A.* B = ⎢ 11 ⎥. ∗ ⎢ ⎥=⎢ ⎥
⎣ a21 a22 ⎦ ⎣b21 b22 ⎦ ⎣ a21b21 a22b22 ⎦

The transpose of the matrix A, AT, is defined as an interchange of the rows and
columns of A, and is denoted by A' in MATLAB. If A = A T , then A is called a
symmetric matrix.
An identity, or unit matrix, is one whose diagonal elements are all unity and the rest
of the elements are zero. An identity matrix is always symmetric and square. Thus, a
(3 × 3) order identity matrix is given by:

⎡1 0 0⎤
I = ⎢0 1 0⎥
⎢ ⎥
⎢⎣0 0 1⎥⎦

The MATLAB function eye(n) returns an (n × n) identity matrix.

>> eye(3)
ans =
1 0 0
0 1 0
0 0 1

The inverse of a matrix, A-1, is that matrix which when multiplied by the matrix A
gives the identity matrix,

AA -1 = I
A matrix has an inverse if it is square and nonsingular, or equivalently, if the
determinant of the matrix is non-zero. To check if a matrix B is nonsingular, the
determinant of the matrix can be readily computed with a MATLAB command, as
shown below:
>> B=[1,3,5;2,0,4;1,5,6]
B =
1 3 5
2 0 4
1 5 6

>> det(B)
ans =
6
520 APPENDIX C: REVIEW OF LINEAR ALGEBRA

Similarly, the inverse of matrix can be easily computed in MATLAB, as shown

below for the matrix B defined above:
>> inv(B)
ans =
-3.3333 1.1667 2.0000
-1.3333 0.1667 1.0000
1.6667 -0.3333 -1.0000

In MATLAB, the division of two matrices, A divided by B, is equivalent to the

product of matrix A and the inverse of B. Thus, A/B = A B-1, while A\B is equivalent
to A-1B. Note also that the inverse of the product of two matrices is the reverse
product of the inverses of the two matrices:

( AB) −1 = B −1A −1

The rank of a matrix A is defined as the order of the largest nonsingular square
matrix within A. Consider the (m × n) matrix, where n ≥ m. Then, the rank of the
largest square submatrix within A is m, provided the (m × m) submatrix is
nonsingular, that is, the determinant of the (m × m) submatrix is nonzero. The
MATLAB command rank(A) gives the rank of the matrix:
>> A=[1,2,1,0;1,0,0,1;2,1,0,1]
A =
1 2 1 0
1 0 0 1
2 1 0 1
>> rank(A)
ans =
3

Each square matrix has its characteristic scalar properties, called the eigenvalues, and
corresponding vectors, called the eigenvectors. The matrix A is said to have
eigenvalue λ and eigenvector x if and only if:
Ax = λ x

which can also be expressed as:

( A − λ I)x = 0

To find λ and the corresponding vector x requires the solution of the homogeneous
set of equations shown above. The homogeneous problem above has nontrivial
solutions if the determinant of ( A − λ I), also called the characteristic matrix, equals
zero. For example, for a (2 × 2) matrix, A, shown below, the values of λ can be found
readily:
C.2 MATRIX FACTORIZATION 521

⎡1 0⎤
A=⎢ ⎥
⎣1 1⎦

Since the determinant of the characteristic matrix equals zero:

1− λ 0
A − λI = = (1 − λ )2 = 0 ⇒ λ1 = 1, λ2 = 1
1 1− λ

In MATLAB, eig(A) finds the values of λ. The output vector of eigenvalues can be
named, as m=eig(A).
The command [Z,D]=eig(A) generates a diagonal matrix D of eigenvalues and a full
matrix Z whose columns contain the corresponding eigenvectors:
>> B
B =
1 3 5
2 0 4
1 5 6
>> m=eig(B)
m =
9.6540
-0.2596
-2.3944
>> [Z,D]=eig(B)
Z =
0.5587 0.8176 0.0067
0.4136 0.3791 -0.8595
0.7189 -0.4334 0.5111
D =
9.6540 0 0
0 -0.2596 0
0 0 -2.3944

C.2 Matrix Factorization

The purpose of LU factorization of a matrix A is to find a lower triangular matrix, L,
and an upper triangular matrix U, such that A = LU. For any matrix, A (certain
conditions may apply; see text for discussion), the problem is to find L and U such
that LU = A:

⎡ l11 0 0⎤ ⎡ u11 u12 u13 ⎤ ⎡ a11 a12 a13 ⎤

⎢l l 0⎥ ⎢0 u u23 ⎥ = ⎢ a21 a22 a23 ⎥
⎢ 21 22 ⎥ ⎢ 22 ⎥ ⎢ ⎥
⎣⎢ l31 l32 l33 ⎦⎥ ⎣⎢ 0 0 u33 ⎦⎥ ⎣⎢ a31 a32 a33 ⎦⎥
522 APPENDIX C: REVIEW OF LINEAR ALGEBRA

A lower triangular matrix L with 1s on the diagonals can be constructed from the
multipliers used in Gaussian elimination (reviewed in Chapter 4). The elimination
process transforms the original matrix A into an upper triangular matrix U. The lower
triangular matrix L is formed by placing the negatives of the multipliers in the
appropriate positions, as illustrated below.
LU factorization from basic Gaussian elimination (no pivoting)
Input
A n-by-n matrix to be factored
n dimension of A
Initialize
L=I n-by-n identity matrix
U=A
Operate
For k = 1 to n-1
For i = k + 1 to n each row of matrix U after the kth row
m(i, k ) = −U (i, k ) / U ( k , k )
For j = k to n transform row i
U (i , j ) = U (i , j ) + m (i , k ) * U ( k , j )
End
L (i , k ) = − m (i , k ) update L matrix
End
End
L lower triangular matrix
U upper triangular matrix

Direct LU factorization
There are two alternative approaches to Gaussian elimination for finding the LU
factorization of matrix A: Doolittle factorization, wherein the diagonal elements of
matrix L are 1s; and Cholesky factorization for symmetric matrix A, wherein the
upper triangular matrix U is the transpose of the lower triangular matrix L.
Doolittle factorization
For a three-by-three matrix A, the problem is to find matrices L and U so that
L U = A
⎡1 0 0⎤ ⎡ u11 u12 u13 ⎤ ⎡ a11 a12 a13 ⎤
⎢l 1 0⎥ ⎢ 0 u22 u23 ⎥ = ⎢ a21 a22 a23 ⎥
⎢ 21 ⎥⎢ ⎥ ⎢ ⎥
⎢⎣ l31 l32 1⎥⎦ ⎢⎣ 0 0 u33 ⎥⎦ ⎢⎣ a31 a32 a33 ⎥⎦
C.2 MATRIX FACTORIZATION 523

One begins by finding u11 = a11 and then solving for the remaining elements in the
first row of U and the first column of L. At the second stage, we find u22 and then the
remainder of the second row of U and the second column of L. This procedure is
continued till all the elements of U and L are determined. A general algorithm is
summarized below: in what follows, the MATLAB colon (:) notation is used to
represent a row or column.
Input
A n-by-n matrix to be factorized
n dimension of A
Initialize
U = zeros(n) initialize U to n-by-n zero matrix
L = I(n) initialize L to identity matrix
Operate
For k = 1 to n
U ( k , k ) = A( k , k ) − L( k ,1: k − 1) * U (1: k − 1, j )
For j = k + 1 to n
U ( k , j ) = A( k , j ) − L( k ,1: k − 1) * U (1: k − 1, j )
L( j, k ) = ( A( j, k ) − L( j ,1: k − 1) * U (1: k − 1, k )) / U ( k , k )
End
End

Cholesky LU factorization
If the matrix A is symmetric, there is a convenient form of LU factorization called
the Cholesky factorization. In this method, the upper triangular matrix U is the
transpose of the lower triangular matrix L; that is, A = LLT. The basic steps for the
Cholesky factorization are summarized below:
Input
A symmetric n-by-n matrix ( A = AT )
Initialize
L=I(n) n-by-n identity matrix
Operate
For k = 1 to n
x = L( k ,1: k − 1) columns 1 to k-1 of the kth row of L
L( k , k ) = A( k , k ) − xx T
For j = k + 1 to n
y = L( j,1: k − 1) columns 1 to k-1 of the jth row of L
524 APPENDIX C: REVIEW OF LINEAR ALGEBRA

End
End
Transpose
U=LT

MATLAB contains several convenient commands for LU factorization.

The MATLAB function lu can be used to find the LU decomposition of a square

matrix A. The function call [L, U] = lu(A) returns an upper triangular matrix in U,
and a product of lower triangular matrices in L such that A = LU. For example:
>> a=[4,12,8,4;1,6,25,9;2,9,30,30;3,11,20,18]
a =
4 12 8 4
1 6 25 9
2 9 30 30
3 11 20 18
>>
>> [L, U]=lu(a)
L =
1.0000 0 0 0
0.2500 1.0000 0 0
0.5000 1.0000 1.0000 0
0.7500 0.6667 -0.4444 1.0000

U =
4.0000 12.0000 8.0000 4.0000
0 3.0000 23.0000 8.0000
0 0 3.0000 20.0000
0 0 0 18.5556

The MATLAB function for the Cholesky factorization is chol. This function uses
only the diagonal and upper triangular part of A, as demonstrated below. Note that
the Cholesky factorization only works on positive definite matrices: A symmetric
matrix A is positive definite if xTAx > 0; all eigenvalues of A are positive and all
diagonal elements are positive. In general, A is positive definite if it is nonsingular,
diagonally dominant ( aii > ∑ aij ), and each diagonal element is positive
j ≠i

( aii > 0 for i = 1,..n ) .

>> A=[1,4,5;4,20,40;5,40,128]
A =
1 4 5
4 20 40
5 40 128
>>
C.2 MATRIX FACTORIZATION 525

>> chol(A)
ans =
1.0000 4.0000 5.0000
0 2.0000 10.0000
0 0 1.7321

This concludes our brief review of selected topics from linear algebra. The interested
student is encouraged to read more on the subject in any of the numerous texts
available on linear algebra.