INTRODUCTION TO MATRIX ALGEBRA

1. D EFINITION OF A MATRIX AND A VECTOR

1.1. Definition of a matrix. A matrix is a rectangular array of numbers arranged into rows and
columns. It is written as
 
a11 a12 . . . a1n
 a21 a22 . . . a2n 
 
 . . . . 

 .
 (1)
 . . . 

 . . . . 
am1 am2 . . . amn
The above array is called an m by n (m × n) matrix since it has m rows and n columns. Typically
upper-case letters are used to denote a matrix and lower case letters with subscripts the elements.
The matrix A is also often denoted

A = k aij k (2)
Consider the following 3×3 example.
 
2 3 −2
 −1 4 13  (3)
2
4 3 −1
In this matrix a31 = 4 and a23 = 13.

1.2. Definition of a vector. A vector is a n-tuple of numbers. In two dimensional space or R2 , a

vector would be an ordered pair of numbers { x, y}. In three dimensional space or R3 , a vector is
a 3-tuple, i.e., { x1 , x2 , x3 }. Similarly for Rn . Vectors are usually denoted by lower case letters such
as a or b, or more formally ~a or ~b.

1.3. Row and column vectors.

1.3.1. Row vector. A matrix with one row and n columns (1×n) is called a row vector. It is usually
written ~x ′ or


~x = x1 x2 x3 . . . xn (4)
The use of the prime ’ symbol indicates we are writing the n-tuple horizontally as if it were the
row of a matrix. Note that each row of a matrix is a row vector. A row vector might be as follows


~z = −1 4 13 (5)
where z2 = 4.

Date: 4 March 2008.

1
2 INTRODUCTION TO MATRIX ALGEBRA

1.3.2. Column vector. A matrix with one column and n rows (n×1) is called a column vector.
It is written as
 
x1
 x2 
 
 x3 
 
~x = 
 . 
 (6)
 . 
 
 . 
xn
A column vector might be as follows
 
−2
~p =  13  (7)
−1
Note that each column of a matrix is a column vector. It is common to write the columns of a
matrix as a1 , a2 , . . . an where each column vector a j is of length m. As an example a2 is given by
 
a12
 a22 
 
 a32 
 
~a2 =  . 
 (8)
 . 
 
 . 
am2
In equation 3, a2 is given by
 
3
~a2 =  4  (9)
2
3

2. VARIOUS TYPES OF M ATRICES AND V ECTORS

2.1. Square matrices. A square matrix is a matrix with an equal number of rows and columns, i.e.
m=n.

2.2. Transpose of a matrix. The transpose of a matrix A is a matrix formed from A by interchanging
rows and columns such that row i of A becomes column i of the transposed matrix. The transpose
is denoted by A′ or A T and

A′ = k a ji k when A = k aij k (10)

If a′ij is the ijth element of A′ , then a′ij
= a ji . If the matrix A is given by
 
3 2 5 7
1 4 6 3 
A = 
5 10 −2 0 
 (11)
1 1 15 −2
then A′ is given by
 INTRODUCTION TO MATRIX ALGEBRA 3

 
3 1 5 1
′
2 4 10 1 
A =   (12)
5 6 −2 15 
7 3 0 −2

2.3. Symmetric matrix. A symmetric matrix is a square matrix A for which

A = A′ (13)
An example of a symmetric matrix is
 
3 1 5 1
1 4 10 1 
T =  
5 10 −2 15 
1 1 15 −2
  (14)
3 1 5 1
1 4 10 1 
T′ = 


5 10 −2 15 
1 1 15 −2

2.4. Identity matrix. The identity matrix of order n written I or In , is a square matrix having ones
along the main diagonal (the diagonal running from upper left to lower right and zeroes elsewhere).
 
1 0 0 ... 0
0 1 0 ... 0
 
0 0 1 ... 0
 
. . . . . (15)
 
. . . . .
 
. . . . .
0 0 0 ... 1
If we write I = k δij k then
(
1, i=j
δij = (16)
0, i 6= j
The symbol δij is called the Kronecker delta.

2.5. Scalar matrix. For any scalar λ, the square matrix

S = k λ δij k = λI (17)
is called a scalar matrix. An example is
 
3 0 0 0
0 3 0 0

0
 (18)
0 3 0
0 0 0 3
4 INTRODUCTION TO MATRIX ALGEBRA

2.6. Diagonal matrix. A square matrix

D = k λi δij k (19)
is called a diagonal matrix. Notice that λi varies with i. An example is
 
13 0 0 0
0 2 0 0

 0 0 −4 0 
 (20)
0 0 0 56
If a system of equations in four variables was written with this coefficient matrix, we could
solve the system by solving each equation individually because each variable would appear in
each equation only once.
2.7. Null or zero matrix. The null or zero matrix is a matrix with each element being zero. It is
denoted as 0.
 
0 0 0 ... 0
0 0 0 ... 0
 
0 0 0 ... 0
 
0 =  . . . . . (21)
. . . . .
 
. . . . .
0 0 0 ... 0
2.8. Upper triangular matrix. A matrix with all elements below the main diagonal equal to zero is
called an upper triangular matrix.
 
a11 a12 a13 . . . a1n
 0 a22 a23 ... a2n 
 
 0
 0 a33 . . . a3n  
A =   . . . . .  (22)
 . . . . . 
 
 . . . . . 
0 0 0 . . . amn
Specifically aij = 0 if i > j as long as i < m and j < n.

2.9. Lower triangular matrix. A matrix with all elements above the main diagonal equal to zero is
called a lower triangular matrix.
 
a11 0 0 ... 0
 a21 a22 0 ... 0 
 
 a31 a32 a33 . . . 0 
 
A =   . . . . . 
 (23)
 . . . . . 
 
 . . . . . 
am1 am2 am3 . . . amn
Specifically aij = 0 if i < j as long as i < m and j < n.

The following two matrices are upper triangular and lower triangular respectively.
 INTRODUCTION TO MATRIX ALGEBRA 5

   
3 −1 7 6 3 0 0 0
0 1 4 5  4 −1 0 0 
 2
 
 −3 1
 (24)
0 0 2 3 3 −4 0 
0 0 0 6 5 2 −1 −2

3. A NOTE ON SUMMATION NOTATION

3.1. Single sums.

3.1.1. Definition of a single sum.
n
∑ ai = am + am+1 + am+2 + ... + an (25)
i =m
For example, suppose we have a vector with the following elements


~a = a1 a2 a3 a4 . . . a9

(26)
= 1 3 −2 6 2 −1 4 3 5
Then
6
∑ a i = −2 + 6 + 2 + −1 = 5 (27)
i =3

3.1.2. Properties of a single sum.

n n
∑ kai = k ∑ ai
i =1 i =1
n
∑ k = k + k + k + ... + k = nk (28)
i =1
n n n
∑ ( a i + bi ) = ∑ a i + ∑ bi
i =1 i =1 i =1

3.2. Double sums.

3.2.1. Definition of a double sum.
n m m m m
∑ ∑ aij = ∑ a1j + ∑ a2j + ... + ∑ anj
i =1 j =1 j =1 j =1 j =1

= a11 + a12 + a13 + ... + a1m

+ a21 + a22 + a23 + ... + a2m (29)
+...
+...
+...
+ an1 + an2 + an3 + ... + anm
Consider the following matrix
6 INTRODUCTION TO MATRIX ALGEBRA

 
4 −1 5 7
 1 4 −2 3 
A =  
 0 10 −2 0
−11 1 6 2
We compute ∑4i=2 ∑3j=2 aij as follows
S = a22 + a23 + a32 + a33 + a42 + a43
= 4 + −2 + 10 + −2 + 1 + 6
= 17
3.2.2. Properties of a double sum.
! !
n n n
∑ aj ∑ ai = ∑ a2i + 2 ∑∑ ai a j
j =1 i =1 i =1 i< j
(30)
n
= ∑ a2i + ∑∑ ai a j
i =1 i6= j
 
For example let a = c d e . Then

[c + d + e] × [c + d + e] = c2 + d2 + e2 + 2cd + 2ce + 2de

 INTRODUCTION TO MATRIX ALGEBRA 7

4. M ATRIX OPERATIONS

4.1. Scalar multiplication (matrix). Given a matrix A and a scalar λ, the product of λ and A, writ-
ten λA, is defined to be
 
λa11 λa12 . . . λa1n
 λa21 λa22 . . . λa2n 
λA =  . (31)
 
.. .. .. 
 .. . . . 
λam1 λam2 . . . λamn

4.2. Scalar multiplication (vector). Given a column vector ~a and a scalar λ, the product of λ and
~a, written λ ~a, is defined to be
 
λa1
 λa2 
λ ~a =  ..  (32)
 
 . 
λam
For the second column of a matrix we could write
 
λa12
 λa22 
 
λ~a2 =  λa32  (33)
 
 .. 
 . 
λam2

4.3. Trace of a square matrix. The trace of a matrix is the sum of the diagonal elements and is
denoted tr A. Consider the matrix C below.

 
3 1 5 1
1 4 10 1 
C =   (34)
5 10 −2 15 
1 1 15 −2
The trace of C is [3 + 4 + -2 + -2] = 3.

4.4. Addition of vectors. - The sum of a vector a with m elements and a vector b having m elements
is a vector c with m elements and whose elements are given by

c j = a j + bj ∀ j (35)
This gives
       
c1 a1 b1 a1 + b1
 c2   a2   b2   a2 + b2 
~c =  .  =  .  +  .  =  (36)
       
..
 ..   ..   .. 

 . 
cm am bm am + bm
8 INTRODUCTION TO MATRIX ALGEBRA

4.5. Linear combinations of vectors. If a and b are two n-vectors and s and t are two real numbers,
ta + sb is said to be the linear combination of a and b. In symbols we write,
     
a1 b1 t a1 + s b1
 a2   b2   t a2 + s b2 
t  .  + s  .  =  .. (37)
     
 ..   .. 

 . 
am bm t am + s bm

   
a1 b1
4.5.1. Example. Let ~a = a2  and let ~b = b2 . Let t = 2 and s =4. Then we obtain
a3 b3
       
a1 b1 2a1 + 4b1 c1
2  a2  + 4 b2  = 2a2 + 4b2  = c2 
a3 b3 2a3 + 4b3 c3
 
c1
where c2  represents the linear combination.
c3

4.5.2. Numerical Example.

       
5 −3 15 + −6 9
3  −2  + 2  4  =  −6 + 8  =  2 
4 1 12 + 2 14

4.5.3. Writing a system of equations as a linear combination of vectors.

Consider three vectors, each with two elements. Call the vectors ~a1 , ~a2 and ~b. Call the elements
of the first one a11 and a21 , the elements of the second one a12 and a22 and the elements of ~b, b1 and
b2 . Now consider two scalars denoted x1 and x2 . Now multiply ~a1 by x1 and ~a2 by x2 and add the
products. We obtain
         
a11 a12 a11 x1 a12 x2 a11 x1 + a12 x2
x1 + x2 = + = (38)
a21 a22 a21 x1 a22 x2 a21 x1 + a22 x2

If set this expression equal to ~b we obtain

   
a11 x1 + a12 x2 b1
= (39)
a21 x1 + a22 x2 b2
which is a linear system of 2 equations in 2 unknowns. We can write a general system of m
equations in n unknowns as

x1 ~a1 + x2 ~a2 + . . . + x n ~an = ~b (40)

where xi are a series of scalar unknowns and each a j is a column of the A matrix of coefficients.
 INTRODUCTION TO MATRIX ALGEBRA 9

4.6. Addition of matrices. The sum C of a matrix A having m rows and n columns and a matrix B
having m rows and n columns is a matrix having m rows and n columns whose elements are given
by

cij = aij + bij ∀ i, j (41)

This gives
     
c11 c12 ... c1n a11 a12 ... a1n b11 b12 ... b1n
 c21 c22 ... c2n   a21 a22 ... a2n   b21 b22 ... b2n 
C =  . .. .. ..  =  .. .. .. ..  +  .. .. .. .. 
     
 .. . . .   . . . .   . . . . 
cm1 cm2 ... cmn am1 am2 ... amn bm1 bm2 ... bmn
(42)
 
a11 + b11 a12 + b12 ... a1n + b1n
 a21 + b21 a22 + b22 ... a2n + b2n 
=  .. .. .. ..
 

 . . . . 
am1 + bm1 am2 + bm2 ... amn + bmn
For example
     
1 2 4 2 −1 −2 3 1 2
2 3 −4  +  2 1 4  = 4 4 0 
5 2 1 −2 1 3 3 3 4
4.7. Inner (dot) product of two vectors. The inner (scalar or dot) product to two vectors u,v of
length n is the scalar quantity denoted by
n
u · v = ∑ ui vi = u1 v1 + u2 v2 + ... + un vn (43)
i =1
It is easiest to see how to multiply two vectors if we write the first one as a row vector and the
second one as a column vector. For example
 
  2
3 4 6 · 3 = 3 × 2 + 4 × 3 + 6 × 5 = 6 + 12 + 30 = 48 (44)
5
4.8. Multiplication of a matrix and a column vector. We can multiply a matrix and column vector
if the matrix has the same number of columns as there are elements in the column vector. The result
of this multiplication is a column vector with the same number of elements as the matrix has rows.
The ith element of the resulting column vector is obtained as the dot product of the ith row of the
matrix and the column vector. Specifically for an m×n matrix A and an n×1 column vector b,
n
ci = ∑ aik bk , i = 1, . . . , m. (45)
k =1
For example
     
1 2 4 −2 18
2 3 −4   4  =  −4 
5 2 1 3 1
10 INTRODUCTION TO MATRIX ALGEBRA

Specifically 1 × -2 + 2 × 4 + 4 × 3 = 18 and so on.

Or consider the example

     
1 2 4 −1 5
2 3 −4   1  =  −3  .
5 2 1 1 −2
4.9. Multiplication of a row vector and a matrix. We can multiply a row vector and a matrix if
the matrix has the same number of rows as there are elements in the row vector. The result of this
multiplication is a row vector with the same number of elements as the matrix has columns. The
ith element of the resulting row vector is obtained as the dot product of the row vector and the ith
column of A. Specifically for an m×n matrix A and an m×1 column vector b,
m
ci = ∑ bk aki , i = 1, . . . , n. (46)
k =1
For example
 

2 −1 −2

1 2 4  2 1 4  = −2 5 18
−2 1 3
Specifically 1 × 2 + 2 × 2 + 4 × -2 = -2 and so on.

Here is a second example.

 
2
−1 −2

5 2 1  2 1 4  = 12 −2 1
−2 1 3
4.10. Multiplication of matrices. Given an m×n matrix A and an n×r matrix B, the product AB is
defined to be an m × r matrix C, whose elements are computed from the elements of A,B according
to
n
cij = ∑ aik bkj , i = 1, . . . , m, j = 1, . . . , r. (47)
k =1
In other words to obtain the ijth element of c we take the ith row of A and jth column of B and
form the inner product.
4.10.1. Example 1. Consider multiplying the following two matrices A and B.
   
1 2 4 2 −1 −2
A =  2 3 −4  , B =  2 1 4  (48)
5 2 1 −2 1 3
We obtain the first element of the product by multiplying the first row of A by the first column
of B.
   
  2 −2
1 2 4 ·  2  = c11 =   (49)
−2
 INTRODUCTION TO MATRIX ALGEBRA 11

We obtain the second element of the first row of the product by multiplying the first row of A by
the second column of B.
   
  −1 5
1 2 4 ·  1  = c12 =   (50)
1
We obtain the third element of the first row of the product by multiplying the first row of A by
the third column of B.
   
  −2 18
1 2 4 ·  4  = c13 =   (51)
3
Combining operations for the first row of A and the matrix B we obtain
   

2 −1 −2 −2 5 18
1 2 4  2 1 4  =  
−2 1 3
Now consider the second row of A and the matrix B. Performing these operations we obtain
   

2 −1 −2
2 3 −4  2 1 4  = 18 −3 −4
−2 1 3
Completing the operations we obtain
     
1 2 4 2 −1 −2 −2 5 18
2 3 − 4   2 1 4  =  18 −3 −4
5 2 1 −2 1 3 12 −2 1
4.10.2. Example 2. As a second example consider the matrices below
   
3 4 7 1 0 1
A =  2 5 2 B =  2 1 1 (52)
1 0 4 1 4 1
The element c11 comes from multiplying the first row of A with the first column of B as follows:
 

1
c11 = 3 4 7 2 = 3 + 8 + 7 = 18 (53)
1
Similarly the element c32 comes from multiplying the third row of A with the second column of
B as follows:
 

0
c32 = 1 0 4 1 = 0 + 0 + 16 = 16 (54)
4
Multiplying out the rest of the entries gives
 
18 32 14
C = 14 13 9  (55)
5 16 5
12 INTRODUCTION TO MATRIX ALGEBRA

4.11. Writing a system of equations as a matrix product. Consider an m×n matrix, an n×1 vector
and an m×1 vector. The case of a square matrix is handled by setting m=n. Call the matrix A
and the vectors ~x and ~b. Call the elements of ~x, x1 ,x2 ,. . . , xn , and the elements of ~b, b1 ,b2 ,. . . , bn .
Consider a case where A is a 4×3 matrix and x is an 3×1 vector and b is a 4×1 vector. Multiply the
matrix A by the column vector x and set it equal to the vector b as follows.
   
a11 a12 a13   b1
 a21 x 1
a22 a23  b2 
  x2  =  

 a31 (56)
a32 a33  b3 
x3
a41 a42 a43 b4
If we then carry out the multiplication we obtain

a11 x1 + a12 x2 + a13 x3 = b1

a21 x1 + a22 x2 + a23 x3 = b2
(57)
a31 x1 + a32 x2 + a33 x3 = b3
a41 x1 + a42 x2 + a43 x3 = b4

which is a linear system of 4 equations in 3 unknowns. The general system of m equations in n

unknowns can be written

a11 x1 + a12 x2 + ··· + a1n xn = b1

a21 x1 + a22 x2 + ··· + a2n xn = b2
a31 x1 + a32 x2 + ··· + a3n xn = b3 (58)
.. .. .. ..
. + . + ··· + . = .
am1 x1 + am2 x2 + ··· + amn xn = bm
In this system, the aij ’s and bi ’s are given real numbers; aij is the coefficient for the unknown x j
in the ith equation. We call the set of all aij ’s arranged in a rectangular array the coefficient matrix
of the system. Using matrix notation we can write the system as

Ax = b
 
a11 a12 ··· a1n    
x1 b1
 a21 a22 ··· a2n 
   x2   b2  (59)
 a31 a32 ··· a3n 
 .  = . 
   
..   ..   .. 

 .. .. ..
 . . . . 
xn bm
am1 am2 ··· amn
Consider the following matrix A and vector b
   
1 2 1 3
A =  2 5 2 , b =  8  (60)
−3 −4 −2 −4
We can then write
 INTRODUCTION TO MATRIX ALGEBRA 13

Ax = b
    

1 2 1 x1 3 (61)
 2 5 2   x2  =  8 
−3 −4 −2 x3 −4
for the linear equation system

x1 + 2x2 + x3 = 3
2x1 + 5x2 + 2x3 = 8 (62)
−3x1 − 4x2 − 2x3 = −4

4.12. A system of equations with an identity coefficient matrix. Consider a system of n variables
and n equations. The coefficient matrix is square. If the coefficient matrix is an identity matrix then
the solution is obvious upon inspection.

Ix = b
    
1 0 0 ··· 0 x1 b1
0 1 0 ··· 0   x2   b2 
   
 (63)
0 0
 1 ··· 0   x3  =  b3 
   
 .. .. .. .. ..   ..   .. 
. . . . .  .   . 
0 0 0 ··· 1 xn bm
Consider the following 3 × 3 example.
     
1 0 0 x1 −2
0 1 0  x 2  =  2  (64)
0 0 1 x3 1
It is obvious that x1 = -2 and so on as 1 × x1 + 0 × x2 + 0 × x3 = -2. If one were to use Gaussian
elimination to solve a system of n equations in n unknowns and rewrite the system as a matrix
equation at each step, it is clear one would end up with a system where the coefficient matrix was
an identity matrix.

4.13. A system of equations with a diagonal coefficient matrix. Consider a system of n variables
and n equations. The coefficient matrix is square. If the coefficient matrix is a diagonal matrix then
the solution can be obtained by solving each equation individually by one simple division.

Ix = b
   

a11 0 0 ··· 0 x1 b1
 0 a22 0 ··· 0   x2   b2 
     (65)
 0 0 a33 ··· 0   x3   b3 
  = 
 .. .. .. .. ..   ..   .. 

 . . . . .  .   . 
0 0 0 ··· ann xn bm
b1 b2
It is clear that a11 x1 = b1 which implies that x1 = a11 . Similarly x2 = a22 and so on.
Consider the following 3 × 3 example.
14 INTRODUCTION TO MATRIX ALGEBRA

     
−3 0 0 x1 6
 0 4 0   x 2  =  8 (66)
0 0 5 x3 5
6
It is obvious that (-3)x1 = 6 which implies that x1 = −3 = -2 and so on.

4.14. Some properties of matrix operations. Let α and β denote real numbers (scalars), ~a,~b, ~c de-
note n-vectors, and A, B, C denote matrices. The properties are conditional on the operations being
defined for the case in point.
4.14.1. Equality.
vectors: Two n-vectors a and b are said to be equal if all their corresponding components are
equal. Equality is only possible for vectors of the same dimension.
matrices: Two m x n matrices A and B are said to be equal if all their corresponding compo-
nents are equal. Equality is only possible for matrices of the same dimension.
4.14.2. Multiplication by a scalar.
a: ( α + β ) A = αA + βA
b: α(A + B) = αA + αB
c: α (βA) = (αβ) A
Note that A and B above can be replaced by a and b as in (1)( a) = a
4.14.3. Addition.
a: ~a + ~b = ~b +~a
b: ~a + 0 = ~a
c: (~a + ~b ) +~c = ~a + (~b +~c)
d: ~a + (−~a) = 0
e: A + B = B + A
f: A + ( B + C ) = ( A + B) + C
g: A + 0 = 0 + A = A
h: A + (− A) = 0
4.14.4. Multiplication.
a: ~a~b = ~b ~a
b: AB 6= BA
c: A(BC) = (AB)C
d: α(~b +~c) = α~b + α~c
e: A( B + C ) = AB + AC
f: ( B + C ) A = AB + CA
g: (α~a)~b = ~a(α~b) = α(~a~b)
h: ~a ·~a > 0 ⇔ ~a 6= 0
i: ~a · 0 = 0 ·~a = 0
j: A0 = 0A = 0
k: AI = I A = A
4.14.5. Transposes.
a: ( A′ )′ = A
b: ( ABC )′ = C ′ B′ A′
c: ( A + B)′ = A′ + B′
 INTRODUCTION TO MATRIX ALGEBRA 15

4.14.6. Properties of the trace.

a: trace (I) = n
b: trace (ABC) = trace (CAB) = trace (BCA)
c: trace (A + B) = trace (A) + trace (B)
d: tr(AB) = tr(BA) if both AB and BA are defined
e: tr(kA) = ktr(A) where k is a scalar
4.15. Idempotent matrices. - A matrix is called idempotent if

A2 = A (67)
For example the identity matrix is idempotent. Consider the matrix M below.
 
0.8 −0.2 −0.2 −0.2 −0.2
−0.2 0.8 −0.2 −0.2 −0.2
 
M = −0.2 −0.2 0.8 −0.2 −0.2 (68)

−0.2 −0.2 −0.2 0.8 −0.2
−0.2 −0.2 −0.2 −0.2 0.8
We can verify that it is idempotent by carrying out the multiplication.
   
0.8 −0.2 −0.2 −0.2 −0.2 0.8 −0.2 −0.2 −0.2 −0.2
−0.2 0.8 −0.2 −0.2 −0.2 −0.2 0.8 −0.2 −0.2 −0.2
   
−0.2 −0.2 −0.2 −0.2
MM =  0.8 −0.2 −0.2 0.8 −0.2 −0.2 (69)
  
−0.2 −0.2 −0.2 0.8 −0.2 −0.2 −0.2 −0.2 0.8 −0.2
−0.2 −0.2 −0.2 −0.2 0.8 −0.2 −0.2 −0.2 −0.2 0.8
Consider the multiplication of the first row and first column
 
0.8

−0.2
 
0.8 −0.2 −0.2 −0.2 −0.2 −0.2

 = 0.64 + 0.4 + 0.4 + 0.4 + 0.4 = 0.8 (70)
−0.2
−0.2
Or consider the multiplication of the first row and second column
 
−0.2

 0.8 
 
0.8 −0.2 −0.2 −0.2 −0.2  − 0.2  = −0.16 + −0.16 + 0.4 + 0.4 + 0.4 = −0.2 (71)

−0.2
−0.2
Later we will discuss an important concept called the rank of a matrix. For an idempotent matrix
A, tr(A) = rank of A .