Introduction to Matrix Algebra for Statistics

Contents
1 Definitions 1

2 Basic matrix operations 10

2.1 Addition and subtraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.1.1 Properties of the matrix addition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2 Transpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2.1 Properties of the transpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.3 Multiplication by scalar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.3.1 Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.4 Inner product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.5 Multiplying matrix by a vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.6 Outer product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.7 Matrix multiplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.7.1 Comparison to scalar algebra . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.8 The trace of a matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.8.1 Properties of the trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.9 Exercises: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

3 Special types of matrices 31

3.1 Symmetric matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.2 Orthogonal and orthonormal vectors and matrices . . . . . . . . . . . . . . . . . . . . . . . . 32
3.2.1 Vector norm and unit vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.3 Ortogonal vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.3.1 Orthogonal matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.4 Idempotent matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.5 Matrices having all elements equal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.6 Exercises: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

4 Determinant 46
4.1 2 ◊ 2 matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4.2 Larger matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4.3 Sarrus’ rule for 3 ◊ 3 matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4.4 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.5 Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

5 Inverse of a matrix 51
5.1 Calculating the matrix inverse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
5.2 Useful properties of the inverse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
5.3 Solving the system of linear equations using the inverse . . . . . . . . . . . . . . . . . . . . . 57
5.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

6 Linear dependence and matrix rank 60

6.1 Linear dependence and independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
6.2 Matrix rank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
6.3 Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
6.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

7 Partitioned matrices (Block matrix) 65

7.1 Multiplication of partitioned matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
7.2 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

8 Eigenvalues and eigenvectors 71

 8.1 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

9 Generalized inverse 74
9.1 Exercises: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

10 solutions 79

2
1 Definitions
Matrix is a rectangular array of numbers arranged into rows and columns. A general matrix consisting r
rows and c columns is of the form

S T
a11 a12 a13 . . . a1c
Wa21 a22 a23 . . . a2c X
W X
W . . . a3c X
A = Wa31 a32 a33 X,
W .. .. .. .. X
U . . . . V
ar1 ar2 ar3 . . . arc

where the numbers aij are called elements or entries of the matrix A.Three dots indicate, for example in the
first row, that the elements a11 , a12 , a13, continue in sequence up to a1c . The elements are often denoted by
lower case letters with a subscripts, e.g. aij refers to the element in row i and column j of the matrix A. For
example a13 is the element in the first row and the third column of the matrix A.

Previous form of writing a matrix specifies its entries and also the number of rows and columns that is
referred to as the dimension (or order or size) of the matrix. Dimension of the previous matrix A is r ◊ c
(i.e. matrix A has r rows and c columns). Sometimes it is useful to use subscript notation to denote the
dimension of a matrix. For example Ar◊c is a matrix having r rows and c columns.

Example: R dataset trees provides measurements of the girth, height and volume of timber in 31 felled
black cherry trees. The following table presents the first 7 rows of the data.

Girth Height Volume

8.3 70 10.3
8.6 65 10.3
8.8 63 10.2
10.5 72 16.4
10.7 81 18.8
10.8 83 19.7
11.0 66 15.6

This table can be presented using 7 ◊ 3 matrix

S T
8.3 70 10.3
W 8.6 65 10.3X
W X
W 8.8 63 10.2X
W X
W10.5 72 16.4X
W X
W10.7 81 18.8X
W X
U10.8 83 19.7V
11.0 66 15.6

Note: The word order is sometimes used for other characteristics of a matrix, but in this handout order
always refers to the number of rows and columns.