Solving a system of linear equations

Shivangi Chandel

Jindal School of Government and Public Policy

Advanced Mathematical Tools

Shivangi Chandel (JSGP) Mathematics Advanced Mathematical Tools 1 / 28

Solving a system of linear equations

1 Some Matrix Algebra

2 Solving Ax = b

Shivangi Chandel (JSGP) Mathematics Advanced Mathematical Tools 2 / 28

What is a Matrix?

A matrix is a rectangular array of numbers where an m × n matrix has m

rows and n columns. We can write this as
 
a11 a12 a13 . . . a1n
 a21 a22 a23 . . . a2n 
A = [aij ]m×n ≡ [aij ] =  .
 
 .. 

am1 am2 am3 . . . amn

where aij represents the element in the the ith row and the jth column.
The structure of a matrix comprises of row and column vectors.
The vector Ri = ai1 , ai2 , ai3 , · · · , ain ∈ Rn is called the ith

row vector of A for any i = 1, 2, · · · m.

The vector Cj = a1j , a2j , a3j , · · · , amj ∈ Rm is called the jth

column vector of A for any j = 1, 2, · · · n.

Shivangi Chandel (JSGP) Mathematics Advanced Mathematical Tools 3 / 28

Matrix Operations

Equality of Matrices:
Let A = [aij ] and B = [bij ] be any two m × n matrices. Then
A = B ⇐⇒ aij = bij for all i = 1, 2, · · · m and j = 1, 2, · · · n.
Matrix Addition/Subtraction:
Let A = [aij ] and B = [bij ] be any two m × n matrices. Then
A ± B = [aij ± bij ].
Scalar Multiplication:
Let A = [aij ] is an m × n matrix and c is a scalar. Then,
cA = c[aij ] = [caij ].
Matrix Multiplication: Let A = [aij ] is an m × n matrix and
B = [bij ] be a n × r matrices.Then,the
" n # matrix product AB is an
X
m × r matrix given by aik bkj , i = 1, 2, · · · m and j = 1, 2, · · · r .
k=1

Shivangi Chandel (JSGP) Mathematics Advanced Mathematical Tools 4 / 28

Properties of Matrix Operations

Matrix addition is commutative i.e A + B = B + A

Matrix addition is associative i.e. (A + B) + C = A + (B + C)
Matrix multiplication is associative i.e. (AB)C = A(BC) = ABC
Matrix multiplication is distributive over
addition:A(B + C) = AB + AC ; (B + C)A = BA + CA

It is important to note that matrix multiplication

 isnot commutative
 i.e.
2 3 0 −1
AB 6= BA. For example, let A = and B = . Then
4 0 2 1
   
6 1 −4 0
AB = and BA = , both clearly not equal.
0 −4 8 6

Shivangi Chandel (JSGP) Mathematics Advanced Mathematical Tools 5 / 28

 If A = [aij ] is an m × n matrix , then the n × m matrix B defined by
bij = aji , i = 1, 2, · · · m and j = 1, 2, · · · n is called the transpose of
A and is denoted by AT .
Properties of Transpose:
1 (AT )T = A
2 (A + B)T = AT + BT
3 (AB)T = BT AT
4 (cA)T = cAT , c is a scalar
Some Special Matrices:
1 A m × n matrix is a square matrix if m = n.
2 A n × n matrix is symmetric if aij = aji for i 6= j.
3 A n × n matrix is a diagonal matrix if aij = 0 for i 6= j.
4 A n × n matrix is an identity matrix (denoted by In or I) if aii = 1 for
i = 1, 2, · · · , n; aij = 0 for i 6= j.
5 A m × n matrix is a null matrix (denoted by 0) if aij = 0 for all
i = 1, 2, · · · m and j = 1, 2, · · · n.

Shivangi Chandel (JSGP) Mathematics Advanced Mathematical Tools 6 / 28

 Properties of null matrix:
1 A + 0 = A = 0 + A.
2 0A = 0 = A0
Properties of Identity matrix: IA = A = AI.
Give an example where AB = 0 is possible with neither A nor B
being the null matrix.
Give an example where CD = CE with C not a null matrix is possible
without D and E being equal.

Shivangi Chandel (JSGP) Mathematics Advanced Mathematical Tools 7 / 28

Inverse of a matrix

Let A be a n × n matrix. If B is a n × n matrix such that AB = I;

and BA = I, then A is invertible and B is called the inverse of A
(denoted by A−1 ).
Properties of inverse:
1 (A−1 )−1 = A
2 (AB)−1 = B−1 A−1
3 (AT )−1 = (A−1 )T

Shivangi Chandel (JSGP) Mathematics Advanced Mathematical Tools 8 / 28

Rank of a matrix

For any m × n matrix A, the sets spanned by its row vectors and the
column vectors are indeed vector spaces. We call the space spanned
by row vectors, {R1 , R2 , · · · , Rm }, as the Row Space of A or Row (A)
and the space spanned by column vectors, {C 1 , C 2 , · · · , C n }, as the
Column Space of A or Col(A).
The dimensions of these spaces hold valuable information for us.
Define: Row Rank of A = dimension of Col(A) and
Column Rank of A = dimension of Row (A)

Theorem 1.1 (Rank Theorem)

For any m × n matrix A, Row Rank of A = Column Rank of A.

Shivangi Chandel (JSGP) Mathematics Advanced Mathematical Tools 9 / 28

 Therefore, simply refer to it as the rank of A (denoted by rank(A)).
 
4 1 −5 6
Verify Rank theorem for A = 0 3 0 6 .
0 0 2 0
If A is n × n matrix then A is called non-singular if rank(A) = n.
A is called singular if rank(A) < n.
An n × n matrix A is invertible iff A is non-singular.
 
2 1 4
Is A = 3 2 1 invertible?
1 3 3

Shivangi Chandel (JSGP) Mathematics Advanced Mathematical Tools 10 / 28

Solving a system of linear equations

1 Some Matrix Algebra

2 Solving Ax = b

Shivangi Chandel (JSGP) Mathematics Advanced Mathematical Tools 11 / 28

Solution possiblilites

If we have a linear equation ax = b in which the unknown is x and a and b

are real numbers then there are just three possibilities:
b
1 a 6= 0 then x = = a−1 b. In this case the equation ax = b has a
a
unique solution for x.
2 a = 0, b = 0 then the equation ax = b becomes 0 = 0 and any value
of x will do. In this case, there are infinitely many solutions.
3 a = 0 and b 6= 0 then ax = b becomes 0 = b which is a
contradiction. In this case, there is no solution for x.
What happens if we have more than one equation and more than one
unknown?

Shivangi Chandel (JSGP) Mathematics Advanced Mathematical Tools 12 / 28

A general system of m linear equations in n variables
Suppose we need to solve the following set of equations in n variables;

a11 x1 + a12 x2 + · · · + a1n xn = b1

a21 x1 + a22 x2 + · · · + a2n xn = b2
.. ..
. . (1)
am1 x1 + am2 x2 + · · · + amn xn = bm

Define
 
a11 a12 a13 ... a1n  
x1
 
b1
 a21 a22 a23 ... a2n   x2   b2 
A= . ,x = 
· · · , and b = · · · .
    
 .. 
am1 am2 am3 . . . amn xn bm

We can write the system using matrix notation as Ax = b.

Shivangi Chandel (JSGP) Mathematics Advanced Mathematical Tools 13 / 28

 The system of equations in (1) is called homogeneous if b = 0; and
non-homogeneous if b 6= 0.
The obvious generalization would be to check whether the system
solves in a unique solution, infinitely many solutions, or no solution.
If m ≥ n, either can happen.
If m < n, there can never be a unique solution to the system (Why?).
If the system of equations is homogeneous, there always is a trivial
solution, namely, x = 0.
If An×n is non-singular, the system Ax = b always has a unique
solution.

Shivangi Chandel (JSGP) Mathematics Advanced Mathematical Tools 14 / 28

Rank of a matrix using Row-echelon form

Recall, that rank of a matrix is defined as the dimension of the row

space (column space) of that matrix.
An alternative definition: Rank of a matrix Am×n is equal to the
number of non-zero rows in the row echelon form of the matrix A.

Definition 2.1
A matrix C is said to be in row echelon form if the following conditions
are satisfied:
1 All non-zero rows are above any rows of all zeroes.
2 The first non-zero coefficient of any row (also called leading
coefficient or pivot) is always placed to the right of the leading
coefficient of the row above it.

Denote row-echelon form of the coefficient matrix A as Ar .

Shivangi Chandel (JSGP) Mathematics Advanced Mathematical Tools 15 / 28

 A matrix will be called to be in row reduced echelon form if, in
addition to conditions (1) and (2),
resume All pivots are equal to 1.
resume All entries in the same column as the pivots are equal to 0.
Denote row-reduced echelon form of the coefficient matrix A as Arr .
Writethe row echelon and
 row reduced echelon form of the matrix
4 1 −5 6
A =  8 5 −10 8 and find the rank of the matrix.
−4 2 7 5
To answer above, we need to first define a method using which a
matrix can be written in its row echelon (and row-reduced) form.

Shivangi Chandel (JSGP) Mathematics Advanced Mathematical Tools 16 / 28

Convert A into Ar

To do this, we use three elementary row operations which can be

applied repeatedly to generate the desired matrix. For some l 6= k,
l, k = 1, 2, ..m, we can define
1 Row switching: Rl ↔ Rk
2 Row multiplication: Rl ↔ cRl for some c ∈ R
3 Row replacement: Rl ↔ Rl + cRk , for some c ∈ R
It is worth noticing that the rank of the coefficient matrix is the same
as that of the matrix generated after an elementary row operation is
applied (Why?).
This basically guarantees that as long as we use only the above three
elementary row operations to rewrite a matrix in its row echelon form,
we will have the same rank as the original matrix.

Shivangi Chandel (JSGP) Mathematics Advanced Mathematical Tools 17 / 28

Existence and Uniqueness of Solution

Consider the system Ax = b. The m × n + 1 matrix

 
a11 a12 a13 . . . a1n b1
 a21 a22 a23 . . . a2n b2 
Ac =  .
 
 ..


am1 am2 am3 . . . amn bm

is called the augmented matrix.

Theorem 2.1
Let A be an m × n matrix and b be a vector in Rm . Then the system of
equations Ax = b has a solution if and only if rank(A) = rank(Ac ).
The system has a unique solution iff rank(A) = rank(Ac ) = n.

Shivangi Chandel (JSGP) Mathematics Advanced Mathematical Tools 18 / 28

 Consider the following system of equations:

2x1 + x2 + 4x3 = 16
3x1 + 2x2 + x3 = 10
x1 + 3x2 + 3x3 = 16

Set this system as Ax = b. Verify that rank(A) = rank(Ac ) = 3.

What is the solution to this system?
   
2 3 1 4 3
Let A = 2 3 7 9  and b =  9 . Is there a solution for
2 3 13 14 15
this system Ax = b? Is the solution unique?
Notice that, another way of checking whether the system
Ax = b has a solution or not is to check whether b ∈ Col(A).

Shivangi Chandel (JSGP) Mathematics Advanced Mathematical Tools 19 / 28

Revisiting Linear Independence using Rank

We can talk about linear independence of n vectors using a matrix

picture.
To see whether vectors {x1 , x2 , · · · , xn } are linearly independent
where xi ∈ Rm ∀i, is equivalent to checking whether the homogenous
system Aλ = 0 has λ = 0 as the only solution where A is the matrix
with column vectors xi ’s and λ = (λ1 , λ2 , · · · , λn ) ∈ Rn .
If m < n, the vectors are linearly dependent.
If m = n, the vectors are linearly independent iff A is non-singular or
rank(A) = n = m.
If m > n, the vectors are linearly independent iff rank(A) = n ( ∵
Saying vectors are linearly independent is equivalent to saying that
the column space has the dimension n.).

Shivangi Chandel (JSGP) Mathematics Advanced Mathematical Tools 20 / 28

Then, for n × n matrix A , the following statements become equivalent:
A is invertible;
A is non-singular;
detA 6= 0;
rank(A) = n;
Column vectors of A are linearly independent.

Shivangi Chandel (JSGP) Mathematics Advanced Mathematical Tools 21 / 28

Revisiting Row(A) and Col(A)

Now that you know how to convert a matrix into its row-echelon form,
we can use it to identify Row space and Column Space of that matrix.
The row space of A is the same as the row space of Ar .
The column space of A need not be the same as the column space of
Ar .    
1 2 r 1 2
Consider matrix A = . Then A = .
2 4 0 0
The row space of A contains linear combinations of vectors (1, 2) and
(2, 4) which are linearly dependent. Therefore, the Row (A) is nothing
but the linear span of (1, 2). This is the same as Row (Ar ).
The column space of A is the vector space containing scalar multiples
of (1, 2) i.e. line passing through origin with equation y = 2x. The
column space of Ar , however, is the space containing scalar multiples
of (1, 0) i.e. line passing through origin with equation y = 0 or x-axis.

Shivangi Chandel (JSGP) Mathematics Advanced Mathematical Tools 22 / 28

 This does not mean the row-echelon form is useless; we can indeed
look at it to identify columns of A that are linearly independent.
A column in A is the basis column if the corresponding column of its
row echelon form Ar contains a pivot. The basis columns of A,
together form the basis of Col(A).
   
1 2 3 0 1 2 3 0
Consider matrix A = . Then Ar = .
2 4 7 3 0 0 1 3
 
1
The basis columns of A are column 1 and 3. Therefore columns
2
 
3
and form the basis of Col(A).
7
Verify that Rank(A) = Rank(Ar ) = 2 = number of pivots = number
of non-zero rows in Ar .

Shivangi Chandel (JSGP) Mathematics Advanced Mathematical Tools 23 / 28

What does the solution space look like?

To be able to identify solution space of any general system of m linear

equations and n unknowns, we need to first define a couple of things;
For Am×n xn×1 = bm×1 , define Nullspace of A or Null(A) as
{x ∈ Rn |Ax = 0}.
In other words, Null(A) is the subspace of solutions to the
homogeneous system Ax = 0.
The dimension of Null(A) is called the Nullity of A.
 
1 2 3 0
For matrix A = , show that
2 4 7 3
     

 −2 9 

1 0
   
Null(A) = a   + b   where a, b ∈ R
   

 0 −3 

0 1
 

Shivangi Chandel (JSGP) Mathematics Advanced Mathematical Tools 24 / 28

  
1 2 3 0
How are we writing this? Notice that using Ar = 0 0 1 3 , the system
Ax = 0 can be written as

x1 + 2x2 + 3x3 = 0
x3 + 3x4 = 0

Since there are two equations in 4 variables, the system solves itself in a solution
space. How is the space identified? Should we write x3 in the form of x4 or the
other way around?
The convention we follow is easy; call variables corresponding to basis columns as
basis variables and the variables not corresponding to the basis columns as free
variables. Now write basis variables as expressions of free variables.
In the example, x1 and x3 are basis variables and x2 and x4 are free variables.
Therefore, we say, x3 = −3x4 and x 1 = −2x2 + 9x4 . For any given x2 , x4 , the
−2x2 + 9x4
 x2 
solution of Ax = 0 would look like  −3x4 . Take x2 = a and x4 = b for


x4
any a, b ∈ R, to get the desired expression.

Shivangi Chandel (JSGP) Mathematics Advanced Mathematical Tools 25 / 28

 The solution sets to non-homogeneous system of linear equations are
not subspaces (Check?).
However, they do have a linear structure. In fact, these are translates
of a subspace, knowns as affine subspaces.
Let c ∈ Rn be a fixed vector. A subset in the form of c + S, where S
is a subspace of Rn , is called an affine subspace of Rn .
Example: (1, 0) + S where S = {(x1 , x2 ) ∈ R2 |x1 = x2 } is not a
subspace of R2 . What does it look like? It is a line passing through
(1, 0) which is parallel to the 45◦ line passing through origin.
In the next theorem, we state that the solution sets of Ax = b can be
written as a affine subspace of the solution set of Ax = 0 or Null(A).

Shivangi Chandel (JSGP) Mathematics Advanced Mathematical Tools 26 / 28

Theorem 2.2
Let Ax = b be a m × n system of linear equations. Let x0 ∈ Rn be a
particular solution of this system. Then every other solution x of Ax = b
can be written as x0 + w, where w ∈ Null(A). In other words, the solution
set of Ax = b is the affine subspace x0 + Null(A).

What this tells us is that the dimension of solution space of Ax = b is

the same as Null(A).
   
1 2 0 −2 0 3
Verify this for A = 2 4 1 −1 0 and b = 1.
1 2 1 1 1 2

Shivangi Chandel (JSGP) Mathematics Advanced Mathematical Tools 27 / 28

Rank-Nullity Theorem

In the example, Nullity of A = 2 = number of free variables = n−

number of basis variables = n − Rank(A).
Verify that Rank(A) = 3.
- This is a celebrated result, known as the Rank-Nullity Theorem,
sometimes even referred to as the Fundamental Theorem of Linear
Algebra.
 
1
1
 
In the same example, verify that x0 + Null(A) where x0 = −5 is

0
4
the solution space of the system.

Shivangi Chandel (JSGP) Mathematics Advanced Mathematical Tools 28 / 28