DSA2102 Essential Data Analytics Tools: Numerical Computation

Lecture 5: Systems of Linear Equations

Tim Wertz

Department of Mathematics, NUS

Wertz Lecture 5 1 / 30
Introduction to linear systems

General notations for linear systems:

a11 x1 + a12 x2 + · · · + a1n xn = b1 ,

a21 x1 + a22 x2 + · · · + a2n xn = b2 ,
··· ··· ··· ···
am1 x1 + am2 x2 + · · · + amn xn = bm .

Let A = (aij )m⇥n , x = (x1 , · · · , xn )T , and b = (b1 , · · · , bm )T . Then

Ax = b.

Question:
Does the solution exist?
If solutions exist, how many solutions are there?

Wertz Lecture 5 2 / 30
Examples
Example 1: m = n = 1:
a11 x1 = b1 .
x1 = b1 /a11 if a11 6= 0.
The solution does not exist if a11 = 0 but b1 6= 0.
There are infinite solutions if a11 = b1 = 0.

In general, the number of solutions can only be zero, one or

infinity.Example 2: m = 1, n = 2:
a11 x1 + a12 x2 = b1 .
If a11 6= 0, we can choose any x2 , and the solution of x1 is
(b1 a12 x2 )/a11 .
if a11 = 0 and a12 6= 0, we can choose any x1 , and the solution of x2
is (b1 a11 x1 )/a12 .
if a11 = a12 = 0, there are infinite solutions if b1 = 0, and there are
no solutions if b1 6= 0.

In general, if m < n, the numberWertz

of solutions
Lecture 5
can only be zero or 3 / 30
Economics
Economy has n sectors
Internal demand
External demand
Model
total amount produced = internal demand + external demand
x = f (x) + d () x f (x) = d

Four sectors: agriculture, labor, manufacturing, and service

xA = 0.2xA + 0.5xL + 0.2xM + 0.1xS
xL = 0.4xA + 0.1xL + 0.1xM + 0.4xS
xM = 0.1xA + 0.5xL + 0.5xM + 0.1xS
xS = 0.1xA + 0.3xL + 0.1xM + 0.3xS

0.8xA 0.5xL 0.2xM 0.1xS = dA

0.4xA + 0.9xL 0.1xM 0.4xS = dL
Wertz Lecture 5 4 / 30
Statistics
Fit a linear model to data
Height and weight
Height Weight
160 60
170 70
180 75
System of equations
60 = x1 · 160 + x0
70 = x1 · 170 + x0
75 = x1 · 180 + x0
Instead
a = x1 · 160 + x0
b = x1 · 170 + x0
c = x1 · 180 + x0

Wertz Lecture 5 5 / 30
GPS
Model of the earth = {x 2 + y 2 + z 2 = 1}
Signals received Satellite Position Time
A (1,1,1) 3
B (2,1,1) 4
C (1,0,2) 5
D (1,3,2) 6
Express distance q
dA = c(t 3) = (x 1)2 + (y 1)2 + (z 1)2
2x + 2y + 2z + 6c 2 t = x 2 + y 2 + z 2 + 3 c 2t 2 9c 2

System of equations
4x + 2y + 2x 8c 2 t = 6 16c 2 + 2x + 2y + 2z 6c 2 t 3 + 9c 2
2x + 0y + 4z 10c 2 t = 5 25c 2 + 2x + 2y + 2z 6c 2 t 3 + 9c 2
2x + 6y + 4z 12c 2 t = 14 36c 2 + 2x + 2y + 2z 6c 2 t 3 + 9c 2 .

Wertz Lecture 5 6 / 30
Errors and Conditioning

Relative forward error = relative error in outputs

Relative backward error = relative error in inputs
System of linear equations Ax = b
Question: what is the output? Answer: x.
Question: what are the inputs? Answer: b and A.
We consider each one separately
How to measure the size of x, b, and A?

Wertz Lecture 5 7 / 30
Vector Norms
Euclidean norm v
u n
p uX
kxk2 = hx, xi = t xi2
i=1

One-norm
n
X
kxk1 = |xi |
i=1

Infinity norm
kxk1 = max |xi |
i

p-norms
n
!1/p
X
kxkp = |xi |p
i=1

limp!1 kxkp = kxk1

Wertz Lecture 5 8 / 30
Vector Norms

x = (2, 3, 6)T
kxk2 = 7
kxk1 = 11
kxk1 = 6

Wertz Lecture 5 9 / 30
Matrix Norms

Entry-wise norms
For systems Ax = b, we can think of A as a function A : Rn ! Rn
What is a good way to measure the size of a linear function?

f (x) = 3x g (x) = 2x h(x) = 4x

For linear functions from Rn ! Rn , we take

kAxk
max = max kAxk
x6=0 kxk kxk=1

Wertz Lecture 5 10 / 30
Induced Matrix Norms

kAk := max kAxk

kxk=1

For p = 1 and p = 1 we have

m
X
kAk1 = max |A(i, j)| = max absolute column sum
j
i=1

For p = 1 we have
n
X
kAk1 = max |A(i, j)| = max absolute row sum
i
j=1

For p = 2 we have kAk2 = max (A) = largest singular value of A

Submultiplicative and consistent norms
kABk  kAkkBk and kAxk  kAk · kxk
Wertz Lecture 5 11 / 30
Induced Matrix Norms

✓ ◆
1 2
A=
0 2

kAk1 = 4

kAk1 = 3

kAk2 = 2

Wertz Lecture 5 12 / 30
Conditioning
We have a non-singular system Ax = b
Instead of b we have a perturbed input b̂, with b = b̂ b
Let x̂ be the solution to the perturbed system Ax̂ = b̂, with
x = x̂ x
Then we have A( x) = A(x̂ x) = Ax̂ Ax = b̂ b = b,
equivalently x = A 1 ( b)
relative output error k xk/kxk k xkkbk kA 1 ( b)kkAxk
= = =
relative input error k bk/kbk k bkkxk k bkkxk

relative output error kA 1 kkbkkAkkxk 1

 = kA kkAk
relative input error k bkkxk

Equivalently, we have
k xk 1 k bk
 kA kkAk
kxk kbk
Wertz Lecture 5 13 / 30
Conditioning

We define the condition number of the system Ax = b to be

(A) = kAkkA 1 k
What if we perturb A?
A similar analysis yields

k xk 1 k Ak
 kA kkAk
kx̂k kAk

With even more work, we can get

k xk kA 1 kkAk k Ak

kxk (1 kA 1 kk Ak) kAk

Wertz Lecture 5 14 / 30
Solve linear systems

We only consider the case A 2 Rn⇥n .

Example for n = 2:
x1 + 2x2 = 3,
2x1 + 3x2 = 8.
Multiply the first equations by 2, and add the result to the second
equation:

( x1 + 2x2 ) ⇥ 2 + 2x1 + 3x2 = 3 ⇥ 2 + 8

Wertz Lecture 5 15 / 30
Solve linear systems

Example for n = 4:

E1 : x1 + x2 + 3x4 = 4,
E2 : 2x1 + x2 x3 + x 4 = 1,
E3 : 3x1 x2 x3 + 2x4 = 3,
E4 : x1 + 2x2 + 3x3 x4 = 4,

Multiply E1 by 2 and subtract the result from

E2 :E2 2 ⇥ E1 :E2 E2 2 ⇥ E1 :
(2x1 +x2 x3 +x4 ) 2(x1 +x2 +3x4 ) = 1 2⇥4. x2 x3 5x4 = 7.
E3 E3 3 ⇥ E1 :
E3 : 4x2 x3 7x4 = 15.
E4 E4 + E1 :
E4 : 3x2 + 3x3 + 2x4 = 8.

Wertz Lecture 5 16 / 30
Solve the smaller linear system

A smaller linear system:

E2 : x2 x3 5x4 = 7,
E3 : 4x2 x3 7x4 = 15,
E4 : 3x2 + 3x3 + 2x4 = 8.

E3 E3 4 ⇥ E2 :
E3 : 3x3 + 13x4 = 13.
E4 E4 + 3 ⇥ E2 :
E4 : 13x4 = 13.

Wertz Lecture 5 17 / 30
Find the values of all the unknowns

The system after elimination:

E1 : x 1 + x2 + 3x4 = 4,
E2 : x2 x3 5x4 = 7,
E3 : 3x3 + 13x4 = 13,
E4 : 13x4 = 13.

Backward substitution:
x4 = 13 ÷ ( 13) = 1
x3 = (13 13x4 ) ÷ 3 = (13 13 ⇥ 1) ÷ 3 = 0
x2 = ( 7 + x3 + 5x4 ) ÷ ( 1) = ( 7 + 0 + 5 ⇥ 1) ÷ ( 1) = 2
x1 = 4 x2 3x4 = 4 2 3⇥1= 1

Wertz Lecture 5 18 / 30
How to transform it to a computer program?

A computer program can only deal with numbers.

Every symbol in the computer program must have a value.
A computer program cannot represent equations like 3x3 + 13x4 = 13.
We need to find a way to represent the process of solving equations
using only numbers.

Representation of the equations:

0 1
E1 : x1 + x2 + 3x4 = 4, 1 1 0 3 4
E2 : 2x1 + x2 x3 + x4 = 1, B
B 2 1 1 1 1C
C
E3 : 3x1 x2 x3 + 2x4 = 3, @ 3 1 1 2 3A
E4 : x1 + 2x2 + 3x3 x4 = 4, 1 2 3 1 4

Wertz Lecture 5 19 / 30
Augmented matrix
In general, the linear system
a11 x1 + a12 x2 + · · · + a1n xn = b1 ,
a21 x1 + a22 x2 + · · · + a2n xn = b2 ,
··· ··· ··· ···
an1 x1 + an2 x2 + · · · + ann xn = bn ,
can be stored as
0 1
a11 a12 · · · a1n b1
Ba21 a22 · · · a2n b2 C
B C
B .. .. .. .. C
@ . . . .A
an1 an2 · · · ann bn

Definition
Given a linear system Ax = b for A 2 Rn⇥n and b 2 Rn , the augmented
matrix of this linear system is an n ⇥ (n + 1) matrix (A | b), where the
vertical line is placed in the matrix
Wertz
to divide
Lecture 5
the coefficient matrix and 20
the/ 30
Existence and uniqueness

Theorem
Let à = (A | b) be the augmented matrix of the linear system Ax = b,
where A 2 Rm⇥n and b 2 Rm . Then the solution x exists if and only if
rank A = rank Ã. The solution x is unique if and only if
rank A = rank à = n.
Rank of a matrix: Let A = (aij )m⇥n . Then the rank of A is the minimum
dimenision of the hyperplane that includes all the following vectors:
0 1 0 1 0 1
a11 a12 a1n
B a21 C B a22 C B a2n C
B C B C B C
B .. C , B .. C , · · · , B .. C .
@ . A @ . A @ . A
am1 am2 amn

Wertz Lecture 5 21 / 30
Elimination of variables on the augmented matrix
Consider the operation E2 E2 2 ⇥ E1 :

E1 : x1 + x2 + 3x4 =
4
E1 : x 1 + x2 + 3x4 = 4 E : 2x1 + x2 x3 + x 4 =
1
2
E2 : 2x1 + x2 x3 + x4 = 1 E : 3x1 x2 x3 + 2x4 =
3
3
E3 : 3x1 x2 x3 + 2x4 = 3 E : x1 + 2x2 + 3x3 x4 =
4
4
E4 : x1 + 2x2 + 3x3 x4 = 4 2
2 ⇥ E1 : 2x1 + 2x2 + 6x4 = 8
E2 2
0 1
1 1 0 3 4
B 0 1 1 5 7C
B C
@ 3 1 1 2 3A
1 2 3 1 4

Wertz Lecture 5 22 / 30
Elimination of variables on the augmented matrix
E2 E2 2 ⇥ E1 :
0 1 0 1
1 1 0 3 4 1 1 0 3 4
B 2 1 1 1 1C (2) (2) 2⇥(1) B 0 1 1 5 7C
B C !B C
@ 3 1 1 2 3A @ 3 1 1 2 3A
1 2 3 1 4 1 2 3 1 4

E3 E3 3 ⇥ E1 :
0 1 0 1
1 1 0 3 4 1 1 0 3 4
B 0 1 1 5 7C (3) (3) 3⇥(1) B 0 1 1 5 7C
B C !B C
@ 3 1 1 2 3A @ 0 4 1 7 15A
1 2 3 1 4 1 2 3 1 4

E4 E4 + E1 :
0 1 0 1
1 1 0 3 4 1 1 0 3 4
B 0 1 1 5 7C (4) (4)+(1) B0 1 1 5 7C
B C !B C
@ 0 4 1 7 15A @0 4 1 7 15A
1 2 3 1 4 0 3 3 2 8

Wertz Lecture 5 23 / 30
Elimination of variables on the augmented matrix

0 1 (2) (2) 2⇥(1)

0 1
1 1 0 3 4 (3) (3) 3⇥(1)
1 1 0 3 4
B 2 1 1 1 1C B0 1 1 5 7C
B C (4) (4)+(1)
!B C
@ 3 1 1 2 3A @0 4 1 7 15A
1 2 3 1 4 0 3 3 2 8
0 1
(3) (3) 4⇥(2)
1 1 0 3 4
(4) (4)+3⇥(2) B0 1 1 5 7C
!B @0
C
0 3 13 13A
0 0 0 13 13

The left part of the final augmented matrix is upper triangular!

Wertz Lecture 5 24 / 30
Backward substitution

0 1
1 1 0 3 4 x 1 + x2 + 3x4 = 4
B0 1 1 5 7C x2 x3 + 5x4 = 7
B C
@0 0 3 13 13A 3x3 + 13x4 = 13
0 0 0 13 13 13x4 = 13

x4 = 13 ÷ ( 13) = 1
x3 = (13 13x4 ) ÷ 3 = (13 13 ⇥ 1) ÷ 3 = 0
x2 = ( 7 + x3 + 5x4 ) ÷ ( 1) = ( 7 + 0 + 5 ⇥ 1) ÷ ( 1) = 2
x1 = 4 x2 3x4 = 4 2 3⇥1= 1
The method using elimination and backward substitution to solve a linear
system is called Gaussian elimination.

Wertz Lecture 5 25 / 30
Elimination for a general system
Elimination of the first unknown:
0 1 (2) (2) (a21 /a11 )⇥(1) 0 1
a11 a12 · · · a1n b1 (3) (3) (a31 /a11 )⇥(1) a11 a12 · · · a1n b1
Ba21 ··· ··· ··· ···
B a22 · · · a2n b2 CC (n) (n) (an1 /a11 )⇥(1)
B0
B ã22 · · · ã2n b̃2 C
C
B . .. .. .. C !B . .. .. .. C
@ .. . . .A @ .. . . .A
an1 an2 · · · ann bn 0 ãn2 · · · ãnn b̃n
where for j = 2, · · · , n and k = 2, · · · , n,
aj1 aj1
ãjk = ajk a1k , b̃j = bj b1 .
a11 a11
For simplicity, we remove the tildes on the variables.Elimination of the
second unknown:
0 1 (3) (a32 /a22 )⇥(2) 0a 1
a11 a12 a13 · · · a1n b1 (3)(4) (4) (a42 /a22 )⇥(2)
11 a12 a13 · · · a1n b1
B0 a22 a23 · · · a2n b2 C B0 a22 a23 · · · a2n b2 C
B C (n)··· (n)···(a ··· ··· B C
B0 a32 a33 · · · a3n b3 C n2 /a22 )⇥(2) B 0 0 ã33 · · · ã3n b̃3 C
B C !B C
B . .. .. .. .. C B . .. .. .. .. C
@ .. . . . .A @ .. . . . .A
0 an2 an3 · · · ann bn 0 0 ãn3 · · · ãnn b̃n
where for j = 3, · · · , n and k = 3, · · · , n,
aj2Wertz ˜ 5
Lecture
aj2 26 / 30
Backward substitution for a general system

Final augmented matrix:

0 1
a11 a12 a13 · · · a1,n 1 a1n b1
B 0 a22 a23 · · · a2,n 1 a2n b2 C
B C
B 0 0 a33 · · · a3,n 1 a3n b3 C
B C
B .. .. . . .. .. .. .. C
B . . . . . . . C
B C
@ 0 0 ··· 0 an 1,n 1 an 1,n bn 1
A
0 0 ··· 0 0 ann bn

xn = bn /ann n
X
xn = (bn an bi aij xj
1 1 1,n xn )/an 1,n 1
j=i+1
··· ··· xi =
aii
x1 = (b1 a12 x2 ··· a1n xn )/a11

Wertz Lecture 5 27 / 30
Elimination on the same matrix
Current notation may be a waste of paper/memory:
0 1 (2) (2) 2⇥(1)
0 1
1 1 0 3 4 (3) (3) 3⇥(1)
1 1 0 3 4
B 2 1 1 1 1C B0 1 1 5 7C
B C (4) (4)+(1)
!B C
@ 3 1 1 2 3A @0 4 1 7 15A
1 2 3 1 4 0 3 3 2 8
0 1
(3) (3) 4⇥(2)
1 1 0 3 4
(4) (4)+3⇥(2) B0 1 1 5 7C
!B @0
C
0 3 13 13A
0 0 0 13 13
0 1
1 1 0 3 4
B 2 1 1 1 10C
B C
B 1 1 5 70 1C
B C
B 1 5 7 C
B C
B 3 1 1 2 30C
B C
B 4 1 7 4C
B C
B 150 C
B C
B C
B 1 7 0 3C
B C
B Wertz 150Lecture 5 C 28 / 30
Elimination on the same matrix
We can make the elimination process more economic by avoid changing
numbers to zero:
0 1
1 1 0 3 4
B 2 1 1 1 12C
B C
B 1 1 5 72 1C
B C
B 1 5 7 C
B C
B 3 1 1 2 33C
B C
B 4 1 7 4C
B C
B 153 C
B C
B C
B 1 7 4 3C
B C
B 153 C
B C
B 13 133 4 3 13C
B C
B 13 C
B C
B 1 2 3 1 4 1C
B C
B 3 3 2 8 1 3C
B C
B 3 2 3 0C
B C
B 8 1 C
B Wertz Lecture 5 C 29 / 30
Summary

Definition of a linear system

Number of solutions for a linear system
Vector and matrix norms
Conditioning of linear systems
Use Gaussian elimination to solve a linear system
An economic way to present Gaussian elimination

Wertz Lecture 5 30 / 30