ENG1014 Engineering Numerical Analysis Course Notes

ENG1014 Course notes

Week 7. Advanced Program Control

WEEK 7. ADVANCED PROGRAM CONTROL.......................................................................................... 1

7.1. ALGORITHMS AND FLOWCHARTS ................................................................................................................. 2
7.2. ADVANCED LOOP CONCEPTS ..................................................................................................................... 5
7.3. EFFICIENCY AND TIMING ............................................................................................................................ 7
7.4. EXAMPLE – SOLVING SYSTEMS OF LINEAR EQUATIONS ................................................................................... 13

Overview
The greater the complexity of the tasks that we wish to perform with the programs we write, the
more important it becomes to understand the instructions that we employ within those
programs and their interactions with one another. These relationships and the ordering of
different program instructions is referred to as an algorithm. Flowcharts can be very useful in
expressing algorithms visually, which is easier to digest than a sequence of descriptive text.
This week, we will cover the complexity of programs, exploring nested loops and the
consequence of program complexity on the time it takes to execute a program. The usefulness
of nesting loops is illustrated through the example of solving simultaneous equations using
matrix algebra, which is relatively easy to program and scalable.

1
ENG1014 Engineering Numerical Analysis Course Notes

7.1. Algorithms and Flowcharts

Previously, we introduced the concept of a flowchart. We used them to plan our code, so that
we can visualise each step in the programs we’re writing. Our flowcharts represent what we call
algorithms, which we will explore in further depth this week.

7.1.1. What is an algorithm?

Algorithms are well-defined sets of instructions that we use to solve a complex problem.
Algorithms can often be represented as a flowchart, which describes the steps taken at each
point. While algorithms are clearly important in computer programming, they are also
applicable for non-computing tasks. For example, we can write an algorithm for baking a cake,
or any other planned process.

Figure 7.1 – Example flowchart for cooking rice on a stove.1 Note that the “decision” points are
expressed as yes/no questions; this is required if a computer will control the process.

1
AB Chaudhuri “Flowchart and Algorithms Basics: The Art of Programming (2020)

2
ENG1014 Engineering Numerical Analysis Course Notes

Another example might be the process we used to fit the various non-linear models we
encountered in W5 – Models and Curve Fitting.

Figure 7.2 – Flowchart describing the process of fitting non-linear models.

3
ENG1014 Engineering Numerical Analysis Course Notes

A good algorithm should:

• Work for all possible reasonable inputs.
• Reliably reach the intended endpoints for each combination of inputs.
• Be a complete set of instructions, with every step being fully defined, and not
requiring human judgement.
• Always produce its result in a finite time.

A good way to ensure that your algorithm is complete is to break the problem down into multiple
different levels, while still ensuring that the algorithm is complete at each level. At the final level,
each part of the algorithm should be simple enough to attack successfully. The process of
breaking down a complex program is shown graphically in Figure 7.3.

Figure 7.3 – Process of breaking down a complex program into small, manageable parts.

Breaking down complex problems into manageable pieces, as described in Figure 7.3, is an
important abstract skill for all engineers. It seems like a simple thing to do, but it is very easy to
miss details that end up being crucial. Always check that your breakdown is complete.

4
ENG1014 Engineering Numerical Analysis Course Notes

7.2. Advanced Loop Concepts

As we saw last week, loops are extremely powerful tools that are very flexible and can be applied
to a very wide variety of different problems. In this section we will explore how to further utilise
loops in more complicated situations.

7.2.1. Nested loops

Nested loops are the situation where one loop is contained within another loop. The inner loop
will run in its entirety for every iteration of the outer loop. Nested loops are frequently (but not
always) used when you have multi-dimensional objects, such as a matrix (2-dimensional),
rather than a vector (1-dimensional). A simple example of this is shown in Figure 7.4.

Script rows, cols = 5, 6

A = np.zeros([rows, cols])

for i in range(rows):
for j in range(cols):
A[i,j] = i*j

print(A)

Output [[ 0. 0. 0. 0. 0. 0.]
[ 0. 1. 2. 3. 4. 5.]
[ 0. 2. 4. 6. 8. 10.]
[ 0. 3. 6. 9. 12. 15.]
[ 0. 4. 8. 12. 16. 20.]]

Figure 7.4 – Use of nested for loops to create multiplication tables

Figure 7.5 shows the corresponding flowchart for this algorithm. You can see that the presence
of nested loops can start to make flowcharts quite complicated since you must manage the exit
conditions of all the loops present, as well as anything that happens inside those loops.

5
ENG1014 Engineering Numerical Analysis Course Notes

Figure 7.5 – Flowchart for creating multiplication tables.

Refer to Example 1 from the W7 – Advanced Program Control – Course Notes Examples

6
ENG1014 Engineering Numerical Analysis Course Notes

7.2.2. Pre-allocation
When writing for loops in ENG1014 to build new arrays, we usually already know the shape of
the new array we want to create. In this case, we should pre-allocate the array. Pre-allocation
means that we create an “empty” array and then fill it up later with the values we want. In
ENG1014, the functions np.zeros() and np.ones() can be used to pre-allocate NumPy arrays.

Consider the example where we want to calculate 𝑟𝑐𝑜𝑠(𝜃) for various values of 𝑟 and 𝜃:
r = np.arange(0, 10)
theta = np.arange(0, 2*np.pi, 0.1)

rows, cols = len(r), len(theta)

A = np.zeros([rows, cols])

for i in range(rows):
for j in range(cols):
A[i][j] = r[i]*np.cos(theta[j])

Here, we start with some data stored in r and theta. Using the np.zeros() function, we pre-
allocate an array A that has one row for each item in r and one column for each item in theta.
Following that, we loop over the rows and the columns, and calculate the value of A[i][j] for
each combination of i and j.

It is possible to append (add new values at the end) values to the end of an array, changing its
size each time you calculate a new value, but it should be avoided where possible. This is
because it will make your code run slowly, because the way Python appends to an array is quite
inefficient. In some limited circumstances this is necessary, but these are relatively rare.

Refer to Example 2 from the W7 – Advanced Program Control – Course Notes Examples

Complete Practice Questions 1-7 from the W7 – Advanced Program Control Course Notes
Practice Questions

7
ENG1014 Engineering Numerical Analysis Course Notes

7.3. Efficiency and Timing

When writing programs, it is important to be able to write efficient code, and to be able to
understand why our code is efficient or inefficient in different ways. In this section, we will
explore different tools that we can use to analyse and measure the efficiency of our code.

7.3.1. Timing
The simplest way that we can think about how efficient our code is by timing it. An efficient
program will run faster than an inefficient program when run on the same hardware. The built-in
Python module timeit contains a function default_timer() that returns the system time in
seconds. We can use this twice to get a start and an end time for anything that we want to time.
Thus, if we do this for each program we want to compare, we can compare the efficiency of our
code.
You should note that the time taken will change a little bit every time you run the program.
Sometimes one approach might be faster than the other, but at other times that might change
just because your computer was allocating resources to something else. If this happens, how
can we know which approach we should be taking?

Refer to Example 3 from the W7 – Advanced Program Control – Course Notes Examples

7.3.2. Computational complexity and Big-O notation

While timing is an important way to measure how well your code performs, there are many other
factors that affect how long it takes your code to run. How powerful is your computer? How
many other processes are running on it? These sorts of things are going to vary a lot based on
what computer you’re running it on. Is there a way that we could compare efficiency more
objectively between different programs on different computers?
To analyse the efficiency of our code without those interfering factors, we turn to another tool,
Computational Complexity. It is an abstract idea that asks, “how many operations will need to
be performed?”. In particular, it relates to how many operations are needed as our inputs grow
to larger sizes.

Big-O notation: If our input is “n” times larger, roughly how much longer will the program take
to run?

Big-O notation won’t tell you exactly how long a program will take; it is a rule-of-thumb, that
becomes more accurate when inputs become larger.

8
ENG1014 Engineering Numerical Analysis Course Notes

Example 1: 𝑶(𝒏)
Let’s consider an example of finding the maximum value in an array.
We can find the maximum value by assuming that the first thing we see is the maximum, then
comparing it to the next value we see. If the next value is higher, we replace that as the maximum
and then keep comparing that new maximum to each new value.
x = np.array([5, 10, 1, 3, 2, 20, 9, 17])
max_val = x[0]
for i in range(len(x)):
new_val = x[i]
if new_val > max_val:
max_val = new_val
How many iterations of this loop do we go through? There are 8 items in x, so we go through the
loop 8 times. However, if we had an array with 50 items, we would go through it 50 times. So, if
we have an array with 𝑛 items, we will go through the loop 𝑛 times.
We describe this computational complexity as being 𝑂(𝑛) , meaning that the number of
computations is some constant multiplied by n, otherwise described as on the order of 𝑛.

Example 2: 𝑶(𝟏), 𝒊. 𝒆. 𝑶(𝒏𝟎 )

Sometimes the number of operations we do doesn’t vary at all. For example, let’s find the first
element of x:
print(x[0])
The first element of x is always going to be in the same place, and we’re never going to have to
do any extra work to find it, no matter how big the rest of x is. This is a constant amount of work,
and we describe it as being 𝑂(1).

Example 3: 𝑶(𝒏𝟐 )
Let’s look at a more complicated example. Let’s say we had a list of numbers, and we wanted
to pair up every number with every other number. We can do this by picking the first element in
the list, then going through every element in the list and pairing them, then doing the same for
the second, third etc elements. This looks like a nested loop.
vals = np.array([9, 6, 2, 7, 3])

for i in range(len(vals)):
for j in range(len(vals)):
print((vals[i], vals[j]))

9
ENG1014 Engineering Numerical Analysis Course Notes

How many operations did we do? The outer loop ran 5 times, and the inner loop ran 5 times for
each time the outer loop ran. So, we did 52 = 25 calculations. If we had a list of length 𝑛, we
would do 𝑛 2 calculations, so we would have a 𝑂(𝑛 2 ) algorithm. In general, the complexity of a
nested piece of code is the complexity of the inside multiplied by the complexity of the outside,
in this case 𝑂(𝑛 ∗ 𝑛) = 𝑂(𝑛 2 ).

Example 4: 𝑶(𝒏)
It is important to note that sometimes you can end up with something that behaves like a loop,
even though you haven’t written one. This often happens when you are doing operations on
entire arrays.
x = np.arange(5,10)
print(x)
In this case, print(x) is an 𝑂(𝑛) operation, because it must go through and print each element
of the entire array x.

Example 5: 𝑶(𝒏𝟐 )
Likewise, if we nested an operation on array within a loop, we would have an 𝑂(𝑛 2 ) algorithm:
x = np.arange(6)
y = np.zeros(np.shape(x))

for i in range(len(x)):
y += x
In this example, we went through the loop 𝑂(𝑛) times. Each time, we added the entirety of x to
y. In doing so, we had to add each element of x to y, which also takes 𝑂(𝑛) time. Therefore, we
have an 𝑂(𝑛 2 ) algorithm. You should be careful when thinking about the efficiency of vector
operations, because they can implicitly use loops in this way. Sometimes it’s necessary, but at
other times you may unknowingly give your algorithms a poor complexity.

7.3.3. Scaling and Big-O notation

For small programs and small inputs, we often don’t worry about efficiency. The difference
between running a program in 1 millisecond or 10 milliseconds is not important for a human.
We typically only care about how the algorithm behaves as the inputs become very large, which
is where computers are most used.
If you tripled the input size on an 𝑂(𝑛) algorithm, you would only triple the runtime. But if you
tripled the input size on an 𝑂(𝑛 2 ) algorithm, you would multiply the runtime by 32 = 9. Since a

10
ENG1014 Engineering Numerical Analysis Course Notes

higher order complexity means more computation as we scale up, we generally want to make
our algorithms as low a complexity as possible. Ideally, all our algorithms would be 𝑂(1) –
constant! But this is obviously not possible, so we must deal with linear, quadratic or other
higher order complexities.
Because we only care about scaling as our inputs become very large, in Big-O we only write the
factor that contributes most to the scaling. If you have constant factors and/or lower order
terms, we simply ignore them. Let’s make our previous example a little bit more complicated to
illustrate this point.

Example 6: 𝑶(𝒏𝟐 )

vals = np.array([9, 5, 2, 7, 3])

for i in range(len(vals)):
print(vals[i])

for i in range(len(vals)):
for j in range(len(vals)):
print(f"\n{vals[i]*vals[j]}")
print((vals[i], vals[j]))

We have added two new steps here: we print out every number, and we also calculate the
product of each pair. So, the steps we are doing are:
• Printing out every number. This is an 𝑂(𝑛) operation, because we go through every
element in the list once.
• Printing out the product of each pair. This happens once every time we go through the
inner loop, which happens 𝑛 2 times, just like it did for printing out the pair, so it is an
𝑂(𝑛 2 ) operation.
• Printing out the pair of numbers. As we discussed earlier, this is an 𝑂(𝑛 2 ) operation.

So, we have 𝑂(𝑛) + 𝑂(𝑛 2 ) + 𝑂(𝑛 2 ). Let’s simplify this now.

𝑂(𝑛) + 𝑂(𝑛 2 ) + 𝑂(𝑛 2 )
= 𝑂(𝑛 + 2𝑛 2)
= 𝑂(𝑛 2 )

11
ENG1014 Engineering Numerical Analysis Course Notes

As we said earlier, we only care about how the amount of computation scales as the inputs get
very large. When 𝑛 is very large, 𝑛 2 will dwarf 𝑛 to the point where it is negligible, which is why
we ignore it. Similarly, since we only care about how much the amount of computation changes
as inputs grow, we simply ignore any constant factors that appear. The reason that we only care
about how our algorithms behave on large inputs is because they are the ones that can be
affected the most by a small change. Small inputs are always going to run fast, and large inputs
are always going to run slowly.
At a basic level, we can judge the complexity of an algorithm by how deep the nested loops go.
Example 1 had 1 loop, and it was 𝑂(𝑛). Both examples 3 and 6 had 2 loops nested and were
𝑂(𝑛 2 ) . Similarly, if you nested 3, or 4 or 5 loops, you would get 𝑂(𝑛 3) , 𝑂(𝑛 4 ) and 𝑂(𝑛 5 )
complexities. However, there are many cases where this isn’t true, and you need to be more
careful. It is common to see 𝑂(log(𝑛)) and 𝑂(2𝑛 ) complexities in some contexts, but we will
not go into detail about them in this unit. Figure 7.6 shows a hierarchy of some common
complexities, with 𝑂(1) being the best and 𝑂(𝑛!) being the worst.

Figure 7.6 – Graph showing the performance of different complexities 2.

2
https://adrianmejia.com/how-to-find-time-complexity-of-an-algorithm-code-big-o-notation/

12
ENG1014 Engineering Numerical Analysis Course Notes

7.3.4. Complexities on multiple input sizes

In many cases, we will also encounter inputs that have multiple parameters. If you had a matrix
that could be any dimension, 3x3, 4x5, 9x7 etc, you would have two different inputs dimensions
to consider for your complexity. Let’s take another look at our first nested loops example:
A = np.zeros((5,6))
rows, cols = np.shape(A)

for i in range(rows):
for j in range(cols):
A[i,j] = i*j
We note that A is a 5 × 6 matrix, so the outer loop runs 5 times and the inner loop runs 6 times,
and we get 5*6=30 operations. You could run the same algorithm on any 𝑚 × 𝑛 matrix, and it
would take 𝑂(𝑚𝑛) time to run.
The principles we discussed earlier about simplifying complexities still apply, but you should be
careful to note that you can’t always cancel out the other variables down to one term. For
example, if we had a much more complicated algorithm we might get:
𝑂(𝑚2 𝑛) + 𝑂(𝑚𝑛 2 ) + 𝑂(𝑚𝑛) + 𝑂(2𝑚2 )
= 𝑂(𝑚2 𝑛 + 𝑚𝑛 2 + 𝑚𝑛 + 2𝑚2 )
= 𝑂(𝑚2 𝑛 + 𝑚𝑛(𝑛 + 1) + 2𝑚2 )
= 𝑂(𝑚2 𝑛 + 𝑚𝑛 2 + 2𝑚2 )
= 𝑂(𝑚2 (𝑛 + 2) + 𝑚𝑛 2 )
= 𝑂(𝑚2 𝑛 + 𝑚𝑛 2 )

You could try to simplify this differently with some different working out steps, but you will
always end up with two terms that you can’t simplify out. In this unit, we won’t see many
algorithms with complexities that are this annoying to deal with, but you will certainly run into
many algorithms that are 𝑂(𝑚𝑛).

Complete Practice Questions 8-13 from the W7 – Advanced Program Control Course Notes
Practice Questions

13
ENG1014 Engineering Numerical Analysis Course Notes

7.4. Example – Solving Systems of Linear Equations

In this section, we will go through an example algorithm that uses advanced program control
(nested loops and conditionals) to perform a task. Solving systems of linear equations is critical
to many areas of engineering, such as optimising the best combination of conditions for a
chemical reaction or controlling the motion of a robotic arm.
Assuming you have solved a lot of simultaneous equations in the past, you can probably
appreciate that the process is quite repetitive. Additionally, while solving for 2-4 unknowns with
2-4 independent equations is something you could do on paper, if you had 100 unknowns and
100 equations, this becomes extremely tedious to solve by hand. This makes solving systems
of linear equations perfect for a computer to solve instead.
In accordance with good programming principles, we would ideally like to define a process that
is general; that can solve any system of equations, of any size. But to create such an algorithm,
we must be able to describe the process in a consistent manner.

14
ENG1014 Engineering Numerical Analysis Course Notes

7.4.1. Introduction to Gaussian elimination

From your previous studies, you should be familiar with the elimination method for solving
simultaneous equations. Here is an example with four variables, 𝑤, 𝑥, 𝑦, and 𝑧:
2𝑤 + 2𝑥 + 𝑧 = 10 [𝐸𝑞. 1]
𝑤 + 3𝑥 + 𝑦 + 𝑧 = 14 [𝐸𝑞. 2]
2𝑤 + 𝑥 + 𝑦 + 𝑧 = 11 [𝐸𝑞. 3]
3𝑤 + 3𝑥 + 3𝑦 + 3𝑧 = 30 [𝐸𝑞. 4]
This can be solved by a simple process:

Step Description Equations

1 Multiply 𝐸𝑞. 1 by 1/2 and subtract from 𝐸𝑞. 2 2𝑤 + 2𝑥 + 0𝑦 + 𝑧 = 10 [𝐸𝑞. 1]

1
0𝑤 + 2𝑥 + 𝑦 + 𝑧 = 9 [𝐸𝑞. 2 ]
2
2𝑤 + 𝑥 + 𝑦 + 𝑧 = 11 [𝐸𝑞. 3]
3𝑤 + 3𝑥 + 3𝑦 + 3𝑧 = 30 [𝐸𝑞. 4]

2 Multiply 𝐸𝑞. 1 by 2/2 = 1 and subtract from 𝐸𝑞. 3 2𝑤 + 2𝑥 + 0𝑦 + 𝑧 = 10 [𝐸𝑞. 1]

1
0𝑤 + 2𝑥 + 𝑦 + 𝑧 = 9 [𝐸𝑞. 2 ]
2
0𝑤 − 𝑥 + 𝑦 + 0𝑧 = 1 [𝐸𝑞. 3]
3𝑤 + 3𝑥 + 3𝑦 + 3𝑧 = 30 [𝐸𝑞. 4]

3 Multiply 𝐸𝑞. 1 by 3/2 = 1 and subtract from 𝐸𝑞. 4 2𝑤 + 2𝑥 + 0𝑦 + 𝑧 = 10 [𝐸𝑞. 1]

1
0𝑤 + 2𝑥 + 𝑦 + 𝑧 = 9 [𝐸𝑞. 2 ]
2
0𝑤 − 𝑥 + 𝑦 + 0𝑧 = 1 [𝐸𝑞. 3]
3
0𝑤 + 0𝑥 + 3𝑦 + 𝑧 = 15 [𝐸𝑞. 4]
2

4 Multiply 𝐸𝑞. 2 by −1/2 and subtract from 𝐸𝑞. 3 2𝑤 + 2𝑥 + 0𝑦 + 𝑧 = 10 [𝐸𝑞. 1]

1
0𝑤 + 2𝑥 + 𝑦 + 𝑧 = 9 [𝐸𝑞. 2 ]
2
3 1 11
0𝑤 + 0𝑥 + 𝑦 + 𝑧 = [𝐸𝑞. 3]
2 4 2
3
0𝑤 + 0𝑥 + 3𝑦 + 𝑧 = 15 [𝐸𝑞. 4]
2

15
ENG1014 Engineering Numerical Analysis Course Notes

5 Multiply 𝐸𝑞. 3 by 3/(3/2) = 2 and subtract from 2𝑤 + 2𝑥 + 0𝑦 + 𝑧 = 10 [𝐸𝑞. 1]

𝐸𝑞. 4 1
0𝑤 + 2𝑥 + 𝑦 + 𝑧 = 9 [𝐸𝑞. 2 ]
2
3 1 11
0𝑤 + 0𝑥 + 𝑦 + 𝑧 = [𝐸𝑞. 3]
2 4 2
0𝑤 + 0𝑥 + 0𝑦 + 𝑧 = 4 [𝐸𝑞. 4]

6 Solve 𝐸𝑞. 4 for 𝑧 𝑧=4

7 Substitute 𝑧 into 𝐸𝑞. 3 and solve for 𝑦 3 1 11

𝑦 + (4) =
2 4 2
11/2 − 1/4(4)
𝑦=
3/2
𝑦=3

8 Substitute 𝑦 and 𝑧 into 𝐸𝑞. 2 and solve for 𝑥 1

2𝑥 + 3(1) + (4) = 9
2
1
9 − (3(1) + (4))
𝑥= 2
2
𝑥=2

9 Substitute 𝑥, 𝑦, and 𝑧 into 𝐸𝑞. 1 and solve for 𝑤 2𝑤 + 2(2) + 0(3) + 1(4) = 10
10 − (2(2) + 0(3) + 1(4))
𝑤=
2
𝑤=1

We can also express systems of linear equations in the form of a matrix equation, 𝐴𝑥 = 𝑏 .
Finding 𝑥 will give us the solution of the linear system.

Figure 7.7 – Expressing a system of linear equations as a matrix equation, 𝐴𝑥 = 𝑏

16
ENG1014 Engineering Numerical Analysis Course Notes

Here is the same process for solving the four simultaneous equations, but using matrices:

Step Description Matrices

0 Express the equations as matrices in the form 2 2 0 1 𝑤 10

𝐴𝑥 = 𝑏 (Hint: expand them out using matrix 1 3 1 1 𝑥 14
[ ][ ] = [ ]
2 1 1 1 𝑦 11
multiplication to check that this is correct).
3 3 3 3 𝑧 30

1 Multiply row 1 by 1/2 and subtract from row 2 2 0 1 𝑤 10

0 2 1 1/2 𝑥 9
[ ][ ] = [ ]
2 1 1 1 𝑦 11
3 3 3 3 𝑧 30

2 Multiply row 1 by 2/2 = 1 and subtract from row 3 2 2 0 1 𝑤 10

0 2 1 1/2 𝑥 9
[ ][ ] = [ ]
0 −1 1 0 𝑦 1
3 3 3 3 𝑧 30

3 Multiply row 1 by 3/2 = 1 and subtract from row 4 2 2 0 1 𝑤 10

0 2 1 1/2 𝑥 9
[ ][ ] = [ ]
0 −1 1 0 𝑦 1
0 0 3 3/2 𝑧 15

4 Multiply row 2 by −1/2 and subtract from row 3 2 2 0 1 𝑤 10

0 2 1 1/2 𝑥 9
[ ][ ] = [ ]
0 0 3/2 1/4 𝑦 11/2
0 0 3 3/2 𝑧 15

5 Multiply row 3 by 3/(3/2) = 2 and subtract from 2 2 0 1 𝑤 10

row 4 0 2 1 1/2 𝑥 9
[ ][ ] = [ ]
0 0 3/2 1/4 𝑦 11/2
0 0 0 1 𝑧 4

6 Solve row 4 for 𝑧 𝑧 = 4, 𝑥 = [4]

7 Substitute 𝑧 into row 3 and solve for 𝑦 11/2 − [1/4][4]

𝑦=
3/2
3
𝑦 = 3, 𝑥 = [ ]
4

8 Substitute 𝑦 and 𝑧 into row 2 and solve for 𝑥 9 − [11/2] [3]

𝑥= 4
2
2
𝑥 = 2, 𝑥 = [3]
4

17
ENG1014 Engineering Numerical Analysis Course Notes

9 Substitute 𝑥, 𝑦, and 𝑧 into row 1 and solve for 𝑤 2

10 − [2 0 1] [ 3]
𝑤= 4
2
1
2
𝑤 = 1, 𝑥 = [ ]
3
4

We mentioned in 7.1.1. What is an algorithm? that breaking down complex problems into
manageable pieces is important to ensuring that your algorithm is complete. On examination of
both processes above, you may note that it can be broken down into two smaller pieces: steps
1-5 eliminate terms, starting from the top-left element working towards the bottom-right
element, and steps 6-9 substitute known values to solve for the unknown values, starting from
the bottom-right element and working backwards towards the top-left element. Steps 1-5 are
collectively known as “forward elimination” and steps 6-9 are known as “back substitution”. If
we follow these exact processes every single time, then we will always get to a final answer, and
there is no need for any human judgement during the process. As discussed earlier, these are
two of the important requirements for an algorithm.

7.4.2. Naïve Gaussian algorithm

The algorithm described above is known as “Naïve Gaussian”. In this section, we will break
down the algorithm into pseudocode that we can use to create a function.
There are a couple of key matrix concepts you should be familiar with when expressing systems
of linear equations in the form of a set of matrices:
• Main diagonal of a square matrix - elements where the row and column indices are the same
• Upper triangular matrix - square matrix where all elements below the main diagonal are zero
• Elements of a matrix 𝐴 can be described as 𝐴𝑟𝑐 where 𝑟 is the row and 𝑐 is the column

Forward elimination
We can break steps 1-5 down even further. Steps 1-3 eliminated the terms in the first column
below the main diagonal, step 4 eliminated the terms in the second column below the main
diagonal, and step 5 eliminated the term in the third column below the main diagonal.
To eliminate the term 𝐴𝑟𝑐 , the row 𝑐 was multiplied by a factor and then subtracted from row 𝑟.
That factor was specifically chosen by dividing the element 𝐴𝑟𝑐 by 𝐴𝑐𝑐 . As 𝐴𝑐𝑐 is used to
eliminate all terms in column 𝑐 below the main diagonal, 𝐴𝑐𝑐 is known as the “pivot”.
This process is then repeated until we have an upper triangular matrix.

18
ENG1014 Engineering Numerical Analysis Course Notes

Back substitution
Once we have an upper triangular matrix, we then need to solve for the unknown values.
Step 6 solved for the last unknown, dividing the element in the last row of 𝐵 by the element in
the last row and column of 𝐴.
In steps 7-8, to solve for the element 𝑥𝑟 , the traditional multiplication of the already-solved
variables (𝐴𝑟,𝑟+1 to 𝐴𝑟,𝑛 multiplied by 𝐵𝑟+1 to 𝐵𝑛 , where 𝑛 is the number of rows/column in 𝐴)
was subtracted from 𝐵𝑟 , and the result divided by 𝐴𝑟𝑟 .

Pseudocode for the naïve Gaussian algorithm

Description:
Uses naive Gaussian elimination to solve a system of linear equations represented as the
matrix equation 𝐴𝑥 = 𝑏.

Arguments: Returns:
• 𝐴 – 2D array containing the coefficients 𝑥 – a 1D array containing the unknowns
• 𝑏 – 1D array or 2D row/column vector
containing the solutions

Algorithm:
1. Find the shape of 𝐴, let this be known as 𝑚 × 𝑛.
2. Create the augmented matrix [𝐴 𝑏] as 𝐴𝑢𝑔 (this simplifies step 7).
3. Pre-allocate an array of 0s for 𝑥.
Forward elimination:
4. Start at the first column, let this be known as 𝑐.
5. Start at the first row below the main diagonal, let this be known as 𝑟.
6. Determine the normalisation factor (𝐴𝑟𝑐 /𝐴𝑐𝑐 ).
7. Replace row 𝑟 of 𝐴𝑢𝑔 with row 𝑟 of 𝐴𝑢𝑔 – factor × row 𝑐 of 𝐴𝑢𝑔.
8. Is 𝑟 the last row? If yes, continue. If no, move to the next row and repeat steps 6-8.
9. Is 𝑐 the second last column? If yes, continue. If no, move to the next column and
repeat steps 5-9.
Back substitution:
10. Solve the last row as 𝑏𝑛 /𝐴𝑛 .
11. Start at the second last row, let this be known as 𝑟.
12. Determine the value of 𝑥𝑟 according to the following equation:
𝑏𝑟 −𝐴𝑟𝑞 𝑥𝑞
𝑥𝑟 = , where 𝑞 = 𝑟 + 1: 𝑛
𝐴𝑟𝑟

13. Is 𝑟 the last row? If yes, continue. If no, move to the next row and repeat steps 12-13.
14. Return 𝑥. Algorithm is complete.

19
ENG1014 Engineering Numerical Analysis Course Notes

Complete Steps 1-4 from the W7 – Advanced Program Control – ENG1014 Module
Instructions

Refer to Examples 4-5 from the W7 – Advanced Program Control – Course Notes Examples

7.4.3. Testing Naïve Gaussian algorithm

In 3.6.1. Input testing, we discussed the idea that pieces of code are often written so that it only
works for particular inputs. We also emphasised the importance of checking that these
requirements are fulfilled, and that the piece of code should print an error message when the
requirements are not met.

Requirement 1
For the Naïve Gaussian elimination algorithm above, it is obvious that one requirement of this
algorithm is that 𝐴 is a square matrix. Thus, 𝐴 must have 2 dimensions, and must have the same
number of rows and columns. The vector 𝑏 must have 1 dimension or have 2 dimensions but
just 1 row or column. Additionally, 𝑏 must have the same number of elements as there are
rows/columns in 𝐴. These are all simple requirements that we can check and raise an error if
they are not met.

Input testing for the naïve Gaussian algorithm

Before step 1 of the naïve Gaussian algorithm, check the following conditions. If they are true,
continue. If they are false, raise an error.
• 𝐴 has 2 dimensions
• 𝐴 has the same number of rows and columns
• 𝑏 has either 1 or 2 dimensions
o If 𝑏 has 2 dimensions, 𝑏 is a column vector
• 𝑏 has the same number of elements as there are rows/columns in 𝐴

Requirement 2
From your previous studies, you should be familiar with the three types of solutions to systems
of linear equations described in Figure 7.8: one solution, no solution, and infinite solutions.

20
ENG1014 Engineering Numerical Analysis Course Notes

Figure 7.8 – Types of solutions to systems of linear equations

Here are two examples with two variables, 𝑎 and 𝑏:
Example 1
2𝑥 + 4𝑦 = 8
𝑥 + 2𝑦 = 4

Matrix representation:
2 4 𝑥 8
[ ] [𝑦] = [ ]
1 2 4
After applying forward elimination:
2 4 𝑥 8
[
] [𝑦] = [ ]
0 0 0
The second row indicates redundancy, 0 = 0, which means that there are infinite solutions.
Example 2
𝑥 + 2𝑦 = 3
𝑥 + 2𝑦 = 5
Matrix representation:
1 2 𝑥 3
[ ] [𝑦] = [ ]
1 2 5
After applying forward elimination:
1 2 𝑥 3
[][ ] = [ ]
0 0 𝑦 2
The second row indicates inconsistency, 0 ≠ 2, which means there are no solutions.
These two examples make it clear that the Naïve Gaussian elimination algorithm above does
not work when the system of linear equations either has infinite or no solutions.
In 3.6.1. Input testing, we also mentioned that if it is not possible to check whether the input
requirements are fulfilled, the limitations should be clearly noted in the function’s
documentation. Checking this requirement is more complicated than just checking the shape
of the inputs, so we will just note the limitations of this algorithm in the function documentation.

21
ENG1014 Engineering Numerical Analysis Course Notes

Requirement 3
The Naïve Gaussian elimination algorithm involves two division operators – one at step 6 and
one in step 11, where both divide a value by the pivot. As you cannot divide a value by 0, the
algorithm will fail when there is a zero in the pivot. In this case, hopefully you know that the
solution is to perform a row-swap, also known as “partial pivoting”.

Complete Steps 5-6 from the W7 – Advanced Program Control – ENG1014 Module
Instructions

7.4.4. Gaussian elimination with partial pivoting

Partial pivoting can be implemented by inserting an if statement into our Naïve Gaussian
algorithm above, between steps 3 and 4. If the pivot is zero, a row swap is performed. If the pivot
is non-zero, the algorithm continues as before. The algorithm above was called “Naïve
Gaussian” because we did not account for zeros in the main diagonal.

Pseudocode for the Gaussian algorithm

Same as naïve Gaussian algorithm above, except between steps 4 and 5:
1. Check if the pivot (𝐴𝑐𝑐 ) is equal to 0. If yes, continue. If no, skip to step 5 in the original
naïve Gaussian algorithm.
2. Find the index of the maximum of the elements directly below the pivot.
3. Adjust the index such that it refers to the row index of 𝐴 instead of just the index of the
elements directly below the pivot.
4. Swap the row containing the maximum with the row of the original pivot.
5. Continue to step 5 in the original naïve Gaussian algorithm.

Complete Steps 7-9 from the W7 – Advanced Program Control – ENG1014 Module
Instructions

Refer to Example 6 from the W7 – Advanced Program Control – Course Notes Examples

Complete Practice Questions 14-15 from the W7 – Advanced Program Control Course Notes
Practice Questions

You should now be able to answer Part B from W7 – Weekly Quiz

22