INT102

Algorithmic Foundations And

Problem Solving
Algorithm efficiency + Searching/Sorting

Dr Yushi Li
Department of Computer Science
 Learning outcomes
• See some examples of polynomial time and
exponential time algorithms
• Able to carry out simple asymptotic analysis of
algorithms
• Able to apply searching/sorting algorithms and
derive their time complexities

2
More polynomial time algorithms -
searching …
 Searching
Input: a sequence of n numbers a0, a1, …, an-1; and a
number X
Output: determine whether X is in the sequence or
not
Algorithm (Linear search):
1. Starting from i=0, compare X with ai one by one as long
as i < n.
2. Stop and report "Found!" when X = ai .
3. Repeat and report "Not Found!" when i >= n.

4
 To find 7
Linear Search
➢ 12 34 2 9 7 5 six numbers
7 number X
➢ 12 34 2 9 7 5
7
➢ 12 34 2 9 7 5
7
➢ 12 34 2 9 7 5
7
➢ 12 34 2 9 7 5
7 found!

5
 To find 10
Linear Search (2)
➢ 12 34 2 9 7 5
10
➢ 12 34 2 9 7 5
10
➢ 12 34 2 9 7 5
10
➢ 12 34 2 9 7 5
10
➢ 12 34 2 9 7 5
10
➢ 12 34 2 9 7 5
10 not found!

6
 Linear Search (3)
i=0
while i < n do
begin
if X == a[i] then
report "Found!" and stop
else
i = i+1
end
report "Not Found!"

7
 Time Complexity
i = 0
Important operation of while i < n do
begin
searching: comparison if X == a[i] then
report "Found!" & stop
How many comparisons this else
i = i+1
algorithm requires? end
report "Not Found!"

Best case: X is the 1st no., 1 comparison, O(1)

Worst case: X is the last no. OR X is not found, n
comparisons, O(n)
8
 Improve Time Complexity?
If the numbers are pre-sorted, then we
can improve the time complexity of
searching by binary search.

9
 Binary Search
more efficient way of searching when the sequence of
numbers is pre-sorted
Input: a sequence of n sorted numbers a0, a1, …, an-1 in
ascending order and a number X
Idea of algorithm:
– compare X with number in the middle
– then focus on only the first half or the second half (depend
on whether X is smaller or greater than the middle number)
– reduce the amount of numbers to be searched by half

10
 To find 24
Binary Search (2)
3 7 11 12 15 19 24 33 41 55
24
19 24 33 41 55
24
19 24
24
24
24 found!

11
 Binary Search (3)
To find 30

3 7 11 12 15 19 24 33 41 55 10 nos
30 X

19 24 33 41 55
30
19 24
30
24
30 not found!

12
 Binary Search (4)
first=0, last=n-1
while (first <= last) do   is the floor function,
truncate the decimal part
begin
mid = (first+last)/2
if (X == a[mid])
report "Found!" & stop
else
if (X < a[mid])
last = mid-1
else
first = mid+1
end
report "Not Found!"
13
 Time Complexity
Best case: first=0, last=n-1
X is the number in the middle while (first <= last) do
=> 1 comparison, O(1)-time begin
mid = (first+last)/2
if (X == a[mid])
Worst case: report "Found!" & stop
at most log2n+1 comparisons, else
if (X < a[mid])
O(log n)-time last = mid-1
else
Why? Every comparison first = mid+1
end
reduces the amount of report "Not Found!"
numbers by at least half
E.g., 16 => 8 => 4 => 2 => 1
14
 Binary search vs Linear search
Time complexity of linear search is O(n)
Time complexity of binary search is O(log n)
Therefore, binary search is more efficient than linear
search

15
 Search for a pattern
We’ve seen how to search a number over a sequence
of numbers
What about searching a pattern of characters over
some text?
Example
text: N O B O D Y _ N O T I C E _ H I M
pattern: N O T
substring: N O B O D Y _ N O T I C E _ H I M

16
 String Matching
Given a string of n characters called the text and a string
of m characters (mn) called the pattern.
We want to determine if the text contains a substring
matching the pattern.

Example
text: N O B O D Y _ N O T I C E _ H I M
pattern: N O T
substring: N O B O D Y _ N O T I C E _ H I M

17
 Example
T[ ]: N O B O D Y _ N O T I C E _ H I M
P[ ]: N O T
N O T
N O T
bolded: match
N O T underlined: not match
N O T un-bolded: not considered
N O T
N O T
N O T

18
 The algorithm
The algorithm scans over the text position by position.
For each position i, it checks whether the pattern
P[0..m-1] appears in T[i..i+m-1]
If the pattern exists, then report found
Else continue with the next position i+1
If repeating until the end without success, report not
found

19
 Match pattern with T[i..i+m-1]
j=0
while (j < m && P[j]==T[i+j]) do
j=j+1
if (j == m) then
report "found!" & stop

T[i] T[i+1] T[i+2] T[i+3] … T[i+m-1]

P[0] P[1] P[2] P[3] … P[m-1]

20
 Match for all positions
for i = 0 to n-m do
begin

// check if P[0..m-1] match with T[i..i+m-1]

end
report "Not found!"

21
 Match for all positions
for i = 0 to n-m do
begin
j=0
while (j < m && P[j]==T[i+j]) do
j=j+1
if (j == m) then
report "found!" & stop
end
report "Not found!"

22
 Time Complexity
How many comparisons this algorithm requires?
Best case: for i = 0 to n-m do
pattern appears in begin
j = 0
the beginning of the while j < m & P[j]==T[i+j] do
j = j + 1
text, O(m)-time if j == m then
report "found!" & stop
Worst case: end
pattern appears at report "Not found!"

the end of the text

OR pattern does not
exist, O(nm)-time

23
More polynomial time algorithms -
sorting …
 Sorting
Input: a sequence of n numbers a0, a1, …, an-1
Output: arrange the n numbers into ascending order,
i.e., from smallest to largest
Example: If the input contains 5 numbers 132, 56, 43,
200, 10, then the output should be 10, 43, 56, 132,
200

There are many sorting algorithms:

insertion sort, selection sort, bubble sort,
merge sort, quick sort 25
 Selection Sort
➢find minimum key from the input sequence
➢delete it from input sequence
➢append it to resulting sequence
➢repeat until nothing left in input sequence

26
 Selection Sort - Example
➢ sort (34, 10, 64, 51, 32, 21) in ascending order
Sorted part Unsorted part Swapped
34 10 64 51 32 21 10, 34
10 34 64 51 32 21 21, 34
10 21 64 51 32 34 32, 64
10 21 32 51 64 34 51, 34
10 21 32 34 64 51 51, 64
10 21 32 34 51 64 --
10 21 32 34 51 64

27
 Selection Sort Algorithm
for i = 0 to n-2 do
begin

// find the index of the minimum number

// in the range a[i] to a[n-1]

swap a[i] and a[min]

end

28
 Selection Sort Algorithm
for i = 0 to n-2 do
begin
how to swap
min = i
two entries of
for j = i+1 to n-1 do an array?
if a[j] < a[min] then using a temp
min = j variable?
swap a[i] and a[min] not using a
end temp variable?

29
 Algorithm Analysis
The algorithm consists of a for i = 0 to n-2 do
nested for-loop. begin
min = i
For each iteration of the outer i- for j = i+1 to n-1 do
if a[j] < a[min] then
loop, min = j
swap a[i] and a[min]
there is an inner j-loop. end

Total number of comparisons i # of comparisons

= (n-1) + (n-2) + … + 1 in inner loop
= n(n-1)/2 0 n-1
1 n-2
O(n2)-time
… ...
n-2 1 30
 Bubble Sort
starting from the last element, swap adjacent items if
they are not in ascending order
when first item is reached, the first item is the
smallest
repeat the above steps for the remaining items to
find the second smallest item, and so on

31
 Bubble Sort - Example
(34 10 64 51 32 21)
round
34 10 64 51 32 21
1 34 10 64 51 21 32
34 10 64 21 51 32
34 10 21 64 51 32 don’t need to
swap
34 10 21 64 51 32
10 34 21 64 51 32
2 10 34 21 64 32 51
10 34 21 32 64 51 don’t need to
swap
10 34 21 32 64 51
10 21 34 32 64 51
underlined: being considered
italic: sorted 32
 Bubble Sort - Example (2)
round
10 21 34 32 64 51
3 10 21 34 32 51 64 don’t need
to swap
10 21 34 32 51 64
10 21 32 34 51 64 don’t need
to swap
4 10 21 32 34 51 64 don’t need
to swap
10 21 32 34 51 64 don’t need
to swap
5 10 21 32 34 51 64

underlined: being considered

italic: sorted 33
 Bubble Sort Algorithm
the smallest will be moved to a[i]
for i = 0 to n-2 do
for j = n-1 downto i+1 do
if (a[j] < a[j-1])
swap a[j] & a[j-1] start from a[n-1],
check up to a[i+1]

34 10 64 51 32 21
i =0 j=5
j=4
j=3
j=2
j=1
i =1 j=5
j=4
j=3
j=2 34
 Algorithm Analysis
The algorithm consists of a nested for-loop.
for i = 0 to n-2 do
for j = n-1 downto i+1 do
if (a[j] < a[j-1])
swap a[j] & a[j-1]

Total number of comparisons

= (n-1) + (n-2) + … + 1 i # of comparisons
= n(n-1)/2 in inner loop
O(n2)-time 0 n-1
1 n-2
… ...
n-2 1 35
 Sorting
Input: a sequence of n numbers a0, a1, …, an-1
Output: arrange the n numbers into ascending order,
i.e., from smallest to largest

We have learnt these sorting algorithms:

selection sort, bubble sort
Next: insertion sort (optional, self-study)

36
 Insertion Sort (optional, self-study)
look at elements one by one
build up sorted list by inserting the element at the
correct location

37
optional
 Example
➢ sort (34, 8, 64, 51, 32, 21) in ascending order
Sorted part Unsorted part int moved
34 8 64 51 32 21
34 8 6451 32 21 -
8 34 64 51 32 21 34
8 34 64 51 32 21 -
8 34 51 64 32 21 64
8 32 34 51 64 21 34, 51, 64
8 21 32 34 51 64 32, 34, 51, 64

38optional
 Insertion Sort Algorithm
for i = 1 to n-1 do
using linear search to find
begin the correct position for key
key = a[i]
pos = 0
while (a[pos] < key) && (pos < i) do
pos = pos + 1
shift a[pos], …, a[i-1] to the right
a[pos] = key
end
i.e., move a[i-1] to a[i], a[i-2]
finally, place key (the to a[i-1], …, a[pos] to
original a[i]) in a[pos] a[pos+1]

39
optional
 Algorithm for
Analysis
i = 1 to n-1 do
begin
Worst case input key = a[i]
– input is sorted in descending pos = 0
while (a[pos] < key) && (pos < i) do
order pos = pos + 1
Then, for a[i] shift a[pos], …, a[i-1] to the right
a[pos] = key
– finding the position takes i end
comparisons
i # of comparisons
in the while loop
total number of comparisons 1 1
= 1 + 2 + … + n-1 2 2
= (n-1)n/2 O(n2)-time
… ...
n-1 n-1
40
optional
 Selection, Bubble, Insertion Sort
All three algorithms have time complexity O(n2) in
the worst case.
Are there any more efficient sorting algorithms? YES,
we will learn them later.
What is the time complexity of the fastest
comparison-based sorting algorithm?
O(n log n)

41
Some exponential time algorithms –
Traveling Salesman Problem,
Knapsack Problem …
 Traveling Salesman Problem
Input: There are n cities.
Output: Find the shortest tour from a particular city
that visit each city exactly once before returning to
the city where it started.

This is known as
Hamiltonian circuit

43
 2
Example
a b
To find a Hamiltonian
7 8 3 circuit from a to a
5

c d
1
Tour Length
a -> b -> c -> d -> a 2 + 8 + 1 + 7 = 18
a -> b -> d -> c -> a 2 + 3 + 1 + 5 = 11
a -> c -> b -> d -> a 5 + 8 + 3 + 7 = 23
a -> c -> d -> b -> a 5 + 1 + 3 + 2 = 11
a -> d -> b -> c -> a 7 + 3 + 8 + 5 = 23
a -> d -> c -> b -> a 7 + 1 + 8 + 2 = 18
44
 Idea and Analysis
A Hamiltonian circuit can be represented by a
sequence of n+1 cities v1, v2, …, vn, v1, where the
first and the last are the same, and all the others
are distinct.
Exhaustive search approach: Find all tours in this
form, compute the tour length and find the
shortest among them.

How many possible tours to consider?

(n-1)! = (n-1)(n-2)…1
N.B.: (n-1)! is exponential in terms of n
45
 Knapsack Problem
Input: Given n items with weights w1, w2, …, wn and
values v1, v2, …, vn, and a knapsack with capacity W.
Output: Find the most valuable subset of items that
can fit into the knapsack.
Application: A transport plane is to deliver the most
valuable set of items to a remote location without
exceeding its capacity.

46
 Example
capacity = 10

w=7 w=3 w=4 w=5

v = 42 v = 12 v = 40 v = 25
item 1 item 2 item 3 item 4 knapsack
total total total total
subset weight value subset weight value
 0 0 {2,3} 7 52
{1} 7 42 {2,4} 8 37
{2} 3 12 {3,4} 9 65
{3} 4 40 {1,2,3} 14 N/A
{4} 5 25 {1,2,4} 15 N/A
{1,2} 10 54 {1,3,4} 16 N/A
{1,3} 11 N/A {2,3,4} 12 N/A
{1,4} 12 N/A {1,2,3,4} 19 N/A
47
 Idea and Analysis
Exhaustive search approach: Try every subset of the
set of n given items, compute total weight of each
subset and compute total value of those subsets
that do NOT exceed knapsack's capacity.

How many subsets to consider?

2n, why?

48
 Exercises (1)
Suppose you have forgotten a password with 5
characters. You only remember:
– the 5 characters are all distinct
– the 5 characters are B, D, M, P, Y
If you want to try all possible combinations, how
many of them in total?
What if the 5 characters can be any of the 26 upper
case alphabet?

49
 Exercises (2)
Suppose the password also contains 2 digits, i.e., 7
characters in total
– all characters are all distinct
– the 5 alphabets are B, D, M, P, Y
– the digit is either 0 or 1
How many combinations are there?

50
 Exercises (3)
What if the password is in the form adaaada?
– a means alphabet, d means digit
– all characters are all distinct
– the 5 alphabets are B, D, M, P, Y
– the digit is either 0 or 1
How many combinations are there?

51
 Learning outcomes
• Able to carry out simple asymptotic analysis of
algorithms
• Know some examples of polynomial time and
exponential time algorithms
• Able to apply searching/sorting algorithms and
derive their time complexities

52