Department of Computer Science, University of Jos

Algorithms and Complexity Analysis

An Introductory Course on the Design & Analysis of Algorithms

Lecture Handout

May, 2017
 DESIGN & ANALYSIS OF ALGORITHMS

An Introductory course on the Design and Analysis of Algorithms.

Department of Computer Science, University of Jos.

1.0 PREAMBLE
Informally, an algorithm is a sequence of unambiguous instructions for solving a
problem, i.e., for obtaining a required output for any legitimate input in a finite
amount of time.

Notion of algorithm

problem

algorithm

input “computer” output

Example of computational problem: sorting

 Statement of problem:
• Input: A sequence of n numbers <a1, a2, …, an>
• Output: A reordering of the input sequence <a´1, a´2, …, a´n> so
that a´i ≤ a´j whenever i < j

 Algorithms:
• Selection sort
• Insertion sort
• Heap sort
• Bubble sort
• (many others)

Page 1
Selection Sort

 Input: array a[1],…,a[n]

 Output: array a sorted in non-decreasing order
Algorithm:
for i=1 to n
swap a[i] with smallest of a[i],…a[n]

This will be treated in details later in the course.

Some Well-known Computational Problems

 Sorting
 Searching
 Shortest paths in a graph
 Traveling salesman problem
 Towers of Hanoi
 etc.

Basic Issues Related to Algorithms

 How to design algorithms

 Proving correctness
 Efficiency
o Theoretical analysis
o Empirical analysis
 Optimality

Page 2
Algorithm Design Strategies

 Divide and conquer

 Decrease and conquer
 Transform and conquer
 Space and time tradeoffs
 etc.

Complexity, time-space tradeoff

 How good is the algorithm? In terms of:

 Correctness
 Time efficiency
 Space efficiency
 Optimality
 The efficiency of an algorithm is measured by the Time and Space it
uses.
 The complexity of an algorithm is the function which gives the running
time and/or space in terms of the input size.
 Does “there exist” a better algorithm? (Optimality)

What is an algorithm?

Recipe, process, method, technique, procedure, routine,… with the

following requirements:
1. Finiteness: terminates after a finite number of steps
2. Definiteness: rigorously and unambiguously specified
3. Input (not always required, why?): valid inputs are clearly specified
4. Output: can be proved to produce the correct output given a
valid input
5. Effectiveness: steps are sufficiently simple and basic

Page 3
Why study algorithm?

 Theoretical importance
o the core of computer science
 Practical importance
o A practitioner’s toolkit of known algorithms
o Framework for designing and analyzing algorithms for new
problems

2.0 ANALYSIS OF ALGORITHMS

Theoretical analysis of time efficiency
Time efficiency is analyzed by determining the number of repetitions of
the basic operation as a function of input size

Basic operation: the operation that contributes most towards the running
time of the algorithm.

Page 4
Input size and basic operation examples

Empirical analysis of time efficiency

 Select a specific (typical) sample of inputs

 Use physical unit of time (e.g., milliseconds)

OR

 Count actual number of basic operations

 Analyze the empirical data

Best-case, Average-case, Worst-case

For some algorithms efficiency depends on type of input:

 Worst case: W(n) – maximum over inputs of size n
 Best case: B(n) – minimum over inputs of size n
 Average case: A(n) – “average” over inputs of size n
• Number of times the basic operation will be executed on typical
input
• NOT the average of worst and best case
• Expected number of basic operations repetitions considered as a
random variable under some assumption about the probability
distribution of all possible inputs of size n

Page 5
 Example on best, average, and worst case analyses: Sequential search
 Problem: Given a list of n elements and a search key K, find an element
equal to K, if any.
 Algorithm: Scan the list and compare its successive elements with K until
either a matching element is found (successful search) of the list is
exhausted (unsuccessful search)

Algorithm LinearSearch(DATA, N, ITEM, LOC)

//Here, DATA is a linear array with N elements, and ITEM is a given item of
//information. This algorithm finds the location LOC of ITEM in DATA, or
//sets LOC=0 if the search is unsuccessful.
1: Set DATA[N+1] = ITEM //Insert ITEM at the end of DATA
2: Set LOC=1 //initialize counter
3: //Search for ITEM
Repeat while DATA[LOC] <> ITEM
Set Loc = Loc + 1
//End Loop
4: If LOC = N+1 then Set LOC = 0
5: Exit

The complexity of the search algorithm is measured by the number f(n) of

comparisons required to find ITEM in DATA, where DATA contains n elements.
 Best case

The best case is when the item is located at the first location DATA[1], i.e.,

f(n) = 1

 Worst case

Clearly, the worst case occurs when one must search through the entire array
DATA, i.e., when ITEM does not appear in DATA. In this case, the algorithm
requires f(n) = n+1 comparisons. Thus, in the worst case, the running time is
proportional to n.

Page 6
  Average case

Here, we assume that ITEM does appear in DATA, and that it is equally likely to
appear at any position in the array. Accordingly, the number of comparisons
can be any of the numbers 1, 2, 3, 4, ………, n, and each number occurs with
probability p = 1/n. Then
1 1 1 1 1
f(n) = 1 x 𝑛 + 2 x 𝑛 + 3 x 𝑛+ ……… + (n-1) x 𝑛 + n x 𝑛
1
= (1 + 2 + 3 + ……… + n-1 + n) x 𝑛

Let Sn = (1 + 2 + 3 + ……… + n-1 + n).

1
==> f(n) = Sn x 𝑛 ................................................... (1)

suppose n is even: the sum of the first and the last numbers in Sn is n+1; the sum
of the second and the second-last elements is 2+n-1, = n+1; the sum of the third
𝑛
and the third-last elements is 3+n-2 = n+1, and so on. Now there are such
2
𝑛
pairs. Therefore, Sn = 2(n+1) ................................................... (2)

suppose n is odd: ==> Sn = Sn-1 + n ................................................... (3)

(where n-1 is even)
Substituting n-1 into equation (2),
𝑛−1 𝑛(𝑛−1)
we get Sn-1 = (n-1+1) = ................................................... (4)
2 2

substituting equation (4) into equation (3) yields

𝑛(𝑛−1) 𝑛
Sn = + n = 2(n+1) Ξ equation (2), where n is even. ………… (5)
2

𝑛
Substituting Sn = 2(n+1) into equation (1) therefore we get
𝑛 1 1
f(n) = 2(n+1) x 𝑛 = 2(n+1)

𝟏
Therefore f(n) = 𝟐(n+1)
Page 7
This agrees with our intuitive feeling that the average number of comparisons
needed to find the location of ITEM is approximately equal to half the number
of elements in the DATA list.

3.0 TYPES OF FORMULAS FOR BASIC OPERATION COUNT

 Exact formula
e.g., f(n) = n(n-1)/2
 Formula indicating order of growth with specific multiplicative constant
e.g., f(n) ≈ 0.5 n2
 Formula indicating order of growth with unknown multiplicative constant
e.g., f(n) ≈ cn2

Asymptotic growth rate (Order of growth)

Suppose M is an algorithm, and suppose n is the size of the input data. Clearly
the complexity f(n) of M increases as n increases. It is usually the rate of
increase of f(n) that we want to examine. This is usually done by comparing f(n)
with some standard functions, such as log2n, n, nlog2n, n2, n3, 2n.

The rates of growth for these standard functions are represented in the figure
below, which gives their approximate values for certain values of n. Observe
that the functions are listed in the order of their rates of growth: the logarithmic
function log2n, grows most slowly, the factorial function n! grows most rapidly,
and the polynomial functions nc grow according to the exponent c.

Page 8
One way to compare the function f(n) with these standard functions is to use
the functional O notation, called the “big-oh”, defined as follows:

Suppose f(n) and g(n) are functions defined on the positive integers with the
property that f(n) is bounded by some multiple of g(n) for almost all n. That is,
suppose there exist a positive integer n0 and a positive number M such that, for
all n > n0, we have

|f(n)| ≤ M |g(n)|

Then we may write

f(n) = O(g(n))

which is read “f(n) is of order g(n)”

Generally, this is a way of comparing functions that ignores constant factors

and small input sizes:

 O(g(n)): class of functions f(n) that grow no faster than g(n)

 Θ(g(n)): class of functions f(n) that grow at same rate as g(n)
  (g(n)): class of functions f(n) that grow at least as fast as g(n)

Page 9
See figures below:

Page 10
Basic Asymptotic Efficiency classes

O(1) Constant
O(log log n) Double logarithmic
O(log n) Logarithmic
O(nc) 0 < c < 1 Fractional power
O(n) Linear
O(n log n) = O(log n!) Near linear, linearithmic, loglinear, or quasilinear
O(n2) Quadratic
O(n3) Cubic
O(nc) c > 1 Polynomial or algebraic
O(2n) Exponential
O(cn), c > 1 Exponential
O(n!) Factorial

Time efficiency of nonrecursive algorithms

Steps in mathematical analysis of nonrecursive algorithms:

 Decide on parameter n indicating input size

 Identify algorithm’s basic operation
 Determine worst, average, and best case for input of size n
 Set up summation for C(n) reflecting algorithm’s loop structure
 Simplify summation using standard formulas

Page 11
Examples on analysis of algorithm using order of growth: Matrix multiplication,
Selection sort, Insertion sort, Bubble sort.

Example 1: Matrix multiplication

Exercise: find the complexity of the matrix multiplication algorithm.

Example 2: Selection sort

The method is based on the following principles:

1. Select the item with the least key.
2. Exchange it with the first item Aj.

Then repeat these operations with the remaining n-1 items, then with n-2 items,
…, until only one item – the largest – is left.

Algorithm (Observe this very well):

For i := 1 To n-1 Do
Begin
“Assign the index of the least item of A(i) . . . A(n) to k”
“Exchange Ai and Ak
End.

Page 12
This method is the opposite of Insertion Sort. Insertion Sort considers in each
step only the one next item of the source sequence and all items of the
destination array to find the insertion point. Selection Sort considers all items of
the source array to find the one with the least key and to deposit it as the one
next item of the destination sequence.

44 55 12 42 94 18 06 67
06 55 12 42 94 18 44 67
06 12 55 42 94 18 44 67
06 12 18 42 94 55 44 67
06 12 18 42 94 55 44 67
06 12 18 42 44 55 94 67
06 12 18 42 44 55 94 67
06 12 18 42 44 55 67 94

Procedure SelectionSort(A, N);

Begin
For i := 1 To n-1 Do
Begin
k := i;
x := A(i);
For j := i+1 To n Do
If A(j ) < x Then
Begin
k := j;
x := A(j);
End;
A(k) := A(i);
A(i) := x;
End;
End.

Complexity of Selection Sort Algorithm

First, note that the number f(n) of comparisons in the selection sort algorithm is
independent of the original order of the elements, so the average case and
the worst case are the same. Observe that there are n-1 comparisons during

Page 13
pass 1 to find the smallest element, there are n-2 comparisons during pass 2 to
find the second smallest element, and so on. Accordingly,

𝑛(𝑛−1)
f(n) = (n-1) + (n-2) + ………. + 2 + 1 = = O(n2)
2

𝑛(𝑛−1)
Therefore, Average case = Worst case = = O(n2)
2

Example 3: Insertion sort

Algorithm (Observe this very well):

For i := 2 To n Do
Begin
X := A(i)
“Insert x at the appropriate place in Ai and Ak; k ≤ i
End.

In the process of actually finding the appropriate place, it is convenient

to alternate between comparisons and moves, i.e., to let x “shift down” by
comparing x with the next item Aj to the right and proceed to the left. Note
that there are two distinct conditions that may cause the termination of the
“shift down” process:

1. An item Aj is found with a key less than the key x

2. The left end of the destination sequence is reached. Because of this, we
have to pass a sentinel Ao. We choose Ao such that it is less than any
possible number. A good choice is Ao = -

Page 14
Pass A(1) A(2) A(3) A(4) A(5) A(6) A(7) A(8)
I=1 44 55 12 42 94 18 06 67
I=2 44 55 12 42 94 18 06 67
I=3 12 44 55 42 94 18 06 67
I=4 12 42 44 55 94 18 06 67
I=5 12 42 44 55 94 18 06 67
I=6 12 18 42 44 55 94 06 67
I=7 06 12 18 42 44 55 94 67
I=8 06 12 18 42 44 55 67 94

Procedure InsertionSort(A, N);

Begin
A(0) := -;
For i := 2 To n Do
Begin
x := A(i);
j := i – 1;

While x < A(j) Do

Begin
A(j + 1) := A(j);
j := j – 1;
End;
A(j + 1) := x;
End;
End.

Complexity of Insertion Sort Algorithm

The number f(n) of comparisons in the insertion sort algorithm can be easily
computed. First of all, the worst case occurs when the array A is in reverse
order and the inner loop must use the maximum number i-1 of comparisons.
Hence

f(n) = 1 + 2 + 3 + ……… + (n-1);

𝑛
but from equations (2) and (5), Sn = (1 + 2 + 3 + ……… + n) = 2(n+1).

Page 15
 𝑛(𝑛−1)
Therefore f(n) = = O(n2)
2

𝑛(𝑛−1)
Worst case = = O(n2)
2

Furthermore, one can show that on the average, there will be approximately
(i-1)/2 (half of worst case) comparisons on the inner loop. Accordingly, for the
average case,

1 2 3 𝑛−1 𝑛(𝑛−1)
f(n) = 2 + 2 + 2 + ……… + = = O(n2)
2 4

𝑛(𝑛−1)
Average case = = O(n2)
2

Example 4: Bubble Sort

This method is based on the principle of comparing and exchanging pairs of
adjacency items until all items are sorted.

In this method, we make repeated passes over the array, each time shifting the
least item of the remaining set to the left end of the array. If, for a change, we
view the array to be in a vertical position instead of horizontal position – and
with the help of some imagination the items as bubbles in a water tank with
“weights” according to their keys, - then each pass over the array results in the
ascension of a bubble to its appropriate level of weight.

c=1 c=2 c=3 c=4 c=5 c=6 c=7 c=8

44 06 06 06 06 06 06 06
55 44 12 12 12 12 12 12
12 55 44 18 18 18 18 18
42 12 55 44 42 42 42 42
94 42 18 55 44 44 44 44
18 94 42 42 55 55 55 55
06 18 94 67 67 67 67 67
67 67 67 94 94 94 94 94

Page 16
 Procedure BubbleSirt(A, n);
Begin
For i := 2 To n Do
Begin
For j := n DownTo i Do
If A(j – 1) > A(j) Then
Begin
x := A(j – 1);
A(j – 1) := A(j);
A(j) := x;
End;
End;
End.

Complexity of the Bubble sort Algorithm

In the bubble sort algorithm, there are n-1 comparisons during the first pass,
which places the largest element in the last position; there are n-2 comparisons
in the second step, which places the second largest element in the next-to-last
position, and so on.

𝑛(𝑛−1)
Therefore f(n) = (n-1) + (n-2) + ………. + 2 + 1 = = O(n2)
2

4.0 ALGORITHM DESIGN STRATEGIES

Divide and Conquer Strategies

The most well-known algorithm design strategy:

1. Divide instance of problem into two or more smaller instances

2. Solve smaller instances recursively

3. Obtain solution to original (larger) instance by combining these solutions

Page 17
Divide-and-conquer technique:

A problem of size n

Subproblem Subproblem
1 2
of size n/2 of size n/2

A solution to A solution to
subproblem 1 subproblem 2

A solution to
the original problem

Examples on Divide and Conquer Algorithms:

 Binary search
 Sorting – quicksort, heap/heapsort
 Tree traversals

Binary Search

While performing a binary search, the search space is halved in every iteration.
The number of steps is Logarithmic, but you must also consider the complexity
of the function used to move low and high pointers in each iteration.

Consider the binary search algorithm below. [give a suitable example in class
before the complexity analysis]

Page 18
Merge Sort

Give example and let the students perform the complexity analysis

Page 19
Quicksort Algorithm

Algorithm and Complexity Analysis: pages 173 – 176 of Schaum’s Outline Series
of Theory and Problems on Data Structures, by SEYMOUR Lipschutz (2002).

Heap/Heapsort Algorithm

Algorithm and Complexity Analysis: pages 243 – 249 of Schaum’s Outline Series
of Theory and Problems on Data Structures, by SEYMOUR Lipschutz (2002).

Binary Search Algorithm

Algorithm and Complexity Analysis: pages 78 – 81 of Schaum’s Outline Series of

Theory and Problems on Data Structures, by SEYMOUR Lipschutz (2002).

Decrease and Conquer Strategies

1. Reduce problem instance to smaller instance of the same problem and

extend solution

2. Solve smaller instance

3. Extend solution of smaller instance to obtain solution to original problem

Also referred to as inductive or incremental approach

Examples: Binary Search Tree (BST) and Insertion Sort

Transform and Conquer Strategies

Solve problem by transforming into:

 a more convenient instance of the same problem (instance

implification)
o presorting
o Gaussian elimination
 a different representation of the same instance (representation change)
Page 20
 o balanced search trees
o heaps and heapsort
o polynomial evaluation by Horner’s rule
o Fast Fourier Transform
 a different problem altogether (problem reduction)
o reductions to graph problems
o linear programming

Instance simplification - Presorting

Solve instance of problem by transforming into another simpler/easier instance

of the same problem

Presorting:

Many problems involving lists are easier when list is sorted.

 searching
 computing the median (selection problem)
 computing the mode
 finding repeated elements

Space-time tradeoffs

For many problems some extra space really pays off:

 extra space in tables (breathing room?) – hashing

Treat Hashing and collision resolution techniques.

Page 21