Introduction to Algorithms

Chapter 02

Mirza Mohammad Lutfe Elahi

CSE 373 Design and Analysis of Algorithms ECE@NSU

Outline
Algorithm

Random Access Model

Insertion Sort – Design and Analysis

Merge Sort – Design and Analysis

CSE 373 Design and Analysis of Algorithms ECE@NSU

 3

Algorithm
An algorithm-
well defined computational procedure
takes set of values as input
produces set of values as output
in a finite amount of time
A sequence of computational steps that transform the input into
output

CSE 373 Design and Analysis of Algorithms ECE@NSU

Algorithm - example
Human Genome Project
identifying genes in human DNA
determining chemical base pairs
Internet
finding good routes
search engines to quickly find pages
E-commerce
Manufacturing
Medical Science
…. Many more

CSE 373 Design and Analysis of Algorithms ECE@NSU

 5

Insertion Sort
Input: A sequence of n numbers
Output: A permutation (reordering) of the
input sequence such that .

The numbers to be sorted are known as keys.

CSE 373 Design and Analysis of Algorithms ECE@NSU

Insertion Sort - Algorithm

CSE 373 Design and Analysis of Algorithms ECE@NSU

 7

Random Access Machine

Predicting the resources that algorithm requires
Such as memory, communication bandwidth, or computer hardware
are of primary concern.
It is computational time that we want to measure.
Random Access Machine (RAM)
Based on one processor model of computation.
Instructions are executed one after another, with no concurrent operations.
arithmetic, data movement, and control - each instruction takes a constant
amount of time.
Data type - integer and float.
Do not model the memory hierarchy.
Analyzing even a simple algorithm in the RAM model can be a
challenge.
CSE 373 Design and Analysis of Algorithms ECE@NSU

Insertion Sort – Running Time

n = A.length and 𝑡! denote the number of times the while
loop test is executed for the value of j

CSE 373 Design and Analysis of Algorithms ECE@NSU

 9

Insertion Sort – Analysis

The running time of the algorithm is the sum of running
times for each statement executed;
A statement that takes 𝑐" steps to execute and executes n
times will contribute 𝑐" 𝑛 to the total running time.
For an input of n values, T(n) - the running time of
insertion sort:

CSE 373 Design and Analysis of Algorithms ECE@NSU

10

Insertion Sort – Best Case

When the array is already sorted.
For each j = 2, 3, ..., n, A[i] ≤ key and
𝑡! = 1 for j = 2, 3, ..., n
Running time

This running time can be expressed as a linear function of

n : an + b where a and b are constants that depend on the
statement costs 𝑐" .
CSE 373 Design and Analysis of Algorithms ECE@NSU
 11

Insertion Sort – Worst Case

When the array is in reserve sorted order.
Must compare each element A[j] with each element in the entire
sorted subarray A[1..j − 1] and 𝑡! = j for j = 2, 3, ..., n

Running time

This running time can be expressed as a quadratic function of n :

𝑎𝑛" + 𝑏𝑛 + 𝑐 where a, b and c are constants that depend on the
statement costs 𝑐# .
CSE 373 Design and Analysis of Algorithms ECE@NSU

12
Insertion Sort – Worst Case and Average Case
Analysis
Worst-case
Gives an upper bound on the running time for any input. Provides a guarantee
that the algorithm will never take any longer.
For some algorithms, the worst case occurs fairly often. In some applications,
searches for absent information may be frequent.

Average-case
Often roughly as bad as the worst case.
Suppose that randomly choose n numbers and apply insertion sort.
On average, half the elements in A[1..j − 1] are less than A[j], and half the
elements are greater.
Check half of the subarray A[1..j − 1], and so 𝑡! is about j/2.
Running time turns out to be a quadratic function of the input size, just like the
worst-case running time.
CSE 373 Design and Analysis of Algorithms ECE@NSU
 13

Insertion Sort – Order of Growth

We ignored the actual cost of each statement, using the constants 𝑐#
to represent these costs.
We expressed the worst-case running time as 𝑎𝑛" + 𝑏𝑛 + 𝑐 for some
constants a, b, and c that depend on the statement costs 𝑐# .
Real interest is on the rate of growth or order of growth.
Consider only the leading term of a formula (e.g., an2),
Lower-order terms are relatively insignificant for large values of n.
Ignore the leading term’s constant coefficient.
Worst-case running time of insertion sort is Ɵ(n2).
We usually consider one algorithm to be more efficient than another
if its worst-case running time has a lower order of growth.

CSE 373 Design and Analysis of Algorithms ECE@NSU

14

Divide & Conquer

Many useful algorithms are recursive in structure.
To solve a given problem, they call themselves recursively
one or more times to deal with closely related subproblems.
These algorithms typically follow a divide-and-conquer
approach.
Divide the problem into one or more subproblems that are smaller
instances of the same problem.
Conquer the subproblems by solving them recursively.
Combine the subproblem solutions to form a solution to the
original problem.

CSE 373 Design and Analysis of Algorithms ECE@NSU

 15

Merge Sort – Divide & Conquer

Divide the subarray A[p : r] to be sorted into two adjacent
subarrys, each of half the size. Compute the midpoint q of
A[p : r] and divide A[p : r] into subarrays A[p : q] and
A[q + 1 : r]

Conquer by sorting each of the two subarrays A[p : q] and

A[q + 1 : r] recursively using merge sort.

Combine by merging the two sorted subarrays A[p : q] and

A[q + 1 : r] back into A[p : r], producing the sorted answer.

CSE 373 Design and Analysis of Algorithms ECE@NSU

16

Merge Sort – Divide & Conquer

The recursion bottoms out when the sequence to be sorted has
length 1.
Every sequence of length 1 is already in sorted order.
The key operation of the merge sort algorithm is the merging of
two sorted sequences in the combine step.
It merges by calling an auxiliary procedure Merge(A, p, q, r),
where A is an array and p, q, and r are indices into the array
such that p ≤ q < r.
The procedure assumes that the subarrays A[p : q] and
A[q + 1 : r] are in sorted order.
It merges them to form a single sorted subarray that replaces the
current subarray A[p : r].
CSE 373 Design and Analysis of Algorithms ECE@NSU
 17

Merge Sort – Combine Algorithm

CSE 373 Design and Analysis of Algorithms ECE@NSU

18

Merge Sort – Operations

Figure: The operation of merge (A, 9, 12, 16)

CSE 373 Design and Analysis of Algorithms ECE@NSU
 19

Merge Sort - Algorithm

CSE 373 Design and Analysis of Algorithms ECE@NSU

20

Merge Sort – Merge Operations

Figure: Operation of Merge-Sort(A, 1, A.length) on the array A = 12, 3, 7, 9, 14, 6, 11, 2

CSE 373 Design and Analysis of Algorithms ECE@NSU
 21

Divide & Conquer - Analysis

Running time of an algorithm contains a recursive call to itself, can be
described by a recurrence equation or recurrence.
It describes the overall running time on a problem of size n in terms of
the running time on smaller inputs.
Suppose that our division of the problem yields a subproblems, each of
which is 1/b the size of the original.
It takes time T(n/b) to solve one subproblem of size n/b, and so it takes
time aT(n/b) to solve a of them.
D(n) time to divide the problem into subproblems.
C(n) time to combine the solutions to the subproblems into the solution
to the original problem.

CSE 373 Design and Analysis of Algorithms ECE@NSU

22

Merge Sort - Analysis

Assume the original problem size is a power of 2.
Merge sort on just one element takes constant time.
When we have n > 1 elements, we break down the running time as
follows:
Divide: The divide step just computes the middle of the subarray, which takes
constant time. Thus, D(n) = Ɵ(1).
Conquer: We recursively solve two subproblems, each of size n/2, which
contributes 2T(n/2) to the running time.
Combine: We have already noted that the MERGE procedure on an n-element
subarray takes time Ɵ(n), and so C(n) = Ɵ(n).
Worst-case running time T(n) of merge sort:

CSE 373 Design and Analysis of Algorithms ECE@NSU

 23

Merge Sort - Analysis

Let us rewrite recurrence as

Constant c represents the time required to solve problems

of size 1 as well as the time per array element of the divide
and combine steps.

CSE 373 Design and Analysis of Algorithms ECE@NSU

24

Merge Sort - Analysis

Figure: Merge Sort Analysis using Recursion Tree Method

CSE 373 Design and Analysis of Algorithms ECE@NSU

 25

Merge Sort - Analysis

Figure: Merge Sort Analysis using Recursion Tree Method

CSE 373 Design and Analysis of Algorithms ECE@NSU

26

Merge Sort - Analysis

To compute total cost, simply add up the costs of all the
levels.

The recursion tree has lg n + 1 levels, each costing cn, for a

total cost of cn(lg n + 1) = cn lg n + cn.

Ignoring the low-order term and the constant c gives the

desired result of Ɵ(n lg n).

CSE 373 Design and Analysis of Algorithms ECE@NSU