CSC 311 DESIGN AND ANALYSIS OF ALGORITHMS

IMPORTANCE OF LEARNING DESIGN AND ANALYSIS OF ALGORITHMS

➢ Augment problem solving skills- algorithm design and their

implementations
➢ Strengthen formal thinking- proofs; analysis
COMPETENCIES THAT YOU WILL DEVELOP
You will be able to:
● Strengthen the recal of the basic data structures;
● Describe core algorithms;
● Apply algorithm design paradigms: divide & conquer, greedy
algorithms, dynamic programming;
● Analyze the correctness and runtime performance of a
given algorithm;
● Apply techniques in solving practical problems;
● Describe the inherent complexity (lower bounds &
intractability) of some problems.
Weeks-1,2,3 1
CSC 311 DESIGN AND ANALYSIS OF ALGORITHMS

REST OF THE SESSION SCHEDULE

Introduction – why ADA?
Algorithm
The Input and Outputs
Experimental values on some input functions
Analysis of Algorithm – time and space complexities
Interesting measures- best cases, worst case, average case
Notations
Growth of functions

Elementary Data Structures

Arrays
Lists
Queues
Priority Queues
Binary Trees
Heaps
Hash Tables

Weeks-1,2,3 2
Introduction – why Analysis and Design of Algorithms?

Defining Analysis and Design of Algorithms

It is the determination of the amount of time, storage and/or other resources
necessary to execute the algorithm ( also called computational complexity of
algorithms).

Determine: function that relates the length of an algorithm's input to the number
of steps it takes (its time complexity)

OR
Determine: the function that relates its input to the number of storage locations it
uses (its space complexity).

Efficient Algorithm
One whose function's values are small, or grow slowly compared to a growth in
the size of the input.

Weeks-1,2,3 3
Introduction – why Analysis and Design of Algorithms?

History
The term "Analysis of algorithms" was coined by Donald Knuth.

Importance of Analysis of algorithms

➔ Algorithm analysis is an important part of a broader computational complexity
theory.

➔ Computational complexity theory provides theoretical estimates for the

resources needed by any algorithm which solves a given computational
problem.

➔ These estimates provide an insight into reasonable directions of search for

efficient algorithms.

Use of Analysis of Algorithms

➢ Determine characteristics of the algorithm.
➢ Evaluate suitability of an algorithm to an application.
➢ Compare algorithm with other algorithms for the same application.
● Helps in understanding the algorithm and enables its improvement.

Weeks-1,2,3 4
ALGORITHM- WHAT IS IT?

Algorithm: is a step-by-step procedure for solving a problem in a finite amount of

time with finite amout of effort.

General steps in exhaustive analysis of algorithms

1)Implement the algorithm completely.

2)Determine the time required for each basic operation.
3)Identify unknown quantities that can be used to describe the frequency of
execution of the basic operations.
4)Develop a realistic model for the input to the program.
5)Analyze the unknown quantities, assuming the modelled input.
6)Calculate the total running time by multiplying the time by the frequency for
each operation, then adding all the products.

● Classical algorithm analysis on early computers could result in exact

predictions of running times.

● Modern systems and algorithms are much more complex, but modern analyses
are informed by the idea that exact analysis of this sort could be performed in
principle.

Weeks-1,2,3 5
The Input and Outputs
Isolated values
Single or countable isolated values for input
X, y, z : 3 input values; can be varied to any number of finite inputs
(input size usually associated by some constant say C or c)
Arrays- one dimensional
Index 0 1 2 3 4 5 6 7 8 9
-1 17 -30 50 6 2 28 11 12 -6

Input size may vary but is assoicated with size of array denoted as n

Arrays – two dimensional

Index 0 1 2 3 4 5 6 7 8 9

0 -1 17 36 50 6 2 28 11 12 -6
33 4 2 6 8 99 71 -8 11 2
1

Weeks-1,2,3 6
The Input and Outputs
Isolated values
input size usually associated by some constant say C or c
Input size = f(n) = C, where C is a constant, n Is a non negative
integer

Arrays- one dimensional

Input size may vary but is assoicated with size of array denoted as n

Input size = f(n), where n varies, n is a non negative integer

Arrays – two dimensional (can be translated to one dimensional array)

Input size = f(2*n) = g(n), where n varies, n is a non negative integer

Input size = f(k*n) = h(n), where n varies, n is a non negative integer for
k-dimensional array
Weeks-1,2,3 7
The Input and Outputs
SOME TYPICAL INPUT FUNCTIONS
T(n) = n, n Is a natural number

T(n) = log2n, n Is a natural number

T(n) = nlog2n, n Is a natural number

T(n) = n2, n Is a natural number

T(n) = n3, n Is a natural number

T(n) = 2n, n Is a natural number

T(n) = n!, n Is a natural number

Weeks-1,2,3 8
CSC 311 DESIGN AND ANALYSIS OF ALGORITHMS

Numeric values of some possible input functions

Weeks-1,2,3 9
SOME TYPICAL SEQUENCES
ARITHMETIC SEQUENCE

Weeks-1,2,3 10
SOME TYPICAL SEQUENCES

Weeks-1,2,3 11
LOGARITHMIC RELATED FORMULAE

Weeks-1,2,3 12
RECURSION
Recursive function
A function defined in terms of itself via self-referential
expressions.
The function will continue to call itself and repeat its
behavior until some condition is met to return a result.
Example (python):
def fact(n): #recursive function to calculate factorial
""" Function to find factorial """ # another recursive funtion
def printRev( n ):
if n == 1:
if n > 0 :
return 1 print( n )
printReverse( n-1 )
else:
return (n * fact(n-1))
print ("3! = ",fact(3))

Weeks-1,2,3 13
RECURRENCES
Recurrence relation
Is an equation that recursively defines a sequence or
multidimensional array of values, once one or more initial
terms are given;
Each further term of the sequence or array is defined as a
function of the preceding terms.
Examples

is a function, where X is a set to which the elements of a

sequence must belong. For any u0 ∈ X this defines a
unique sequence with u0 as its first element, called the
initial value.

Weeks-1,2,3 14
RECURRENCES
Recurrence relations
Factorial: defined by the recurrence relation
n ! = n ( n − 1 ) ! for n > 0 , and the initial condition 0 ! = 1
Logistic map: An example of a recurrence relation is the
x n+1 = r x n ( 1 − x n ) , with a given constant r; given the initial
term x 0 each subsequent term is determined by this relation.

Fibonacci numbers: a type of a homogeneous linear recurrence

relation with constant coefficients, see below.
F n = F n-1 + F n-2 with initial conditions (seed values) F 0 = 0,

F 1 = 1.

Solving a recurrence relation means obtaining a : a non-recursive

function of n.

Weeks-1,2,3 15
ANALYSIS OF ALGORITHM – TIME AND SPACE COMPLEXITIES

Time complexity of an algorithm (T(n))

- the amount of time taken by an algorithm to run as a
function of the length(size) of the input.

Space complexity of an algorithm

- the amount of space or memory taken by an algorithm to
run as a function of the length (size) of the input.

Example: Search for a value x in an array A

for i← 1 to length of A
if A[i] is equal to x
return TRUE
return FALSE
Time Complexity: varies from c to c+size of A, c=some other
operations; depends on where x occurs in A; op-time=const.
Space Complexity: size of A + number of temporary variables
Most attention: on time complexity; that will be our focus
Weeks-1,2,3 16
ANALYSIS OF ALGORITHM – TIME COMPLEXITY
INTERESTING MEASURES OF TIME COMPLEXITY
Best case- a function; gives the minimum number of steps taken on any
instance of size n.

Worst case- a function; gives the maximum number of steps taken on any
instance of size n.

Average case- is the amount of some computational resource (typically time)

used by the algorithm, averaged over all possible inputs.The analysis of such
algorithms leads to the related notion of an expected complexity that is based on
probabilities.

Weeks-1,2,3 17
NOTATIONS
ORDER OF GROWTH/ GROWTH OF FUNCTIONS
-how the time of execution depends on the length of the input. Consider
the length of input increasing to large values.

-Ignore the lower order terms, since the lower order terms are relatively
insignificant for large input.
Different notations are used to describe the limiting behavior of a
function, that is as the input size increases and becomes very large.

The O- notations are used to

describe the growth of
functions

Weeks-1,2,3 18
NOTATION - GROWTH OF FUNCTIONS

T(n) = O(g(n)), compute the

estimate of the function
g(n) as n approaches
infinity to find the actual
values dominating terms
Example:
g(n) = 500+ 10n2
As n becomes large only
10n2 counts so T(n) = O(n2)

Weeks-1,2,3 19
NOTATION - GROWTH OF FUNCTIONS

T(n) = Ω(g(n)), compute

the estimate of the
function g(n) as n
approaches infinity to find
the actual values
dominating terms
Example:
g(n) = 500+n+ 10n2
As n becomes large only n
is lower bound so
Weeks-1,2,3 20
T(n) = Ω(n)
NOTATION - GROWTH OF FUNCTIONS

T(n) = Θ(g(n)), find a

function g(n) that lies
between some two
functions as n approaches
infinity
Example: g(n) =2n+3
Check k1*n<2n+3<k2*n
T(n) = Θ(n)

Weeks-1,2,3 21
 NOTATION - GROWTH OF FUNCTIONS
Exercise: Find about the small O, small omega and small theta.

Weeks-1,2,3 22
 NOTATION - GROWTH OF FUNCTIONS
Exercise: Use the table below to discuss your preferred
algotithms for searching for sorting when the input size is
small and when the input size is very large.

Weeks-1,2,3 23
ELEMENTARY DATA STRUCTURES: ARRAY ABSTRACT DATA
TYPES

➔ Arrays store and retrieve items using an integer index.

➔ An item is stored in a given index and can be retrieved at a later

time by specifying the same index.

Weeks-1,2,3 24
ELEMENTARY DATA STRUCTURES: ARRAY ABSTRACT DATA
TYPES

def sum_elements():
➔ my_array = [None,] * 3 # declaring and using a list in this way isn't Pythonic

➔ my_array[0] = 1 # instead you'd make an empty list and use the append method
➔ my_array[1] = 3 # but if we did that here, then we wouldn't be showing off its
➔ my_array[2] = 5 # array functionality very much
➔ sum = 0
➔ for i in range(0, len(my_array)):
➔ sum += my_array[i]
➔ print(sum)
➔ Operations: insert element; delete element; asign element to a
cell; read element from a cell; read values in all cells; display
values of the whole array
➔ Caution: may have fixed size;

Weeks-1,2,3 25
ELEMENTARY DATA STRUCTURES: LIST ABSTRACT DATA TYPES

Linked Lists
A general purpose structure that can be used for linear storage;
Compared to array it requires a smaller memory allocation and no
element shifts for insertions and deletions.

It has no constant time direct element access like in array. So it is not

suitable for every data storage problem.

Types of linked lists: The singly linked list; circularly linked, the doubly
linked, and the circularly doubly linked lists.

Weeks-1,2,3 26
ELEMENTARY DATA STRUCTURES: LIST ABSTRACT DATA TYPES

Linked Lists and set ups

class ListNode :
def __init__( self, data ) :
self.data = data

a = ListNode( 11 )
b = ListNode( 52 )
c = ListNode( 18 )

class ListNode :
def __init__( self, data ) :
self.data = data
self.next = None
a.next = b
b.next = c
a.next = b
print( a.data )
print( a.next.data )
print( a.next.next.data )

Weeks-1,2,3 27
ELEMENTARY DATA STRUCTURES: LIST ABSTRACT DATA TYPES

Singly Linked
➔ a linked list;

➔ each node contains a single link field;

➔ allows for a complete traversal from a distinctive first node to the last.

Operations: traverse a list; display elements; add a node; delete a

node;search for a node; inserting element;
def traversal( head ): def unorderedSearch( head, target ):
curNode = head curNode = head
while curNode is not None : while curNode is not None and
print curNode.data curNode.data != target :
curNode= curNode.next
curNode = curNode.next return curNode is not None

Weeks-1,2,3 28
ELEMENTARY DATA STRUCTURES: QUEUES
A queue:
A specialized list with a limited number of operations;
Items can only be added to one end and removed from the other.
A queue is also known as a first-in, first-out (FIFO) list.

Served Enter

Operations
Queue(): Creates a new empty queue, which is a queue containing no items.
isEmpty(): Returns a boolean value indicating whether the queue is empty.
length (): Returns the number of items currently in the queue.
enqueue( item ): Adds the given item to the back of the queue.
dequeue(): Removes and returns the front item from the queue. An item can-
not be dequeued from an empty queue.

Weeks-1,2,3 29
ELEMENTARY DATA STRUCTURES: QUEUES
# Implementation of the Queue ADT using a Python list.
class Queue : # Creates an empty queue.
def __init__( self ):
self._qList = list()

def isEmpty( self ): # Returns True if the queue is empty.

return len( self ) == 0

def __len__( self ):# Returns the number of items in the queue.
return len( self._qList )

def enqueue( self, item ):# Adds the given item to the queue.
self._qList.append( item )

def dequeue( self ):# Removes and returns the first item in the queue.
assert not self.isEmpty(), "Cannot dequeue from an empty queue."
return self._qList.pop( 0 )

Weeks-1,2,3 30
ELEMENTARY DATA STRUCTURES: PRIORITY QUEUES

A priority queue:
is simply an extended version of the basic queue with the exception
that a priority p must be assigned to each item at the time it is enqueued.

Basic types of priority queues: bounded and unbounded.

The bounded priority queue: allows a small limited range of p priorities over the
interval of integers [0 . . . p).

The unbounded priority queue: no limit on the range of integer values that can be
used as priorities.

For same priority, the FIFO rule applies.

Weeks-1,2,3 31
ELEMENTARY DATA STRUCTURES: PRIORITY QUEUES
A priority queue implementation:
# Implementation of the unbounded Priority Queue ADT using a Python list
# with new items appended to the end.
class PriorityQueue : #Create an empty unbounded priority queue.
def __init__( self ):
self._qList = list()
def isEmpty( self ): # Returns True if the queue is empty.
return len( self ) == 0
def __len__( self ):#Returns the number of items in the queue.
return len( self._qList )
def enqueue( self, item, priority ): # Adds the given item to the queue.
# Create a new instance of the storage class and append it to the list.
entry = _PriorityQEntry( item, priority )
self._qList.append( entry )
def dequeue( self ) : #Removes and returns the first item in the queue.
assert not self.isEmpty(), "Cannot dequeue from an empty queue."
# Find the entry with the highest priority.
highest = self._qList[i].priority
for i in range( self.len() ) :
# See if the ith entry contains a higher priority (smaller integer).
If self._qList[i].priority < highest :
Highest = self._qList[i].priority
#Remove the entry with the highest priority and return the item.
entry = self._qList.pop( highest )
return entry.item
class _PriorityQEntry( object ):#Private storage class for associating queue items with their priority.
def __init__( self, item, prioity ):
self.item = item
self.priority = priority
Weeks-1,2,3 32
ELEMENTARY DATA STRUCTURES: BINARY TREES
A binary tree
A tree in which each node can have at most two children
One child is identified as the left child
Other as the right child

Binary trees have many shapes

Weeks-1,2,3 33
ELEMENTARY DATA STRUCTURES: BINARY TREES
A binary tree
Levels

Weeks-1,2,3 34
ELEMENTARY DATA STRUCTURES: BINARY TREES
A binary tree
Implementations

# The storage class for creating binary tree nodes.

class _BinTreeNode :
def __init__( self, data ): def postorderTrav( subtree ):
self.data = data if subtree is not None :
self.left = None postorderTrav( subtree.left )
self.right = None postorderTrav( subtree.right )
print( subtree.data )
Traversals
def preorderTrav( subtree ): def breadthFirstTrav( bintree ):
# Create a queue and add the root node to it.
if subtree is not None : Queue q
print( subtree.data ) q.enqueue( bintree )
preorderTrav( subtree.left ) # Visit each node in the tree.
preorderTrav( subtree.right ) while not q.isEmpty() :
# Remove the next node from the queue and
visit it.
def inorderTrav( subtree ): node = q.dequeue()
if subtree is not None : print( node.data )
inorderTrav( subtree.left ) # Add the two children to the queue.
print( subtree.data ) if node.left is not None :
q.enqueue( node.left )
inorderTrav( subtree.right ) if node.right is not None :
q.enqueue( node.right )

Weeks-1,2,3 35
ELEMENTARY DATA STRUCTURES: HEAPS
A heap is a complete binary tree in which the nodes are organized based on their
data entry values
There are two variants of the heap structure: max-heap and min-heap

The max-heap:
 Has the heap order property;
 For each non-leaf node V , the value in V is greater than the value of its two
children;
 The largest value in a max-heap will always be stored in the root while the
smallest values will be stored in the leaf nodes.

MAX-HEAP

Weeks-1,2,3 36
ELEMENTARY DATA STRUCTURES: HEAPS
A heap is a complete binary tree in which the nodes are organized based on their
data entry values
There are two variants of the heap structure: max-heap and min-heap
The min-heap:
For each non-leaf node V , the value in V is smaller than the value of its two
children.
MIN-HEAP

Operations on heaps
A heap is a specialized structure with limited operations
One can insert a new value into a heap
Once can extract and remove the root node’s value from the heap.

Weeks-1,2,3 37
 ELEMENTARY DATA STRUCTURES: HEAPS IMPLEMENTATIONS
# An array-based implementation of the max-heap. value = self._elements[0]
class MaxHeap : self._count -= 1
# Create a max-heap with capacity of maxSize. self._elements[0] = self._elements[ self._count ]
def __init__( self, maxSize ): # Sift the root value down the tree.
self._elements = Array( maxSize ) self._siftDown( 0 )
self._count = 0 # Sift the value at the ndx element up the tree.
# Return the number of items in the heap. def _siftUp( self, ndx ):
If ndx > 0 :
def __len__( self ):
parent = ndx // 2
return self._count if self._elements[ndx] > self._elements[parent] :
# Return the maximum capacity of the heap. tmp = self._elements[ndx]
def capacity( self ): self._elements[ndx] = self._elements[parent]
return len( self._elements ) self._elements[parent] = tmp
# Add a new value to the heap. self._siftUp( parent )
def add( self, value ): # swap elements
assert self._count < self.capacity(), "Cannot # Sift the value at the ndx element down the tree.
add to a full heap." def _siftDown( self, ndx ):
# Add the new value to the end of the list. left = 2 * ndx + 1
self._elements[ self._count ] = value right = 2 * ndx + 2
self._count += 1 # Determine which node contains the larger value.
# Sift the new value up the tree. largest = ndx
self._siftUp( self._count - 1 ) if left < count and self._elements[left] >=
# Extract the maximum value from the heap. self._elements[largest] :
def extract( self ): largest = left
elif right < count and self._elements[right] >=
assert self._count > 0, "Cannot extract f self._elements[largest]:
rom an empty heap." largest = right
# If the largest value is not in the current node (ndx),
swap it with
# the largest value and repeat the process.
if largest != ndx :
Weeks-1,2,3 swap( self._elements[ndx], 38
self._elements[largest] )
_siftDown( largest )
ELEMENTARY DATA STRUCTURES: HASH TABLES

Hash Table
Stores data into an array format
It uses a hashing function that generates a slot or an index
to store/insert any element or value.

Weeks-1,2,3 39
ELEMENTARY DATA STRUCTURES: HASH TABLES

Hashing Function
➔ Generates a slot or index to any “key” value.

➔ Perfect hashing or perfect hash function is the one which

assigns a unique slot for every key value.

➔ Sometimes, there can be cases where the hash function

generates the same index for multiple key values.

➔ The size of the hash table can be increased to improve the

perfection of the hash function.

Creating a hash_table- modulo can be a hashing function

hash_table = [None] * 10
print (hash_table)
# Output:
# [None, None, None, None, None, None, None, None, None, None]

Weeks-1,2,3 40
ELEMENTARY DATA STRUCTURES: HASH TABLES

Hashing Function
Creating a hash_table- modulo can be a hashing function
hash_table = [None] * 10
print (hash_table)
# Output:
# [None, None, None, None, None, None, None, None, None, None]

def hashing_func(key): # module used as a hashing function

return key % len(hash_table)

print (hashing_func(10)) # Output: 0

print (hashing_func(20)) # Output: 0
print (hashing_func(25)) # Output: 5

More implementations
hash_table = [[] for _ in range(10)]
print (hash_table)
# Output:
# [[], [], [], [], [], [], [], [], [], []]

Weeks-1,2,3 41
ELEMENTARY DATA STRUCTURES: HASH TABLES

More implementations
def insert(hash_table, key, value): def search(hash_table, key):
hash_key = hash(key) % hash_key = hash(key) % len(hash_table)
len(hash_table) bucket = hash_table[hash_key]
key_exists = False
bucket = hash_table[hash_key]
for i, kv in enumerate(bucket):
for i, kv in enumerate(bucket): k, v = kv
k, v = kv if key == k:
if key == k: return v
key_exists = True
break print (search(hash_table, 10)) # Output: Nepal
if key_exists: print (search(hash_table, 20)) # Output: India
bucket[i] = ((key, value)) print (search(hash_table, 30)) # Output: None
else:
bucket.append((key, value))

insert(hash_table, 10, 'Nepal')

insert(hash_table, 25, 'USA')
insert(hash_table, 20, 'India')
print (hash_table)
# Output:
# [[(10, 'Nepal'), (20, 'India')], [], [], [],
[], [(25, 'USA')], [], [], [], []]

Weeks-1,2,3 42
ELEMENTARY DATA STRUCTURES: HASH TABLES
More implementations
def delete(hash_table, key):
hash_key = hash(key) % len(hash_table)
key_exists = False
bucket = hash_table[hash_key]
for i, kv in enumerate(bucket):
k, v = kv
if key == k:
key_exists = True
break
if key_exists:
del bucket[i]
print ('Key {} deleted'.format(key))
else:
print ('Key {} not found'.format(key))

delete(hash_table, 100)
print (hash_table)
# Output:
# Key 100 not found
# [[(10, 'Nepal'), (20, 'India')], [], [], [], [], [(25, 'USA')], [], [], [], []]

delete(hash_table, 10)
print (hash_table)
# Output:
# Key 10 deleted
Weeks-1,2,3
# [[(20, 'India')], [], [], [], [], [(25, 'USA')], [], [], [], []] 43
ELEMENTARY DATA STRUCTURES- COMPLEXITIES

Weeks-1,2,3 44
CSC 311 DESIGN AND ANALYSIS OF ALGORITHMS
EXERCISES
(1)Define the term algorithm
(2)Define the term ‘design and analysis of algorithm’
(3)Describe the features of a an efficient algorithm
(4)What is the origin of ‘design and analyis of
algorithms’?
(5) Why is analysi of algorithms important?
(6) What is the use of analysis of algorithms?
(7) Describe the steps in analysis of algorithms
(8) Describe the role of inputs in analysis of algorithms
(9)State some typical input functions
(10)Discuss why it is still necessary to perform analysis
of algorithms when computers are now very fast.
(11)Describe some arithmetic sequences
(12)Describe some geometric sequences
(13)Describe some logarithmic formulae
(14)Discus the importance of ‘recursion’
Weeks-1,2,3 45
CSC 311 DESIGN AND ANALYSIS OF ALGORITHMS
EXERCISES
(1)Discus the importance of ‘recurrence relations’ giving
examples
(2)Describe the term ‘time complexity of an algorithm’
(3)Describe the term ‘space complexity of an algorithm’
(4)State the important features to consider in time
complexity of an algorithm
(5)Discuss the order of growth of functions
(6)Describe the big-oh
(7)Describe the big-omega
(8)Describe the bg-theta
(9)Find out how various algorithms perform in terms of
the big-o
(10)Which algorithms would you prefer in sorting and
serching?
(11)Describe arrays and their implementations
(12)Describe linked-lists Weeks-1,2,3
and their implementations 46
CSC 311 DESIGN AND ANALYSIS OF ALGORITHMS
EXERCISES
(1)Describe queues and their implementations
(2)Describe priority queues
(3)Describe binary trees and their implementations
(4)Describe heaps and their implementations
(5)Describe hash tables and hashing functions
(6)Discuss space and time complexity associated with
operations on some data structures.

Weeks-1,2,3 47