CSC 301 – Design and Analysis of Algorithms

Lecture# 01: Introduction

1
Algorithm Definition
• An algorithm is a step-by-step procedure for solving a particular
problem in a finite amount of time.

• More generally, an algorithm is any well defined computational

procedure that takes collection of elements as input and produces
a collection of elements as output.

Input ALGORITHM
Some mysterious Output
X processing Y=
F: X→Y
F(X) 2
Algorithm -- Examples
• Repairing a lamp
• A cooking recipe
• Calling a friend on the phone
• The rules of how to play a game
• Directions for driving from A to B
• A car repair manual
• Human Brain Project
• Internet & Communication Links (Graph)
• Matrix Multiplication

3
Algorithm vs. Program
• A computer program is an instance, or concrete representation,
for an algorithm in some programming language

• Set of instructions which the computer follows to solve a problem

Problem Algorithm: A sequence of

instructions describing how
to do a task

4
High Level
Language
Program
Solving Problems (1)
When faced with a problem:
1. First clearly define the problem

2. Think of possible solutions

3. Select the one that seems the best under the prevailing
circumstances

4. And then apply that solution

5. If the solution works as desired, fine; else go back to step 2

5
Solving Problems (2)
• It is quite common to first solve a problem for a particular case
• Then for another
• And, possibly another
• And watch for patterns and trends that emerge
• And to use the knowledge from these patterns and trends in
coming up with a general solution
• And this general solution is called …………….
“Algorithm”

6
One Problem, Many Algorithms
Problem
• The statement of the problem specifies, in general terms, the
desired input/output relationship.
Algorithm
• The algorithm describes a specific computational procedure for
achieving input/output relationship.
Example
• Sorting a sequence of numbers into non-decreasing order.
Algorithms
• Various algorithms e.g. merge sort, quick sort, heap sorts etc.

7
Problem Instances
• An input sequence is called an instance of a Problem

• A problem has many particular instances

• An algorithm must work correctly on all instances of the

problem it claims to solve

• Many interesting problems have infinitely many instances

– Since computers are finite, we usually need to limit the number and/or
size of possible instances in this case
– This restriction doesn’t prevent us from doing analysis in the abstract

8
Properties of Algorithms
• It must be composed of an ordered sequence of precise steps.

• It must have finite number of well-defined instructions /steps.

• The execution sequence of instructions should not be ambiguous.

• It must be correct.

• It must terminate.

9
Syntax & Semantics
An algorithm is “correct” if its: WARNINGS:
• Semantics are correct
• Syntax is correct 1. An algorithm can be
syntactically correct, yet
semantically incorrect –
Semantics: dangerous situation!
Colorless
• The conceptgreen
embeddedideas
in an sleep furiously!
2. Syntactic correctness is
algorithm (the soul!)
easier to check as
compared to semantic
Syntax: correctness
• The actual representation of
an algorithm (the body!)
10
Algorithm Summary
• Problem Statement
• Relationship b/w input and output

• Algorithm
• Procedure to achieve the relationship

• Definition
• A sequence of steps that transform the input to output

• Instance
• The input needed to compute solution

• Correct Algorithm 11
• for every input it halts with correct output
Brief History
• The study of algorithms began with mathematicians and was a
significant area of work in the early years. The goal of those early
studies was to find a single, general algorithm that could solve all
problems of a single type.

• Named after 9th century Persian Muslim mathematician

Abu Abdullah Muhammad ibn Musa al-Khwarizmi who
lived in Baghdad and worked at the Dar al-Hikma

• Dar al-Hikma acquired & translated books on science & philosophy,

particularly those in Greek, as well as publishing original research.

• The word algorism originally referred only to the rules of performing 12

arithmetic using Hindu-Arabic numerals, but later evolved to include
all definite procedures for solving problems.
Al-Khwarizmi’s Golden Principle
All complex problems can be and must be solved
using the following simple steps:

1. Break down the problem into small, simple sub-problems

2. Arrange the sub-problems in such an order that each of them
can be solved without effecting any other
3. Solve them separately, in the correct order
4. Combine the solutions of the sub-problems to form the
solution of the original problem

13
Why Algorithms are Useful?
• Once we find an algorithm for solving a problem, we do not need
to re-discover it the next time we are faced with that problem

• Once an algorithm is known, the task of solving the problem

reduces to following (almost blindly and without thinking) the
instructions precisely

• All the knowledge required for solving the problem is present in

the algorithm

14
Why Write an Algorithm Down?
• For your own use in the future, so that you don’t have spend the
time for rethinking it

• Written form is easier to modify and improve

• Makes it easy when explaining the process to others

15
Designing of Algorithms
• Selecting the basic approaches to the solution of the problem
• Choosing data structures
• Putting the pieces of the puzzle together
• Expressing and implementing the algorithm
• clearness, conciseness, effectiveness, etc.

Major Factors in Algorithms Design

• Correctness: An algorithm is said to be correct if for every input, it
halts with correct output. An incorrect algorithm might not halt at
all OR it might halt with an answer other than desired one. Correct
algorithm solves a computational problem
• Algorithm Efficiency: Measuring efficiency of an algorithm
• do its analysis i.e. growth rate. 16
• Compare efficiencies of different algorithms for the same problem.
Designing of Algorithms
• Most basic and popular algorithms are search and sort algorithms
• Which algorithm is the best?
• Depends upon various factors, for example in case of sorting
• The number of items to be sorted
• The extent to which the items are already sorted
• Possible restrictions on the item values
• The kind of storage device to be used etc.

17
Important Designing Techniques
• Brute Force–Straightforward, naive approach–Mostly expensive

• Divide-and-Conquer –Divide into smaller sub-problems

• e.g merge sort

• Iterative Improvement–Improve one change at a time.

• e.g greedy algorithms

• Decrease-and-Conquer–Decrease instance size

• e.g fibonacci sequence

• Transform-and-Conquer–Modify problem first and then solve it

• e.g repeating numbers in an array
18
• Dynamic programming–Dependent sub-problems, reuse results
 Algorithm Efficiency
• Several possible algorithms exist that can solve a particular problem
• each algorithm has a given efficiency
• compare efficiencies of different algorithms for the same problem

• The efficiency of an algorithm is a measure of the amount of resources

consumed in solving a problem of size n
• Running time (number of primitive steps that are executed)
• Memory/Space

• Analysis in the context of algorithms is concerned with predicting the required

resources

• There are always tradeoffs between these two efficiencies

• allow one to decrease the running time of an algorithm solution by increasing space
to store and vice-versa

• Time is the resource of most interest 19

• By analyzing several candidate algorithms, the most efficient one(s) can be
identified
 Algorithm Efficiency
Two goals for best design practices:
1. To design an algorithm that is easy to understand, code, debug.
2. To design an algorithm that makes efficient use of the computer’s
resources.

How do we improve the time efficiency of a program?

The 90/10 Rule
• 90% of the execution time of a program is spent in executing 10% of the
code. So, how do we locate the critical 10%?
• software metrics tools
• global counters to locate bottlenecks (loop executions, function calls)

Time Efficiency improvement

• Good programming: Move code out of loops that does not belong there
• Remove any unnecessary I/O operations
• Replace an inefficient algorithm (best solution) 20

Moral - Choose the most appropriate algorithm(s) BEFORE

program implementation
Analysis of Algorithms
• Two essential approaches to measuring algorithm efficiency:
• Empirical analysis:
• Program the algorithm and measure its running time on example
instances
• Theoretical analysis
• Employ mathematical techniques to derive a function which relates
the running time to the size of instance

• In this cousre our focus will be on Threoretical Analysis.

21
Analysis of Algorithms
• Many criteria affect the running time of an algorithm, including
• speed of CPU, bus and peripheral hardware
• design time, programming time and debugging time
• language used and coding efficiency of the programmer
• quality of input (good, bad or average)
• But
• Programs derived from two algorithms for solving the same
problem should both be
• Machine independent
• Language independent
• Amenable to mathematical study
• Realistic
22
Analysis of Algorithms
• The following three cases are investigated in algorithm analysis:

• A) Worst case: The worst outcome for any possible input

• We often concentrate on this for our analysis as it provides a clear upper
bound of resources
• an absolute guarantee

• B) Average case: Measures performance over the entire set of

possible instances
• Very useful, but treat with care: what is “average”?
• Random (equally likely) inputs vs. real-life inputs

23
• C) Best Case: The best outcome for any possible input
• provides lower bound of resources
Analysis of Algorithms
• An algorithm may perform very differently on different example
instances. e.g: bubble sort algorithm might be presented with data:
• already in order
• in random order
• in the exact reverse order of what is required

• Average case analysis can be difficult in practice

• to do a realistic analysis we need to know the likely distribution of instances
• However, it is often very useful and more relevant than worst case; for
example quicksort has a catastrophic (extremly harmful) worst case, but in
practice it is one of the best sorting algorithms known

• The average case uses the following concept in probability theory.

Suppose the numbers n1, n2 , …, nk occur with respective probabilities p1,
p2,…..pk. Then the expectation or average value E is given by E = n1p1 + 24
n2p2 + ...+ nk.рk
Empirical Analysis
• Most algorithms transform input
objects into output objects

• The running time of an algorithm

typically grows with the input
size

• Average case time is often

difficult to determine

• We focus on the worst case

running time
25
• Easier to analyze
• Crucial to applications such as
games, finance and robotics
Empirical Analysis
• Write a program implementing
the algorithm

• Run the program with inputs of

varying size and compositions

• Use timing routines to get an

accurate measure of the actual
running time e.g.
System.currentTimeMillis()

• Plot the results

26
Limitations of Empirical Analysis
• Implementation dependent
• Execution time differ for different
implementations of same program

• Platform dependent
• Execution time differ on different
architectures

• Data dependent
• Execution time is sensitive to
amount and type of data
minipulated.

• Language dependent
• Execution time differ for same 27
code, coded in different languages
∴ absolute measure for an algorithm is not appropriate
Theorerical Analysis
• Data independent
• Takes into account all possible inputs

• Platform independent

• Language independent

• Implementatiton independent
• not dependent on skill of programmer
• can save time of programming an inefficient solution

• Characterizes running time as a function of input size, n. Easy to

28
extrapolate without risk
Why Analysis of Algorithms?
• For real-time problems, we would like to prove that an
algorithm terminates in a given time.

• Algorithmics may indicate which is the best and fastest

solution to a problem without having to code up and test
different solutions

• Many problems are in a complexity class for which no

practical algorithms are known
• better to know this before wasting a lot of time trying to develop a
”perfect” solution: verification

29
But Computers are So Fast These Days??
• Do we need to bother with algorithmics and complexity
any more?
• computers are fast, compared to even 10 years ago...

• Many problems are so computationally demanding that no

growth in the power of computing will help very much.

• Speed and efficiency are still important.

30
Importance of Analyzing Algorithms
• Need to recognize limitations of various algorithms for solving a
problem

• Need to understand relationship between problem size and running

time
• When is a running program not good enough?

• Need to learn how to analyze an algorithm's running time without

coding it

• Need to learn techniques for writing more efficient code

• Need to recognize bottlenecks in code as well as which parts of

31
code are easiest to optimize
Importance of Analyzing Algorithms
• An array-based list retrieve operation takes at most one operation,
a linked-list-based list retrieve operation at most “n” operations.

• But insert and delete operations are much easier on a linked-list-

based list implementation.

• When selecting the implementation of an Abstract Data Type (ADT),

we have to consider how frequently particular ADT operations
occur in a given application.

• For small problem size, we can ignore the algorithm’s efficiency.

32
• We have to weigh the trade-offs between an algorithm’s time
requirement and memory requirements.
What do we analyze about Algorithms?
• Algorithms are analyzed to understand their behavior and to
improve them if possible

• Correctness
• Does the input/output relation match algorithm requirement?
• Amount of work done
• Basic operations to do task
• Amount of space used
• Memory used
• Simplicity, clarity
• Verification and implementation.
• Optimality
33
• Is it impossible to do better?
Problem Solving Process
• Problem
• Strategy
• Algorithm
• Input
• Output
• Steps
• Analysis
• Correctness
• Time & Space Optimality
• Implementation
• Verification
34
Computation Model for Analysis
• To analyze an algorithm is to determine the amount of
resources necessary to execute it. These resources include
computational time, memory and communication bandwidth.

• Analysis of the algorithm is performed with respect to a

computational model called RAM (Random Access Machine)

• A RAM is an idealized uni-processor machine with an infinite

large random-access memory
• Instruction are executed one by one
• All memory equally expensive to access
• No concurrent operations
35
• Constant word size
• All reasonable instructions (basic operations) take unit time
Complexity of an Algorithm
• The complexity of an algorithm is the amount of work the algorithm
performs to complete its task. It is the level of difficulty in solving
mathematically posed problems as measured by:
• Time (time complexity)
• No. of steps or arithmetic operations (computational complexity)
• Memory space required (space complexity)

• Complexity is a function T(n) which yields the time (or space)

required to execute the algorithm of a problem of size ‘n’.

36
Pseudocode
• High-level description of an algorithm
• More structured than English prose but Less detailed than a
program
• Preferred notation for describing algorithms
• Hides program design issues

ArrayMax(A, n)
Input: Array A of n integers
Output: maximum element of A

1. currentMax  A[0];
2. for i = 1 to n-1 do
3. if A[i] > currentMax then 37
4. currentMax  A[i]
5. return currentMax;
Pseudocode
• Indentation indicates block structure. e.g body of loop
• Looping Constructs while, for and the conditional if-then-else
• The symbol // indicates that the reminder of the line is a comment.
• Arithmetic & logical expressions: (+, -,*,/, ) (and, or and not)
• Assignment & swap statements: a b , ab c, a  b
• Return/Exit/End: termination of an algorithm or block
ArrayMax(A, n)
Input: Array A of n integers
Output: maximum element of A

1. currentMax  A[0];
2. for i = 1 to n-1 do
3. if A[i] > currentMax then 38
4. currentMax  A[i]
5. return currentMax;
Pseudocode
• Local variables mostly used unless global variable explicitly defined
• If A is a structure then |A| is size of structure. If A is an Array then
n =length[A], upper bound of array. All Array elements are
accessed by name followed by index in square brackets A[i].
• Parameters are passed to a procedure by values
• Semicolons used for multiple short statement written on one line
ArrayMax(A, n)
Input: Array A of n integers
Output: maximum element of A

1. currentMax  A[0]
2. for i = 1 to n-1 do
3. if A[i] > currentMax then 39
4. currentMax  A[i]
5. return currentMax
Elementary Operations
• An elementary operation is an operation which takes constant time
regardless of problem size.
• The running time of an algorithm on a particular input is determined by
the number of “Elementary Operations” executed.
• Theoretical analysis on paper from a description of an algorithm

• Defining elementary operations is a little trickier than it appears

• We want elementary operations to be relatively machine and language
independent, but still as informative and easy to define as possible

• Example of elementary operations include

• variable assignment
• arithmetic operations (+, -, x, /) on integers
• comparison operations (a < b)
• boolean operations 40
• accessing an element of an array

• We will measure number of steps taken in term of size of input

Components of an Algorithm
• Variables and values
• Instructions
• Sequences
• Selections
• Repetitions

41
Instruction and Sequence
• A linear sequence of elementary operations is also performed in
constant time.
• More generally, given two program fragments P1 and P2 which run
sequentially in times t1 and t2
• use the maximum rule which states that the larger time dominates
• complexity will be max(t1,t2)

e.g. Assignment Statements

x=a .....................1
x= a+b*c/h-u ......1 T(n) = 1+1+1
a>b ......................1 T(n) = 3

42
Sequences
• Analysing a group of consecutive statements
• The statement taking the maximum time will be the one counted
• use the maximum rule

Block #1 t1

T(n) = max(t1,t2)
Block #2 t2

• e.g. a fragment with single for-loop followed by double for- loop

• T(n) = n2 43
• Always analyze function calls first
Selection
• If <test> then P1 else P2 structures are a little harder;
conditional loops.
• The maximum rule can be applied here too:
• max(t1, t2), assuming t1, t2 are times of P1, P2
• However, the maximum rule may prove too conservative
• if <test> is always true the time is t1
• if <test> is always false the time is t2
e.g. if (a>b) then ...........1
a=2......................1
b=b+3*t .............1
else
x=x+2*3 ................1
Block #1 t1 Block #2 t2 Max(t1,t2)
44
T(n)= 1+max(2,1) = 3
Repetition (Loops)
• Analyzing loops: Any loop has two parts:
• How many iterations are performed?
• How many steps per iteration?

for i = 1 to n do
P(i);

• Assume that P(i) takes time t, where t is independent of i

• Running time of a for-loop is at most the running time of the
statements inside the for-loop times number of iterations

T(n) = nt

• This approach is reasonable, provided n is positive 45

• If n is zero or negative the relationship T(n) = nt is not valid
Repetition (Loops)
Analysing Nested Loops

for i = 0 to n do
for j = 0 to m do
P(j);
• Assume that P(j) takes time t, where t is independent of i and j
• Start with outer loop:
• How many iterations? n
• How much time per iteration? Need to evaluate inner loop

• Analyze inside-out. Total running time is running time of the

statement multiplied by product of the sizes of all the for-loops
46
T(n) = nmt
Repetition (Loops)
Analysing Nested Loops
for i = 0 to n do
for j = 0 to i do
P(j);
• Assume that P(j) takes time t, where t is independent of i and j
• How do we analyze the running time of an algorithm that has
complex nested loop?
• The answer is we write out the loops as summations and then
solve the summations. To convert loops into summations, we
work from inside-out.

T(n) = n +
i 0 ri + t 
n n
i 0 i
r 47
= n + n(n+1)/2 + tn(n+1)/2
 Analysis Example
Algorithm: Number of times executed
1. n = read input from user 1
2. sum = 0 1
3. i = 0 1
4. while i < n n

n 1
5. number = read input from user n or i 0
1

n 1
6. sum = sum + number n or i 0
1


n 1
7. i = i + 1 n or 1
i 0
8. mean = sum / n 1

The computing time for this algorithm in terms on input size n is:
T(n) = 1 + 1 + 1 + n + n + n + n + 1 48

T(n) = 4n + 4
Another Analysis Example
i=1 ...............................1
while (i < n)................n-1
a=2+g...............n-1
i=i+1 ................n-1
if (i<=n).......................1
a=2 ....................1
else
a=3.....................1

T(n) = 1 + 3(n-1) + 1 + 1
49
=3n
Another Analysis Example
i=1...............................? 1
while (i<=10)................?
i=i+1...................?
10
i=1 ...............................? 10
while (i<=n)..................? 1
a=2+g .................?
i=i+1 ...................?
n
if (i<=n)........................ ? n
a=2 .................... ? n
else
a=3..................... ?
1
1

50
T ( n) = ? T ( n) = 3n +24
Asymptotic Growth Rate
• Changing the hardware/software environment
• Affects T(n) by constant factor, but does not alter the growth rate of T(n)

• Algorithm complexity is usually very complex. The growth of the

complexity functions is what is more important for the analysis and is a
suitable measure for the comparison of algorithms with increasing input
size n.
• Asymptotic notations like big-O, big-Omega, and big-Theta are used to
compute the complexity because different implementations of algorithm
may differ in efficiency.
• The big-Oh notation gives an upper bound on the growth rate of a
function.
• The statement “f(n) is O(g(n))” means that the growth rate of f(n) is no
more than the growth rate of g(n). 51

• We can use the big-Oh notation to rank functions according to their

growth rate.
Asymptotic Growth Rate
Two reasons why we are interested in asymptotic growth rates:

• Practical purposes: For large problems, when we expect to have

big computational requirements

• Theoretical purposes: concentrating on growth rates frees us

from some important issues:
• fixed costs (e.g. switching the computer on!), which may
dominate for a small problem size but be largely irrelevant
• machine and implementation details
• The growth rate will be a compact and easy to understand the
function
52
Properties of Growth-Rate Functions
Example: 5n + 3
Estimated running time for different values of n:
n = 10 => 53 steps
n = 100 => 503 steps
n = 1,000 => 5003 steps
n = 1,000,000 => 5,000,003 steps

As “n” grows, the number of steps grow in linear proportion to n for

this function “Sum”

What about the “+3” and “5” in 5n+3?

As n gets large, the +3 becomes insignificant
5 is inaccurate, as different operations require varying amounts of
time and also does not have any significant importance 53

What is fundamental is that the time is linear in n.

Asymptotic Algorithm Analysis
• The asymptotic analysis of an algorithm determines the running
time in big-Oh notation
• To perform the asymptotic analysis
• We find the worst-case number of primitive operations executed as a
function of the input size n
• We express this function with big-Oh notation

• Example: An algorithm executes T(n) = 2n2 + n elementary

operations. We say that the algorithm runs in O(n2) time
• Growth rate is not affected by constant factors or lower-order terms
so these terms can be dropped
• The 2n2 + n time bound is said to "grow asymptotically" like n2
• This gives us an approximation of the complexity of the algorithm 54
• Ignores lots of (machine dependent) details
Algorithm Efficiency
Measuring efficiency of an algorithm
• do its analysis i.e. growth rate.
• Compare efficiencies of different algorithms for the same problem.

As inputs get larger, any algorithm of a smaller order will be more

efficient than an algorithm of a larger order

0.05 N2 = O(N2)

3N = O(N)
Time (steps)

55

Input (size)
N = 60
Important Functions
These functions often appear in algorithm analysis:

56
A comparison of Growth-Rate Functions
Size does Matter:
What happens if we increase the input size N?

57
 A comparison of Growth-Rate Functions
250

f(n) = n
f(n) = log(n)
f(n) = n log(n)
f(n) = n^2
f(n) = n^3
f(n) = 2^n

0
58
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
 A comparison of Growth-Rate Functions

500

f(n) = n
f(n) = log(n)
f(n) = n log(n)
f(n) = n^2
f(n) = n^3
f(n) = 2^n

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 59
 A comparison of Growth-Rate Functions

1000

f(n) = n
f(n) = log(n)
f(n) = n log(n)
f(n) = n^2
f(n) = n^3
f(n) = 2^n

0
1 3 5 7 9 11 13 15 17 19 60
 A comparison of Growth-Rate Functions

5000

4000
f(n) = n
f(n) = log(n)
3000
f(n) = n log(n)
f(n) = n^2
2000 f(n) = n^3
f(n) = 2^n

1000

0
61
11
1

13

15

17

19
 A comparison of Growth-Rate Functions

10000000

1000000

100000

10000

1000

100

10

1
1 4 16 64 256 1024 4096 16384 65536 62
Performance Classification
f(n) Classification

1
:
Constant run time is fixed, and does not depend upon n . Most instructions are
executed once, or only a few times, regardless of the amount of information being processed

log n Logarithmic: when n increases, so does run time, but much slower . Common in
programs which solve large problems by transforming them into smaller problems.

n Linear: run time varies directly with n. Typically, a small amount of processing is done on
each element.

n log n When n doubles, run time slightly more than doubles . Common in programs
which break a problem down into smaller sub-problems, solves them independently, then combines
solutions

n2 Quadratic: when n doubles, runtime increases fourfold. Practical only for small
problems; typically the program processes all pairs of input (e.g. in a double nested loop).

n3 Cubic: when n doubles, runtime increases eightfold

63
2n Exponential: when n doubles, run time squares. This is often the result of a natural, “brute
force” solution.
Running Time vs. Time Complexity
• Running time is how long it takes a program to run.

• Time complexity is a description of the asymptotic behavior of

running time as input size tends to infinity.

• The exact running time might be 2036.n2 + 17453.n + 18464 but

you can say that the running time "is" O(n2), because that's the
formal(idiomatic) way to describe complexity classes and big-O
notation.

• Infact, the running time is not a complexity class, IT'S EITHER A

DURATION, OR A FUNCTION WHICH GIVES YOU THE DURATION.
"Being O(n2)" is a mathematical property of that function, not a
full characterization of it.
64
Example:
Running Time to Sort Array of 2000 Integers
Computer Type Desktop Server Mainframe Supercomputer
Time (sec) 51.915 11.508 0.431 0.087

Array Desktop Server

Size
125 12.5 2.8
250 49.3 11.0
500 195.8 43.4
1000 780.3 172.9
2000 3114.9 690.5

65
Analysis of Results
f(n) = a n2 + b n + c
where a = 0.0001724, b = 0.0004 and c = 0.1

n f(n) a n2 % of n2
125 2.8 2.7 94.7
250 11.0 10.8 98.2
500 43.4 43.1 99.3
1000 172.9 172.4 99.7
2000 690.5 689.6 99.9 66
Model of Computation
Drawbacks:
• poor assumption that each basic operation takes constant time
• Adding, Multiplying, Comparing etc.

Finally what about Our Model?

• With all these weaknesses, our model is not so bad because
• We have to give the comparison, not absolute analysis of any algorithm.
• We have to deal with large inputs not with the small size

• Model seems to work well describing computational power of

modern nonparallel machines
Can we do Exact Measure of Efficiency ?
• Exact, not asymptotic, measure of efficiency can be sometimes
computed but it usually requires certain assumptions concerning
implementation
67
Complexity Examples
What does the following algorithm compute?

procedure who_knows(a1, a2, …, an: integers)

m := 0
for i := 1 to n-1
for j := i + 1 to n
if |ai – aj| > m then m := |ai – aj|

{m is the maximum difference between any two numbers in the

input sequence}

Comparisons: n-1 + n-2 + n-3 + … + 1

= n*(n – 1)/2 = 0.5n2 – 0.5n
68
Time complexity is O(n2).
Complexity Examples
Another algorithm solving the same problem:

procedure max_diff(a1, a2, …, an: integers)

min := a1
max := a1
for i := 2 to n
if ai < min then min := ai
else if ai > max then max := ai
m := max - min

Comparisons: 2n + 2
Time complexity is O(n).
69
Prerequisite Review :
Mathematics
Data Structures

70
Notations
• Floor x and Ceiling x  3.4 = 3 and 3.4 = 4
• open interval (a, b) is {x   | a < x < b}

• closed interval [a, b] is {x   | a  x  b}

• Set is a collection of not ordered, not repeated elements e.g {a, b, c}
• Operations: union, intersection, difference, complement

• Membership: x ϵ X  x is a member of X e.g. a ϵ {a, b, c}

• Existential Quantifier (Ǝ)  Ǝ x (x≥x2) is true since x=0 is a solution

• Universal Quantifier (ꓯ)  ꓯ x (x2 ≥ 0) is true for all values of x 71

Exponent Review
x a x b  x a b
xa a b
 x
xb
( x a ) b  x ab
x0  1
xn  xn  2xn
2 n  2 n  2 * 2 n  2 n 1
( xy ) a  x a y a
m 1 72
x  m
x
Logarithm Review
x  log b y  bx  y
log c b
log a b 
log c a
log ab  log a log b
a
log  log a  log b
b
log(a b )  b log a
lg 1  0, lg 2  1, lg 1024  10
log y n log y x
x n
log a b
b a
c *log a b 73
b ac

log x 1  0
74
Summation Algebra
N

x
i 1
i
The sum of numbers from 1 to N; e.g 1 + 2 + 3 … + N

 i
x 2

i 1
Suppose our list has 5 number, and they are 1, 3, 2, 5, 6.
Then resulting summation will be 12 + 32 + 22 + 52 + 62 = 75
y N

 a  ( y  x  1)aor
ix
 a TheNaFirst constant Rule
i 1

N N

 ax
i 1
i a  xi
i 1
The Second constant Rule

N N N

 (x
i 1
i  y i )   xi   y i The Distributive Rule
i 1 i 1
3 3
75
 x
i 1 j 1
ij  x11  x12  x13  x 21  x 22  x 23  x31  x32  x33
Double Summation
Summation Algebra: Practice Questions
6

2
i 0
14
N N

 6x y
i 1
i 6 y  xi
i 1

2 3


i 0 j 0
2i  3 j 78

76
Arithmetic Series/Sequence
• A sequence in which the difference between one term and the
next is a constant. (add a fixed value each time ... on to infinity)
• Generalized form: {a, a+d, a+2d, a+3d, ... }
• where a is the first term, and d is the common difference between
the two terms

• Example: 1, 4, 7, 10, 13, 16, 19, 22, 25, ...

• Nth term would be xn = a + d(n-1)
n
• The summation of Arithmetic Sequence is S N  [2a1  (n  1)d ]
2
• Sn is the sum of the first n terms in a sequence
• a1 is the first term in the sequence
77
• d is the common difference in the arithmetic sequence
• n is the number of terms you are adding up
Geometric Series/Sequence
• A sequence in which each term is found by multiplying the
previous term by a constant.
• Generalized form: {a, ar, ar2, ar3, ... }
• where a is the first term, and r is the common ratio (r ≠ 0)

• Example: 2, 4, 8, 16, 32, 64, 128, 256, ...

• Nth term would be xn = ar(n-1)
• The summation of geometric Sequence is S N  a1
[1  r n
]
1 r
• Sn is the sum of the first n terms in a sequence
• a1 is the first term in the sequence
• r is the common ratio in the geometric sequence 78
• n is the number of terms you are adding up
Series Review

79
Series Examples

80
Permutation and Combination
Permutation
Set of n elements is an arrangement of the elements in given order
e.g. Permutation for elements a, b, c are abc, acb, bac, bca, cab, cba
- n! permutation exist for a set of elements
5! = 120 permutation for 5 elements
Combination
Set of n elements is an arrangement of the elements in any order
e.g. Combination for elements a, b, c is abc

81
Sample Space
A set “S” consisting of all possible outcomes that can result from a
random experiment (real or conceptual), can be defined as the
sample space for that experiment.
• Each possible outcome is called a sample point in that space.

Example: The sample space for an experiment of tossing a coin is

expressed as S = {H, T}, as two possible outcomes are possible: a
head (H) or a tail (T). ‘H’ and ‘T’ are the two sample points.

Example: The sample space for tossing two coins at once (or tossing
a coin twice) will contain four possible outcomes and is denoted by
S = {HH, HT, TH, TT}.
In this example, clearly, S is the Cartesian product A  A, where A =
{H, T}. 82
 Events
Any subset of a sample space S of a random experiment, is called an event. In
other words, an event is an individual outcome or any number of outcomes
(sample points) of a random experiment.
Simple event is an event that contains exactly one sample point.
Compound event is an event that contains more than one sample point, and
is produced by the union of simple events.

Occurrence of an event: An event A is said to occur if and only if the outcome

of the experiment corresponds to some element of A.

Example: The occurrence of a 6 when a die is thrown, is a simple event,

while the occurrence of a sum of 10 with a pair of dice, is a compound event,
as it can be decomposed into three simple events (4, 6), (5, 5) and (6, 4).

Example: Suppose an event A={2, 4, 6} represents occurrence of an even 83

number when the dice is thrown. If a dice shows ‘2’, ‘4’ or ‘6’, we say that the
event A of our interest has occurred.
 Events
Complementary Event is the event “not-A”, denoted by Ā or Ac, and is called
the negation of A.
Example: If we toss a coin once, then the complement of “heads” is “tails”. If
we toss a coin four times, then the complement of “at least one head” is “no
heads”.
A sample space consisting of n sample points can produce 2n different
subsets of simple and compound events.
Example: Consider a sample space S containing 3 sample points, i.e.
S = {a, b, c}. Then the 23 = 8 possible subsets are:
, {a}, {b}, {c}, {a, b}, {a, c}, {b, c}, {a, b, c}.
Each of these subsets is an event.
The subset {a, b, c} is the sample space itself and is also an event. It always
occurs and is known as the certain or sure event. 84
The empty set  is also an event, sometimes known as impossible event,
because it can never occur.
Events
Mutually Exclusive Events Two events A and B of a single experiment are
said to be mutually exclusive or disjoint if and only if they cannot both
occur at the same time; i.e. they have no points in common.

Example: When we toss a coin, we get either a head or a tail, but not
both at the same time. The two events head and tail are therefore
mutually exclusive.

85
 Probability Theory
If a random experiment can produce n mutually exclusive and
equally likely outcomes, and if m out to these outcomes are
considered favorable to the occurrence of a certain event A, then
the probability of the event A, denoted by P(A), is defined as the
ratio m/n.
m Number of favourable outcomes
P  A  
n Total number of possible outcomes

Formal Definition: Let S be a sample space with the sample points

E1, E2, … Ei, …En. Each sample point is assigned a real number,
denoted by the symbol P(Ei), and called the probability of Ei, that
must satisfy the following basic axioms:
• Axiom 1: For any event Ei, 0 < P(Ei) < 1.
86
• Axiom 2: P(S) = 1 for the sure event S.
• Axiom 3: If A and B are mutually exclusive events (subsets of S), then P
(A  B) = P(A) + P(B).
 Probability Theory
Example: If a card is drawn from an ordinary deck of 52 playing cards, find
the probability that
i) the card is a red card,
ii) the card is a 10.
Solution: The total number of possible outcomes is 13+13+13+13 = 52, and
we assume that all possible outcomes are equally likely.
(It is well-known that an ordinary deck of cards contains 13 cards of
diamonds, 13 cards of hearts, 13 cards of clubs, and 13 cards of spades.)
(i) Let A represent the event that the card drawn is a red card. Then the
number of outcomes favorable to the event A is 26 (since the 13 cards
of diamonds and the 13 cards of hearts are red).

m Number of favourable outcomes 26 1

P  A    
n Total number of possible outcomes 52 2
(ii) Let B denote the event that the card drawn is a 10. Then the number
of outcomes favorable to B is 4 as there are four 10’s 87
4 1
P B    .
52 13
 Probability Theory
Example: A fair coin is tossed three times. What is the probability that at
least one head appears?

Solution: The sample space for this experiment is:

S = {HHH, HHT, HTH, THH, HTT, THT, TTH, TTT}
and thus the total number of sample points is n(S) = 8.
Let A denote the event that at least one head appears. Then:
A = {HHH, HHT, HTH, THH, HTT, THT, TTH}
and therefore n(A) = 7.

n A 7
P A    .
n  S 8 88
 Probability Theory
Example: Four items are taken at random from a box of 12 items and
inspected. The box is rejected if more than 1 item is found to be faulty. If
there are 3 faulty items in the box, find the probability that the box is
accepted.
Solution: The sample space for this experiment is number of possible
combinations for selecting 4 out of 12 items from the box.
12  n! 12!
     495
4
  ( n  k )! k ! (8 )!4!

The box contains 3 faulty and 9 good items. The box is accepted if there is
(i) no faulty items, or (ii) one faulty item in the sample of 4 items selected.
Let A denote the event the number of faulty items chosen is 0 or 1. Then:
 3   9   3  9 
n A             126  252  378 sample po int s.
 0   4  1   3 
m 378 89
 P  A    0.76
n 495
Hence the probability that the box is accepted is 76%.
Data Structures Review
• Data structure is the logical or mathematical model of a
particular organization of data.
• Data structures let the input and output be represented in a
way that can be handled efficiently and effectively.
• Data may be organized in different ways.

Array

Linked list

90
Queue
Graph/Tree Stack
Arrays

Customer Customer Salesperson

Jamal 1 Jamal Tony
Sana 2 Sana Tony
Saeed 3 Saeed Nadia
Farooq 4 Farooq Owais
Salman 5 Salman Owais
Danial 6 Danial Nadia

Linear Arrays Two Dimensional Arrays 91

Example: Linear Search Algorithm
• Given a linear array A containing n elements, locate the position of an
Item ‘x’ or indicate that ‘x’ does not appear in A.

• The linear search algorithm solves this problem by comparing ‘x’, one by
one, with each element in A. That is, we compare ITEM with A[1], then
A[2], and so on, until we find the location of ‘x’.

LinearSearch(A, x) Number of times executed

i←1 1
while (i ≤ n and A[i] ≠ x) n
i ← i+1 n
if i ≤ n 1
T(n) = 2n+3
return true 1
92
else
return false 1
Best/Worst Case
Best case: ‘x’ is located in the first location of the array and loop
executes only once
T(n) = 1 + n + n + 1 +1
= 1+1+0+1+1
= O(1)

Worst case: ‘x’ is located in last location of the array or is not there
at all.
T(n) = 1 + n + n + 1 +1
= 2n +3
= O(n)

93
Average case
Average Case: Assume that it is equally likely for ‘x’ to appear at
any position in array A,. Accordingly, the number of comparisons
can be any of the numbers 1,2,3,..., n, and each number occurs
with probability p = 1/n.

T(n) = 1.1/n + 2. 1/n +……….+ n.1/n

= (1+2+3+……+n).1/n
= [n(n+1)/2] 1/n = n+1/2
= O(n)

This agrees with our intuitive feeling that the average number of
comparisons needed to find the location of ‘x’ is approximately
equal to half the number of elements in the A list. 94
Linked List
A B C 

Head

• A series of connected nodes

• Each node contains a piece of data and a pointer to the next node

Operations Average Case Worst Case

Insert O(1) O(1)
Delete O(1) O(1)
Search O(N) O(N)
95
Stack IN OUT

• LIFO
• Implemented using linked-list or arrays 2132

123

123

123

123

123

123

Operations Average Case Worst Case

Push O(1) O(1)
Pop O(1) O(1) 96
IsEmpty O(1) O(1)
Queue IN

• FIFO
• Implemented using linked-list or arrays 2132

123

123

2544

33

Operations Average Case Worst Case OUT

Enqueue O(1) O(1)
Dequeue O(N) O(N) 97
 Graphs/Trees (2x+y)(a-7b)3
*
+ ^

* y - 3

2 x a *
56
Tokyo Seattle
7 b
Seoul 128

16 181 New York

30 140
98

L.A.
Sydney
Graphs
• A graph (network) G = (V, E) is a data structure containing a set of
vertices (points/nodes) V, and a set of edges E, where an edge in E
represents a connection between a pair of vertices in V.

V (G)= { A, B, C, D}
E (G)= { (A, B), (A, C), (B, C), (B, D) }
A
C
• Examples B
•
•
Web pages with links
Methods in a program that call each other
D
• Road maps (e.g., Google maps)
• Airline routes
• Facebook friends
• Course pre-requisites 99
• Family trees
• Paths through a maze
Definitions
e7
1. Vertex set = {v1, v2, v3, v4, v5, v6}
v6
2. Edge set = {e1, e2, e3, e4, e5, e6, e7}
v4 e5
3. e1, e2, and e3 are incident on v1
4. v2 and v3 are adjacent to v1
v5
5. e2,, e3 and e4 are adjacent to e1
e6
6. v5 and v6 are adjacent to themselves
7. v4 is an isolated vertex
v1
8. e6 and e7 are loops
9. e2 and e3 are parallel e1 e2 e3

10. Endpoint(e5) = (v5, v6) 100

v2 v3
e4
Degree of vertices in a Graph
• The degree of a vertex “v” in a graph G, written deg (v), is
equal to the number of edges in G incident on a vertex “v”.

• Each edge is counted twice in counting the degrees of the

vertices of G. Therefore, sum of the degrees of the vertices
of a graph G is equal to twice the number of edges in G.

• A vertex is said to be even/odd if its degree of that vertex is

an even/odd number. A vertex of degree zero is called an
isolated vertex.

101
 Definitions: Path
A path of length k is a sequence v0, v1, …, vk of vertices such
that (vi , vi+1) for i = 0, 1, …, k – 1 is an edge of G.
Vertex a is reachable from b if a path exists from a to b.
b, c, d not a path
a, e, b is a closed path a b c d
Simple path:
a, e, k, p, l, q
m, h, d, c, g e f g h
Non-simple path:
(no repeated vertices)
a, b, e, f, g, b, g, l
j k l m
Length of a path
is the number of 102
edges in the path.
n o p q
Definitions: Cycle
A cycle is a path that starts and ends at the same vertex.
A simple cycle has no repeated vertices.
An acyclic graph does not contain any cycles

a b c d

e f g h
k, j, n, k, p, o, k
is not simple.
j k l m
103

n o p q
 Definitions: Subgraph
A subgraph H of G
• is a graph;
• its edges and vertices are subsets of those of G.
V(H) = {b, d, e, f, g, h, l, p, q} E(H) = {(b, e), (b, g), (e, f), (d, h), (l, p), (l, q)}

a b c d

e f g h

j k l m
104

n o p q
Directed Graphs
• A graph is directed (digraph) if it contains a finite set of vertices V
and a set A of ordered pair of distinct vertices called arcs or direct
edges. Edge (u,v) notated as uv.
• An undirected edge is treated as two directed edges in opposite
directions
• Outdeg(v) is the number of arcs
beginning at v
• Indeg(v) is the number of arcs
ending at v.
• Each arc begins (source) and ends
(destination) at a vertex.
• Theorem: The sum of the outdegrees of the vertices of a digraph G
105
equals the sum of the indegrees of the vertices, which equals the
number of edges in G.
More Definitions 5
• A simple graph is a graph: 0 -2
• that is undirected 7
• that contains no parallel edges
4
• that contains no loop of length one

• A weighted graph is a graph whose edges are weighted

• A complete graph has every vertex connected to every other vertex

• In a regular graph (k-regular) every vertex has degree k

• A bipartite graph has its vertices partitioned into two subsets M&N
such that each edge of G connects a vertex of M to a vertex of N.
• In a connected graph every vertex is reachable from any other. 106
Representations of Graphs
Two techniques to represent graphs:
• Adjacency matrix
• Adjacency List

Given a graph G(V, E), where V = {1, 2, …, n} , then G is represented

as an n x n adjacency matrix A[I, j] = (aij)

1 if (i, j)  E
aij = { 0 otherwise
Given a graph G(V, E), An adjacency list represents the graph by array
Adj of |V| lists. For each u  V, the adjacency list Adj[u] consists of
all the vertices adjacent to vertex u
107
Adjacency Matrix Example
1 2 3 4 5
1 2 1 0 1 0 0 1
3
2 1 0 1 1 1
3 0 1??
0 1 0
5 4 4 0 1 1 0 1
5 1 1 0 1 0

1 2 3 4 5 6
1 0 1 0 1 0 0
1 2 3 2 0 0 0 0 1 0
3 0 0 0 0 1 1
4 0 1
??
0 0 0 0
108
5 0 0 0 1 0 0
4 5 6 6 0 0 0 0 0 1
Adjacency Matrix Example
1 2 3 4 5
1 2 1 0 1 0 0 1
3
2 1 0 1 1 1
3 0 1 0 1 0
5 4 4 0 1 1 0 1
5 1 1 0 1 0

1 2 3 4 5 6
1 0 1 0 1 0 0
1 2 3 2 0 0 0 0 1 0
3 0 0 0 0 1 1
4 0 1 0 0 0 0
109
5 0 0 0 1 0 0
4 5 6 6 0 0 0 0 0 1
Adjacency List Example
Adj
2 1 2 3 5
2 1 3
1 3
3 1 2 4 5
4 3 5
5 4
5 1 3 4

If G is directed, the total length of all the adjacency lists is |E|.

If G is undirected, the total length is 2|E|.
110
Space requirement: (|V| + |E|).
Adjacency Matrix
• Storage required for adjacency matrix is O(V2)

• In an undirected graph, (u, v) & (v, u) represents the same edge,

adjacency matrix of an undirected graph is its own transpose A = AT

• For undirected graph, needed memory can be cut down by only

saving entries on and above diagonal
• It can be adapted to represent weighted graphs

• The adjacency matrix is a dense representation

• Usually too much storage for large graphs
• But can be very efficient for small graphs

• Preferred when graph is dense 111

• |E| is close to |V|2
Adjacency List
• How much storage is required?
• The degree of a vertex v = # incident edges
• Directed graphs have in-degree, out-degree
• For directed graphs, # of items in adjacency lists is
 out-degree(v) = |E|
takes (V + E) storage
• For undirected graphs, # items in adj lists is
 degree(v) = 2 |E|
also (V + E) storage
• So an Adjacency lists takes O(V+E) storage

• Most large interesting graphs are sparse

• E.g., planar graphs, in which no edges cross, have |E| = O(|V|)
112
• Preferred when graph is sparse
Time and Space Complexity
Operation Adjacency List Adjacency Matrix

Scan incident edges (deg(v)) (|V|)

Scan outgoing edges (outdeg(v)) (|V|)

Scan incoming edges (indeg(v)) (|V|)

Test adjacency (min(deg(u), deg(v)) O(1)

of u and v
Space (|V|+|E|) (|V|2 )
113
Definitions: Trees
• A connected graph T without any cycles/loops is called a Tree

• A unique simple path exists between any two nodes U & V in T

• A group of trees is a Forest

• For an undirected graph G = (V, E)

• If G is connected, then | E | ≥ | V | – 1
• If G is a tree, then | E | = | V | – 1
• If G is a forest, then | E | < | V | – 1.

114
Tree: Node Types
A

Sub-tree
B F K

C H D L X

Q G M I N P

Root (no parent)

Interior/Intermediate nodes (has a parent and at least one child)

115
Leaf nodes (0 children)
Tree: Node Degree (Number of children)
A

B F K

C H D L X

Q G M I N P

Degree 1 Degree 3
116
Degree 0 Degree 2
Tree: Node Level (Distance from the Root)
A

B F K

C H D L X

Q G M I N P

Level 0 Level 2

Level 1 Level 3

117
Height/Depth of the Tree = Maximum Level + 1
 Special Trees: Binary Tree
A A

B K B

C H L X C

Q M P Q

• The binary Tree is unweighted, directed, acyclic graph (DAG)

• Each node's in-degree is at most 1, and out-degree is at most 2
• There is exactly one path from the root to every node
• The maximum number of nodes on level i of a binary tree is 2i 118
• The maximum number of nodes in a binary tree of height k is 2k – 1
Special Trees: Full Binary Tree
• A full binary tree is a tree in which every node other than the
leaves has two children. 

2 3

4 5 6 7

8 9 10 11 12 13 14 15

119
Special Trees: Complete Binary Tree
• A complete binary tree is a binary tree in which every level,
except possibly the last, is completely filled, and all nodes are
as far left as possible. 
1

2 3

4 5 6 7

8 9 10 11 12
• Heap is a complete binary tree
• Height of the tree = lg n 120
• No. of leaves = n/2
Approaches to Algorithms Analysis
(Nested Loops)
i) Top-Down Approach

ii) Bottom-Up Approach

121

121
Example: Top-down vs. Bottom-up
for i  1 to n do
for j  1 to m do
ab

Bottom-up Solution: Inner Loop Top-down Solution:

for j  1 to m do .......... m for i  1 to n do .......... n
ab .......... m for j  1 to m do .......... nm
Inner(m)= m+m = O(m) ab ..........
nm
for i  1 to n do .......... n
Inner(m) .......... m T(n) = n + 2mn
= O(nm)
T(n) = n+Inner(m)(n)  = n+O(m)(n)
= O(n2) if m>=n
= O(nm) 122
= O(n2) if m>=n
122
 Example: LOOP Analysis (Bottom Up Approach)
Step-3: Outer For Loop (Line 1)
n n

T(n) =  for (i ) =  (2i

i 1
2
 3i )
i 1
n n
= 2 i 2
 3 i
i 1 i 1

= 2 [(2n3+3n2+n)/6] + 3 [n(n+1)/2]
= O (n3)
Step-1: Bottom while Loop (line 5 and 6)
j

while(j) = 1 
k 0
j 1 Quadratic Series

Step-2: Inner For

2 i Loop (line 3 and 4)
2i

for(i) =  while ( j )   j  1
j 1 j 1
2i 2i

=
j 1
j  1
j 1
123

123
= (2i(2i+1)/2) + 2i  2i2 +3i