Algorithm Analysis

Course Introduction

Dr. Aimal Tariq Rextin

Department of Computing

© Algorithms
 Textbook

Introduction to Algorithms
Third Edition

Thomas H. Cormen
Charles E. Leiserson
Ronald L. Rivest
Clifford Stein

The MIT Press

© Algorithms
 Evaluation
5% Assignments
▶ 3–5 assignments
▶ grades added together, thus resulting in a weighted average
▶ In order to deter copying in assignments, I sometimes take quiz from
the assignment on the day of submission.
15% Quizzes
▶ Taken to ensure students study regularly
▶ Can be announced or unannounced.

30% midterm exam

50% final exam

© Algorithms
 Evaluation
5% Assignments
▶ 3–5 assignments
▶ grades added together, thus resulting in a weighted average
▶ In order to deter copying in assignments, I sometimes take quiz from
the assignment on the day of submission.
15% Quizzes
▶ Taken to ensure students study regularly
▶ Can be announced or unannounced.

30% midterm exam

50% final exam

−100% plagiarism penalties

© Algorithms
Plagiarism

© Algorithms
 Plagiarism
A student should never take someone else’s material and present it as his
or her own. Doing so means committing plagiarism.

© Algorithms
 Plagiarism
A student should never take someone else’s material and present it as his
or her own. Doing so means committing plagiarism.

“material” means ideas, words, code, suggestions, corrections on

one’s work, etc.
Using someone else’s material may be appropriate
▶ e.g., software libraries
▶ always clearly identify the external material, and acknowledge its
source. Failing to do so means committing plagiarism.
▶ the work will be evaluated based on its added value

© Algorithms
 Plagiarism
A student should never take someone else’s material and present it as his
or her own. Doing so means committing plagiarism.

“material” means ideas, words, code, suggestions, corrections on

one’s work, etc.
Using someone else’s material may be appropriate
▶ e.g., software libraries
▶ always clearly identify the external material, and acknowledge its
source. Failing to do so means committing plagiarism.
▶ the work will be evaluated based on its added value

Plagiarism on an assignment or an exam will result in

▶ failing that assignment or that exam
▶ loosing one or more points in the final note!

Penalties will be according to extent of plagiarism and serial copiers

may be referred to department for serious action.
© Algorithms
Deadlines

© Algorithms
 Deadlines
Deadlines are firm.

© Algorithms
 Deadlines
Deadlines are firm.

Exceptions may be granted

▶ at the instructor’s discretion
▶ only for documented medical conditions or other documented
emergencies

© Algorithms
 Deadlines
Deadlines are firm.

Exceptions may be granted

▶ at the instructor’s discretion
▶ only for documented medical conditions or other documented
emergencies

Each late day will reduce the assignment’s grade by one third of the
total value of that assignment

© Algorithms
 Deadlines
Deadlines are firm.

Exceptions may be granted

▶ at the instructor’s discretion
▶ only for documented medical conditions or other documented
emergencies

Each late day will reduce the assignment’s grade by one third of the
total value of that assignment
▶ Corollary 1: The grade of an assignment turned in more than two
days late is 0

© Algorithms
 Deadlines
Deadlines are firm.

Exceptions may be granted

▶ at the instructor’s discretion
▶ only for documented medical conditions or other documented
emergencies

Each late day will reduce the assignment’s grade by one third of the
total value of that assignment
▶ Corollary 1: The grade of an assignment turned in more than two
days late is 0
▶ The proof of Corollary 1 is left as an exercise

© Algorithms
Now let’s move on to the real
interesting and fun stuff. . .

© Algorithms
 Automating Arithmetic
Human brain find many things easy that are difficult to implemented on
a computer. While tasks that are difficult for humans are easy on
computers ( arithmetic)
Humans have always been trying to make arithmetic easier and faster

© Algorithms
 Automating Arithmetic
Human brain find many things easy that are difficult to implemented on
a computer. While tasks that are difficult for humans are easy on
computers ( arithmetic)
Humans have always been trying to make arithmetic easier and faster

© Algorithms
Automating Arithmetic

© Algorithms
An Islamic Example

© Algorithms
 Babbage Difference Engine
Charles Babbage made a Mechanical Computer in 1822. It calculated
polynomial equations.

© Algorithms
 Babbage Difference Engine
Charles Babbage made a Mechanical Computer in 1822. It calculated
polynomial equations.

© Algorithms
Rigid versus Flexible Computing

© Algorithms
 Rigid versus Flexible Computing
Rigid Computing Machines are computing machines that can
perform a limited number computations

© Algorithms
 Rigid versus Flexible Computing
Rigid Computing Machines are computing machines that can
perform a limited number computations
Flexible Computing Machines are able to perform unlimited
number of computations. Today’s computer are flexible one can
load and execute various types of programs.

© Algorithms
 Rigid versus Flexible Computing
Rigid Computing Machines are computing machines that can
perform a limited number computations
Flexible Computing Machines are able to perform unlimited
number of computations. Today’s computer are flexible one can
load and execute various types of programs.

Alan Turing in early 1920s proposed first time mathematically how a

flexible computing machine will work. These are called Turing Machines
© Algorithms
 The First Computers Mid 1940s

ENIAC was used to calculate artillery projectiles and feasibility of

thermo-nuclear bombs.
It was reprogrammed by setting the various wires in a new way. This
step took a few days once the ”program” was figured on a paper.
© Algorithms
Digital Computers

© Algorithms
Modern Computers

© Algorithms
The Name and Concept of Algorithms

© Algorithms
The Name and Concept of Algorithms
The decimal numbering system (India,
circa 600 AD)

© Algorithms
The Name and Concept of Algorithms
The decimal numbering system (India,
circa 600 AD)

Persian mathematician Khwārizmı̄ writes

a book (Baghdad, circa 830 AD)
Book name was Aljabar Wal Muqabala:
Aljabar → Algebra

Muhammad ibn Musa

al-Khwārizmı̄

© Algorithms
The Name and Concept of Algorithms
The decimal numbering system (India,
circa 600 AD)

Persian mathematician Khwārizmı̄ writes

a book (Baghdad, circa 830 AD)
Book name was Aljabar Wal Muqabala:
Aljabar → Algebra

▶ methods for adding, multiplying, and

dividing numbers (and more)

Muhammad ibn Musa

al-Khwārizmı̄

© Algorithms
The Name and Concept of Algorithms
The decimal numbering system (India,
circa 600 AD)

Persian mathematician Khwārizmı̄ writes

a book (Baghdad, circa 830 AD)
Book name was Aljabar Wal Muqabala:
Aljabar → Algebra

▶ methods for adding, multiplying, and

dividing numbers (and more)

▶ these procedures were precise,

unambiguous, mechanical, efficient, and
correct
Muhammad ibn Musa
al-Khwārizmı̄

© Algorithms
The Name and Concept of Algorithms
The decimal numbering system (India,
circa 600 AD)

Persian mathematician Khwārizmı̄ writes

a book (Baghdad, circa 830 AD)
Book name was Aljabar Wal Muqabala:
Aljabar → Algebra

▶ methods for adding, multiplying, and

dividing numbers (and more)

▶ these procedures were precise,

unambiguous, mechanical, efficient, and
correct
Muhammad ibn Musa
▶ they were algorithms! al-Khwārizmı̄

© Algorithms
So what are Algorithms?

© Algorithms
 So what are Algorithms?
All software running on the computer is like tape of the turing
machines. They are the programs

© Algorithms
 So what are Algorithms?
All software running on the computer is like tape of the turing
machines. They are the programs
All program are implementations of an algorithm.

© Algorithms
 So what are Algorithms?
All software running on the computer is like tape of the turing
machines. They are the programs
All program are implementations of an algorithm.
One can imagine an algorithm as flow chart we created before we
wrote the program in our first programming courses.

© Algorithms
 So what are Algorithms?
All software running on the computer is like tape of the turing
machines. They are the programs
All program are implementations of an algorithm.
One can imagine an algorithm as flow chart we created before we
wrote the program in our first programming courses.

© Algorithms
Algorithms are
the essence
of computer programs

© Algorithms
Algorithms are the
logic
of computer programs

© Algorithms
 Implementations of Algorithms
Two programs are different implementations of the same algorithms if
anyone of the following changes:

© Algorithms
 Implementations of Algorithms
Two programs are different implementations of the same algorithms if
anyone of the following changes:
The programmer
the hardware running the program
a different programming language Java versus C++

© Algorithms
 Implementations of Algorithms
Two programs are different implementations of the same algorithms if
anyone of the following changes:
The programmer
the hardware running the program
a different programming language Java versus C++
Question: In what aspect way would two implementation of the same
algorithm differ

© Algorithms
 Implementations of Algorithms
Two programs are different implementations of the same algorithms if
anyone of the following changes:
The programmer
the hardware running the program
a different programming language Java versus C++
Question: In what aspect way would two implementation of the same
algorithm differ

Time it takes to execute the program

© Algorithms
 One Problem Many Algorithms
There can be many problems for a single problem

© Algorithms
 One Problem Many Algorithms
There can be many problems for a single problem
Example
Sorting an array of numbers is one of the most common function. There
are various algorithms for it for example insertion sort is one and merge
sort is another

© Algorithms
 One Problem Many Algorithms
There can be many problems for a single problem
Example
Sorting an array of numbers is one of the most common function. There
are various algorithms for it for example insertion sort is one and merge
sort is another
Should we have any preference regarding implementing one another
algorithm ?
Yes Some are better than others

© Algorithms
Better Algorithms?

© Algorithms
 Better Algorithms?
The algorithm that finish faster are better

© Algorithms
 Better Algorithms?
The algorithm that finish faster are better
For small input (e.g. array size) it does not matter. But when the
input size is very very large size this time difference can be
significant.
▶ Example: think about searching the Call Record Database of Ufone

© Algorithms
 Time Complexity
Time an algorithm takes can be estimated by a mathematical expression
of the following form:
=c ×E
T (n)
I

© Algorithms
 Time Complexity
Time an algorithm takes can be estimated by a mathematical expression
of the following form:
=c ×E
T (n)
I
c is a constant like 2, 10, 100 etc.

© Algorithms
 Time Complexity
Time an algorithm takes can be estimated by a mathematical expression
of the following form:
=c ×E
T (n)
I
c is a constant like 2, 10, 100 etc.
it estimates the various implementation specific factors

© Algorithms
 Time Complexity
Time an algorithm takes can be estimated by a mathematical expression
of the following form:
=c ×E
T (n)
I
c is a constant like 2, 10, 100 etc.
it estimates the various implementation specific factors
E is a polynomial expression in terms of the input size, e.g.
▶ n
▶ n2
▶ nlogn
▶ etc
▶ it gives an estimate on the number instructions that need to be
executed
I is Instructions per sec

© Algorithms
 Which is important?
Question
So we now have the following factors that can influence the time taken
by some particular.
The programmer
Hardware
Programming Language
The Algorithm

© Algorithms
 Which is important?
Question
So we now have the following factors that can influence the time taken
by some particular.
The programmer
Hardware
Programming Language
The Algorithm

Question: Which of these factors is most important in performance of

the software?

© Algorithms
Algorithms are most important!

© Algorithms
Lets assume we implement two algorithms insertion sort and merge sort
We need to sort an array of 10 million numbers. i.e

n = 107

© Algorithms
Lets assume we implement two algorithms insertion sort and merge sort
We need to sort an array of 10 million numbers. i.e

n = 107

Insertion Sort Merge Sort

Num of Instructions(E) : n2 n × log n
Constant c=2 Higher with c = 50
CPU Speed(Instructions/sec) 10
10 (faster) 107

Note: Insertion sort is implemented on a computer that is 1000 times

faster than the other computer

© Algorithms
Time Taken

© Algorithms
 Time Taken
Insertion Sort
2
2 × 107
T = 10 = 2 × 104 secs = 20000secs
10 Intruc/sec

or about 5.5 hours

© Algorithms
 Time Taken
Insertion Sort
2
2 × 107
T = 10 = 2 × 104 secs = 20000secs
10 Intruc/sec

or about 5.5 hours

Merge Sort

50 × 107 log2 107

T = = 50 × 23.25 = 1163secs
107 Instruc/sec

or about 0.5 hours

© Algorithms
 Time Taken
Insertion Sort
2
2 × 107
T = 10 = 2 × 104 secs = 20000secs
10 Intruc/sec

or about 5.5 hours

Merge Sort

50 × 107 log2 107

T = = 50 × 23.25 = 1163secs
107 Instruc/sec

or about 0.5 hours

Conclusion
Algorithm is more important at large input sizes

© Algorithms
 Time Taken
Insertion Sort
2
2 × 107
T = 10 = 2 × 104 secs = 20000secs
10 Intruc/sec

or about 5.5 hours

Merge Sort

50 × 107 log2 107

T = = 50 × 23.25 = 1163secs
107 Instruc/sec

or about 0.5 hours

Conclusion
Algorithm is more important at large input sizes

This example is covered in Section 1.2 of the textbook

© Algorithms
 Example of Algorithm
A sequence of numbers

0, 1, 1, 2, 3, 5, 8, 13, 21, 34, . . .

© Algorithms
 Example of Algorithm
A sequence of numbers

0, 1, 1, 2, 3, 5, 8, 13, 21, 34, . . .

The well-known Fibonacci sequence

Leonardo da Pisa (ca. 1170–ca. 1250)

son of Guglielmo “Bonaccio”
a.k.a. Leonardo Fibonacci

© Algorithms
 The Fibonacci Sequence

0
 if n = 0
Mathematical definition: Fn = 1 if n = 1

Fn−1 + Fn−2 if n > 1


© Algorithms
 The Fibonacci Sequence

0
 if n = 0
Mathematical definition: Fn = 1 if n = 1

Fn−1 + Fn−2 if n > 1


Implementation on a computer:
Scheme
(define (F n)
(cond
((= n 0) 0)
((= n 1) 1)
(else (+ (F (- n 1)) (F (- n 2))))))

© Algorithms
 The Fibonacci Sequence

0
 if n = 0
Mathematical definition: Fn = 1 if n = 1

Fn−1 + Fn−2 if n > 1


Implementation on a computer:
Java
public class Fibonacci {
public static int F(int n) {
if (n == 0) {
return 0;
} else if (n == 1) {
return 1;
} else {
return F(n-1) + F(n-2);
} }
}

© Algorithms
 The Fibonacci Sequence

0
 if n = 0
Mathematical definition: Fn = 1 if n = 1

Fn−1 + Fn−2 if n > 1


Implementation on a computer:
C or C++
int F(int n) {
if (n == 0) {
return 0;
} else if (n == 1) {
return 1;
} else {
return F(n-1) + F(n-2);
}
}

© Algorithms
 The Fibonacci Sequence

0
 if n = 0
Mathematical definition: Fn = 1 if n = 1

Fn−1 + Fn−2 if n > 1


Implementation on a computer:
Ruby
def F(n)
case n
when 0
return 0
when 1
return 1
else
return F(n-1) + F(n-2)
end
end
© Algorithms
 The Fibonacci Sequence

0
 if n = 0
Mathematical definition: Fn = 1 if n = 1

Fn−1 + Fn−2 if n > 1


Implementation on a computer:
Python
def F(n):
if n == 0:
return 0
elif n == 1:
return 1
else:
return F(n-1) + F(n-2)

© Algorithms
 The Fibonacci Sequence

0
 if n = 0
Mathematical definition: Fn = 1 if n = 1

Fn−1 + Fn−2 if n > 1


Implementation on a computer:

very concise C/C++ (or Java)

int F(int n) { return (n<2)?n:F(n-1)+F(n-2); }

© Algorithms
 The Fibonacci Sequence

0
 if n = 0
Mathematical definition: Fn = 1 if n = 1

Fn−1 + Fn−2 if n > 1


Implementation on a computer:
“pseudo-code”
Fibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else return Fibonacci(n − 1) + Fibonacci(n − 2)

© Algorithms
How The Algorithm Works?

© Algorithms
 Questions on Our First Algorithm

Fibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else return Fibonacci(n − 1) + Fibonacci(n − 2)

© Algorithms
 Questions on Our First Algorithm

Fibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else return Fibonacci(n − 1) + Fibonacci(n − 2)

1. Is the algorithm correct?

▶ for every valid input, does it terminate?
▶ if so, does it do the right thing?

© Algorithms
 Questions on Our First Algorithm

Fibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else return Fibonacci(n − 1) + Fibonacci(n − 2)

1. Is the algorithm correct?

▶ for every valid input, does it terminate?
▶ if so, does it do the right thing?

2. How much time does it take to complete?

© Algorithms
 Questions on Our First Algorithm

Fibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else return Fibonacci(n − 1) + Fibonacci(n − 2)

1. Is the algorithm correct?

▶ for every valid input, does it terminate?
▶ if so, does it do the right thing?

2. How much time does it take to complete?

3. Can we do better?
© Algorithms
Correctness

© Algorithms
 Correctness

Fibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else return Fibonacci(n − 1) + Fibonacci(n − 2)


0
 if n = 0
Fn = 1 if n = 1

Fn−1 + Fn−2 if n > 1


© Algorithms
 Correctness

Fibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else return Fibonacci(n − 1) + Fibonacci(n − 2)


0
 if n = 0
Fn = 1 if n = 1

Fn−1 + Fn−2 if n > 1


The algorithm is clearly correct

▶ assuming n ≥ 0
© Algorithms
 Performance
How long does it take?

© Algorithms
 Performance
How long does it take?

Let’s try it out. . .

© Algorithms
 Results

60
Ruby
Java
running time (seconds)

Python
40 C
C-wiz
C-gcc

20

0
20 25 30 35 40 45 50
n

© Algorithms
Comments

© Algorithms
 Comments
Different implementations perform differently
▶ it is better to let the compiler do the optimization
▶ simple language tricks don’t seem to pay off

© Algorithms
 Comments
Different implementations perform differently
▶ it is better to let the compiler do the optimization
▶ simple language tricks don’t seem to pay off

However, the differences do not seem to be substantial

▶ all implementations sooner or later seem to hit a wall. . .

© Algorithms
 Comments
Different implementations perform differently
▶ it is better to let the compiler do the optimization
▶ simple language tricks don’t seem to pay off

However, the differences do not seem to be substantial

▶ all implementations sooner or later seem to hit a wall. . .

Conclusion: the problem is with the algorithm

© Algorithms
Comments

© Algorithms
 Comments
Different implementations perform differently
▶ it is better to let the compiler do the optimization
▶ simple language tricks don’t seem to pay off

© Algorithms
 Comments
Different implementations perform differently
▶ it is better to let the compiler do the optimization
▶ simple language tricks don’t seem to pay off

However, the differences do not seem to be substantial

▶ all implementations sooner or later seem to hit a wall. . .

© Algorithms
 Comments
Different implementations perform differently
▶ it is better to let the compiler do the optimization
▶ simple language tricks don’t seem to pay off

However, the differences do not seem to be substantial

▶ all implementations sooner or later seem to hit a wall. . .

Conclusion: the problem is with the algorithm

© Algorithms
 A Better Algorithm
Again, the sequence is 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, . . .

© Algorithms
 A Better Algorithm
Again, the sequence is 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, . . .

Idea: we can build Fn from the ground up!

© Algorithms
 A Better Algorithm
Again, the sequence is 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, . . .

Idea: we can build Fn from the ground up!

SmartFibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else pprev = 0
6 prev = 1
7 for i = 2 to n
8 f = prev + pprev
9 pprev = prev
10 prev = f
11 return f
© Algorithms
 Results

60
Ruby
Scheme
running time (seconds)

Python
40 C
C-wiz
Java
C-gcc
(Python) SmartFibonacci
20

0
20 40 60 80 100 120 140 160 180 200
n

© Algorithms