CS203: Analysis & Design of Algorithms

Dr. Amr A.HAssanain

Lecture# 02 : Algorithm Analysis

1
Computation Model for Analysis
• To analyze an algorithm is to determine the amount of resources
necessary to execute it. These resources include computational
time, memory and communication bandwidth.

• Analysis of the algorithm is performed with respect to a

computational model called RAM (Random Access Machine)

• A RAM is an idealized uni-processor machine with an infinite large

random-access memory
• Instruction are executed one by one
• All memory equally expensive to access
• No concurrent operations
• Constant word size 2

• All reasonable instructions (basic operations) take unit time

Complexity of an Algorithm
• The complexity of an algorithm is the amount of work the algorithm
performs to complete its task. It is the level of difficulty in solving
mathematically posed problems as measured by:
• Time (time complexity)
• No. of steps or arithmetic operations (computational complexity)
• Memory space required (space complexity)

• Complexity is a function T(n) which yields the time (or space)

required to execute the algorithm of a problem of size ‘n’.

3
Pseudocode
• High-level description of an algorithm
• More structured than English prose but Less detailed than a
program
• Preferred notation for describing algorithms
• Hides program design issues

ArrayMax(A, n)
Input: Array A of n integers
Output: maximum element of A

1. currentMax  A[0];
2. for i = 1 to n-1 do
3. if A[i] > currentMax then 4
4. currentMax  A[i]
5. return currentMax;
Pseudocode
• Indentation indicates block structure. e.g body of loop
• Looping Constructs while, for and the conditional if-then-else
• The symbol // indicates that the reminder of the line is a comment.
• Arithmetic & logical expressions: (+, -,*,/, ) (and, or and not)
• Assignment & swap statements: a b , ab c, a → b
• Return/Exit/End: termination of an algorithm or block
ArrayMax(A, n)
Input: Array A of n integers
Output: maximum element of A

1. currentMax  A[0];
2. for i = 1 to n-1 do
3. if A[i] > currentMax then 5
4. currentMax  A[i]
5. return currentMax;
Pseudocode
• Local variables mostly used unless global variable explicitly defined
• If A is a structure then |A| is size of structure. If A is an Array then
n =length[A], upper bound of array. All Array elements are
accessed by name followed by index in square brackets A[i].
• Parameters are passed to a procedure by values
• Semicolons used for multiple short statement written on one line

ArrayMax(A, n)
Input: Array A of n integers
Output: maximum element of A

1. currentMax  A[0]
2. for i = 1 to n-1 do
3. if A[i] > currentMax then 6
4. currentMax  A[i]
5. return currentMax
Elementary Operations
• An elementary operation is an operation which takes constant time
regardless of problem size.
• The running time of an algorithm on a particular input is
determined by the number of “Elementary Operations” executed.
• Theoretical analysis on paper from a description of an algorithm

• Defining elementary operations is a little trickier than it appears

• We want elementary operations to be relatively machine and language
independent, but still as informative and easy to define as possible

• Example of elementary operations include

• variable assignment
• arithmetic operations (+, -, x, /) on integers
• comparison operations (a < b)
• boolean operations
• accessing an element of an array 7

• We will measure number of steps taken in term of size of input

Components of an Algorithm
• Variables and values
• Instructions
• Sequences
• Selections
• Repetitions

8
Instruction and Sequence
• A linear sequence of elementary operations is also performed in
constant time.
• More generally, given two program fragments P1 and P2 which run
sequentially in times t1 and t2
• use the maximum rule which states that the larger time dominates
• complexity will be max(t1,t2)

e.g. Assignment Statements

x=a .....................1
x= a+b*c/h-u ......1 T(n) = 1+1+1
a>b ......................1 T(n) = 3

9
Sequences
• Analysing a group of consecutive statements
• The statement taking the maximum time will be the one counted
• use the maximum rule

Block #1 t1

T(n) = max(t1,t2)
Block #2 t2

• e.g. a fragment with single for-loop followed by double for- loop

• T(n) = n2 10
• Always analyze function calls first
Selection
• If <test> then P1 else P2 structures are a little harder;
conditional loops.
• The maximum rule can be applied here too:
• max(t1, t2), assuming t1, t2 are times of P1, P2
• However, the maximum rule may prove too conservative
• if <test> is always true the time is t1
• if <test> is always false the time is t2

e.g. if (a>b) then ...........1

a=2......................1
b=b+3*t .............1
else
x=x+2*3 ................1
Block #1 t1 Block #2 t2 Max(t1,t2)
11
T(n)= 1+max(2,1) = 3
Repetition (Loops)
• Analyzing loops: Any loop has two parts:
• How many iterations are performed?
• How many steps per iteration?

for i = 1 to n do
P(i);

• Assume that P(i) takes time t, where t is independent of i

• Running time of a for-loop is at most the running time of the
statements inside the for-loop times number of iterations

T(n) = nt
12
• This approach is reasonable, provided n is positive
• If n is zero or negative the relationship T(n) = nt is not valid
Repetition (Loops)
Analysing Nested Loops

for i = 0 to n do
for j = 0 to m do
P(j);

• Assume that P(j) takes time t, where t is independent of i and j

• Start with outer loop:
• How many iterations? n
• How much time per iteration? Need to evaluate inner loop

• Analyze inside-out. Total running time is running time of the

statement multiplied by product of the sizes of all the for-loops
13
T(n) = nmt
Repetition (Loops)
Analysing Nested Loops

for i = 0 to n do
for j = 0 to i do
P(j);

• Assume that P(j) takes time t, where t is independent of i and j

• How do we analyze the running time of an algorithm that has
complex nested loop?
• The answer is we write out the loops as summations and then
solve the summations. To convert loops into summations, we
work from inside-out.

i =0 ri + t 
n n
T(n) = n + i =0 i
r 14

= n + n(n+1)/2 + tn(n+1)/2
 Analysis Example
Algorithm: Number of times executed
1. n = read input from user 1
2. sum = 0 1
3. i = 0 1
4. while i < n n
n or i =0 1
n −1
5. number = read input from user
n or i =0 1
n −1
6. sum = sum + number
7. i=i+1
8. mean = sum / n

n or n −11
1 i =0

The computing time for this algorithm in terms on input size n is:
T(n) = 1 + 1 + 1 + n + n + n + n + 1 15

T(n) = 4n + 4
Another Analysis Example
i=1 ...............................1
while (i < n)................n-1
a=2+g...............n-1
i=i+1 ................n-1
if (i<=n).......................1
a=2 ....................1
else
a=3.....................1

T(n) = 1 + 3(n-1) + 1 + 1
16
=3n
Another Analysis Example
i=1...............................? 1
while (i<=10)................? 10
i=i+1...................? 10
i=1 ...............................? 1
while (i<=n)..................? n
a=2+g .................? n
i=i+1 ...................? n
if (i<=n)........................ ? 1
a=2 .................... ? 1
else
a=3..................... ? 1

17
T ( n) = ? T ( n) = 3n +24
Asymptotic Growth Rate
• Changing the hardware/software environment
• Affects T(n) by constant factor, but does not alter the growth rate of T(n)

• Algorithm complexity is usually very complex. The growth of the

complexity functions is what is more important for the analysis and
is a suitable measure for the comparison of algorithms with
increasing input size n.
• Asymptotic notations like big-O, big-Omega, and big-Theta are used
to compute the complexity because different implementations of
algorithm may differ in efficiency.
• The big-Oh notation gives an upper bound on the growth rate of a
function.
• The statement “f(n) is O(g(n))” means that the growth rate of f(n) is
no more than the growth rate of g(n). 18

• We can use the big-Oh notation to rank functions according to their

growth rate.
Asymptotic Growth Rate
Two reasons why we are interested in asymptotic growth rates:

• Practical purposes: For large problems, when we expect to have

big computational requirements

• Theoretical purposes: concentrating on growth rates frees us

from some important issues:
• fixed costs (e.g. switching the computer on!), which may
dominate for a small problem size but be largely irrelevant
• machine and implementation details
• The growth rate will be a compact and easy to understand the
function
19
Properties of Growth-Rate Functions
Example: 5n + 3
Estimated running time for different values of n:
n = 10 => 53 steps
n = 100 => 503 steps
n = 1,000 => 5003 steps
n = 1,000,000 => 5,000,003 steps

As “n” grows, the number of steps grow in linear proportion to n for

this function “Sum”

What about the “+3” and “5” in 5n+3?

As n gets large, the +3 becomes insignificant
5 is inaccurate, as different operations require varying amounts of
time and also does not have any significant importance 20

What is fundamental is that the time is linear in n.

Asymptotic Algorithm Analysis
• The asymptotic analysis of an algorithm determines the running
time in big-Oh notation
• To perform the asymptotic analysis
• We find the worst-case number of primitive operations executed as a
function of the input size n
• We express this function with big-Oh notation

• Example: An algorithm executes T(n) = 2n2 + n elementary

operations. We say that the algorithm runs in O(n2) time

• Growth rate is not affected by constant factors or lower-order terms

so these terms can be dropped
• The 2n2 + n time bound is said to "grow asymptotically" like n2
• This gives us an approximation of the complexity of the algorithm 21
• Ignores lots of (machine dependent) details
Algorithm Efficiency
Measuring efficiency of an algorithm
• do its analysis i.e. growth rate.
• Compare efficiencies of different algorithms for the same problem.

As inputs get larger, any algorithm of a smaller order will be more

efficient than an algorithm of a larger order

0.05 N2 = O(N2)

3N = O(N)
Time (steps)

22

Input (size)
N = 60
Important Functions
These functions often appear in algorithm analysis:

23
A comparison of Growth-Rate Functions
Size does Matter:
What happens if we increase the input size N?

24
 A comparison of Growth-Rate Functions

250

f(n) = n
f(n) = log(n)
f(n) = n log(n)
f(n) = n^2
f(n) = n^3
f(n) = 2^n

0
25
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
 A comparison of Growth-Rate Functions

500

f(n) = n
f(n) = log(n)
f(n) = n log(n)
f(n) = n^2
f(n) = n^3
f(n) = 2^n

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 26
 A comparison of Growth-Rate Functions

1000

f(n) = n
f(n) = log(n)
f(n) = n log(n)
f(n) = n^2
f(n) = n^3
f(n) = 2^n

0
1 3 5 7 9 11 13 15 17 19 27
 A comparison of Growth-Rate Functions

5000

4000
f(n) = n
f(n) = log(n)
3000
f(n) = n log(n)
f(n) = n^2
2000 f(n) = n^3
f(n) = 2^n

1000

0
28
1

11

13

15

17

19
 A comparison of Growth-Rate Functions

10000000

1000000

100000

10000

1000

100

10

1
1 4 16 64 256 1024 4096 16384 65536 29
Performance Classification
f(n) Classification

1
:
Constant run time is fixed, and does not depend upon n. Most instructions are
executed once, or only a few times, regardless of the amount of information being processed

log n Logarithmic: when n increases, so does run time, but much slower. Common in
programs which solve large problems by transforming them into smaller problems.

n Linear: run time varies directly with n. Typically, a small amount of processing is done on
each element.
n log n When n doubles, run time slightly more than doubles. Common in programs
which break a problem down into smaller sub-problems, solves them independently, then combines
solutions
n2 Quadratic: when n doubles, runtime increases fourfold. Practical only for small
problems; typically the program processes all pairs of input (e.g. in a double nested loop).

n3 Cubic: when n doubles, runtime increases eightfold

2n
30
Exponential: when n doubles, run time squares. This is often the result of a natural, “brute
force” solution.
Running Time vs. Time Complexity
• Running time is how long it takes a program to run.

• Time complexity is a description of the asymptotic behavior of

running time as input size tends to infinity.

• The exact running time might be 2036.n2 + 17453.n + 18464 but

you can say that the running time "is" O(n2), because that's the
formal(idiomatic) way to describe complexity classes and big-O
notation.

• Infact, the running time is not a complexity class, IT'S EITHER A

DURATION, OR A FUNCTION WHICH GIVES YOU THE DURATION.
"Being O(n2)" is a mathematical property of that function, not a 31
full characterization of it.
Model of Computation
Drawbacks:
• poor assumption that each basic operation takes constant time
• Adding, Multiplying, Comparing etc.

Finally what about Our Model?

• With all these weaknesses, our model is not so bad because
• We have to give the comparison, not absolute analysis of any algorithm.
• We have to deal with large inputs not with the small size

• Model seems to work well describing computational power of

modern nonparallel machines

Can we do Exact Measure of Efficiency ?

• Exact, not asymptotic, measure of efficiency can be sometimes 32
computed but it usually requires certain assumptions concerning
implementation
Complexity Examples
What does the following algorithm compute?

procedure who_knows(a1, a2, …, an: integers)

m := 0
for i := 1 to n-1
for j := i + 1 to n
if |ai – aj| > m then m := |ai – aj|

{m is the maximum difference between any two numbers in the

input sequence}

Comparisons: n-1 + n-2 + n-3 + … + 1

= n*(n – 1)/2 = 0.5n2 – 0.5n
33
Time complexity is O(n2).
Complexity Examples
Another algorithm solving the same problem:

procedure max_diff(a1, a2, …, an: integers)

min := a1
max := a1
for i := 2 to n
if ai < min then min := ai
else if ai > max then max := ai
m := max - min

Comparisons: 2n + 2
Time complexity is O(n).
34