Algorithms

Analysis

Data Structures
Problem Solving
Main Steps:
1. Problem definition
2. Algorithm design / Algorithm
specification
3. Algorithm analysis
4. Implementation
5. Testing
6. [Maintenance]
1. Problem Definition
 What is the task to be
accomplished?
 Calculate the average of the grades for
a given student
 What are the time / space / speed /
performance requirements ?
 2. Algorithm Design /
Specifications
 Algorithm: Finite set of instructions that, if
followed, accomplishes a particular task.
 Describe: in natural language / pseudo-code /
diagrams / etc.
 Criteria to follow:
 Input: Zero or more quantities
 Output: One or more quantities
 Definiteness: Clarity, precision of each instruction
 Finiteness: The algorithm has to stop after a finite
(may be very large) number of steps
 Effectiveness: Each instruction has to be basic
enough and feasible
Computer Algorithm
A procedure (a finite set of well-defined
instructions) for accomplishing some
tasks which,
 given an initial state
 terminate in a defined end-state

The computational complexity and

efficient implementation of the
algorithm are important in computing,
and this depends on suitable data
structures.
4,5,6: Implementation,
Testing, Maintenance
 Implementation
 Decide on the programming language to use
• C, C++, Lisp, Java, Perl, Prolog, assembly, etc. , etc.
 Write clean, well documented code

 Test, test, test

 Integrate feedback from users, fix bugs, ensure

compatibility across different versions 
Maintenance
3. Algorithm Analysis
 Space complexity
 How much space is required
 Time complexity
 How much time does it take to run the
algorithm

Often, we deal with estimates!

Space Complexity
 Space complexity = The amount
of memory required by an algorithm
to run to completion
 Some algorithms may be more
efficient if data completely loaded
into memory
 Need to look also at system limitations
 E.g. Sorting records of a 2GB file. can I
afford to load the entire file into
memory?
Time Complexity
 Often more important than space complexity
 space available (for computer programs!) tends to be
larger and larger
 time is still a problem for all of us

 3-4GHz processors on the market

 still …
 Some problems would take years to run to completion
 Algorithms running time is an important issue
Running time
5 ms worst-case
4 ms

3 ms
}
average-case?
best-case
2 ms

1 ms

A B C D E F G
Input
Suppose the program includes an if-then statement that
may execute or not:  variable running time

Typically algorithms are measured by their worst case

 Running time
 The running time of an algorithm best case
varies with the inputs, and
typically grows with the size of average case
the inputs. worst case
120
 To evaluate an algorithm or to 100
compare two algorithms, we focus
on their relative rates of growth

Running Time
80
wrt the increase of the input size.
60
 The average running time is
difficult to determine. 40

20
 We focus on the worst case
running time 0
 Easier to analyze 1000 2000 3000 4000
Input Size

09:20 AM
 Algorithm analysis:
Experimental Approach
 Write a program to 9000
implement the algorithm.
8000

 Run this program with 7000

inputs of varying size and 6000

Time (ms)
composition.
5000

4000
 Get an accurate measure
of the actual running time 3000
(e.g. system call date). 2000

1000
 Plot the results.
0
 Problems?
0 50 100
Input Size

09:20 AM
Limitations of Experimental
Approach

 The algorithm has to be

implemented, which may take a long
time and could be very difficult.
 Results may not be indicative for the
running time on other inputs that are
not included in the experiments.
 In order to compare two algorithms,
the same hardware and software
must be used.
Algorithm analysis:
Theoretical Approach
 Based on high-level description of
the algorithms, rather than language
dependent implementations
 Makes possible an evaluation of the
algorithms that is independent of
the hardware and software
environments
 Generality
 Algorithm Description
Pseudocode
Example: find the max element of
 High-level
description of an an array
algorithm.
Algorithm arrayMax(A, n)
 More structured than Input array A of n integers
plain English. Output maximum element of
A
 Less detailed than a
program. currentMax  A[0]
 Preferred notation for i  1 to n  1 do
for describing if A[i]  currentMax
algorithms. then
currentMax
 A[i]
return currentMax
 Pseudocode
 Control flow  Method call
 if … then … [else …] var.method (arg [, arg…])
 while … do …
 repeat … until …  Return value
 for … do … return expression
 Indentation replaces
braces  Expressions
¬ Assignment (equivalent
 Method declaration to )
Algorithm method (arg [, arg…])
Input … = Equality testing
(equivalent to )
Output …
n2 Superscripts and other
mathematical
formatting allowed
 Primitive Operations
 The basic computations  Examples:
performed by an algorithm  Evaluating an
expression
 Identifiable in pseudocode
 Assigning a value
 Largely independent from the to a variable
programming language
 Calling a method
 Exact definition not important  Returning from a
 Use comments method
 Instructions have to be basic
enough and feasible!
Low-Level Algorithm Analysis
 Based on primitive operations (low-level
computations independent from the
programming language)
 E.g.:
 Make an addition = 1 operation
 Calling a method or returning from a method
= 1 operation
 Index in an array = 1 operation
 Comparison = 1 operation etc.
 Method: Inspect the pseudo-code and
count the number of primitive operations
executed by the algorithm
Counting Primitive Operations
 By inspecting the code, we can determine the
number of primitive operations executed by
an algorithm, as a function of the input size.

Algorithm arrayMax(A, n) # operations

currentMax  A[0] 2
for i  1 to n  1 do 2+n
if A[i]  currentMax then 2(n  1)
currentMax  A[i] 2(n  1)
increment counter i 2(n  1)
return currentMax 1 .
Total 7n
1
 Estimating Running Time
 Algorithm arrayMax executes 7n  1 primitive
operations.
 Let’s define
a:= Time taken by the fastest primitive operation
b:= Time taken by the slowest primitive operation

 Let T(n) be the actual running time of arrayMax.

We have
a (7n  1)  T(n)  b(7n  1)
 Therefore, the running time T(n) is bounded by
two linear functions.
Growth Rate of Running Time
 Changing computer hardware /
software
 Affects T(n) by a constant factor
 Does not alter the growth rate of T(n)

 The linear growth rate of the

running time T(n) is a property of
algorithm arrayMax
Growth Rate
 On a graph, as
you go to the

Value of function 
fA(n)=30n+8
right, a faster
growing
function
fB(n)=n2+1
eventually
becomes
larger... Increasing n 
 Constant Factors
 The growth rate is not affected by
 constant factors or
 lower-order terms

 Examples
 102n + 105 is a linear function
 103n2 + 105n is a quadratic function
Understanding Rate of Growth
 Consider the example of buying elephants
and a fish:

Cost: (cost_of_elephants) + (cost_of_fish)

or Cost ~ cost_of_elephants (approximation)

 The low order terms of a function are relatively

insignificant for large n

n4 + 100n2 + 10n + 50  n4

 The highest order term of a function

determines its rate of growth!
Example
 A website to process user data (e.g.,
financial records).
 To process n records:
 program A takes fA(n)=30n+8 microseconds
 program B takes fB(n)=n2+1 microseconds

 Which program would you choose,

knowing you’ll want to support millions of
users?
Compare rates of growth
(30n + 8)  n (n2 + 1)  n2
Algorithm Analysis
 We know:
 Experimental approach – problems
 Low level analysis – count operations

 Characterize an algorithm as a
function of the “problem size”
 E.g.
 Input data = array  problem size is N
(length of array)
 Input data = matrix  problem size is N x M
Asymptotic Notation
 A way to describe behavior of functions
 Abstracts away low-order terms and constant
factors
 How we indicate running times of algorithms
 Describe the running time of an algorithm as
n grows to 
 Asymptotic Notation
 Goal: to simplify analysis by getting
rid of unneeded information
 We want to say in a formal way 3n2
≈ n2
 The “Big-Oh” Notation:
 given functions f(n) and g(n), we say
that f(n) is O(g(n)) if and only if there
are positive constants c and n0 such
that f(n) ≤ c g(n) for n ≥ n0
 Graphic Illustration
 f(n) = 2n+6
 According to definition: c g(n) = 4n
 Need to find a
function g(n) and a
const. c and a f(n) = 2n + 6
constant n0 such as
f(n) ≤ cg(n) when n
≥ n0
 g(n) = n and c = 4 g(n) = n
and n0=3
  f(n) is O(n)
 The order of f(n) is n
 2n+6<=cn
 cn-2n>=6
 n(c-2)>=6
n0
n
 n>=6/c-2
 More examples
 O(g(n)) = the set of functions with a
smaller or same order of growth as g(n)
 50n3 + 20n + 4 is O(n3)
 Would be correct to say is O(n3+n) ?
• Not useful, as n3 exceeds by far n, for large
values
 Would be correct to say is O(n5)
• OK, but g(n) should be as close as possible to
f(n)
 3log(n) + log (log (n)) = O( ? )
• Simple Rule: Drop lower order terms and constant factors
Big-Oh and Growth Rate
 The big-Oh notation gives an upper
bound on the growth rate of a function.
 The statement “f (n) is O(g (n))” means
that the growth rate of f (n) is no more
than the growth rate of g (n).
 We can use the big-Oh notation to rank
functions according to their growth rate.
Big-Oh Rules
 If f (n) is a polynomial of degree d, then f
(n) is O(nd), i.e.,
1. Drop lower-order terms
2. Drop constant factors

 Use the smallest possible class of

functions
 Say “2n is O(n)” instead of “2n is O(n 2)”

 Use the simplest expression of the class

 Say “3n + 5 is O(n)” instead of “3n + 5 is
O(3n)”
 Properties of Big-Oh
 log nx is O(log n), for x > 0 – how?
 nx vs. an, for any fixed x > 0 and a > 1
 An algorithm of order n to a certain power is
better than an algorithm of order a ( > 1) to the
power of n
 logx n vs. ny, for x > 0 and y > 0
 An algorithm of order log n (to a certain power) is
better than an algorithm of n raised to a power y.
Asymptotic analysis -
terminology
 Special classes of algorithms:
logarithmic: O(log n)
linear: O(n)
quadratic: O(n2)
polynomial: O(nk), k ≥ 1
exponential: O(an), n > 1, a > 1
 Polynomial vs. exponential ?
 Logarithmic vs. polynomial ?
 Some Numbers
2 3 n
log n n n log n n n 2
0 1 0 1 1 2
1 2 2 4 8 4
2 4 8 16 64 16
3 8 24 64 512 256
4 16 64 256 4096 65536
5 32 160 1024 32768 4294967296
More Examples …
 n4 + 100n2 + 10n + 50 is of the
order of O(n4)
 10n3 + 2n2 is O(n3)
 n3 - n2 is O(n3)
 10 is O(1),
 1270 is O(1)
Running Time Calculations
General Rules
1. FOR loop
• The number of iterations times the time of the inside
statements.
2. Nested loops
• The product of the number of iterations times the
time of the inside statements.
3. Consecutive Statements
• The sum of running time of each segment.
4. If/Else
• The testing time plus the larger running time
of the cases.
 Some Examples
Case1: for (i=0; i<n; i++)
for (j=0; j<n; j++) O(n2)
k++;

Case 2: for (i=0; i<n; i++)

k++;
for (i=0; i<n; i++) O(n2)
for (j=0; j<n; j++)
k++;

Case 3: for (int i=0; i<n-1; i++)

for (int j=0; j<i; j++) O(n2)
int k+=1;
Some Math
 properties of logarithms:
logb(xy) = logbx + logby
logb (x/y) = logbx - logby
logbxa= alogbx
logba= logxa/logxb
 properties of exponentials:
a(b+c) = abac abc = (ab)c
ab /ac = a(b-c)
b= alogab bc = ac*logab
Problem:
 Order the following functions by
their asymptotic growth rates
 nlogn
 logn3
 n2
 2logn
 log(logn)
 Sqr(logn)