AlgorithmsAnalysis

and Big-O notations

2015 1
 What is an algorithm?

 An algorithm is a finite set of precise

instructions for performing a computation or for
solving a problem.

 AnAlgorithm is step-wise solution to the given

problem.

2015 2
 Algorithms : An Algorithm is any well-defined
computational procedure that takes some value, or
set of values, as input and produces some value, or
set of values, as output.
 An algorithm is thus a sequence of computational
steps that transform the input into the output.
 We can also view an algorithm as a tool for solving a
well-specified computational problem.
 For example, Sorting Problem
 Input: A sequence of n numbers(a1, a2,….,an)
 Output: A permutation (reordering) (a’1,a’2,….a’n) of
the input sequence such that a’1 <=,a’2 <=.a’n
 For example, given the input sequence (31,
41,59,26,41,58), a sorting algorithm returns as
output the sequence (26, 31, 41, 41, 58, 59.)
 An algorithm is said to be Correct if, for every input
instance, it halts with the correct output
2015 3
 Properties of algorithms:

• Input from a specified set,

• Output from a specified set (solution),

• Definiteness of every step in the computation,

• Correctness of output for every possible

input,
• Finiteness of the number of calculation steps,

• Effectiveness of each calculation step and

• Generality for a class of problems.

2015 4
 Wewill use a pseudo code to specify
algorithms, which slightly reminds us of
C/C++/VB/Java and etc.
 Example:an algorithm that finds the maximum
element in a finite sequence

 procedure max(a1, a2, …, an: integers)

 max := a1
 for i := 2 to n
 if max < ai then max := ai
 {max is the largest element}

2015 5
 Another example: a linear search algorithm,
that is, an algorithm that linearly searches a
sequence for a particular element.
 procedure linear_search(x: integer; a1, a2, …,
an: integers)
 i := 1
 while (i  n and x  ai)
 i := i + 1
 if i  n then location := i
 else location := 0
 {location is the subscript of the term that
equals x, or is zero if x is not found}

2015 6
 If
the terms in a sequence are ordered, a binary
search algorithm is more efficient than linear
search.

 Thebinary search algorithm iteratively restricts

the relevant search interval until it closes in on
the position of the element to be located.

2015 7
 binary search for the letter ‘j’

search interval

a c d f g h j l m o p r s u v x z

center element

2015 8
 binary search for the letter ‘j’

search interval

a c d f g h j l m o p r s u v x z

center element

2015 9
 binary search for the letter ‘j’

search interval

a c d f g h j l m o p r s u v x z

center element

2015 10
 binary search for the letter ‘j’

search interval

a c d f g h j l m o p r s u v x z

center element

2015 11
 binary search for the letter ‘j’

search interval

a c d f g h j l m o p r s u v x z

center element

found !
2015 12
The sequential search algorithm is very slow. If we
have an array of 1 million elements, we must do 1
million comparisons in the worst case.

It is suggested that you consider binary searches

whenever the list contains more than 50 elements.
The Binary Search is applied only on sorted List, So if
we want to search an element form the List (Array),
Then we will have to first sort the Array then search it
using the binary search.
Binary search starts by testing the data in the element
at the middle of the array. This determines if the target
is in the first half or the second half of the list.

2015 13
 If
it is in the first half, we do not need to check
the second half. If it is in the second half, we do
not need to check the first half. In other words,
either ways we eliminate half the list from further
consideration. We repeat this process until we
find the target or satisfy ourselves that it is not in
the list.

 To find the middle of the list, we need three

variables, one to identify the beginning of the
list, one to identify the middle of the list, and one
to identify the end of the list.
 We will call our three indexes first, mid and last.
◦ Mid=(first+last)/2

2015 14
 Basic idea: On each step, look at the middle
element of the remaining list to eliminate half
of it, and quickly zero in on the desired
element.

<x <x <x >x

15 2015

Fir mi last Target =85
0 3 7
st d
Index 0 1 2 3 4 5 6 7
23 45 46 58 78 81 85 92

85 > 58
First mid lastTarget =85
4 5 7
Index 0 1 2 3 4 5 6 7
23 45 46 58 78 81 85 92

85 > 81
Firstmid lastTarget =85
6 6 7
Index 0 1 2 3 4 5 6 7
23 45 46 58 78 81 85 92

2015 16
if ( first > last)
return (-1);
mid=(first + last)/2;
if(x== a[mid]
return (mid);
if(x<a[mid])
search for x in a[first] to a[mid-
1];
else
search for x in a[mid+1] to
a[last];
2015 17
 First last Target =85
0 3 7
0 1 2 3 4 5 6 7
23 45 46 58 78 81 85 92

85 > 58
Line 1: if (first > last) ? It is not true, so execute Line 3.
Line 3: Mid=(0+7)/2=3

Line 4: is x==a[3] ? 58!= 85, so execute Line 6.

Line 6: if x<a[3] ? 85<58 it is not true, so per form else
clause at line 8
Repeat the algorithm with first=mid+1=4 and last=7 i.e.
search the upper half of the array.

2015 18
 First mi lastTarget =85
4 5 7
d
0 1 2 3 4 5 6 7
23 45 46 58 78 81 85 92

 Line 1: Is (4> 7) No, so execute line

853.> 81
 Line 3. mid=(4+7)/2=5

 Line 4: Is x==a[5] ? 81 does not equal to 85, so

execute line 6.
 Line 6: Is x< a[5] ? No , Since 85 < 81, so search
for x in a[mid+1] to a[last];
 Line 7: Repeat the algorithm with first=mid+1 and
last=7

2015 19
 Fir mi lastTarget
6 6 7
st d =85
0 1 2 3 4 5 6 7
23 45 46 58 78 81 85 92

Line 1: Is 5> 7 ? No, So execute line 3 85 ==85

Line 3: mid=(6+7)/2=6

Line 4: Since a[6]==85, return mid=6 and the

answer , 85 is indeed the sixth element of the
array.

2015 20
Ingeneral, we are not so much interested in
the time and space complexity for small inputs.

Forexample, while the difference in time

complexity between linear and binary search is
meaningless for a sequence with n = 10, it is
gigantic for n = 230.

2015 21
Forexample, let us assume two algorithms A
and B that solve the same class of problems.
The time complexity of A is 5,000n, the one
for B is 1.1n for an input with n elements.
For n = 10, A requires 50,000 steps, but B
only 3, so B seems to be superior to A.
Forn = 1000, however, A requires 5,000,000
steps, while B requires 2.51041 steps.

2015 22
This means that algorithm B cannot be used
for large inputs, while algorithm A is still
feasible.

Sowhat is important is the growth of the

complexity functions.

The growth of time and space complexity with

increasing input size n is a suitable measure
for the comparison of algorithms.

2015 23
 Comparison: time complexity of algorithms A
and B

Input Size Algorithm A Algorithm B

n 5,000n 1.1n
10 50,000 3
100 500,000 13,781
1,000 5,000,000 2.51041
1,000,000 5109 4.81041392

2015 24
The growth of functions is usually described
using the big-O notation.

Definition:Let f and g be functions from the

integers or the real numbers to the real
numbers.
We say that f(x) is O(g(x)) if there are
constants C and k such that
|f(x)|  C|g(x)|
whenever x > k.

2015 25
 When we analyze the growth of complexity
functions, f(x) and g(x) are always positive.
 Therefore, we can simplify the big-O
requirement to
 f(x)  Cg(x) whenever x > k.

 If
we want to show that f(x) is O(g(x)), we only
need to find one pair (C, k) (which is never
unique).

2015 26
 Theidea behind the big-O notation is to
establish an upper boundary for the growth of a
function f(x) for large x.
 Thisboundary is specified by a function g(x)
that is usually much simpler than f(x).
 We accept the constant C in the requirement
 f(x)  Cg(x) whenever x > k,
 because C does not grow with x.
 We are only interested in large x, so it is OK if
f(x) > Cg(x) for x  k.

2015 27
 Example:

 Show that f(x) = x2 + 2x + 1 is O(x2).

 For x > 1 we have:

 x2 + 2x + 1  x2 + 2x2 + x2
  x2 + 2x + 1  4x2

 Therefore, for C = 4 and k = 1:

 f(x)  Cx2 whenever x > k.

 f(x) is O(x2).

2015 28
 “Popular” functions g(n) are
 n log n, 1, 2n, n2, n!, n, n3, log n

 Listed from slowest to fastest growth:

• 1
• log n
• n
• n log n
• n2
• n3
• 2n
• n!

2015 29
A problem that can be solved with polynomial
worst-case complexity is called
tractable/Traceable.

 Problems of higher complexity are called

intractable/in-Traceable.

 Problemsthat no algorithm can solve are called

unsolvable.

2015 30
 For any polynomial f(x) = anxn + an-1xn-1 + … +
a0, where a0, a1, …, an are real numbers,
 f(x) is O(xn).

 Iff1(x) is O(g1(x)) and f2(x) is O(g2(x)), then

(f1 + f2)(x) is O(max(g1(x), g2(x)))

 Iff1(x) is O(g(x)) and f2(x) is O(g(x)), then

(f1 + f2)(x) is O(g(x)).

 Iff1(x) is O(g1(x)) and f2(x) is O(g2(x)), then

(f1f2)(x) is O(g1(x) g2(x)).

2015 31
What does the following algorithm compute?
procedure who_knows(a1, a2, …, an: integers)
m := 0
for i := 1 to n-1
 for j := i + 1 to n
 if |ai – aj| > m then m := |ai – aj|
{m is the maximum difference between any two
numbers in the input sequence}
Comparisons: n-1 + n-2 + n-3 + … + 1
 = (n – 1)n/2 = 0.5n2 – 0.5n
Time complexity is O(n2).

2015 32
Another algorithm solving the same problem:
procedure max_diff(a1, a2, …, an: integers)
min := a1
max := a1
for i := 2 to n
 if ai < min then min := ai
 else if ai > max then max := ai
m := max - min
Comparisons: 2n - 2

Time complexity is O(n).

2015 33
 To simplify the running time estimation,for
a function f(n), we ignore the constants
and lower order terms.

Example: 10n3+4n2-4n+5 is O(n3).

3
2015 4
More Big-Oh Examples
7n-2
7n-2 is O(n)
need c > 0 and n0  1 such that 7n-2  c•n for n  n0
this is true for c = 7 and n0 = 1

 3n3 + 20n2 + 5
3n3 + 20n2 + 5 is O(n3)
need c > 0 and n0  1 such that 3n3 + 20n2 + 5  c•n3 for n  n0
this is true for c = 4 and n0 = 21
 3 log n + 5
3 log n + 5 is O(log n)
need c > 0 and n0  1 such that 3 log n + 5  c•log n for n  n0
this is true for c = 8 and n0 = 2
3
2015 5
 Growth Rate of Running Time
 Consider a program with time complexity O(n2).
 For the input of size n, it takes 5 seconds.
 If the input size is doubled (2n), then it takes 20
seconds.

 Consider a program with time complexity O(n).

 For the input of size n, it takes 5 seconds.
 If the input size is doubled (2n), then it takes 10
seconds.

 Consider a program with time complexity O(n3).

 For the input of size n, it takes 5 seconds.
 If the input size is doubled (2n), then it takes 40
seconds.
3
2015 6
 Let’s look at the following algorithm
for initializing the values in an array:
final int N = 500;
int [] counts = new int[N];
for (int i=0; i<counts.length; i++)
counts[i] = 0;
 The length of time the algorithm takes to
execute depends on the value of N

3
2015 7
 In that algorithm, we have one loop
that processes all of the elements in
the array
 Intuitively:
◦ If N was half of its value, we would expect the
algorithm to take half the time
◦ If N was twice its value, we would expect the
algorithm to take twice the time
 That is true and we say that the
algorithm efficiency relative to N is
linear 2015
3
8
 Let’s look at another algorithm for initializing the
values in a different array:
final int N = 500;
int [] [] counts = new int[N][N];
for (int i=0; i<counts.length; i++)
for (int j=0; j<counts[i].length; j++)
counts[i][j] = 0;
 The length of time the algorithm takes to execute
still depends on the value of N

3
2015 9
 However, in the second algorithm, we
have two nested loops to process the
elements in the two dimensional array
 Intuitively:
◦ If N is half its value, we would expect the
algorithm to take one quarter the time
◦ If N is twice its value, we would expect the
algorithm to take quadruple the time
 That is true and we say that the
algorithm efficiency relative to N is
quadratic
4
2015 0
 We use a shorthand mathematical notation to
describe the efficiency of an algorithm relative to
any parameter n as its “Order” or Big-O
◦ We can say that the first algorithm is O(n)
◦ We can say that the second algorithm is O(n2)
 For any algorithm that has a function g(n) of the
parameter n that describes its length of time to
execute, we can say the algorithm is O(g(n))
 We only include the fastest growing term and
ignore any multiplying by or adding of constants

4
2015 1
 Eight functions O(n) that occur
frequently in the analysis of
algorithms (in order of increasing rate
of growth relative to n):
◦ Constant  1
◦ Logarithmic  log n
◦ Linear  n
◦ Log Linear  n log n
◦ Quadratic  n2
◦ Cubic  n3
◦ Exponential  2n
◦ Exhaustive Search  n!
4
2015 2
 n=1 n=2 n=4 n=8 n=16 n=32
1 1 1 1 1 1 1
logn 0 1 2 3 4 5
n 1 2 4 8 16 32
nlogn 0 2 8 24 64 160
n2 1 4 16 64 256 1024
n3 1 8 64 512 4096 32768
2n 2 4 16 256 65536 4294967296
n! 1 2 24 40320 20.9T Don’t ask!
4
2015 3
 O(g(n)) for a problem means there is
some O(g(n)) algorithm that solves the
problem
 Don’t assume that the specific algorithm
that you are currently using is the best
solution for the problem
 There may be other correct algorithms
that grow at a smaller rate with
increasing n
 Many times, the goal is to find an 4
2015 4
 That brings up the topic of the structure of
the data on which the algorithm operates
 If we are using an algorithm manually on
some amount of data, we intuitively try to
organize the data in a way that minimizes
the number of steps that we need to take
 Publishers offer dictionaries with the words
listed in alphabetical order to minimize the
length of time it takes us to look up a word

4
2015 5
 We can do the same thing for algorithms
in our computer programs
 Example: Finding a numeric value in a
list
 If we assume that the list is unordered,
we must search from the beginning to
the end
 On average, we will search half the list
 Worst case, we will search the entire list
 Algorithm is O(n), where n is size of 4
2015 6
 Find a match with value in an unordered list
int [] list = {7, 2, 9, 5, 6, 4};

for (int i=0; i<list.length, i++)

if (value == list[i])
statement; // found it
// didn’t find it

4
2015 7
 If we assume that the list is ordered, we can
still search the entire list from the beginning
to the end to determine if we have a match
 But, we do not need to search that way
 Because the values are in numerical order, we
can use a binary search algorithm

 Algorithm is O(log2n), where n is size of array

4
2015 8
 Find a match with value in an ordered list
int [] list = {2, 4, 5, 6, 7, 9};
int min = 0, max = list.length-1;
while (min <= max) {
if (value == list[(min+max)/2])
statement; // found it
else
if (value < list[(min+max)/2])
max = (min+max)/2 - 1;
else
min = (min+max)/2 + 1;
}
statement; // didn’t find it
4
2015 9
 The difference in the structure of the
data between an unordered list and an
ordered list can be used to reduce
algorithm Big-O
 This is the role of data structures and
why we study them
 We need to be as clever in organizing
our data efficiently as we are in figuring
out an algorithm for processing it
efficiently 5
2015 0
 It may take more time to complete one
iteration of the second loop (due to more
code in the body of the loop), but the
growth rate of time taken with increasing
size of the array is less
 As n gets large, the growth rate eventually
dominates the resulting time taken
 That is why we ignore multiplication by or
addition of constants in Big-O notation

5
2015 1
 

Algorithm A Algorithm B
Run time: T = 20000N = O(N) T = 2N2 = O(N2)

■ Answer:

■ Itdepends on the size of the data N.

■ Normally we would say the O(N) algorithm is the better
choice.
■ Setting 20000N = 2N2 gives N=100.
■ So for N<100, Algorithm B is faster.
■ For N>100, Algorithm A is faster.
■ Note that Big O only tells us which is better for large N.

2015 52
 Issue #2: Memory Limitations
■ Question: Which algorithm would you choose?


Algorithm A Algorithm B
Run time: O(N logN) O(N2)
Memory: O(N3) O(N)
■ Answer:

■ It depends on how much memory your computer can

store.
■ Choose Algorithm A only if we have lots of memory
available.
■ Its a general rule in Computer Science that you in
order to make your algorithm faster you have to eat
up more memory.

2015 53


■ Question: Which algorithm would you choose?

Algorithm A Algorithm B
Run time: O(N logN) O(N2)
Programming time: 1 week 1 hour

■ Answer:
■ Itdepends how much time you have to write the program.
■ Another general rule in Computer Science is that the
faster the algorithm is, the more complicated it will be.

2015 54
  Algorithm A Algorithm B

Run time: O(N2N) O(N 2)

Solution: Finds best answer. Finds good answer.

■ Answer:

■ Itdepends which is more important: time or the quality of

the solution.
■ An algorithm that finds a good answer, but not necessarily
the optimal solution, is called an approximation algorithm.
■ Approximation algorithms are generally faster than the
algorithm that is guaranteed to find the best answer.

2015 55
 Suppose a program P is O(n3), and a
program Q is O(3n), and that currently
both can solve problems of size 50 in 1
hour. If the programs are run on
another system that executes exactly
729 times as fast as the original system,
what size problems will they be able
to solve?

2015 56
 n3 = 503 * 729 n = 350 * 729
 n = n = log3 (729 * 350)

3
503 * 3 729 n = log3(729) + log3 350
 n = 50 * 9 n = 6 + log3 350
 n = 50 * 9 = 450 n = 6 + 50 = 56

 Improvement: problem size increased by 9 times

for n3 algorithm but only a slight improvement in
problem size (+6) for exponential algorithm.

2015 57