Data Structures and Algorithms

CHAPTER 2
ANALYSIS OF ALGORITHM

1
MAIN CONTENTS

1 About the algorithms

2 Types of algorithm

3 Time complexity

4 Space complexity

5 Asymptotic notations

2
1. Algorithms

• Algorithms are an essential aspect of data structures.

Data structures are implemented by using algorithms
• Algorithm is a finite sequence of instructions/steps, each
of which is very elementary that must be followed to
solve a problem.
• We can choose any programming language to
implement the Algorithm
• Pseudocode and flowchart are popular ways to
represent an algorithm.

3
 Algorithms (cont.) The result should be
produced after
completion of the job

input Algorithms Output

- The necessary data to

perform algorithms
- Input is externally supplied

4
Algorithms - Criteria
• Definiteness: Each step must be clear, unambiguous, and
precisely defined
• Finiteness: Algorithm should be terminated after a finite
number of steps. Also, each step should be finished in a finite
amount of time.
• Effectiveness: Each step of the algorithm must be feasible
Every instruction must be elementary.

Note: Algorithm should have a step-by-step direction of each

level, which is independent of programming language.

5
2. Types of algorithms

• Divide and conquer

• Greedy Method
• Backtracking
• Branch and Bound
• Dynamic programming
• Deterministic or non-deterministic algorithm
• Serial or parallel or distributed algorithm

6
Types of algorithms:
Divide and conquer

• The aim is to tackle the issue in two sections:

- The first of which divides the problem into sub-problems
that are similar in nature.
- Second, we will approach the more modest issue
autonomously and then add the combined outcome to
complete our final response.

7
Types of algorithms:
Greedy Method

• Greedy algorithm creates the solution portion by portion.

• In selecting the following role, it is determined by the fact
that it provides sudden help and that it never considers
the options that had been assumed recently.

8
Types of algorithms:
Backtracking

• This type of algorithm seeks a steady solution to the

issue by eliminating solutions that fail to meet the
requirements of the issue at any moment.
• As an example, it is an algorithmic procedure for
handling issues recursively by attempting to construct an
answer steadily, eliminating solutions that fail to meet the
criteria at any moment.

9
Types of algorithms:
Branch and Bound

• Branch and bound is an algorithm design paradigm

which is generally used for solving combinatorial
optimization problems.
• These problems are typically exponential in terms of time
complexity and may require exploring all possible
permutations in worst case.
• The Branch and Bound Algorithm technique solves these
problems relatively quickly.

10
Types of algorithms:
Dynamic programming
• Dynamic Programming is mainly an optimization over
plain recursion. Wherever we see a recursive solution
that has repeated calls for same inputs, we can optimize
it using Dynamic Programming.
• The idea is to simply store the results of sub-problems,
so that we do not have to re-compute them when needed
later.
• This simple optimization reduces time complexities from
exponential to polynomial.

11
Types of algorithms:
Deterministic or non-deterministic algorithm

• A deterministic algorithm is one whose behavior is completely

determined by its inputs and the sequence of its instructions.
• A non-deterministic algorithm is one in which the outcome
cannot be predicted with certainty, even if the inputs are
known.

12
Types of algorithms:
Serial or parallel or distributed algorithm
• A sequential algorithm or serial algorithm is an algorithm
that is executed sequentially – once through, from start
to finish, without other processing executing
• A parallel algorithm is an algorithm that can execute
several instructions simultaneously on different
processing devices and then combine all the individual
outputs to produce the final result.
• Distributed algorithms are algorithms designed to run on
multiple processors, without tight centralized control.

13
Algorithm Development Life Cycle: 4 phases

1. Designing

4.Analyzing Algorithms 2.Writing

3. Testing/
Experiment

14
Algorithm Development Life Cycle:
Designing phase
• Algorithm design refers to a method or process of solving a
problem.
• One of the algorithmic design techniques and the data
structure is used.
• Your design techniques are the algorithms you use.
Note that: Multiple algorithms can solve a problem but not all
algorithms can solve it efficiently  using a suitable algorithm
design method.
Example: Design an algorithm to calculate n!
One of the design techniques used to solve this problem is divide
and conquer.
15
Algorithm Development Life Cycle:
Writing phase
Writing algorithm for solving a problem offers these advantages:
 Promotes effective communication between team members
 Enables analysis of problem at hand
 Acts as blueprint for coding
 Assists in debugging
 Becomes part of software documentation for future reference
during maintenance phase
The algorithm is written by using English like language,
pseudocodes (or using flowchart).

16
Algorithm Development Life Cycle:
Writing phase
Initialization Assignment Decision Repetition Output
Input Step
Step Step Step Step Step.

An algorithm is written using the following basic control

statements:
 Sequential: Steps are performed sequentially.
 Selection/Decision: One of the several alternative actions is
selected (condition statements).
 Repetition: One or more than one step(s) are performed
repeatedly (loops)

17
Algorithm Development Life Cycle:
Writing phase
Example: Algorithm of Linear search
Linear search starts from the beginning of the array until the
desired key is found or the end of the array is reached.
Input

Initialization

Repetition
Sequential

Selection/Decision

Output

18
Algorithm Development Life Cycle:
Testing/Implement phase

• This is another step in the development of the algorithm

• Used to check whether the algorithm is correct or not
Perform each step using paper and pencil by some
required valid input, use the algorithm and get the
required output in a finite time
When any error is found, then go back to the algorithm
design phase and redesign the algorithm.

19
Algorithm Development Life Cycle:
Testing/Implement phase

Example: Algorithm of Linear search testing

• Perform the linear search with following input:
- N = 5;
- Key = 23
- Array A: 64, 25, 12, 22, 11

20
Algorithm Development Life Cycle:
Analyzing phase
• Algorithms are to be analyzed to compute the objective
criteria for different input size before actual implementation
• Decide that which algorithm is better
by using some performance measurement systems (time and
space for example).
• Suppose M is an algorithm and n is the size of the input data
The complexity of an algorithm M is the function f(n) which
gives the running time and/or storage space requirement of
the algorithm in terms of the size n of the input data.

21
Complexity of the algorithm

 Some algorithms are more efficient than others.

 Efficiency is preferred; we need the metrics to compare
them.
 An algorithm’s complexity is a function describing the
efficiency of the algorithm in terms of the amount of data
that the algorithm must process.

22
Complexity of the algorithm

Analysis of
algorithm

Apriori Posteriori
Analysis Analysis
(Theoretical) (Empirical)

23
Theoretical / Apriori Analysis

 Apriori analysis is on a theoretical basis,

independent of programming languages and
machine structures.
 Apriori analysis is performed prior to running it
on a specific system.

24
Empirical/Posteriori Analysis

 The actual amount of space and time taken by

the algorithms are recorded during execution.
 We do posteriori analysis of the algorithm only
after running it on the system.

25
Empirical/Posteriori Analysis

The main steps of empirical

analysis:
• Build code for the
algorithm.
• Run this code on different
input data.
• Record actual amount of
space and time taken by
the algorithm.
• Represent obtain results in
form of a graph
26
Empirical/Posteriori Analysis - limitations

1. It’s hard to write a program for some algorithms.

2. It is dependent on the programming language used
and machine structure
3. To compare two algorithms, they need to be
installed on the computers with the same
configuration of hardware and software.

27
Computational Complexity

There are two main complexity measures of

efficiency
• Time complexity describes the amount of time an
algorithm takes in terms of the amount of input
• Space complexity describes the amount of memory
(space) an algorithm takes in terms of the amount of
input

28
3. Time complexity
The factors affecting the execution time

Programmer
skills

Compiler
…..
options

Time
complexity

Hardware
Input size characteristics

Algorithm
used

29
3. Time complexity
The rules for computing running time:
1. Sequence: Add the time of the individual statements.
The maximum is the one that counts.
2. Alternative structures: Time for testing the condition plus
the maximum time taken by any of the alternative paths.
3. Loops: Execution time of a loop is at most the execution
time of the statements of the body (including the
condition tests) multiplied by the number of iterations.
4. Nested loops: Analyze them inside out.

30
3. Time complexity
The rules for computing running time:
5. Subprograms: Analyze them as separate algorithms
and substitute the time wherever necessary.
6. Recursive Subprograms: Generally, the running time
can be expressed as a recurrence relation. The solution
of the recurrence relation yields the expression for the
growth rate of execution time.
Time taken by a program P means:
T (P) = compile time + run time

31
3. Time complexity
While measuring the time complexity of an algorithm, we
concentrate on the frequency count of all key statements
(important statement).

32
3. Time complexity
The frequency count of all key statements:
1 for (i = 0; i < n; i++)
2 {
3 x = a[i] / 2;
4 a[i] = x + 1;
5 }

Analyze
• Two key statements performed in Lines 3 and 4.
• They are repeated n times (for loop).
• Time complexity: f(n) = 2*n.

33
3. Time complexity
The frequency count of all key statements:

34
3. Time complexity

Time Complexity

Average- Worst-
Best-case
case case

The average time of all

The minimum instances taken by an The maximum
time algorithm time

Example: Define the minimum time, the maximum time and

average time in sequentially searching an unordered array to
find a target value 35
3. Time complexity
Best, Average, and Worst Cases
• In sequentially searching an unordered array to find a
target value
• The best and worst cases are straightforward:
- Best case occurs when we find the target in the first cell
- Worst case occurs when we find the target in the last cell, or
not at all (but end up searching the entire array)
• For the average case, we first have to consider the
probability of finding the target.

36
4. Space complexity
The components of space needed by a program:
 Instruction space: Space needed to store the executable
version of the program and it is fixed
 Data Space: Space needed to store all constants,
variable values
 In-build stack space: Space needed to store the
information needed to resume the suspended functions.

37
4. Space complexity

Total
space

Fixed Space Variable Space

Requirement Requirement

Space for Dynamicall

Space for Space for Space for
Space for simple y allocated
component structured recursive
code variables, space
variables variable stack
constants

38
5. Asymptotic Complexity
• For both measures, we are interested in the algorithm’s
asymptotic complexity
• This asks: when n (number of input items) goes to
infinity, what happens to the algorithm’s performance?
• To illustrate this, consider f(n) = n2 + 100n + log10n + 1000.
As the value of n increases, the importance of each term
shifts until for large n, only the n2 term is significant.

39
Big-O Notation

• The most commonly used notation for

asymptotic complexity used is "big-O" notation
• In the previous example we would say
4n = O(n)
(read "big-oh of n ")

40
Big-O Notation
Definition: Let f(n) and g(n) be functions, where n  Z is a
positive integer.
We write f(n) = O(g(n)) (read as "f of n is big-oh of g of n“) if
and only if  c > 0 and  n0 > 0 (c  R +, n0  Z+ ) satisfying
0  f(n)  c*g(n) n  n0
c*g(n)

f(n)

Big-O provides a formal method for

expressing asymptotic upper bounds
(bounding the growth of a function
from above)

n0 f(n)=O(g(n)) 41
Big-O Notation (continued)
• Although the definition of big-O is correct, it lacks
important information
• While c and N exist, it does not tell us how to calculate
them or what to do if multiple candidates exist (and they
often do)
Example 1: Prove that 2n +10 = O(n).
Solution: Find a value of c  R + and a value of n0  Z+ satisfying
0  2n+10 ≤ c*n, n  n0
We have: 2n+10 ≤ c*n 10 ≤ (c-2)*n
(c-2)*n  10
 n  10/(c-2) (constraint: c>2)
Select c=3 and n0=10 or we can select c = 4 and n0 = 5
Conclusion: 2n +10 = O(n)
42
Big-O Notation (continued)
Example 2: Prove that f(n) = n2 + 2n + 1 là O(n2)
Solution:
Find a value of c  R + and a value of N  Z+ satisfying
0  n2 + 2n + 1 ≤ c*n2 n  n0
We have: For all n ≥ 1
2n ≤ 2n2
1 ≤ n2
Thus:
n2 + 2n + 1 ≤ n2 + 2n2 + n2 = 4*n2,  n ≥ 1
We can choose c = 4 and n0 = 1

43
 Properties of Big-O Notation

• The big-O notation is awkward to work with all the time

• The following is a list of useful theorems you can use to
simplify big-O calculations
• Big-O is transitive: if f(n) = O(g(n)) and g(n) is O(h(n)),
then f(n) = O(h(n))
• If f(n) = O(h(n)) and g(n) is O(h(n)), then f(n) + g(n)
= O(h(n))
• A function ank = O(nk) for any a > 0
• Any kth degree polynomial is O(nk+j) for any j > 0

44
Properties of Big-O Notation (continued)

• f(n) = O(g(n)) is true if limn->∞ f(n)/g(n) is a constant. Put

another way, if f(n) = c*g(n), then f(n) = O(g(n))
• logan = O(logb n) for any a, b > 1. This means, except for
a few cases, we don’t care what base our logarithms are
• Given the preceding, we can use just one base and
rewrite the relationship as logan = O(lg n) for positive a ≠
1 and lg n = log2n.

45
 Example of Big-O Notation
• Consider f(n) = n2 + 100n + log10n + 1000. As the value of
n increases, the importance of each term shifts

46
 Example of Big-O Notation
• Consider f(n) = n2 + 100n + log10n + 1000. As the value of
n increases, the importance of each term shifts

52
 Ω Notations
• Big-O only gives us the upper bound of a function
• So if we ignore constant factors and let n get big enough,
some function will never be bigger than some other
function
• This can give us too much freedom
• Consider that selection sort is O(n3), since n2 is O(n3) -
but O(n2) is a more meaningful upper bound
• We need a lower bound, a function that always grows
more slowly than f(n), and a tight bound, a function that
grows at about the same rate as f(n)

53
 Ω Notations (continued)
• Big-Ω is for lower bounds what big-O is for upper bounds
Definition: Let f(n) and g(n) be functions, where n is a
positive integer. We write f(n) = Ω(g(n)) if and only if
g(n) = O(f(n)). We say "f of n is omega of g of n.“
• Or we can state that f(n) = (g(n)) if and only if
 c > 0 and n0 > 0 satisfying f(n)  c*g(n) n  n0

54
 Ω Notations (continued)

• So g is a lower bound for f ; after a certain n, and without

regard to multiplicative constants, f will never go below g

f(n)

cg(n)

n0

f(n)=Ω(g(n))

55
 Ω Notations (continued)

Example: Prove that n2 - 2000n = (n2)

Solution: We need to find c  R + and n0 Z+ satisfying
n2 - 2000n  c*n2 n  n0
From the condition: n2 - 2000n  c*n2, we have:
n2 - c*n2  2000n
n2(1 – c)  2000n
For n>0: 1 – c  2000/n
Choose: c = 0.5, và n0 = 4000

56
 Θ Notations
Finally, theta notation combines upper bounds with lower
bounds to get tight bound
Definition: Let f(n) and g(n) be functions, where n is a
positive integer. We write f(n) = Θ(g(n)) if and only if g(n)
= O(f(n)) and g(n) = (f(n)). We say "f of n is theta of g of
n.“
We can state that: f(n) = (g(n)) if and only if  c1 > 0, c2 > 0
and n0 > 0 satisfying c1*g(n)  f(n)  c2*g(n) n  n0

57
 Θ Notations

c2g(n)

f(n) c1*g(n)  f(n)  c2*g(n) n  n0

c1g(n)

n0
f(n)=Ө(g(n))

58
 O, Ω and Θ Notations

• There are some additional theorems we can consider

when we take Ω and Θ into account;
• f(n) = Ω(g(n)) is true if limn->∞ g(n)/f(n) is a constant
• f(n) = Θ(g(n)) is true if limn->∞ f(n)/g(n) is a non-zero
constant
• nk = O((1+ ε) n)) for any positive k and ε
This means any polynomial is bound from above by any
exponential

59
 O, Ω and Θ Notations (continued)

• So an algorithm that runs in polynomial time is

(eventually) preferable to an algorithm that runs in
exponential time
• (log n)ε = O(n k) for any positive k and ε
• This means a logarithm to any power grows more slowly
than a polynomial
• So an algorithm that runs in logarithmic time is
(eventually) preferable to an algorithm that runs in
polynomial (or from above, exponential) time

60
 O, Ω and Θ Notations (continued)

For any two functions, g(n) and f(n), f(n) = (g(n)) if

and only if
f(n) = O(g(n)) và f(n) = (g(n)).
This means
(g(n)) = O(g(n))  (g(n))

With the function: f(n) = 3n2 + 17 (2th degree polynomial)

- (n), (n2)
- O(n2), O(n3), O(n4),…
- (n2)

61
Growth Functions of Algorithm

62
Growth Functions of Algorithm

63
Growth Functions of Algorithm

64
Growth Functions of Algorithm

65
 Examples of Complexities
• Since we examine algorithms in terms of their time and
space complexity, we can classify them this way, too
• This is illustrated in the next figure

Fig. 2.4 Classes of algorithms and their execution times on a computer executing 1 million operations per
second (1 sec = 106 μsec = 103 msec)

66
 Examples of Complexities (continued)

Fig. 2.4 (concluded)

• The class of an algorithm is the name used to refer to its big-O

notation; it is a more convenient way to describe its behavior
• For example a linear function is O(n); its time increases in
direct proportion to the amount of data processed

67
 Examples of Complexities (continued)

• This relationship can also be expressed graphically:

Fig. 2.5 Typical functions applied in big-O estimates.

• This graph, and the previous chart, show that some
algorithms have no practical application
• Even with today’s supercomputers, cubic order algorithms or
higher are impractical for large numbers of elements
68
 Examples of Complexities (continued)

• This relationship can also be expressed graphically:

Fig. 2.5 Typical functions applied in big-O estimates.

• This graph, and the previous chart, show that some
algorithms have no practical application
• Even with today’s supercomputers, cubic order algorithms or
higher are impractical for large numbers of elements
69
NP problems
NP
Complexity of problems
common problems

“Non-deterministic
“Polynomial” time
Polynomial” time

• Given a particular input, the underlying algorithm has only one

way to decide what step to perform at any given point.
• The NP problems are a set of problems whose solutions are
hard to find but easy to verify and are solved by non-
deterministic machine in polynomial time.
70
NP problems

• Informally, a deterministic algorithm is one that

behaves predictably
• As opposed to this, a nondeterministic algorithm uses
some type of operation to “guess” what to do next when
a decision is made
• Consequently, it can exhibit different behaviors on
different runs
• There are several ways this can happen; for example
concurrent algorithms may experience race conditions

71
 NP problems: NP, NP-Hard, NP-Complete

Halting problem, Vertex

cover problem,

Determine whether a graph

has a Hamiltonian cycle,
Determine whether a Boolean
formula is satisfiable or not,

72
NP problems: NP-Hard, NP-Complete

73
NP problems: NP-Hard, NP-Complete

• Problems that can be solved by a deterministic algorithm

in polynomial time belong to a class P of problems
• If the problem can be solved in polynomial time by a
nondeterministic algorithm, it is of class NP
• Class P problems are tractable; NP problems are
tractable only if a nondeterministic algorithm is used
• Now P ⊆ NP, which simply means that deterministic
algorithms are nondeterministic algorithms that don’t use
nondeterministic decisions
• It is also generally believed P ≠ NP, although this is a
famous open problem in computer science

74
Summary
 The algorithm is a finite sequence of instructions/steps,
each of which is very elementary that must be followed to
solve a problem.
 The space complexity of a program is the amount of
memory it needed to run to completion.
 The time complexity of a program is the amount of
computer time needs to run to completion.
 The asymptotic notations commonly used in performance
analysis to characterize the complexity of an algorithm.
 The big-O notation is the formal method of expressing the
upper bound of an algorithm's running time.
75
Summary
Rule 1: O ( c f (n)) = O (f (n))
Rule 2: O (O (f (n)) = O (f (n))
Rule 3: O (f (n) g (n)) = f (n) O (g (n))
Rule 4: O (f (n) O (g (n)) = O (f (n) g (n))
Rule 5: O (f (n) + g (n)) = O (max (f (n), g (n))
Rule 6: If f(n) = O(g(n)) and g(n) = O(h(n)),
then f(n) = O(h(n)). [Transitivity Rule]
Rule 7: f(n) = O(g(n)) iff g(n) = Ω(f(n)). [Symmetry Rule]

76
78