Analysis And Design Of Algorithms

Associate Prof. Ghada Khoriba

Spring 2021

1
 Issues:

• Correctness
• Time efficiency
Analysis of • Space efficiency
• Optimality
algorithms
Approaches:

• Theoretical analysis
• Empirical analysis

2
Theoretical analysis of time efficiency
Time efficiency is analyzed by determining the number of repetitions of
the basic operation as a function of input size

Basic operation: the operation that contributes most towards the running
time of the algorithm
input size

T(n) ≈ copC(n)
running time
execution time Number of times
for basic operation basic operation is
executed
3
Input size and basic operation examples

Problem Input size measure Basic operation

Searching for key in a Number of list’s items,

Key comparison
list of n items i.e. n

Multiplication of two Matrix dimensions or Multiplication of two

matrices total number of elements numbers

Checking primality of n’size = number of digits

Division
a given integer n (in binary representation)

Visiting a vertex or
Typical graph problem #vertices and/or edges
traversing an edge

4
 • Select a specific (typical) sample of
inputs

• Use physical unit of time (e.g.,

milliseconds)
or
Empirical analysis Count actual number of basic
of time efficiency operation’s executions

• Analyze the empirical data

Empirical implies that the information is based on experience, and data is information we gather about something. Thus the information
5
acquired by scientists through experimentation and observation is called empirical data.
 For some algorithms efficiency depends on form of input:

• Worst case: 𝐶𝑤𝑜𝑟𝑠𝑡(𝑛) – maximum over inputs of size 𝑛

• Best case: 𝐶𝑏𝑒𝑠𝑡(𝑛) – minimum over inputs of size

Best-case,
average-case, • Average case: 𝐶𝑎𝑣𝑔(𝑛) – “average” over inputs of size 𝑛
• Number of times the basic operation will be executed on
worst-case typical input
• NOT the average of worst and best case
• Expected number of basic operations considered as a
random variable under some assumption about the
probability distribution of all possible inputs

6
 Example:
Sequential
search
• Worst case
𝐶𝑤𝑜𝑟𝑠𝑡 (n)=n
largest number of key comparisons

• Best case
𝐶𝑤𝑜𝑟𝑠𝑡 (n)=1
• Average case if 𝑝 = 1, (the search must be successful), the average number of key
𝑝 𝑝 𝑝 𝑝
𝐶𝑤𝑜𝑟𝑠𝑡 (n)=[1. + 2. + ⋯ + 𝑖. + ⋯ + 𝑛. ] + 𝑛. (1 − 𝑝) comparisons made by sequential search is (𝑛 + 1)/2; that is, the
𝑛 𝑛 𝑛 𝑛
=
𝑝
1 + 2 + ⋯+ 𝑖 + ⋯+ 𝑛 + 𝑛 1 − 𝑝 algorithm will inspect, on average, about half of the list’s elements.
𝑛
𝑝 𝑛(𝑛+1) 𝑝(𝑛+1)
If 𝑝 = 0 (the search must be unsuccessful), the average number of
= +𝑛 1−𝑝 = + 𝑛(1 − 𝑝) key comparisons will be n because the algorithm will inspect all 𝑛
𝑛 2 2

elements on all such inputs.

7
 • Exact formula
Types of e.g., C(n) = n(n-1)/2
formulas for
basic • Formula indicating order of growth with
specific multiplicative constant
operation’s
e.g., C(n) ≈ 0.5 n2
count
• Formula indicating order of growth with
unknown multiplicative constant
e.g., C(n) ≈ cn2

8
Order of growth
• Most important: Order of growth within a constant
multiple as 𝑛 → ∞

• Example:
• How much faster will algorithm run on computer that is twice
as fast?

• How much longer does it take to solve problem of double

input size?

9
 Values of some important functions as 𝑛

lower bound avg bound Upper bound

1 < log 𝑛 < 𝑛 < 𝑛 < 𝑛 log 𝑛 < 𝑛2 < 𝑛3 < ⋯ < 2𝑛 < 3𝑛 < ⋯ < 𝑛𝑛

A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 ©2012 Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved. 10
 A way of comparing functions that ignores constant
factors and small input sizes
Asymptotic
• 𝑂(𝑔(𝑛)): (𝑏𝑖𝑔 𝑂) class of functions 𝑓(𝑛) that
order of grow no faster than 𝑔(𝑛) (upper bound)

growth • Θ(𝑔(𝑛)): (𝑏𝑖𝑔 𝑡ℎ𝑒𝑡𝑎) class of functions 𝑓(𝑛) that

grow at same rate as 𝑔(𝑛)

• Ω(𝑔(𝑛)): (𝑏𝑖𝑔 𝑜𝑚𝑒𝑔𝑎) class of functions 𝑓(𝑛) that

grow at least as fast as 𝑔(𝑛) (lower bound)

11
Big-oh

12
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 ©2012 Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved.
Big-
omega

13
Big-theta

14
 lower bound avg bound Upper bound

1 < log 𝑛 < 𝑛 < 𝑛 < 𝑛 log 𝑛 < 𝑛2 < 𝑛3 < ⋯ < 2𝑛 < 3𝑛 < ⋯ < 𝑛𝑛

Establishing Definition: 𝑓(𝑛) is in 𝑂(𝑔(𝑛)) if order of growth of

𝑓(𝑛) ≤ order of growth of 𝑔 𝑛 (within constant

order of multiple),
i.e., there exist positive constant 𝑐 and non-negative integer
𝑛0 such that

growth Examples:
𝑓(𝑛) ≤ 𝑐𝑔(𝑛) for every 𝑛 ≥ 𝑛0

using the • 10n is O(n2)

10n <=…..? Θ 𝑛 ?
definition • 5n+20 is O(n) 𝛺 𝑛 ?
5n+20 <= ...?
5n+20<=100n , n>=1

𝑓(𝑛) c 𝑔(𝑛)
𝑓 𝑛 = 𝑂(𝑛) We choose nearest
𝑓 𝑛 = 𝑂 log 𝑛 𝑋 𝑓 𝑛 = 𝑂(𝑛2 ) 15
Example:
• 𝑓 𝑛 = 𝑛2 log 𝑛 + 𝑛

1 ∗ 𝒏𝟐 𝒍𝒐𝒈 𝒏 ≤ 𝒏𝟐 𝐥𝐨𝐠 𝒏 + 𝑛 ≤ 10 𝒏𝟐 𝒍𝒐𝒈 𝒏

• O(𝒏𝟐 𝐥𝐨𝐠 𝒏)
• Ω (𝒏𝟐 𝐥𝐨𝐠 𝒏)
• Θ (𝒏𝟐 𝐥𝐨𝐠 𝒏)

16
Example
• 𝑓 𝑛 = 𝑛!
𝑛 𝑥 𝑛 − 1 𝑥 𝑛 − 2 𝑥 … 𝑥 3 𝑥2 𝑥 1

1 𝑥 1 𝑥 1 ….≤ 1 𝑥 2 𝑥 3 𝑥 …𝑥 𝑛 ≤ 𝑛 𝑥 𝑛 𝑥 𝑛….

1 ≤ 𝑛! ≤ 𝑛𝑛

O(𝒏𝒏 )
Ω (1)
17
Example

• 𝑓 𝑛 = log 𝑛!

18
Example
• O(n log n)
• Ω (1)
• 𝑓 𝑛 = log 𝑛!

log (1𝑥1 … ) ≤ log 1 𝑥 2 𝑥 3 𝑥 … 𝑥 𝑛 ≤ log(𝑛 𝑥𝑛 𝑥𝑛 … )

1 ≤ log 𝑛! ≤ nlog 𝑛

19
 Some properties of asymptotic order of growth

• 𝑓 𝑛 = 𝑂(𝑓(𝑛))

• 𝑓 𝑛 = 𝑂 𝑔 𝑛 iff 𝑔 𝑛 ≥ (𝑓(𝑛))

• If 𝑓 𝑛 = 𝑂 𝑔 𝑛 𝑎𝑛𝑑 𝑔 𝑛 = 𝑂 ℎ 𝑛 , 𝑡ℎ𝑒𝑛 𝑓 𝑛 = 𝑂(ℎ(𝑛))

Note similarity with a ≤ b

• If 𝑓1 𝑛 = 𝑂 𝑔1 𝑛 𝑎𝑛𝑑 𝑓2 𝑛 = 𝑂(𝑔2(𝑛)) , 𝑡ℎ𝑒𝑛

𝑓1 𝑛 + 𝑓2 𝑛 = 𝑂(max{𝑔1(𝑛), 𝑔2(𝑛)})

20
 • All logarithmic functions loga n belong to the same class
(log n) no matter what the logarithm’s base a > 1 is
Orders of growth of
some important • All polynomials of the same degree k belong to the same class: aknk +
ak-1nk-1 + … + a0  (nk)
functions
• Exponential functions an have different orders of growth for different
a’s

• order log n < order n (>0) < order an < order n! < order nn

lower bound avg bound Upper bound

1 < log 𝑛 < 𝑛 < 𝑛 < 𝑛 log 𝑛 < 𝑛2 < 𝑛3 < ⋯ < 2𝑛 < 3𝑛 < ⋯ < 𝑛𝑛
21
Basic asymptotic efficiency classes
1 constant

log n logarithmic

n linear

n log n n-log-n or linearithmic

n2 quadratic

n3 cubic

2n exponential

n! factorial

22
 Time efficiency of non-recursive algorithms
General Plan for Analysis

• Decide on parameter n indicating input size

• Identify algorithm’s basic operation

• Determine worst, average, and best cases for input of size n

• Set up a sum for the number of times the basic operation is executed

• Simplify the sum using standard formulas and rules

23
 Example 1: Maximum element

Let us denote C(n) the number of times this comparison is executed and try
to find a formula expressing it as a function of size n. The algorithm makes one
comparison on each execution of the loop, which is repeated for each value of the
loop’s variable i within the bounds 1 and n - 1

24
 Example 2: Element uniqueness problem

in the worst case, the algorithm needs to

compare all n(n - 1)/2 distinct pairs of its n elements.

25
Example 3: Matrix multiplication

26
27
 Example 3: Matrix multiplication

the number of multiplications made for every pair of specific

values of variables i and j is

28