Algorithms & data structures

Algorithm analysis
Damjan Strnad
 2

Algorithm
●
in the real world we encounter tasks or problems that
need to be solved
●
a description of a procedure for solving a problem is
called an algorithm:
– an algorithm is a finite sequence of precisely defined
instructions which solves some problem in a finite
number of steps. An algorithm accepts inputs and
returns a solution.
●
we are interested in algorithms that can be executed on
a computer:
– a program is an algorithm that can be executed on a
computer and is written in a programming language
 3

Algorithm
●
we must distinguish between algorithm construction and
algorithm execution; the former is complex intellectual
work, while the latter is usually much simpler
●
algorithm construction consists of the following steps:
1.selection of algorithm description (flow diagram,
pseudo code or programming language)
2.algorithm design (use of known strategies, problem
division into subproblems, conversion to another
problem with known solution)
3.algorithm analysis (time and space complexity)
4.algorithm validation (testing and formal proof of
correctness)
 4

Algorithm analysis
●
with algorithm analysis we determine its efficiency, which
we express using time and space complexity with respect
to problem size n
●
time complexity T(n) defines the algorithm running time
as a number of basic operations with respect to
problem size; every basic operation executes in constant
time
●
space complexity S(n) defines the required memory
size as a number of basic storage units with respect to
problem size
●
in practice we are most interested in complexity order
(logarithmic, polynomial, exponential) of an algorithm
 5

Algorithm analysis
●
algorithm complexity can also depend on input data:
– with least favorable inputs we get worst complexity
Tw(n) and Sw(n); with most favorable inputs we get
best complexity Tb(n) and Sb(n)
– complexity in general case is expressed with average
or expected complexity Ta(n) and Sa(n)
●
complexity orders will be described using special
functions O (big o, rarely big omicron), Ω (big omega)
and Θ (big theta)
●
in the continuation we will focus on time complexity, but
the same theory applies to space complexity
 6

Big O
●
O(g(n)) denotes a set of all T(n), for which positive
constants c and n0 exist such that |T(n)| ≤ c|g(n)| for all
n ≥ n0*
●
we write T(n) = O(g(n)), never the other way (one way
equality)
●
value c is called leading constant
●
function g(n) represents the complexity order in least
favorable case and is an asymptotic upper bound of
function T(n)
●
g(n) is the leading term in function T(n), e.g., the
polynomial term with highest exponent
●
function O is used for expressing the upper bound of
algorithm time complexity
 7

Big O
●
O(g(n)) denotes a set of all T(n), for which positive
constants c and n0 exist such that |T(n)| ≤ c|g(n)| for all
n ≥ n0*
T(n)
●
we write T(n) = O(g(n)), never the other way (one way
equality)
●
value c is called leading constant
●
function g(n) represents the complexity order in least
favorable case and is an asymptotic upper bound of
function T(n)
●
g(n) is the leading term inT(n)
function T(n), e.g., the
polynomial term with highest exponent
●
function O is used for expressing the upper bound of
algorithm time complexity
 8

Big O
●
example: we have calculated that the actual time
complexity of an algorithm is:
1 3 1 2 1 1 3 1 3 1 3 6 3
T (n)= n + n + n ≤ n + n + n = n ( za n≥1)
3 2 6 3 2 6 6
which can be written as T(n) = O(n3)
●
we can similarly write:
3 2 1 2 3 2 2
n + =O(n ) and n =O(n )
5 6 5

●
due to one-way equality we cannot write:
3 2 1 3 2
n+ = n
5 6 5
 9

Big O
●
properties of function O:
– reflexivity: g(n) = O(g(n))
– constant removal: if c>0, then O(cg(n)) = O(g(n))
– sum: O(f(n)) + O(g(n)) = O(max(f(n),g(n)))
– dominant function: if for ∀n it is f(n)>g(n), then
O(f(n)) + O(g(n)) = O(f(n))
– product: O(f(n))·O(g(n)) = O(f(n)·g(n))
– transitivity: if f(n) = O(g(n)) and g(n) = O(h(n)), then
f(n) = O(h(n))
 10

Big O
●
rules for determining the upper bound of algorithm time
complexity from programming code:
– basic operations (e.g., comparison, addition, ...) require
one time unit
– in sequential execution of operations their time
complexity adds
– for conditional statements we take the maximum time
complexity of all choices
– time complexity of a loop is a product of number of
iterations and time complexity of single iteration
– for recursive programs time complexity is expressed as
a recursive function
 11

Big O
● if T(n) = a0+a1n+...+amnm is a polynomial of order m, we
can estimate the time complexity using upper bound
T(n) = O(nm)
●
O(1) means that the number of basic operation executions
is fixed and does not depend on input data size ⇒ time
complexity is constant
●
the following relations exist between complexity orders:
O(1) < O(log2n) < O(n) < O(nlog2n) < O(n2) < O(n3) ≪ O(2n)
●
for large n there is a huge difference between exponential
complexity O(an) (a>1) and polynomial complexity O(nm)
●
there is no such m for which nm would bound 2n
 12

Big O
●
example: the number of executed basic operations for a
problem of size n with different time complexity orders (the
leading constant is c=1)
log2n n nlog2n n2 n3 2n
0 1 0 1 1 2
1 2 2 4 8 4
2 4 8 16 64 16
3 8 24 64 512 256
4 16 64 256 4096 65536
5 32 160 1024 32768 4294967296
 13

Big O
●
example: comparison of running times in μs for a problem
of size n with different time complexity orders, if one
operation takes 1 μs and the leading constant is c=1
T(n)
n n n2 n3 n5 nlog2n 2n 3n
10 0.01 ms 0.1 ms 1 ms 0.1 s 2 ms 1 ms 59 ms
20 0.02 ms 0.4 ms 8 ms 3.2 s 420 ms 1s 58 min
30 0.03 ms 0.9 ms 27 ms 24.3 s 17.3 s 17.9 min 6.5 yrs
40 0.04 ms 1.6 ms 64 ms 1.7 min 5.6 ms 12.7 days 3.9·105 yrs
50 0.05 ms 2.5 ms 125 ms 5.2 min 1.1 h 35.7 years 2.3·1010 yrs
60 0.06 ms 3.6 ms 216 ms 13 min 8.9 h 36.5·103 yrs 1.3 · 1015 yrs
 14

Big O
●
example: estimate of time complexity growth, if problem
size increases by 1
T(n) T(n+1)-T(n)
log2n O(1/n)
n O(1)
nlog2n O(log2n)
n2 O(n)
n3 O(n2)
2n O(2n)
●
for algorithms with polynomial complexity the growth is
one order less than algorithm complexity itself; for
exponential it is equal to algorithm complexity
 15

Big O
●
example: estimate of time complexity growth, if problem
size increases by 1
T(n) T(n+1)-T(n)
log2n O(1/n) T(n+1) = log2(n+1) = log2(n(1+1/n)) =
n O(1) = log2(n) + log2(1/n)=
= T(n) + O(1/n)
nlog2n O(log2n)
n2 O(n)
n3 O(n2)
2n O(2n)
●
for algorithms with polynomial complexity the growth is
one order less than algorithm complexity itself; for
exponential it is equal to algorithm complexity
 16

Big O
●
example: estimate of time complexity growth, if problem
size increases by 1
T(n) T(n+1)-T(n)
log2n O(1/n)
n O(1)
nlog2n O(log2n)
n2 O(n) T(n+1) = (n+1)2 = n2 + 2n + 1 =
n3 O(n2) = T(n) + O(n)
2n O(2n)
●
for algorithms with polynomial complexity the growth is
one order less than algorithm complexity itself; for
exponential it is equal to algorithm complexity
 17

Big O
●
example: estimate of time complexity growth, if problem
size increases by 1
T(n) T(n+1)-T(n)
log2n O(1/n)
n O(1)
nlog2n O(log2n)
n2 O(n)
n3 O(n2)
2n O(2n) T(n+1) = 2n+1 = 2·2n = 2n + 2n =
= T(n) + O(2n)
●
for algorithms with polynomial complexity the growth is
one order less than algorithm complexity itself; for
exponential it is equal to algorithm complexity
 18

Big O
●
example: problem of what size can be solved in given
time, if one operation takes 1 μs
●
example:
– 1 min = 60 s = 6·107 μs ⇒ 6·107 operations

n =6⋅10 →n= √ 6⋅10 =7746

– 2 7 7

n =6⋅10 →n= √ 6⋅10 =35

5 7 5 7
–
log 2 n 7 log 2 n 7
– n =6⋅10 → log 2 n =log 2 (6⋅10 )→
→ log 2 n⋅log 2 n=log 22 n=log 2 (6⋅107 )→ log 2 n= √ log 2 (6⋅107 )→
→n=2 √ log (6⋅10 )
=332
7

W (C⋅ln 2)
– T (n)=n log 2 n→n=e , where W is Lambert function,
which is computed numerically (lambertw in Matlab and
Octave)
 19

Big O
●
example: problem of what size can be solved in given
time, if one operation takes 1 μs
●
example:
T(n) 1s 1 min 1 hour 100 hours
106 √ 6⋅107 =7746
nn2 =6⋅107 →n= 6·107 3.6·109 3.6·1011
nlog2n 5,2617·10 5
72,8014·10
6
1,3338·108 1,0801·1010
n =6⋅10 →n=√ 6⋅10 =35
5 7 5

n2 log 2 n 1000
7 log 2 n 7746 7 6·104 6·105
n =6⋅10 →log 2 n =log 2 (6⋅10 )→
n3 100 391 1533 7114
n
→log
5 2 n⋅log 2
15
n=log 2
2 n=log
35 2 (6⋅107
)→ log 2
81
n= log 2 √
(6⋅107
205
)→
→n=2 √ log (6⋅10 )
7

22 =33
2
n log2n
33 49 73
W (C⋅ln 2)
2nT (n)=n log19 2 n→n=e 25 31 38
3n 12 16 20 24
 20

Big O
●
example: how much bigger problem can we solve in given
time using a faster computer (i.e., execution of basic
operation takes less time)
●
example: on a computer with execution time of basic
operation equal to 1 μs we have solved a problem of size
n and time complexity 2n in time T. Problem of what size n*
can we solve in the same time on a 1000-times faster
computer (i.e., where basic operation takes 1 ns)?
– T = 2n* ns = 2n μs (equalize units and log)
– 2n* ns = 103·2n ns

– n* = log (103·2n) = log (103) + log (2n) = n + 9.97

2 2 2
 21

Big O
●
for polynomial problems the problem size increase is by
some factor, while for exponential problems it is only by a
constant
T(n) 1 μs 1 ns 1 ps
n n1 1000n1 106n1
n2 n2 31.6n2 1000n2
n3 n3 10n3 100n3
n5 n4 3.98n4 15.85n4
2n n5 n5 + 9.97 n5 + 19.93
3n n6 n6 + 6.29 n6 + 12.58
 22

Big omega Ω
●
Ω(g(n)) denotes a set of all T(n), for which positive
constants c and n0 exist such that |T(n)| ≥ c|g(n)| for all
n ≥ n0*
●
we write T(n) = Ω(g(n)), never the other way (one way
equality)
●
function g(n) represents the complexity order in most
favorable case and is an asymptotic lower bound of
function T(n)
●
function Ω is used for expressing the lower bound of
algorithm time complexity
 23

Big omega Ω
●
Ω(g(n)) denotes a set of all T(n), for which positive
constants c and n0 exist such that |T(n)| ≥ c|g(n)| for all
n ≥ n0*
T(n)
●
we write T(n) = Ω(g(n)), never the other way (one way
equality)
●
function g(n) represents the complexity order in most
favorable case and is an asymptotic lower bound of
function T(n)
●
function Ω is used for expressing the lower bound of
algorithm time complexityT(n)
 24

Big theta Θ
●
Θ(g(n)) denotes a set of all T(n) for which positive
constants c1, c2 and n0 exist such that
c1|g(n)| ≤ |T(n)| ≤ c2|g(n)| for all n ≥ n0*
●
we write T(n) = Θ(g(n)), never the other way (one way
equality)
●
function g(n) represents the complexity order in both most
and least favorable case and is an asymptotic tight
bound of function T(n)
●
function Θ is used to express algorithm time complexity
when lower and upper bound have the same order
●
if T(n)=Θ(g(n)), then T(n)=Ω(g(n))=O(g(n))=Θ(g(n))
 25

Big theta Θ
●
Θ(g(n)) denotes a set of all T(n) for which positive
constants c1, c2 and n0 exist such that
c1|g(n)| ≤ |T(n)| ≤ c2|g(n)| for all n ≥ n0*
● T(n) way (one way
we write T(n) = Θ(g(n)), never the other
equality)
●
function g(n) represents the complexity order in both most
and least favorable case and is an asymptotic tight
bound of function T(n)
●
function Θ is used to express algorithm time complexity
T(n) have the same order
when lower and upper bound
●
if T(n)=Θ(g(n)), then T(n)=Ω(g(n))=O(g(n))=Θ(g(n))
 26

Estimation from measurements

●
we need to estimate time complexity from running time
measurements if:
– it cannot be analytically derived
– we want to practically validate theoretical derivation
●
steps:
1) we measure running time for different problem sizes
2) we draw a graph with n on abscissa and T(n) on ordinate
3) we draw an approximation curve through the points
4) we intuitively assume the curve type (e.g., T(n) = cna)
5) we select as many points as there are curve parameters
6) we calculate parameter values from a system of linear equations
7) we draw the curve determined by the computed function
8) if the measured curve matches the computed one to sufficient
degree our assumption is confirmed; otherwise we assume a
different curve type and repeat the computation
 27

Estimation from measurements

●
we need to estimate time complexity from running time
measurements if:
T(n)
– it cannot be analytically derived
– we want to practically validate theoretical derivation
●
steps:
1) we measure running time for different problem sizes
2) we draw a graph with n on abscissa and T(n) on ordinate
3) we draw an approximation curve through the points
4) we intuitively assume the curve type (e.g., T(n) = cna)
5) we select as many points as there are curve parameters
6) we calculate parameter values from a system of linear equations
7) we draw the curve determined by the computed function
8) if the measured curve matches the computed n one to sufficient
degree our assumption is confirmed; otherwise we assume a
different curve type and repeat the computation
 28

Estimation from measurements

●
we need to estimate time complexity from running time
measurements if:
T(n)
– it cannot be analytically derived
– we want to practically validate theoretical derivation
●
steps:
1) we measure running time for different problem sizes
2) we draw a graph with n on abscissa and T(n) on ordinate
3) we draw an approximation curve through the points
4) we intuitively assume the curve type (e.g., T(n) = cna)
measurements
5) we select as many points as there are curve parameters
6) we calculate parameter values from a system of linear equations
7) we draw the curve determined by the computed function
8) if the measured curve matches the computed n one to sufficient
degree our assumption is confirmed; otherwise we assume a
different curve type and repeat the computation
 29

Estimation from measurements

●
we need to estimate time complexity from running time
measurements if:
T(n)
– it cannot be analytically derived
– we want to practically validate theoretical derivation
●
steps:
1) we measure running time for different problem sizes
2) we draw a graph with n on abscissa and T(n) on ordinate
3) we draw an approximation curve through the points
4) we intuitively assume the curve type (e.g., T(n) = cna)
5) we select as many points as there are curve parameters
6) we calculate parameterapproximation
values curve
from a system of linear equations
7) we draw the curve determined by the computed function
8) if the measured curve matches the computed n one to sufficient
degree our assumption is confirmed; otherwise we assume a
different curve type and repeat the computation
 30

Estimation from measurements

●
we need to estimate time complexity from running time
measurements if:
T(n)
– it cannot be analytically derived
– we want to practically validate theoretical derivation
●
steps:
1) we measure running time for different problem sizes
2) we draw a graph with n on abscissa and T(n) on ordinate
3) we draw an approximation curve through the points
4) we intuitively assume the curve type (e.g., T(n) = cna)
5) we select as many points as there are curve parameters
6) we calculate parameter values from a system of linear equations
calculated curve c·n
a

7) we draw the curve determined by the computed function

8) if the measured curve matches the computed n one to sufficient
degree our assumption is confirmed; otherwise we assume a
different curve type and repeat the computation
 31

Estimation from measurements

●
we need to estimate time complexity from running time
measurements if:
T(n)
– it cannot be analytically derived
– we want to practically validate theoretical derivation
●
steps:
1) we measure running time for different problem sizes
2) we draw a graph with n on abscissa and T(n) on ordinate
3) we draw an approximation curve through the points
4) we intuitively assume the curve type (e.g., T(n) = cna)
5) we select as many points as there are curve parameters
6) we calculate parameter values from a system of linear equations
calculated curve c·log n
a

7) we draw the curve determined by the computed function

2

8) if the measured curve matches the computed n one to sufficient

degree our assumption is confirmed; otherwise we assume a
different curve type and repeat the computation
 32

Problem classes P and NP

●
a decision problem is a problem, where the result for a given input
is „yes“ or „no“ (e.g. does a path exist, is the number a prime, does
a subset exist)
●
decision problems that are solvable in polynomial time O(nk) belong
to class P (polynomial)
●
decision problems for which the answer „yes“ can be verified in
polynomial time, belong to class NP (non-deterministic polynomial);
class NP includes class P
●
classes P and NP can also be used to describe optimization
problems, if we treat them as decision problems of form
„does a solution exist with value at most/at least X?“
●
a procedure that converts decision problem A into decision problem
B, such that the answer to B is “yes” if and only if the answer is
“yes” for A, is called a reduction algorithm
 33

NP-complete and NP-hard problems

●
a problem is NP-complete (belongs to class NPC) if it belongs to
class NP, and any other NP problem can be reduced to it in
polynomial time
– examples of NP-complete problems are
subset sum, 0/1 backpack and traveling
salesperson (decision forms)
– example of an NP problem that is not
NP-complete is number factorization
●
a problem is NP-hard if it is not necessarily
in NP, but any other NP problem can be
reduced to it in polynomial time
– example of an NP-hard problem that is
not NP-complete is the halting problem:
for a given program and its input
determine whether the program halts (terminates)
 34

Conclusion
●
out of two algorithms for solving the same problem the
one with lower time complexity order will be more
efficient once the problem size is big enough
●
in algorithm analysis we determine their asymptotic
efficiency, i.e., in problem size limit
●
for small enough problem instances the algorithm with
worse time complexity can still be faster:
– we can use a combined solution, which switches to a
different algorithm for small problem sizes
●
for efficient algorithm implementation we also need to
consider the properties of hardware (cache, branch
prediction)