Data structures and Algorithms

Basic definitions and notations

Pham Quang Dung

Hanoi, 2012

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 1 / 46
Outline

1 First example

2 Algorithms and Complexity

3 Big-Oh notation

4 Pseudo code

5 Analysis of algorithms

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 2 / 46
First example

Find the longest subsequence of a given sequence of numbers

Given a sequence s = ha1 , . . . , an i
a subsequence is s(i, j) = hai , . . . , aj i, 1 ≤ i ≤ j ≤ n
weight w (s(i, j)) =
j
X
ak
k=i

Problem : find the subsequence having largest weight

Example
sequence : -2, 11, -4, 13, -5, 2
The largest weight subsequence is 11, -4, 13 having weight 20

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 3 / 46
Direct algorithm

n n2 +n


Scan all possible subsequences 2 = 2
Compute and keep the largest weight subsequence
C++ code :

Analyzing the complexity by counting the number of basic operations

n3 n2 n
6 + 2 + 3

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 4 / 46
Direct algorithm
Faster algorithm

Pj Pj−1
Observation : k=i a[k] = a[j] + k=i a[k]
C++ code :

n2 n
Complexity : 2 + 2

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 5 / 46
Recursive algorithm
Divide the sequence into 2 subsequences at the middle s = s1 :: s2
The largest subsequence might
be in s1 or
be in s2 or
start at some position of s1 and end at some position of s2
C++ code :

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 6 / 46
Recursive algorithm

Count the number of addition (”+”) operation T (n)


0 if n = 1
T (n) =
T ( n2 ) + T ( n2 ) + n if n > 1

By induction : T (n) = n logn

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 7 / 46
Dynamic programming

General principle
Division : divide the initial problem into smaller similar problems
(subproblems)
Storing solutions to subproblems : store the solution to subproblems
into memory
Aggregation : establish the solution to the initial problem by
aggregating solutions to subproblems stored in the memory

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 8 / 46
Dynamic programming

Largest subsequence
Division :
Let si be the weight of the largest subsequence of a1 , . . . , ai ending at
ai
Aggregation :
s1 = a1
si = max{si−1 + ai , ai }, ∀i = 2, . . . , n
Solution to the original problem is max{s1 , . . . , sn }
Number of basic operations is n (best algorithm)

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 9 / 46
Comparison between algorithms

] operations n = 10 time n = 100 time

logn 3.32 −8
3.3×10 sec. 6.64 6×10−8 sec.
nlogn 33.2 −7
3.3×10 sec. 664 6.6×10−6 sec.
n2 100 10−6 sec. 10000 10−4 sec.
n3 103 −5
10 sec. 10 6 10−2 sec.
] operations n = 104 time n = 106 time
logn 13.3 −6
10 sec. 19.9 < 10−5 sec.
nlogn 1.33×10 5 −3
×10 sec. 1.99×10 7 2×10−1 sec.
n2 108 1 sec. 101 2 2.77 hours
n3 1
10 2 2.7 hours 101 8 115 days

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 10 / 46
Outline

1 First example

2 Algorithms and Complexity

3 Big-Oh notation

4 Pseudo code

5 Analysis of algorithms

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 11 / 46
Algorithms and complexity

Definition
Informally, an algorithm is any well-defined computational procedure that
takes a set of values, as input, and produces a set of values, as output

Input
Output
Precision
Finiteness
Uniqueness
Generality

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 12 / 46
Complexity

Criteria for evaluating the complexity of an algorithm

Memory
CPU time

Definition
The size of input is defined to be the number of necessary bits for
representing it

Definition
The basic operations are the operations which can be performed in a
bounded time, and do not depend on the size of the input

We evaluate the complexity of an algorithm in term of the number of basic

operations it performs

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 13 / 46
Complexity

Worst-case time complexity :

The longest execution time the algorithm takes given any input of size
n
Used to compare the efficiency of algorithms
Average-case time complexity : execution time the algorithm takes on
a random input
Best-case time complexity : The smallest execution time the
algorithm takes given any input of size n

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 14 / 46
Outline

1 First example

2 Algorithms and Complexity

3 Big-Oh notation

4 Pseudo code

5 Analysis of algorithms

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 15 / 46
Big-Oh notation
Given a fucntion g (n), we denote :
Θ(g (n)) = {f (n) : ∃c1 , c2 , n0 s.t. 0 ≤ c1 g (n) ≤ f (n) ≤ c2 g (n), ∀n ≥
n0 }
Example :
10n2 − 3n = Θ(n2 )

source : http ://www.personal.kent.edu/ rmuhamma/Algorithms/MyAlgorithms/intro.htm

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 16 / 46
Big-Oh notation
Given a fucntion g (n), we denote :
O(g (n)) = {f (n) : ∃c, n0 > 0 s.t. f (n) ≤ cg (n), ∀n ≥ n0 }
Example :
2n2 = O(n3 )

source : http ://www.personal.kent.edu/ rmuhamma/Algorithms/MyAlgorithms/intro.htm

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 17 / 46
Big-Oh notation

Given a fucntion g (n), we denote :

Ω(g (n)) = {f (n) : ∃c, n0 > 0 s.t. cg (n) ≤ f (n), ∀n ≥ n0 }
Example :
5n2 = Ω(n)

source : http ://www.personal.kent.edu/ rmuhamma/Algorithms/MyAlgorithms/intro.htm

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 18 / 46
Big-Oh notation

When we say ”the time complexity is O(f (n))” : the time complexity
in the worst case is O(f (n))
When we say ”the time complexity is Ω(f (n))” : the time complexity
in the best case is Ω(f (n))

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 19 / 46
Little-o notation

Given a fucntion g (n), we denote :

o(g (n)) = {f (n) : ∀ constant c > 0, ∃n0 > 0 s.t. 0 ≤ f (n) <
cg (n), ∀n ≥ n0 }
Example :
5n2 = o(n3 )

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 20 / 46
Little-o notation

Given a fucntion g (n), we denote :

ω(g (n)) = {f (n) : ∀ constant c > 0, ∃n0 > 0 s.t. 0 ≤ cg (n) <
f (n), ∀n ≥ n0 }
Example :
5n2 = ω(n1.5 )

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 21 / 46
Big-Oh notation

f (n)
lim < ∞ ⇒ f (n) = O(g (n))
n→∞ g (n)

f (n)
lim > 0 ⇒ f (n) = Ω(g (n))
n→∞ g (n)

f (n)
0 < lim < ∞ ⇒ f (n) = Θ(g (n))
n→∞ g (n)

f (n)
lim = 0 ⇒ f (n) = o(g (n))
n→∞ g (n)

f (n)
lim = ∞ ⇒ f (n) = ω(g (n))
n→∞ g (n)

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 22 / 46
Big-Oh notation

O(lgn) = O(lnn)
lg a n = o(nb ) where a, b are constant
n! = o(nn )
n! = ω(2n )
logn! = Θ(nlogn)

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 23 / 46
Big-Oh notation

Example
A = log3 (n2 ) vs. B = log2 (n3 )
A = 2log3 n = 2lnn/ln3 where we denote lnx = loge (x)
B = 3log2 n = 3lnn/ln2
A
B = constant ⇒ A = Θ(B)

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 24 / 46
Big-Oh notation

Example
A = nlg 4 vs. B = 3lgn where we denote lgx = log2 x
B = 3lgn = nlg 3 (logb a = loga b)
A
B = nlg (4/3) → ∞
A = ω(b)

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 25 / 46
Big-Oh notation

Example
A = lg 2 n vs. B = n1/2 where we denote lgx = log2 x

A lg 2 n
lim = lim 1/2 = 0
n→∞ B n→∞ n

⇒ A = o(B)

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 26 / 46
Big-Oh notation

Properties
f (n) = Θ(g (n)) ∧ g (n) = Θ(h(n)) ⇒ f (n) = Θ(h(n))
f (n) = O(g (n)) ∧ g (n) = O(h(n)) ⇒ f (n) = O(h(n))
f (n) = Ω(g (n)) ∧ g (n) = Ω(h(n)) ⇒ f (n) = Ω(h(n))
f (n) = o(g (n)) ∧ g (n) = o(h(n)) ⇒ f (n) = o(h(n))
f (n) = ω(g (n)) ∧ g (n) = ω(h(n)) ⇒ f (n) = ω(h(n))
f (n) = Θ(g (n)) iff g (n) = Θ(f (n))
f (n) = O(g (n)) iff g (n) = Ω(f (n))
f (n) = o(g (n)) iff g (n) = ω(f (n))

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 27 / 46
Outline

1 First example

2 Algorithms and Complexity

3 Big-Oh notation

4 Pseudo code

5 Analysis of algorithms

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 28 / 46
Pseudo code

Variables declaration :
integer x, y ;
real u,v ;
boolean a,b ;
char c,d ;
datatype x ;

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 29 / 46
Pseudo code

Assignment instruction :
x = expression ;
x ← expression ;
x ← x + 3;

Condition instruction
if condition then
instructions ;
else
instructions ;
endif ;

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 30 / 46
Pseudo code

Loop

while condition do
instructions ;
endwhile ;

repeat instructions ;
until conddition ;

for i = n1 to n2 [step d]
instructions ;
endfor

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 31 / 46
Pseudo code

Case instruction

Case
condition 1 : statement 1 ;
condition 2 : statement 2 ;
...
condition n : statement n ;
endcase

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 32 / 46
Pseudo code

Functions and Procedures

Function name(parameters)
begin
instructions ;
return value ;
end

Procedure name(parameters)
begin
instructions ;
end

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 33 / 46
Pseudo code

Example : Find the maximal value of an array A(1 : n)

Function max(A(1 : n))

begin
datatype x ;
integer i ;
x = A[1] ;
for i = 2 to n do
if x < A[i] then
x = A[i] ;
endif
endfor
return x ;
end

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 34 / 46
Pseudo code

Algorithm 1: max(A)
n ← A.length;
x ← A[1];
foreach i ∈ 2..n do
if x < A[i] then
x ← A[i];

return x;

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 35 / 46
Outline

1 First example

2 Algorithms and Complexity

3 Big-Oh notation

4 Pseudo code

5 Analysis of algorithms

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 36 / 46
Analysis of algorithms

Experiments studies
Write a program implementing the algorithm
Execute the program on a machine with different input sizes
Measure the actual execution times
Plot the results

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 37 / 46
Analysis of algorithms

Shortcomings of experiments studies

Need to implement the algorithm, sometime difficult
Results may not indicate the running time of other input not
experimented
To compare two algorithms, it is required to use the same hardware
and software environments.

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 38 / 46
Analysis of algorithms

Asymptotic algorithm analysis

Use high-level description of the algorithm (pseudo code)
Determine the running time of an algorithm as a function of the input
size
Express this function with Big-Oh notation

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 39 / 46
Analysis of algorithms

Sequential structure : P and Q are two segments of the algorithm

(the sequence P; Q)
Time(P; Q) = Time(P) + Time(Q) or
Time(P; Q) = Θ(max(Time(P), Time(Q)))
for loop : for i = 1 to m do P(i)
t(i) is the time complexity of P(i)P
m
time complexity of the for loop is i=1 t(i)

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 40 / 46
Analysis of algorithms

while (repeat) loop

Specify a function of variables of the loop such that this function
reduces during the loop
To evaluate the running time, we analyze how the function reduces
during the loop

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 41 / 46
Analysis of algorithms

Example : binary search

Function BinarySearch(T [1..n], x)
begin
i ← 1; j ← n;
while i < j do
k ← (i + j)/2 ;
case
x < T [k] : j ← k − 1 ;
x = T [k] : i ← k ; j ← k ; exit ;
x > T [k] : i ← k + 1 ;
endcase
endwhile
end

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 42 / 46
Analysis of algorithms

Example : binary search

Denote
d = j − i + 1 (number of elements of the array to be investigated)
i ∗ , j ∗ , d ∗ respectively the values of i, j, d after a loop
We have
If x < T [k] then i ∗ = i, j ∗ = (i + j)/2 − 1, d ∗ = j ∗ − i ∗ + 1 ≤ d/2
If x > T [k] then j ∗ = j, i ∗ = (i + j)/2 + 1, d ∗ = j ∗ − i ∗ + 1 ≤ d/2
Ifx = T [k] then d ∗ = 1
Hence, the number of iterations of the loop is dlogne

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 43 / 46
Analysis of algorithms

Primitive operations

Function Fib(n)
begin
i ← 0; j ← 1;
for k = 2 to n do
begin
j ←j +i;
i ←j −i;
end
return j ;
end

Primitive operation is j ← j + i, hence, the running time is O(n)

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 44 / 46
Analysis of algorithms
Primitive operations (be careful ! !)

Procedure PigeonholeSorting(T [1..n])

begin
for i = 1 to n do
inc(U[T [i]]) ;
i ← 0;
for k = 1 to s do
while U[k] > 0 do
i ← i + 1;
T [i] ← k ;
U[k] ← U[k] − 1 ;
endwhile
endfor
end
Number of primitive operations is sk=1 U[k] = n. Hence running time is
P
Θ(n) (But not correct !)
Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 45 / 46
Analysis of algorithms

Primitive operations (be careful ! !)

Consider the case T [i] = i 2 , ∀i = 1, . . . , n

1, if k = q 2

U[k] =
0, otherwise

s = n2 , the running time is Θ(n2 ) not Θ(n)

Reason : The primitive operation is not well-chosen. Many null-loop
where U[k] = 0
If the primitive operation is the checking instruction U[k] > 0, then
the running time is Θ(n + s) = Θ(n2 )

Pham Quang Dung () Data structures and Algorithms Basic definitions and notations Hanoi, 2012 46 / 46