Part 2 Complexity and Correctness of Algorithms

Data Structures and Algorithms

Chapter 2

Werner Nutt

Master Informatique Data Structures and Algorithms 1

 Part 2 Complexity and Correctness of Algorithms

Acknowledgments
• The course follows the book “Introduction to Algorithms‘”,
by Cormen, Leiserson, Rivest and Stein, MIT Press
[CLRST]. Many examples displayed in these slides are
taken from their book.
• These slides are based on those developed by
Michael Böhlen for this course.
(See http://www.inf.unibz.it/dis/teaching/DSA/)
• The slides also include a number of additions made by
Roberto Sebastiani and Kurt Ranalter when they taught
later editions of this course
(See http://disi.unitn.it/~rseba/DIDATTICA/dsa2011_BZ//)

Master Informatique Data Structures and Algorithms 2

 Part 2 Complexity and Correctness of Algorithms

DSA, Chapter 2: Overview

• Complexity of algorithms

• Asymptotic analysis

• Correctness of algorithms

• Special case analysis

Master Informatique Data Structures and Algorithms 3

 Part 2 Complexity and Correctness of Algorithms

DSA, Chapter 2: Overview

• Complexity of algorithms

• Asymptotic analysis

• Special case analysis

• Correctness of algorithms

Master Informatique Data Structures and Algorithms 4

 Part 2 Complexity and Correctness of Algorithms

Analysis of Algorithms

• Efficiency:
– Running time
– Space used

• Efficiency is defined as a function of the input size:

– Number of data elements (numbers, points)
– The number of bits of an input number

Master Informatique Data Structures and Algorithms 5

 Part 2 Complexity and Correctness of Algorithms

The RAM Model

We study complexity on a simplified machine model,
the RAM (= Random Access Machine):
– accessing and manipulating data takes a (small)
constant amount of time
Among the instructions (each taking constant time),
we usually choose one type of instruction as a
characteristic operation that is counted:
– arithmetic (add, subtract, multiply, etc.)
– data movement (assign)
– control flow (branch, subroutine call, return)
– comparison
Data types: integers, characters, and floats
Master Informatique Data Structures and Algorithms 6
 Part 2 Complexity and Correctness of Algorithms

Analysis of Insertion Sort

Running time as a function of the input size
(exact analysis)

cost times
for j := 2 to n do c1 n
key := A[j] c2 n-1
// Insert A[j] into A[1..j-1]
i := j-1 c3 n-1
n
while i>0 and A[i]>key do c4 ∑ j= 2 t j
n
A[i+1] := A[i] c5 ∑ j= 2 -t j−1.
n
i-- c6 ∑ j= 2 -t j−1.
A[i+1]:= key c7 n-1

tj is the number of times the while loop is executed, i.e.,

(tj – 1) is number of elements in the initial segment greater than A[j]
Master Informatique Data Structures and Algorithms 7
 Part 2 Complexity and Correctness of Algorithms

Analysis of Insertion Sort/2

• The running time of an algorithm for a given input
is the sum of the running times of each statement.
• A statement
– with cost c
– that is executed n times
contributes c*n to the running time.
• The total running time T(n) of insertion sort is
n
T(n) = c1*n + c2*(n-1) + c3*(n-1) + c4 * ∑ j=2 t j
n n
+ c5 ∑ j=2 -t j −1. + c6 ∑ j=2 -t j −1. + c7*(n - 1)

Master Informatique Data Structures and Algorithms 8

 Part 2 Complexity and Correctness of Algorithms

Analysis of Insertion Sort/3

• The running time is not necessarily equal

for every input of size n

• The performance depends on the details of the input

(not only length n)

• This is modeled by tj

• In the case of Insertion Sort, the time tj

depends on the original sorting of the input array

Master Informatique Data Structures and Algorithms 9

 Part 2 Complexity and Correctness of Algorithms

Performance Analysis
• Often it is sufficient to count the number of iterations
of the core (innermost) part
– no distinction between comparisons, assignments, etc
(that means, roughly the same cost for all of them)
– gives precise enough results
• In some cases the cost of selected operations dominates
all other costs.
– disk I/O versus RAM operations
– database systems

Master Informatique Data Structures and Algorithms 10

 Part 2 Complexity and Correctness of Algorithms

Worst/Average/Best Case
• Analyzing Insertion Sort’s
– Worst case: elements sorted in inverse order, tj=j,
total running time is quadratic (time = an2+bn+c)

– Average case (= average of all inputs of size n):

tj=j/2, total running time is quadratic (time = an2+bn+c)

– Best case: elements already sorted, tj=1,

innermost loop is never executed,
total running time is linear (time = an+b)

• How can we define these concepts formally?

… and how much sense does “best case” make?
Master Informatique Data Structures and Algorithms 11
 Part 2 Complexity and Correctness of Algorithms

Worst/Average/Best Case/2

For a specific size of input size n, investigate running

times for different input instances:

6n worst case =4321

}
running time

5n
average case ???
4n

3n best case =1234

2n

1n

A B C D E F G
input instance

Master Informatique Data Structures and Algorithms 12

 Part 2 Complexity and Correctness of Algorithms

Worst/Average/Best Case/3

For inputs of all sizes:

worst-case

6n average-case
Running time

5n
best-case
4n

3n

2n

1n

1 2 3 4 5 6 7 8 9 10 11 12
…..
Input instance size

Master Informatique Data Structures and Algorithms 13

 Part 2 Complexity and Correctness of Algorithms

Best/Worst/Average Case/4
Worst case is most often used:
– It is an upper-bound
– In certain application domains (e.g., air traffic control,
surgery) knowing the worst-case time complexity is of
crucial importance
– For some algorithms, worst case occurs fairly often
– The average case is often as bad as the worst case

The average case depends on assumptions

– What are the possible input cases?
– What is the probability of each input?

Master Informatique Data Structures and Algorithms 14

 Part 2 Complexity and Correctness of Algorithms

Analysis of Linear Search

INPUT: A[1..n] – an array of integers,
q – an integer.
OUTPUT: j s.t. A[j]=q, or -1 if /j(15j5n): A[j]7q

j := 1
while j 5 n and A[j] != q do j++
if j 5 n then return j
else return -1

• Worst case running time: n

• Average case running time: (n+1)/2 (if q is present)
… under which assumption?

Master Informatique Data Structures and Algorithms 15

 Part 2 Complexity and Correctness of Algorithms

Binary Search: Ideas

Search the first occurrence of q in the sorted array A
• Maintain a segment A[l..r] of A such that
the first occurrence of q is in A[l..r] iff q is in A at all
– start with A[1..n]
– stepwise reduce the size of A[l..r] by one half
– stop if the segment contains only one element
• To reduce the size of A[l..r]
– choose the midpoint m of A[l..r]
– compare A[m] with q
– depending on the outcome,
continue with the left or the right half ...

Master Informatique Data Structures and Algorithms 16

 Part 2 Complexity and Correctness of Algorithms

Binary Search, Recursive Version

INPUT: A[1..n] – sorted (increasing) array of integers, q – integer.
OUTPUT: the first index j such that A[j] = q; -1, if /j (15j5n): A[j] 7 q

int findFirstRec(int q, int[] A)

if A.length = 0 then return -1;
return findFirstRec(q,A,1,A.length)

int findFirstRec(int q, int[] A, int l, int r)

if l = r then
if A[r] = q
then return r
else return -1;
m := ⌊(l+r)/2⌋ ;
if A[m] < q
then return findFirstRec(q,A,m+1,r)
else return findFirstRec(q,A,l,m)

Master Informatique Data Structures and Algorithms 17

 Part 2 Complexity and Correctness of Algorithms

Translate FindFirstRec
into an Iterative Method
Observations:
• FindFirstRec makes a recursive call only at the end
(the method is “tail recursive”)
• In each call, the arguments change
• There is no need to maintain a recursion stack

Idea:
• Instead of making a recursive call, just change the variable values
• Do so, as long as the base case is not reached
• When the base case is reached, perform the corresponding
actions

Result: iterative version of the original algorithm

Master Informatique Data Structures and Algorithms 18
 Part 2 Complexity and Correctness of Algorithms

Binary Search, Iterative Version

INPUT: A[1..n] – sorted (increasing) array of integers, q – integer.
OUTPUT: an index j such that A[j] = q. -1, if /j (15j5n): A[j] 7 q

int findFirstIter(int q, int[] A)

if A.length = 0 then return -1;
l := 1; r := A.length;
while l < r do
m := ⌊(l+r)/2⌋;
if A[m] < q
then l:=m+1
else lr:=m-1
if A[r] = q
then return r
else return -1;

Master Informatique Data Structures and Algorithms 19

 Part 2 Complexity and Correctness of Algorithms

Analysis of Binary Search

How many times is the loop executed?
– With each execution
the difference between l and r is cut in half
• Initially the difference is n = A.length
• The loop stops when the difference becomes 1
– How many times do you have to cut n in half to get 1?
– log n – better than the brute-force approach of linear
search (n).

Master Informatique Data Structures and Algorithms 20

 Part 2 Complexity and Correctness of Algorithms

Linear vs Binary Search

• Costs of linear search: n

• Costs of binary search: log2 n

• Should we care?

• Phone book with n entries:

– n = 200,000, log2 n = log2 200,000 = 8 +10
– n = 2M, log2 2M = 1 + 10 + 10
– n = 20M, log2 20M = 5 + 20

Master Informatique Data Structures and Algorithms 21

 Part 2 Complexity and Correctness of Algorithms

DSA, Part 2: Overview

• Complexity of algorithms

• Asymptotic analysis

• Special case analysis

• Correctness of algorithms

Master Informatique Data Structures and Algorithms 22

 Part 2 Complexity and Correctness of Algorithms

Asymptotic Analysis
• Goal: simplify the analysis of the running time
by getting rid of details, which are
affected by specific implementation and hardware
– “rounding” of numbers: 1,000,001 8 1,000,000
– “rounding” of functions: 3n2 + 2n + 8 8 n2

• Capturing the essence: how the running time of an

algorithm increases with the size of the input in the limit
– Asymptotically more efficient algorithms
are best for all but small inputs

Master Informatique Data Structures and Algorithms 23

 Part 2 Complexity and Correctness of Algorithms

Asymptotic Notation

The “big-Oh” O-Notation

– talks about
c :g ( n)
asymptotic upper bounds
f -n .

Running Time
– f(n) 2 O(g(n)) iff
there exist c > 0 and n0 > 0,
s.t. f(n) 5 c g(n) for n 6 n0
– f(n) and g(n) are functions n0 Input Size
over non-negative integers
Used for worst-case analysis

Master Informatique Data Structures and Algorithms 24

 Part 2 Complexity and Correctness of Algorithms

Asymptotic Notation, Example

f(n) = 2n2 + 3(n+1), g(n) = 3n2

Values of f(n) = 2n2 + 3(n+1):

2+6, 8+9, 18+12, 32+15

Values of g(n) = 3n2:

3, 12, 27, 64

From n0 = 4 onward, we have f(n) 5 g(n)

Master Informatique Data Structures and Algorithms 25

 Part 2 Complexity and Correctness of Algorithms

Asymptotic Notation, Examples

• Simple Rule: We can always drop lower order terms
and constant factors, without changing big Oh:
– 7n + 3 is O(n)
– 8n2 log n + 5n2 + n is O(n2 log n)
– 50 n log n is O(n log n)

• Note:
– 50 n log n is O(n2)
– 50 n log n is O(n100)
but this is less informative than saying
– 50 n log n is O(n log n)

Master Informatique Data Structures and Algorithms 26

 Part 2 Complexity and Correctness of Algorithms

Asymptotic Notation/2
• The “big-Omega” Ω0Notation
– asymptotic lower bound
– f(n) = Ω(g(n)) iff
there exist c > 0 and n0 > 0,

Running Time
f -n.
s.t. c g(n) 5 f(n), for n 6 n0 c :g ( n)
• Used to describe lower bounds
of algorithmic problems
– E.g., searching in n0
Input Size
a sorted array
with linear search is Ω(n),
with binary search is Ω(log n)

Master Informatique Data Structures and Algorithms 27

 Part 2 Complexity and Correctness of Algorithms

Asymptotic Notation/3
• The “big-Theta” Θ0Notation
– asymptotically tight bound
– f(n) = Θ(g(n)) if there exists c 2⋅g- n.

c1>0, c2>0, and n0>0,

Running Time
f -n.
s.t. for n 6 n0 c 1⋅g - n .

c1 g(n) 5 f(n) 5 c2 g(n)

• f(n) = Θ(g(n)) iff n0

Input Size
f(n) = Ο(g(n)) and f(n) = Ω(g(n))
• Note: O(f(n)) is often used
when Θ(f(n)) is meant

Master Informatique Data Structures and Algorithms 28

 Part 2 Complexity and Correctness of Algorithms

Asymptotic Notation/4
• Analogy with real numbers
– f(n) = O(g(n)) 4 f 5 g
– f(n) = Ω(g(n)) 4 f 6 g
– f(n) = Θ(g(n)) 4 f 2 g

• Abuse of notation:
f(n) = O(g(n)) actually means
f(n) 9 O(g(n))

Master Informatique Data Structures and Algorithms 29

 Part 2 Complexity and Correctness of Algorithms

Exercise: Asymptotic Growth

Order the following functions according to their asymptotic growth.
– 2n + n2
– 3n3 + n2 – 2n3 + 5n – n3
– 20 log2 2n
– 20 log2 n2
– 20 log2 4n
– 20 log2 2n
– 3n

Master Informatique Data Structures and Algorithms 30

 Part 2 Complexity and Correctness of Algorithms

Growth of Functions: Rules

• For a polynomial, the highest exponent determines the
long-term growth
Example: n3 + 3 n2 + 2 n + 6 = Θ(n3)
• A polynomial with higher exponent strictly outgrows one
with lower exponent
Example: n2 = Ο(n3) but n3 ≠ Ο(n2)
• An exponential function outgrows every polynomial
Example: n2 = Ο(5n) but 5n ≠ Ο(n2) constant factor)

Master Informatique Data Structures and Algorithms 31

 Part 2 Complexity and Correctness of Algorithms

Growth of Functions: Rules/2

• An exponential function with greater base strictly outgrows
an exponential function with smaller base
Example: 2n = Ο(5n) but 5n ≠ Ο(2n)
• Logarithms are all equivalent
(because identical up to a constant factor)
Example: log2 n = Θ(log5 n)
Reason: loga n = loga b logb n for all a, b > 0
• Every logarithm is strictly outgrown by a function n α,
where α > 0
Example: log5 n = Ο(n0.2) but n0.2 ≠ Ο(log5 n)

Master Informatique Data Structures and Algorithms 32

 Part 2 Complexity and Correctness of Algorithms

Comparison of Running Times

Determining the maximal problem size

Running Time
1 second 1 minute 1 hour
T(n) in µs
400n 2,500 150,000 9,000,000
20n log n 4,096 166,666 7,826,087
2n2 707 5,477 42,426
n4 31 88 244
2n 19 25 31

Master Informatique Data Structures and Algorithms 33

 Part 2 Complexity and Correctness of Algorithms

DSA, Part 2: Overview

• Complexity of algorithms

• Asymptotic analysis

• Special case analysis

• Correctness of algorithms

Master Informatique Data Structures and Algorithms 34

 Part 2 Complexity and Correctness of Algorithms

Special Case Analysis

• Consider extreme cases and make sure

your solution works in all cases.

• The problem: identify special cases.

• This is related to INPUT and OUTPUT specifications.

Master Informatique Data Structures and Algorithms 35

 Part 2 Complexity and Correctness of Algorithms

Special Cases

• empty data structure • zero, empty string

(array, file, list, …) • negative number
• single element data
• border of domain
structure
• completely filled data
structure
• start of loop
• entering a function • end of loop
• termination of a function • first iteration of loop

Master Informatique Data Structures and Algorithms 36

 Part 2 Complexity and Correctness of Algorithms

Sortedness

The following algorithm checks

whether an array is sorted.

INPUT: A[1..n] – an array of integers.

OUTPUT: TRUE if A is sorted; FALSE otherwise

for i := 1 to n
if A[i] 6 A[i+1] then return FALSE
return TRUE

Analyze the algorithm by considering special cases.

Master Informatique Data Structures and Algorithms 37

 Part 2 Complexity and Correctness of Algorithms

Sortedness/2

INPUT: A[1..n] – an array of integers.

OUTPUT: TRUE if A is sorted; FALSE otherwise

for i := 1 to n
if A[i] 6 A[i+1] then return FALSE
return TRUE

• Start of loop, i=1 è OK

• End of loop, i=n è ERROR (tries to access A[n+1])

Master Informatique Data Structures and Algorithms 38

 Part 2 Complexity and Correctness of Algorithms

Sortedness/3

INPUT: A[1..n] – an array of integers.

OUTPUT: TRUE if A is sorted; FALSE otherwise

for i := 1 to n–1
if A[i] 6 A[i+1] then return FALSE
return TRUE

●
Start of loop, i=1 è OK
●
End of loop, i=n-1 è OK
●
A=[1,2,3] è First iteration, from i=1 to i=2 è OK
●
A=[1,2,2] è ERROR (if A[i]=A[i+1] for some i)

Master Informatique Data Structures and Algorithms 39

 Part 2 Complexity and Correctness of Algorithms

Sortedness/4

INPUT: A[1..n] – an array of integers.

OUTPUT: TRUE if A is sorted; FALSE otherwise

for i := 1 to n-1
if A[i] 3 A[i+1] then return FALSE
return TRUE

• Start of loop, i=1 è OK

• End of loop, i=n-1 è OK
• A=[1,2,3] è First iteration, from i=1 to i=2 è OK
• A=[1,1,1] è OK
• Empty data structure, n=0 è ? (for loop)
• A=[-1,0,1,-3] è OK

Master Informatique Data Structures and Algorithms 40

 Part 2 Complexity and Correctness of Algorithms

Binary Search, Variant 1

Analyze the following algorithm
by considering special cases.

l := 1; r := n
do
m := ⌊(l+r)/2⌋
if A[m] = q then return m
else if A[m] > q then r := m-1
else l := m+1
while l < r
return -1

Master Informatique Data Structures and Algorithms 41

 Part 2 Complexity and Correctness of Algorithms

Binary Search, Variant 1

l := 1; r := n
do
m := ⌊(l+r)/2⌋
if A[m] = q then return m
else if A[m] > q then r := m-1
else l := m+1
while l < r
return -1

• Start of loop è OK
• End of loop, l=r è Error! Example: search 3 in [3 5 7]

Master Informatique Data Structures and Algorithms 42

 Part 2 Complexity and Correctness of Algorithms

Binary Search, Variant 1

l := 1; r := n
do
m := ⌊(l+r)/2⌋
if A[m] = q then return m
else if A[m] > q then r := m-1
else l := m+1
while l <= r
return -1

• Start of loop è OK
• End of loop, l=r è OK
• First iteration è OK
• A=[1,1,1] è OK
• Empty array, n=0 è Error! Tries to access A[0]
• One-element array, n=1 è OK
Master Informatique Data Structures and Algorithms 43
 Part 2 Complexity and Correctness of Algorithms

Binary Search, Variant 1

l := 1; r := n
if r < l then return -1;
do
m := ⌊(l+r)/2⌋
if A[m] = q then return m
else if A[m] > q then r := m-1
else l := m+1
while l <= r
return -1
• Start of loop è OK
• End of loop, l=r è OK
• First iteration è OK
• A=[1,1,1] è OK
• Empty data structure, n=0 è OK
• One-element data structure, n=1 è OK
Master Informatique Data Structures and Algorithms 44
 Part 2 Complexity and Correctness of Algorithms

Binary Search, Variant 2

Analyze the following algorithm
by considering special cases

l := 1; r := n
while l < r do
m := ⌊(l+r)/2⌋
if A[m] <= q
then l := m+1 else r := m
if A[l-1] = q
then return l-1 else return -1

Master Informatique Data Structures and Algorithms 45

 Part 2 Complexity and Correctness of Algorithms

Binary Search, Variant 3

Analyze the following algorithm
by considering special cases

l := 1; r := n
while l <= r do
m := ⌊(l+r)/2⌋
if A[m] <= q
then l := m+1 else r := m
if A[l-1] = q
then return l-1 else return -1

Master Informatique Data Structures and Algorithms 46

 Part 2 Complexity and Correctness of Algorithms

Insertion Sort, Slight Variant

• Analyze the following algorithm
by considering special cases
• Hint: beware of lazy evaluations

INPUT: A[1..n] – an array of integers

OUTPUT: permutation of A s.t.
A[1]5 A[2] 5 … 5 A[n]

for j := 2 to n do
key := A[j]; i := j-1;
while A[i] > key and i > 0 do
A[i+1] := A[i]; i--;
A[i+1] := key

Master Informatique Data Structures and Algorithms 47

 Part 2 Complexity and Correctness of Algorithms

DSA, Part 2: Overview

• Complexity of algorithms

• Asymptotic analysis

• Special case analysis

• Correctness of algorithms

Master Informatique Data Structures and Algorithms 48

 Part 2 Complexity and Correctness of Algorithms

Correctness of Algorithms
• An algorithm is correct if for every legal input,
it terminates and produces the desired output.

• Automatic proof of correctness is not possible

(this is one of the so-called “undecidable problems”)

• There are practical techniques and rigorous formalisms

that help one to reason about the correctness of
(parts of) algorithms.

Master Informatique Data Structures and Algorithms 49

 Part 2 Complexity and Correctness of Algorithms

Partial and Total Correctness

n Partial correctness
IF this point is reached, THEN this is the output

every legal input Algorithm output

n Total correctness
INDEED this point is reached, AND this is the output

every legal input Algorithm output

Master Informatique Data Structures and Algorithms 50

 Part 2 Complexity and Correctness of Algorithms

Assertions
• To prove partial correctness we associate a number of
assertions (statements about the state of the execution)
with specific checkpoints in the algorithm.

– E.g., “A[1], …, A[j] form an increasing sequence”

• Preconditions – assertions that must be valid before the

execution of an algorithm or a subroutine (INPUT)

• Postconditions – assertions that must be valid after the

execution of an algorithm or a subroutine (OUTPUT)

Master Informatique Data Structures and Algorithms 51

 Part 2 Complexity and Correctness of Algorithms
Pre- and Postconditions of
Linear Search
INPUT: A[1..n] – a array of integers,
q – an integer.
OUTPUT: j s.t. A[j]=q. -1 if /i(15i5n): A[i]7q

j := 1
while j 5 n and A[j] != q do j++
if j 5 n then return j
else return -1

How can we be sure that

– whenever the precondition holds,
– also the postcondition holds?

Master Informatique Data Structures and Algorithms 52

 Part 2 Complexity and Correctness of Algorithms

Loop Invariant in Linear Search

j := 1
while j 5 n and A[j] != q do j++
if j 5 n then return j
else return -1

Whenever the beginning of the loop is reached, then

A[i] != q for all i where 1 5 i 1 j

When the loop stops, there are two cases

– j = n+1, which implies A[i] != q for all i, 1 5 i 1 n+1
– A[j] = q

Master Informatique Data Structures and Algorithms 53

 Part 2 Complexity and Correctness of Algorithms

Loop Invariant in Linear Search

j := 1
while j 5 n and A[j] != q do j++
if j 5 n then return j
else return -1

Note: The condition

A[i] != q for all i where 1 5 i 1 j

– holds when the loop is entered for the first time

– continues to hold until we reach the loop
for the last time

Master Informatique Data Structures and Algorithms 54

 Part 2 Complexity and Correctness of Algorithms

Loop Invariants
• Invariants: assertions that are valid every time the
beginning of the loop is reached
(many times during the execution of an algorithm)
• We must show three things about loop invariants:
– Initialization: it is true prior to the first iteration.
– Maintenance: if it is true before an iteration,
then it is true after the iteration.
– Termination: when a loop terminates,
the invariant gives a useful property to show the
correctness of the algorithm

Master Informatique Data Structures and Algorithms 55

 Part 2 Complexity and Correctness of Algorithms

Example: Version of Binary Search/1

l:= 1; r:= n;
• We want to show that m:= ⌊(l+r)/2⌋;
while l <= r and A(m) != q do
q is not in A if q < A(m)
if -1 is returned. then r:=m-1
else l:=m+1
• Invariant: m:=⌊(l+r)/2⌋;
/i9[1..l-1]: A[i]<q (ia) if l > r
/i9[r+1..n]: A[i]>q (ib) then return -1
else return m
• Initialization: l = 1, r = n
the invariant holds because
there are no elements to the left of l or to the right of r.
l = 1 yields / i 9 [1..0]: A[i]<q
this holds because [1..0] is empty
r = n yields / i 9 [n+1..n]: A[i]>q
this holds because [n+1..n] is empty
Master Informatique Data Structures and Algorithms 56
 Part 2 Complexity and Correctness of Algorithms

Example: Version of Binary Search/2

l:= 1; r:= n;
m:= ⌊(l+r)/2⌋;
• Invariant: while l <= r and A(m) != q do
/i9[1..l-1]: A[i]<q (ia) if q < A(m)
/i9[r+1..n]: A[i]>q (ib) then r:=m-1
else l:=m+1
m := ⌊(l+r)/2⌋;
if l > r
then return -1
else return m
• Maintenance: 1 ≤ l, r ≤ n, m = ⌊(l+r)/2⌋
We consider two cases:
– A[m] != q & q < A[m]: implies r = m-1
A sorted implies /k9[r+1..n]: A[k] > q (ib)
– A[m] != q & A[m] < q: implies l = m+1
A sorted implies /k9[1..l-1]: A[k] < q (ia)
Master Informatique Data Structures and Algorithms 57
 Part 2 Complexity and Correctness of Algorithms

Example: Version of Binary Search/3

l:= 1; r:= n;
• Invariant: m:= ⌊(l+r)/2⌋;
while l <= r and A(m) != q do
/i9[1..l-1]: A[i]<q (ia) if q < A(m)
/i9[r+1..n]: A[i]>q (ib) then r:=m-1
else l:=m+1
m := ⌊(l+r)/2⌋;
if l > r
then return -1
else return m

• Termination: 1 ≤ l, r ≤ n, l ≤ r
Two cases:
l := m+1 implies lnew = ⌊(l+r)/2⌋+ 1 > lold
r := m-1 implies rnew = ⌊(l+r)/2⌋ - 1 < rold
• The range gets smaller during each iteration and
the loop will terminate when l ≤ r no longer holds
Master Informatique Data Structures and Algorithms 58
 Part 2 Complexity and Correctness of Algorithms

for j := 2 to n do
key := A[j]
i := j-1
while i>0 and A[i]>key do
A[i+1] := A[i]
i--
A[i+1] := key

Master Informatique Data Structures and Algorithms 59

 Part 2 Complexity and Correctness of Algorithms

Example: Insertion Sort/1

for j := 2 to n do
Loop invariants: key := A[j]
External “for” loop i := j-1
while i>0 and A[i]>key do
Let Aorig denote the array at
A[i+1] := A[i]
the beginning of the for loop: i--
A[1..j-1] is sorted A[i+1] := key
A[1..j-1] 9 Aorig[1..j-1]

Internal “while” loop

Let Aorig denote the array at beginning of the while loop:
• A[1..i] = Aorig[1..i]
• A[i+2..j] = Aorig[i+1..j-1]
• A[k] > key for all k in {i+2,...,j}

Master Informatique Data Structures and Algorithms 60

 Part 2 Complexity and Correctness of Algorithms

Example: Insertion Sort/2

External for loop: for j := 2 to n do
(i) A[1...j-1] is sorted key := A[j]
(ii) A[1...j-1] 9 Aorig[1..j-1] i := j-1
Internal while loop: while i>0 and A[i]>key do
A[i+1] := A[i]
– A[1..i] = Aorig[1..i]
i--
– A[i+2..j] = Aorig[i+1..j-1]
A[i+1] := key
– A[k] > key for all k in {i+2,...,j}

Initialization:
External loop: (i), (ii) j = 2: A[1..1] 9 Aorig[1..1] and is trivially sorted
Internal loop: i = j-1:
– A[1...j-1] = Aorig[1..j-1] , since nothing has happend
– A[j+1..j] = Aorig[j..j-1] , since both sides are empty
– A[k] > key holds trivially for all k in the empty set

Master Informatique Data Structures and Algorithms 61

 Part 2 Complexity and Correctness of Algorithms

Example: Insertion Sort/3

External for loop:
for j := 2 to n do
(i) A[1..j-1] is sorted
key := A[j]
(ii) A[1..j-1] 9 Aorig[1..j-1] i := j-1
Internal while loop: while i>0 and A[i]>key do
– A[1..i] = Aorig[1..i] A[i+1] := A[i]
– A[i+2..j] = Aorig[i+1..j-1] i--
– A[k] > key for all k in {i+2,...,j} A[i+1] := key

Maintenance internal while loop

Before the decrement “i--”, the following facts hold:
– A[1..i-1] = Aorig[1..i-1] (because nothing in A[1..i-1] has been changed)
– A[i+1..j] = Aorig[i..j-1] (because A[i] has been copied to A[i+1] and
A[i+2..j] = Aorig[i+1..j-1]
– A[k] > key for all k in {i+1,...,j} (because A[i] has been copied to A[i+1])
After the decrement “i--”, the invariant holds because i-1 is replaced by i.

Master Informatique Data Structures and Algorithms 62

 Part 2 Complexity and Correctness of Algorithms

Example: Insertion Sort/4

External for loop:
for j := 2 to n do
(i) A[1..j-1] is sorted
key := A[j]
(ii) A[1..j-1] 9 Aorig[1..j-1] i := j-1
Internal while loop: while i>0 and A[i]>key do
– A[1..i] = Aorig[1..i] A[i+1] := A[i]
– A[i+2..j] = Aorig[i+1..j-1] i--
– key < A[k] for all k in {i+2,...,j} A[i+1] := key

Termination internal while loop

The while loop terminates, since i is decremented in each round.
Termination can be due to two reasons:
i=0: A[2..j] = Aorig[1..j-1] and key < A[k] for all k in {2,...,j} (because of the invariant)
implies key, A[2..j] is a sorted version of Aorig[1..j]
A[i] 5 key: A[1..i] = Aorig[1..i], A[i+2..j] = Aorig[i+1..j-1], key = Aorig[j]
implies A[1..i], key, A[i+2..j] is a sorted version of Aorig[1..j]

Master Informatique Data Structures and Algorithms 63

 Part 2 Complexity and Correctness of Algorithms

Example: Insertion Sort/5

External for loop:
for j := 2 to n do
(i) A[1..j-1] is sorted
key := A[j]
(ii) A[1..j-1] 9 Aorig[1..j-1] i := j-1
Internal while loop: while i>0 and A[i]>key do
– A[1..i] = Aorig[1..i] A[i+1] := A[i]
– A[i+2..j] = Aorig[i+1..j-1] i--
– key < A[k] for all k in {i+2,...,j} A[i+1] := key

Maintenance of external for loop

When the internal while loop terminates, we have (see previous slide):
A[1..i], key, A[i+2..j] is a sorted version of Aorig[1..j]
After
– assigning key to A[i+1] and
– Incrementing j,
the invariant of the external loop holds again.

Master Informatique Data Structures and Algorithms 64

 Part 2 Complexity and Correctness of Algorithms

Example: Insertion Sort/6

External for loop:
for j := 2 to n do
(i) A[1..j-1] is sorted
key := A[j]
(ii) A[1..j-1] 9 Aorig[1..j-1] i := j-1
Internal while loop: while i>0 and A[i]>key do
– A[1..i] = Aorig[1..i] A[i+1] := A[i]
– A[i+2..j] = Aorig[i+1..j-1] i--
– key < A[k] for all k in {i+2,...,j} A[i+1] := key

Termination of external for loop

The for loop terminates because j is incremented in each round.
Upon termination, j = n+1 holds.
In this situation, the loop invariant of the for loop says:

A[1..n] is sorted and contains the same values as Aorig[1..n]

That is, A has been sorted.

Master Informatique Data Structures and Algorithms 65

 Part 2 Complexity and Correctness of Algorithms

Example: Bubble Sort

INPUT: A[1..n] – an array of integers
OUTPUT: permutation of A s.t. A[1]≤ A[2]≤ … ≤ A[n]

for j := 1 to n-1 do
for i := n downto j+1 do
if A[i-1] > A[i] then
swap(A,i-1,i)

• What is a good loop invariant for the outer loop?

(i.e., a property that always holds at the end of line 1)
• … and what is a good loop invariant for the inner loop?
(i.e., a property that always holds at the end of line 2)

Master Informatique Data Structures and Algorithms 66

 Part 2 Complexity and Correctness of Algorithms

Example: Bubble Sort

A 1 2 3 4 5 7 9 8 6
1 j n

Strategy
• Start from the back
and compare pairs 44 55 12 42 94 18 06 67
of adjacent elements.
• Swap the elements 06 44 55 12 42 94 18 67
if the larger comes 06 12 44 55 18 42 94 67
before the smaller.
06 12 18 44 55 42 67 94
• In each step
the smallest element 06 12 18 42 44 55 67 94
of the unsorted part 06 12 18 42 44 55 67 94
is moved to the beginning
of the unsorted part and the 06 12 18 42 44 55 67 94
sorted part grows by one. 06 12 18 42 44 55 67 94

Master Informatique Data Structures and Algorithms 67

 Part 2 Complexity and Correctness of Algorithms

Exercise
• Apply the same approach that we used for insertion sort
to prove the correctness of bubble sort and
selection sort.

Master Informatique Data Structures and Algorithms 68

 Part 2 Complexity and Correctness of Algorithms

Math Refresher
●
Arithmetic progression
n
∑i= 0 i=0,1,...,n=n -n,1./ 2
●
Geometric progression (for a number a ≠ 1)
n
∑i= 0 a i=1,a 2 ,...,a n=-1−a n,1 ./-1−a.

Master Informatique Data Structures and Algorithms 69

 Part 2 Complexity and Correctness of Algorithms

Induction Principle
We want to show that property P is true
for all integers n 6 n0.
Basis: prove that P is true for n0.
Inductive step: prove that if P is true for all k
such that n0 5 k 5 n – 1 then P is also true for n.

Exercise: Prove that every Fibonacci number

of the form fib(3n) is even

Master Informatique Data Structures and Algorithms 70

 Part 2 Complexity and Correctness of Algorithms

Summary
●
Algorithmic complexity
• Asymptotic analysis
– Big O and Theta notation
– Growth of functions and asymptotic notation
• Correctness of algorithms
– Pre/Post conditions
– Invariants
• Special case analysis

Master Informatique Data Structures and Algorithms 71