Sorting and

Complexity Analysis
Algorithmic Efficiency
• We sometimes say “algorithm A is faster or more efficient than
algorithm B.” But how do we actually measure efficiency?

• We shall discuss how we can qualitatively measure the efficiency of

an algorithm.

• We start with the problem of sorting.

2
The Sorting Problem
• The problem of sorting is to reorder the elements in an array so that
they fall in some defined sequence.

56 25 37 58 95 19 73 30
0 1 2 3 4 5 6 7

Ascending order 19 25 30 37 56 58 73 95
0 1 2 3 4 5 6 7 3
The Selection Sort Algorithm
• One of the simplest sorting algorithms is the selection sort.

• The algorithm goes through each array position and selects a suitable
value for that position.
array[0] : swap with the smallest
value of array[0] to array[n]
array[1] : swap with the smallest
Position 0 56 25 37 58 95 19 73 30 value of array[1] to array[n]

0 1 2 3 4 5 6 7 array[2] : swap with the smallest

value of array[2] to array[n]
:
:
4
The Selection Sort Algorithm
Initial: 56 25 37 58 95 19 73 30 selects a value for:
array[0] 56 : swap with 19
Round 0: 19 25 37 58 95 56 73 30
array[1] 25 : no swap
Round 1: 19 25 37 58 95 56 73 30
array[2] 37 : swap with 30
Round 2: 19 25 30 58 95 56 73 37
array[3] 58 : swap with 37
Round 3: 19 25 30 37 95 56 73 58
array[4] 95 : swap with 56
Round 4: 19 25 30 37 56 95 73 58
array[5] 95 : swap with 58
Round 5: 19 25 30 37 56 58 73 95
array[6] 73 : no swap
Round 6: 19 25 30 37 56 58 73 95

Goes through each array position and

selects a suitable value for that position. 5
The Selection Sort Algorithm
• Round 0: find the smallest element in array[0 … n-1] and exchange it with array[0].
• Round 1: find the smallest element in array[1 … n-1] and exchange it with array[1].
• Round 2: find the smallest element in array[2 … n-1] and exchange it with array[2].
• …
• Round i: find the smallest element in array[i … n-1] and exchange it with array[i].
• …
• Round n-2: find the smallest element in array[n-2 … n-1] and exchange it with
array[n-2].

6
 Selection Sort Implementation
void SelectionSort(int array[], int n) {
int i, j, k; Round i
for (i = 0; i < n - 1; i++) {

(Find the index k such that array[k] is

the smallest in array[i … n-1].)

(Exchange array[i] and array[k].)

}
}

Round i: find the smallest element in array[i … n-1] and exchange it with array[i].

Round 0 to Round n-2

7
 Selection Sort Implementation
void SelectionSort(int array[], int n) {
int i, j, k; Round i
for (i = 0; i < n - 1; i++) {
k = i;
for (j = i + 1; j < n; j++) Find the index k
if (array[j] < array[k]) such that array[k]
k = j; is the smallest in
array[i … n-1]
(Exchange array[i] and array[k])

}
}

Round i: find the smallest element in array[i … n-1] and exchange it with array[i].

8
 Selection Sort Implementation
void SelectionSort(int array[], int n) {
int i, j, k, tmp; Round i
for (i = 0; i < n - 1; i++) {
k = i;
for (j = i + 1; j < n; j++) Find the index k
if (array[j] < array[k]) such that array[k]
k = j; is the smallest in
tmp = array[i]; array[i … n-1]
array[i] = array[k];
array[k] = tmp;
}
Exchange array[i]
}
and array[k]

Round i: find the smallest element in array[i … n-1] and exchange it with array[i].

9
How Efficient is Selection Sort?
• Some empirical measurements
N Time (s)
Observations:
10,000 0.265
Doubling N increases the running
20,000 1.061 time by 4 times roughly.
40,000 4.260
Multiplying N by 10 times increases
100,000 26.797 the running time by 100 times roughly.
110,000 32.557
120,000 38.721
140,000 54.142 CPU: AMD Athlon™ 64 3500+ (2.2GHz)

200,000 110.032
10
Analyzing Selection Sort
Initial: 56 25 37 58 95 19 73 30

• Round 0: must consider all N elements in array[0 … n-1]

Round 0: 19 25 37 58 95 56 73 30
• Round 1: must consider N - 1 elements in array[1 … n-1]
Round 1: 19 25 37 58 95 56 73 30
• Round 2: must consider N - 2 elements in array[2 … n-1]
Round 2: 19 25 30 58 95 56 73 37
• Round 3: must consider N - 3 elements…
Round i: find the smallest element in
array[i … n-1] and exchange it with array[i].

11
Analyzing Selection Sort
• The total running time is roughly proportional to

N + (N-1) + (N-2) + … + 3 + 2 + 1
= N(N+1)/2
= (N2 + N)/2

12
How Big is (N2 + N)/2 ?
N Time (s) F(N)=(N2 + N)/2
10,000 0.265 50,005,000
Observations:
20,000 1.061 200,010,000
40,000 4.260 800,020,000 Doubling N increases F(N) by 4
times roughly.
100,000 26.797 5,000,050,000
Multiplying N by 10 times increases
110,000 32.557 6,050,055,000 F(N) by 100 times roughly.
120,000 38.721 7,200,060,000
140,000 54.142 9,800,070,000
200,000 110.032 20,000,100,000

13
Analyzing an Algorithm
• Precise running time of an algorithm depends on specific computer
hardware.

• The essence of analyzing the selection sort, however, is how the

algorithm responds to changes in the size N of the array.
• That is,

Doubling N increases the

running time by 4 times roughly.

14
Computational Complexity
• The relationship between the problem size N and the performance of
an algorithm as N becomes large is called the computational
complexity (or time complexity) of the algorithm.

• To denote computational complexity, we use the big-O notation.

15
Big-O Notation
• The big-O notation is used to provide a quantitative insight as to how
changes in the problem size N affect the algorithmic performance as
N becomes large.

• For example, as we shall see, the computational complexity of

selection sort is O(N2) (Read as “big-O of N squared.”)

16
Standard Simplifications of Big-O
• Before giving the formal definition, let’s see how we can simplify a
formula when using big-O notation.

• We illustrate the simplifications using the formula obtained from

selection sort:
(N2 + N)/2

17
Simplification Rule 1
• Eliminate any term whose contribution to the total becomes
insignificant as N becomes large.

• Example: (N2 + N)/2

= N2/2 + N/2
= O(N2/2)

18
Simplification Rule 1
N N2/2 N/2 (N2 + N)/2
10 50 5 55
100 5,000 50 5,050
1,000 500,000 500 500,500
10,000 50,000,000 5,000 50,005,000
100,000 5,000,000,000 50,000 5,000,050,000
The term N/2 (comparing with N2/2)
becomes insignificant to the total
value when N becomes large.
19
Simplification Rule 1
• When a formula involves a summation of several terms, the fastest
growing term alone will control the running time of the algorithm for
large N.

• More examples
• N + 1 = O(N)
• N3 + 1000N2 + N = O(N3)

20
Simplification Rule 2
• Eliminate any constant factors.

• Example: (N2 + N)/2

= O(N2/2) Rule 1
= O(N2) Rule 2

21
 Increase by100 times
when N is increased by 10 times
Simplification Rule 2
N N2/2 N2
10 50 100
100 5,000 10,000
1,000 500,000 1,000,000
10,000 50,000,000 100,000,000
100,000 5,000,000,000 10,000,000,000
The constant factor 1/2
has no effect on the growth
rate.
22
Simplification Rule 2
• What we want to capture in computational complexity is how
changes in N affect the algorithmic performance.
• Constant factors have no effect on the growth rate.

• More examples
• 10000N0.5 = O(N0.5)
• 0.0001N3 + 10000N2 + N + 3 = O(N3)

23
Exercises
• 2N9 + N = O(?)
• 7N - 2N1/2 + 4 = O(?)
• N3/2 - 2N1/2 = O(?)
• 2N + 4log N = O(?)
• N2 + Nlog N = O(?)

24
Implications of Computational Complexity
• Recall that the computational complexity of selection sort is O(N2).

• An implication on O(N2) is that the running time grows by the square

of the increase in the problem size.

• This precisely captures the performance of selection sort, which is

doubling N increases the running time by 4 times,
multiplying N by 10 times increases the running time by 100 times

25
Determining Computational Complexity from Code
Structure
• What is the computational complexity of the following function?

double Average(double *array, int n) {

int i;
double total = 0.0;
Loop: for (i = 0; i < n; i++) Each other statement
N iterations total += array[i]; executed once.
return total / n;
}

26
Determining Computational Complexity from Code
Structure
• What is the computational complexity of the following function?

double Average(double *array, int n) {

int i;
double total = 0.0; Each of these statements
for (i = 0; i < n; i++) runs in constant time
total += array[i]; independent totoN).
(independent N).
return total / n;
} Constant is denoted as
O(1) in big-O notation.
• Hence, computational complexity is O(N).
• Commonly called linear time.

27
Determining Computational Complexity from Code
Structure
• In general, we can determine the time complexity simply by finding
the piece of the code that is executed most often.
for (i = 0; i < n; i++)
total += array[i];

• However, if an expression or statement involves function calls, it must

be accounted separately.

28
Determining Computational Complexity from Code
Structure
• What about this one?
double Variance(double array[], int n) { System dependent,
double k, *temp; usually O(1) or O(N)
int i; time.
temp = (double *)malloc(n * sizeof(double));
for (i = 0; i < n; i++) {
Loop: k = array[i] - Average(array, n); O(N) time
N iterations temp[i] = k * k;
} O(1) time
return Average(temp, n);
}
Totally O(N (N + 1)) =
O(N) time O(N2) time for this part.
• O(N2 + N) = O(N2)
• Commonly called quadratic time.
29
Determining Complexity from Code Structure
• With a little bit revision,
double Variance(double array[], int n) {
double k, mean, *temp;
int i;
temp = (double *)malloc(n * sizeof(double));
mean = Average(array, n);
for (i = 0; i < n; i++) { O(N) time

Loop body: k = array[i] – mean;

temp[i] = k * k; O(1) time
N iterations
}
return Average(temp, n);
} Totally O(N × 1) = O(N)
O(N) time time only for this part.
• it improves to O(N + N) = O(N).

30
Selection Sort Revisited
void SelectionSort(int array[], int n) {
int i, j, k;
for (i = 0; i < n - 1; i++) {
k = i;
for (j = i + 1; j < n; j++)
Outer loop: if (array[j] < array[k]) Inner loop: O(N)
O(N) iterations k = j; iterations
j = array[i];
array[i] = array[k];
array[k] = j;
}
}

Each single statement/expression executes in O(1) time.

• O(N × N) = O(N2)
31
Formal Definition of Big-O
• Definition: T(N) = O(f(N)) if and only if
• there are positive constants n0 and c such that for every value of N ≥ n0, the
following condition holds:
T(N) ≤ c × f(N)

• As long as N is “large enough,” T(N) is always bounded by a constant

multiple of f(N).

32
 for N ≥ n0, T(N) ≤ c × f(N)
Example: why (N2 + N)/2 = O(N2)?
• To prove (N2 + N)/2 = O(N2), we need to find constants
n0 and c so that for all values of N ≥ n0, (N2 + N)/2 ≤ cN2

• We know that N ≤ N2 when N ≥ 1

• Therefore, for all N ≥ n0 = 1, we have
(N2 + N)/2 ≤ (N2 + N2)/2
= N2
= 1N2
• Thus, setting n0 = 1 and c = 1 completes the proof

33
 (N2 + N)/2 = O(N2)
2500 f(N) = N2

2000

1500

T(N) = (N2 + N)/2

1000

500

0
0 5 10 15 20 25 30 35 40 45 50

N 34
Examples
(A) 2 + 4 =
for all N ≥ 4, 2 +4≤2 + =3
(n0 = 4 and c = 3)

(B) + 1000 =
for all N ≥ 4, + 1000 ≤ + =2
Note that
when N = 3, N5 = 243 < 1000
when N = 4, N5 = 1024 > 1000 (or 1000 < N5)

(n0 = 4 and c = 2)
35
Polynomials
• In general, given a polynomial P(N) of degree k,

= + +…+ + +

where a0, …, ak and k are constants, we can prove that

= ( )

36
Examples
4N + log N = O(N )
for all N ≥ 1, 4N + log N · 4N + N
= 5N

(n0 = 1 and c = 5)

37