Data Structures and

Algorithms

Lesson 03:
Runtime Analysis and Sorting

Nasir Ali Kalmati

 How Long?
• You want fast algorithms

• You could just run some “experiments”, and check

how long your algorithm takes

• But what if algorithm will need to be run on a larger

problem than I used in the experiments?

• If the problem is twice as big, will my algorithm take

just twice as long???
 public static int sum(int[] array){

int sum = 0;
for(int i=0; i<array.length; i++){
sum += array[i];
}
return sum;
}

• “Cost” can be measured in number of executed statements

• Given size of array N, the loop will be taken N times

• There is some constant cost independent of N, eg creation of

“int sum” variable

• If N doubles, would expect function will be roughly twice as slow

instructions(N)= 3N + 4
instructions(N=3)=13
instructions(N=0)=4
1. int sum = 0;
1. int sum = 0;
2. int i=0;
2. int i=0;
3. i<array.length;
3. i<array.length;
4. sum += array[i];
4. return sum;
5. i++
• Number of instructions 6. i<array.length;
depends on size N of the 7. sum += array[i];
array, plus some constant 8. i++
cost 9. i<array.length;
• Can be represented with a 10. sum += array[i];
function, eg f(N)=3N+4 11. i++
• For large N, constants are 12. i<array.length;
not important 13. return sum;
public static int pairs(int[] array){
int pairs = 0;

for(int i=0; i<array.length; i++){

for(int j=0; j<array.length; j++){
if(i!=j && array[i] ==
array[j]){
pairs++;
}
}
}
return pairs;
}
 /*
On my machine, repeated 100
times:
public static int pairs(int[] array){
int pairs = 0; N=100 seconds=0.005
N=200 seconds=0.005
for(int i=0; i<array.length; i++){ N=400 seconds=0.012
for(int j=0; j<array.length; j++){ N=800 seconds=0.072
if(i!=j && array[i] == array[j]){ N=1600 seconds=0.211
pairs++; N=3200 seconds=0.754
} N=6400 seconds=2.829
N=12800 seconds=11.48
}
*/
}
return pairs;
}

•
 Scalability
• When analyzing algorithms, we will not look at the low level
optimization details

• N as representation of the problem size (eg, length of array

or number of elements in a container)

• How does the algorithm scale for larger sizes???

• Example: if my website works fine with a load of 100

users, what will happen with 2,000??? Will I just need 20
times the resources?
 Wheat/Rice and
Chessboard Problem
• 1 rice grain on first square

• Double at each square

• How many grains on the board?

• 18,446,744,073,709,551,615

• ie, 18 Quintillions
 Analysis of Algorithms

• Mathematically define the cost as a function of the input

size

• Precise functions can be impractical, so we need

approximations

• Usually, we are interested in upper and lower bounds

 Example

•
 Which Is Better?

•
 Large N???
• How do we define large?
• 10? 50? 10000000000000000???

• We can’t really say… however, things grow so fast…

what we think is large today, is likely going to be
considered tiny in few years…

• Today I know how fast my algorithms are, because I

run them. But I want to know how they will scale to
the larger problem instances of tomorrow.
• Eg, when my apps get more users
 FPS… large increase in number
of polygons to render…

Doom (2016)

Doom (1993)
 Scalability

•
 Big O Upper Bound

•
• g(n) = n

• Examples: c = 2, n’ = 1

• n < 2n for n >= 1

•
• g(n) = n

• Examples: c = 11, n’ = 1

• 10n < 11n for n >= 1

• g(n) = n

• Examples: c = 12, n’ = 3

• 10n + 5 < 12n for n >= 3

• Eg: n=3 -> f(n)=35,

g(n)=36

• Note: for n<=2, f(n) is

actually larger
•
• g(n) = n

• Examples: c = 1, n’ = 6

• 5 < n for n >= 6

• g(n) = 7

• Examples: c = 1, n’ = 1

• 5 < 7 for regardless of n

• g(n) = 1

• Examples: c = 7, n’ = 1

• 5 < 7 for regardless of n

• g(n) = 1

• Whatever c I use (eg 5),

f(n)<c will not hold for
n’>c, eg n’=c+1
• g(n) = n

• Whatever c I use (eg 5),

f(n)<cn will not hold for
n’>c, eg n’=c+1
 Big O Examples

•
 Big O Rules of Thumb

•
 Notation

•
 Big Ω Lower Bound

•
•
public static int sum(int[] array){
int sum = 0;
for(int i=0; i<array.length; i++){
sum += array[i];
}
return sum;
}

• In this case, the actual cost in number of instructions is

f(n) = 3n + 4

• Asymptotically, we can say that 3n+4 = Θ(n)

• As the number of instructions does not depend on the

content of the array, the best case and worst case for
the runtime are the same
 Order Of Growth Classification
• 1: constant (best you can have)

• log(N): logarithmic (very, very efficient)

• N: linear (OK for most cases)

• N log(N): linearithmic (OK for most cases)

2
• N : quadratic (bearable, but things start to get expensive)

• N3: cubic (becoming painful)

• 2N: exponential (completely hopeless, time to cry in a corner)

 Which Scales Best?

•
Purple represents possible actual runtimes for those bounds

1 log n n n log n n2 n3 2n
 In Practice

•
 A Big Mistake

•
 public static void doSomething(int[] array){

for(int i=0; i<array.length; i++){

for(int j=i; j<array.length; j++){
//... something
}
}
}

•
Let’s Dig Into the Math…
public static void doSomething(int[] array){

for(int i=0; i<array.length; i++){

for(int j=i; j<array.length; j++){
//... something

•
•
Sorting
Consider a Playlist
 Sort by Title or Artist,
Ascending or Descending
How do you sort when
playing cards?
 Sorting Algorithms
• Many different sorting algorithms, with different properties

• Given two items A and B, just need a comparator that can state which
one is greater or equal
• easy to say that 5 greater than 2, but what does it mean that song A is greater than
song B? e.g., could look at alphabetic ordering of titles or artist names

• Most language APIs provides good defaults

• Unless very large data, default will be fine 99% of the cases

• Sorting is very popular in programming

• Important to understand how it works under the hood

• Tractable mathematically
• So good example to show how to analyze algorithms
 Sorting Algorithms
• Bubble Sort

• Insertion Sort

• Merge Sort (next class)

• Quick Sort (next class)

• There are more, but those are the most famous that you
need to know

• Good way to see a problem been solved in many different

ways
 Bubble Sort

• Easiest sorting algorithms

• From left to right

• Look at adjacent cards, and swap them if not in order

• Repeat from left to right till no more swap

No swap, they are in order

Swap

No swap, they are in order

 Swap

• Restart from beginning.

• At each iteration, at least one card will be in
right position, as it bubbles up to the to top.
 Runtime of Bubble Sort

•
 Insertion Sort
• An array of size 0 or 1 is 4 5 1 3 2 6 K=0
always considered sorted
4 5 1 3 2 6 K=1
• From left to right, till length N
4 5 1 3 2 6 K=2
• K-leftmost values are sorted
4 5 1 3 2 6 K=3
• Position K+1 is not sorted,
4 1 5 3 2 6 swap
insert it in the first K
• by swapping adjacent elements, like
1 4 5 3 2 6 swap
in Bubble Sort
 Cont.
1 4 5 3 2 6 K=4

1 4 3 5 2 6 swap

1 3 4 5 2 6 swap

1 3 4 5 2 6 K=5

1 3 4 2 5 6 swap

1 3 2 4 5 6 swap

1 2 3 4 5 6 swap

1 2 3 4 5 6 K=6
 Best vs. Worst Case

•
 Homework