Week 6: Algorithms

Review
• In the recorded lecture we learned:
– GCD – Euclid’s method; GCD(x,y) = GCD(y, x % y)
– Root finding with binary search
– Learned how to prove correctness

2
Run time Analysis
 How fast will your program run?
• The running time of your program will depend upon:
– The algorithm
– The input
– Your implementation of the algorithm in a programming
language
– The compiler/interpreter you use
– The OS on your computer
– Your computer hardware
– Maybe other things: temperature outside; other programs on
your computer; …
• Our Motivation: analyze the running time of an algorithm
as a function of only simple parameters of the input
4
 Basic idea: counting operations
• Each algorithm performs a sequence of basic operations:
– Arithmetic: (low + high)/2
– Comparison: if x > 0:
– Assignment: temp = x
– Branching: while y>0:
–…
• Count number of basic operations performed on the input
• Difficulties:
– Which operations are basic?
– Not all operations take the same amount of time
– Operations take different times with different hardware or
5 compilers
 Difficulties do not matter “so much”
• Operation counts are only problematic in terms of
“constant factors”
• The general form of the function describing the
running time is invariant over hardware, languages or
compilers!

for i in range(n):
for j in range(n):
x = x + i*j

• Running time is “about” n2

• We use “Big-O” notation, and say that the running
time is O(n2)
6
 Asymptotic behavior of functions

Big difference between these functions. E.g., n=1000, what are

7
 Mathematical Formalization

8
 f = O(g)
There exist a
c and an x0
such that for
all x > x0
f(x) ≤ c g(x)

Source https://en.wikipedia.org/wiki/Big_O_notation#/media/File:Big-O-notation.png
9
 Mathematical Formalization
• f=O(g) should be considered as saying that “f is at
most g, up to constant factors”
• We usually will have f be the running time of an
algorithm, and g a nicely written function.
– For example:
"The running time of the algorithm was O(n2)"

10
 Example

11
 Example
• 4n2 + 2n + 3 = O(n2)

Big-O notation refers to very large n: the contribution

of the terms that grow ‘most quickly’ eventually
make the others irrelevant. Two simplification rules:
1. If f(x) is a sum of several terms, the one with the
largest growth rate should be kept, all others
dropped (i.e., keep only 4n2)
2. If f(x) is a product of several factors, any constants
(i.e., terms in the product that do not depend on
n) should be omitted (i.e., get rid of the 4, keep
the n2)
12
 Asymptotic analysis of algorithms
• We usually embark on an asymptotic worst case analysis
of the running time of the algorithm.
• Asymptotic:
– Formal, exact, depends only on the algorithm
– Ignores constants
– Applicable mostly for large input sizes
• Worst Case:
– Bounds on running time must hold for all inputs
– Thus the analysis considers the worst-case input
– Sometimes the “average” performance can be much better
– Real-life inputs are rarely “average” in any formal sense
13
 What’s our analysis of
Binary search?
• The best case is easy: 1 step

• The worst case is log2 n (i.e., how many times can n be

divided in half before we’re left with a list of length 1?
Starting with 1, how many times can you double a value
until it’s as large as n?)

• The average case requires a more detailed analysis

14
 What’s going on with average case size
• Assume, first of all, that what we are searching for is in
the list (if not, of course, average case of the search
might be affected by how often the item is not in the
array)
• In our search algorithms, the average case size can be
thought of as the sum n

pd
i 0
i i

where pi is the probability of finding the item at a given

15 step, and di is the “amount of work” to reach that step.
 Simple Example
• Given: a 3-element list, 90% chance of finding what we
want in the first location, 5% in the second, 5% in the
third
• We search linearly, left to right
• What’s the expected average amount of work to find
what we are looking for?
90% 5% 5% Probability of finding it after
3 steps
(.9 * 1) + (.05 * 2) + (.05 * 3) = 1.15 steps on average

Probability of finding it after 1 step Probability of finding it after

16 2 steps
 Average Case Size of linear search Algorithm (one end to
the other)
• Average-case size of the search is n/2, assuming
the value might be anywhere with equal
probability
• In other words, our search might be
(1 * [1/n]) + (2 * [1/n]) + (3 * [1/n]) +… + (n * [1/n])
n
i
in other words, 1/n * i1

in other words, 1/n * ([n * (n + 1)] / 2)

17 in other words, n/2 + ½
 Constants Fade in Importance
• So the expected amount of work is:
n/2 + ½
• What really interests us is the “shape” of the function
• The ½ fades away for large n
• Even the division of n by 2 is basically unimportant
(since we didn’t really quantify how much work each
cell of the list took)
• What’s important is that the expected work grows
linearly with the size of the list (i.e., the input)
18 • The expected amount of work is O(n)
 Average Case Size of Binary Search Algorithm

• There is 1 element we can get to in one step…

? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
Average work = (1*(1/n))…

19
Probability of finding it in 1 step
 Average Case Size of Binary Search Algorithm

• There is 1 element we can get to in one step, 2

that we can get to in two steps…

? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
Average work = (1*(1/n)) + (2*(2/n))…

Probability of finding it in 2 steps

20
 Average Case Size of Binary Search Algorithm

• There is 1 element we can get to in one step, 2

that we can get to in two steps, 4 that we can
get to in three steps…

? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
Average work = (1*(1/n)) + (2*(2/n)) + (3*(4/n))…

Probability of finding it in 3 steps

21
 Average Case Size of Binary Search Algorithm

• There is 1 element we can get to in one step, 2

that we can get to in two steps, 4 that we can
get to in three steps, 8 that we can get to in four
steps…

? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

Average work = (1*(1/n)) + (2*(2/n)) + (3*(4/n)) + (4*(8/n))…

Probability of finding it in 4 steps

22
 Average Case Size of Binary Search Algorithm
• There is 1 element we can get to in one step,
2 that we can get to in two steps, 4 that we
can get to in three steps, 8 that we can get to
in four steps…

? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
•In other words, there is 1/n chance of 1 step,
2/n chance of 2 steps, 4/n chance of 3 steps,
8/n chance of 4 steps…
23
 Average Case Size of Binary Search Algorithm

• In other words, we have

1/n * [(1*1)+(2*2)+(4*3)+(8*4)+…+(n/2 *log 2 n)]
• In other words, we have
log2n
1
[
´ å 2i-1 ´ i
n i=1 ]
For large n, this converges to (log2 n) - 1, so
that’s our average case size for binary search
24
 Again, Constants Fade in Importance

• If we calculated that the average case for binary

search is
(log2 n) – 1
• We don’t care how much work we did in each
location
• What’s really interesting is the “shape” of the
function for large inputs (i.e., large n)
• So we would say that the average case for binary
25
search is O(log2 n) – read “Big-O” of log2 n
 Running time of simple GCD algorithm
What is the asymptotic running time?

26
 Running time of simple GCD algorithm
Theorem: The simple GCD algorithm runs in
time

27
 Running time of Euclid’s GCD Algorithm
• How fast does Euclid’s algorithm terminate?

28
 Useful Claim: if then

Proof:
 If then

 Otherwise, when then

Conclusion: the value of decreases by a factor of at least 2 each

iteration (except, maybe, the first one).

E.g., if x is 72 and y is 250, 72%250 is

72, so x * y is unchanged in first
29
iteration – then we look at 250%72
30
 Binary search Square root algorithm

• high - low goes down by a factor of 2 each iteration.

• Starts at max(1, x), finishes when reaching 
• Run time is O(log2(x/)) = O(log x + log (1/))
31
Running Time of Common Python
Operations
 Running time
O(1) means constant time, not
dependent on size of input n

Operation List / Tuple Dict / Set

Read data from index/key O(1) O(1)

Update existing index/key O(1) O(1)

Check membership O(n) O(1) for keys

For values O(n)

Add more items to the end Worst case Amortized

O(n) O(1)
Worst case Amortized O(1)
Insert an item O(n)
O(n)
Delete an item

See also: ‫השקעה לאורך זמן‬

https://wiki.python.org/moin/TimeComplexity
33
 Random Access Memory
34257
34258
34259
34260
a
34261
34262
a[3]
34263
34264

To access a[i], Python takes the start location of

a, and adds i to it to get the address in RAM
34
 Python Implements Lists
a

2
1
3

a = [1, 2, 3]
This results in Python setting
aside, let’s say, 4 places for the
list a, but only using the first 3
35 len(a) == 3
 Python Implements Lists
a

2
1 4
3

a.append(4)
a becomes [1, 2, 3, 4]
Here, append is O(1), free space was available
len(a) == 4
36
 Python Implements Lists
a

2 5
1 4
3

Now do a.append(5)
No contiguous free space. Python finds new (double)
space for the list, updates a, copies pointers to new place
a becomes [1, 2, 3, 4, 5]
Here, append is O(n), no free space was available
37 len(a) == 5
 Python Implements Lists
a

2 5 8
1 4 6
3 7

We do three more O(1) appends, and

a becomes [1, 2, 3, 4, 5, 6, 7, 8]

len(a) == 8
Now do a.append(9)
38 No contiguous free space is available
 Python Implements Lists
a

2 5 8
1 4 6 9
3 7

Python finds new (double) space for the list, updates

a, copies pointers to new place
a becomes [1, 2, 3, 4, 5, 6, 7, 8, 9]
Here, append is again O(n), copying of list occurred
39 len(a) == 9
 List Append – O(1) – Amortized
Setup: lst = list(range(N))
Timing: lst.append('val')

Implementation Logic: If (out of space) Then (copy

40
to a new place that has twice the space)
 Inserting into a list – O(n)
Setup: lst = list(range(N))
Timing: lst.insert(0,'val')

Every insertion requires following values to be

41
pushed forward
 Dictionary Access and Insert – O(1) – Amortized

Setup: my_dict = dict([(i,i) for i in range(N)])

Timing: my_dict['key'] = 'val'

42
 Hash Tables Map to memory
location and store value
We wish to store Dictionary computes hash of key key value
there 0
key value print(hash("hello")) wide 1
print(hash("world"))
hello up print(hash("frank")) 2
7453177550310124801
there wide -9087840708672955053
hello 3
up 4
frank now -2055844616174617975
5
print(hash("hello")%11)
print(hash("there")%11) 6
print(hash("frank")%11) 7
3 frank 8
0 now
8 9
10
• The hash function sends us to the right table location for storing and for
retrieving a key-value pair
• Keys sometimes map into the same cell (a “collision”), but not too often if
43
the table is large enough
 Hash Tables with Collision Map to memory
location and store value
We wish to store Dictionary computes hash of key key value
there 0
key value print(hash("hello")) wide 1
print(hash("world"))
hello up print(hash(”fred")) 2
7453177550310124801
there wide -9087840708672955053
fred
hello
up velma
3
4
fred velma -9162233691798532529
5
print(hash("hello")%11)
print(hash("there")%11) 6
print(hash(”fred")%11) 7
3 8
0
3 9
10
If there is a collision when storing the key-value, we can move
forward and look for an empty location – and when we retrieve
44 the key-value pair, we may need to visit more than one location
Algorithms on Lists

Intro2CS – week 6
 Sorting
• Sorting data is often
done so that
subsequent
searching will be
much easier (e.g.,
binary search)
• An absolutely
fundamental set of
algorithms in
Computer Science
46
 The sorting problem
Input: A list
Output: The same list, with the elements re-ordered from
smallest to largest.

[53, 12, 77, 23, 50]  [12, 23, 50, 53, 77]

["yossi", "david", "anat", "gila"] 

["anat", "david", "gila", "yossi"]
47
 Comparison in Python

48
 Sorting Built Into Python
• Timsort
• “Timsort is a hybrid stable sorting algorithm,
derived from merge sort and insertion sort,
designed to perform well on many kinds of
real-world data…It was implemented by Tim
Peters in 2002 for use in the Python
programming language.”
Wikipedia

49
 sorted()
>>> numbers = [6, 9, 3, 1]
>>> numbers_sorted = sorted(numbers)
>>> numbers_sorted
[1, 3, 6, 9]
>>> numbers
[6, 9, 3, 1]

• The output is a new, sorted list. The original

50
list’s initial values are unchanged
 .sort()
• .sort() is a method of the list class and can only be
used with lists
– It is not a built-in Python function like sorted() with an
iterable passed to it
• .sort() returns None and modifies the values in
place
• Example:
>>> numbers = [6, 9, 3, 1]
>>> numbers.sort()
>>> numbers
51
[1, 3, 6, 9]
 Other Sorting Algorithms
• We’ll look at a classic sorting algorithm
(there are many, many more)
– Selection Sort
• Other(s) in tirgul and in the coming weeks

52
 Selection Sort (large to small sorting)
starting order:

18 35 22 97 84 55 61 10 47
search through list, find largest value, exchange with
first list value:
97 35 22 18 84 55 61 10 47

search through rest of list, find second-largest value,

exchange with second list value:
97 84 22 18 35 55 61 10 47

53
 Continue the Select and Exchange Process (large to small
sorting)
search through rest of list, one less each time:
97 84 61 18 35 55 22 10 47
97 84 61 55 35 18 22 10 47
97 84 61 55 47 18 22 10 35
97 84 61 55 47 35 22 10 18
97 84 61 55 47 35 22 10 18
97 84 61 55 47 35 22 18 10
54
 Selection Sort (small to large sorting)

55
56
 Selection Sort

57
 Proof of Correctness: find_smallest

Theorem: find_smallest() returns the index of the

smallest element in lst[start:]

Invariant: When the iteration with value i ends, best_ind

holds the location of the smallest element in lst[start:]
58
 Proof of Correctness: sort

Theorem: When selection_sort terminates, the list lst

holds the original elements, but in sorted order.

Invariants:
• lst always holds the same elements
• After the i’th iteration, lst[:i+1] holds the smallest
i+1 elements of lst in sorted order
59
 Selection Sort – O(n2) Runtime

O(n)

How many times does

this line run?

O(n)
times
find_smallest

60
 Stable Sorting vs. Unstable Sorting Techniques
• A list might include components with
exactly the same “sorting value” (e.g.,
numbers are in the list, and we’re
sorting on some digit)
• Sorting algorithms that leave such
components in their original order are
called stable, while sorting algorithms
that may change the original order of
those components are called unstable
61
 Merge Sort Intuition

62 https://commons.wikimedia.org/wiki/File:Merge_sort_algorithm_diagram.svg
 QuickSort Intuition

63 https://www.techiedelight.com/quicksort/