CompSci 230 Announcements

Discrete Math • Read for next time Chap. 4.1-4.3

for Computer
Science
• Finish Chapter 3 first, then start Chapter 4,
number theory
October 10, 2013

Prof. Rodger

Slides modified from Rosen

1 2

Chap 3.3 - The Complexity of

Algorithms The Complexity of Algorithms
• Given an algorithm, how efficient is this algorithm for • In this course, focus on time complexity.
solving a problem given input of a particular size? • Measure time complexity in terms of the
– How much time does this algorithm use to solve a problem? number of operations an algorithm uses
– How much computer memory does this algorithm use to solve • Use big-O and big-Theta notation to estimate
a problem? the time complexity
• time complexity - analyze the time the algorithm uses to • Is it practical to use this algorithm to solve
solve the problem given input of a particular size problems with input of a particular size?
• space complexity - analyze the computer memory the • Compare the efficiency of different algorithms
algorithm uses to solve the problem, given input of a for solving the same problem.
particular size
3 4
 Time Complexity Complexity Analysis of Algorithms
• For time complexity, determine the number of Example: Describe the time complexity of the algorithm
for finding the maximum element in a finite sequence.
operations, such as comparisons and arithmetic procedure max(a1, a2, …., an: integers)
operations (addition, multiplication, etc.). max := a1
for i := 2 to n
• Ignore minor details, such as the “house if max < ai then max := ai
keeping” aspects of the algorithm. return max{max is the largest element}
• Focus on the worst-case time complexity of an
algorithm. Provides an upper bound. Solution: Count the number of comparisons.
• Compare max < ai n 1 times.
• More difficult to determine the average case • when i incremented, compare if i n. n 1 times
• One last comparison for i > n.
time complexity of an algorithm (average • 2(n 1 1 2n 1 comparisons are made.
number of operations over all inputs of a
Hence, the time complexity of the algorithm is Θ(n).
particular size) 5 6

Complexity Analysis of Algorithms Worst-Case Complexity of Linear Search

procedure linear search(x:integer,
Example: Describe the time complexity of the algorithm a1, a2, …,an: distinct integers)
for finding the maximum element in a finite sequence.
i := 1
procedure max(a1, a2, …., an: integers) while (i n and x ≠ ai)
max := a1
i := i + 1
for i := 2 to n
if max < ai then max := ai
if i n then location := i
return max{max is the largest element} else location := 0
return location{location is the subscript of the term that
equals x, or is 0 if x is not found}
Solution: Count the number of comparisons.
Solution: Count the number of comparisons.
• Compare max < ai n 1 times.
• At each step two comparisons are made; i n and x ≠ ai .
• when i incremented, compare if i n. n 1 times
• end of loop, one comparison i n is made.
• One last comparison for i > n.
• After loop, one more i n comparison is made.
• 2(n 1 1 2n 1 comparisons are made.
If x = ai , 2i + 1 comparisons are used. If x is not on the list, 2n + 1
comparisons are made. One comparison to exit loop.
Hence, the time complexity of the algorithm is Θ(n).
Worst case 2n + 2 comparisons, complexity is Θ(n).
7 8
Worst-Case Complexity of Linear Search Average-Case Complexity of Linear Search
procedure linear search(x:integer,
a1, a2, …,an: distinct integers)
i := 1 Example: average case performance of linear search
while (i n and x ≠ ai)
i := i + 1 Solution: Assume the element is in the list and that
if i n then location := i the possible positions are equally likely.
else location := 0
return location{location is the subscript of the term that
By the argument on the previous slide, if x = ai , the
equals x, or is 0 if x is not found} number of comparisons is 2i + 1.

Solution: Count the number of comparisons.

• At each step two comparisons are made; i n and x ≠ ai .
• end of loop, one comparison i n is made.
• After loop, one more i n comparison is made.
If x = ai , 2i + 1 comparisons are used. If x is not on the list, 2n + 1
Hence, the average‐case complexity of linear search is Θ n .
comparisons are made. One comparison to exit loop.
Worst case 2n + 2 comparisons, complexity is Θ(n).
9 10

Average-Case Complexity of Linear Search Worst-Case Complexity of Binary Search

procedure binary search(x: integer, a1,a2,…, an: increasing integers)
i := 1 {i is the left endpoint of interval}
j := n {j is right endpoint of interval}
Example: average case performance of linear search while i < j
m := (i + j)/2
Solution: Assume the element is in the list and that if x > am then i := m + 1
the possible positions are equally likely. else j := m
if x = ai then location := i
By the argument on the previous slide, if x = ai , the else location := 0
number of comparisons is 2i + 1. return location{location is the subscript i of the term ai equal to x, or
0 if x is not found}

Solution: Assume n = 2k elements. Note that k = log n.

• Two comparisons are made at each stage; i < j, and x > am .
• Size of list is 2k , then 2k-1. then 2k-2 , … then 21 = 2.
• At the last step, list size is 20 = 1 and single last element compared.
Hence, the average‐case complexity of linear search is Θ n . • Hence, at most 2k + 2 = 2 log n + 2 comparisons are made.
• Therefore, the time complexity is Θ (log n), better than linear search.
11 12
 Worst-Case Complexity of Binary Search Worst-Case Complexity of Bubble Sort
procedure binary search(x: integer, a1,a2,…, an: increasing integers)
i := 1 {i is the left endpoint of interval} procedure bubblesort(a1,…,an: real numbers
j := n {j is right endpoint of interval} with n ≥ )
while i < j
m := (i + j)/2 for i := to n
if x > am then i := m + 1 for j := to n i
else j := m
if x = ai then location := i if aj >aj+1 then interchange aj and aj+1
else location := 0
{a1,…, an is now in increasing order}
return location{location is the subscript i of the term ai equal to x, or
0 if x is not found}

Solution: Assume n = 2k elements. Note that k = log n. Solution: n passes through list. pass n i comparisons
• Two comparisons are made at each stage; i < j, and x > am .
• Size of list is 2k , then 2k-1. then 2k-2 , … then 21 = 2.
• At the last step, list size is 20 = 1 and single last element compared.
• Hence, at most 2k + 2 = 2 log n + 2 comparisons are made. Θ(n2) since
• Therefore, the time complexity is Θ (log n), better than linear search. .
13 14

Worst-Case Complexity of Bubble Sort Worst-Case Complexity of Insertion Sort

procedure bubblesort(a1,…,an: real numbers procedure insertion sort(a1,…,an:
real numbers with n ≥ 2)
with n ≥ ) for j := 2 to n
for i := to n i := 1
while aj > ai
for j := to n i i := i + 1
m := aj
if aj >aj+1 then interchange aj and aj+1
for k := 0 to j i 1
{a1,…, an is now in increasing order} aj-k := aj-k-1
ai := m

Solution: n passes through list. pass n i comparisons

Solution: The total number of comparisons are:
1
Θ(n2) since 2 3 … 1
2
.
15 Therefore the complexity is Θ(n2). 16
Worst-Case Complexity of Insertion Sort Stooge Sort
procedure insertion sort(a1,…,an:
real numbers with n ≥ 2)
for j := 2 to n • n elements are in an array
i := 1 • If the value at the end is smaller than the first
while aj > ai
i := i + 1 element, swap them
m := aj • If there are three of more elements then:
for k := 0 to j i 1
aj-k := aj-k-1 – Stooge sort the first 2/3 of the array
ai := m
– Stooge sort the last 2/3 of the array
– Stooge sort the first 2/3 of the array again
Solution: The total number of comparisons are:
• Else
1
2 3 … 1 – Done
2
Therefore the complexity is Θ(n2). 17
• Worst case time is O( ) 18

Stooge Sort
Worst case of BogoSort
• n elements are in an array
• If the value at the end is smaller than the first
• Also known as “StupidSort”
element, swap them
• If there are three of more elements then:
– Stooge sort the first 2/3 of the array • n elements in an array.
– Stooge sort the last 2/3 of the array • While (not in order)
– Stooge sort the first 2/3 of the array again – Shuffle array
• Else
– Done • Worst case:
.
• Worst case time is O( ) 19
• Average Case: 20
 Matrix Multiplication Algorithm
Worst case of BogoSort • matrix multiplication algorithm; C = A B where C is an
m n matrix that is the product of the m k matrix A and
• Also known as “StupidSort” the k n matrix B.

procedure matrix multiplication(A,B: matrices)

• n elements in an array. for i := to m
• While (not in order) for j := to n
– Shuffle array cij :=
for q := to k
• Worst case: unbounded cij := cij + aiq bqj
return C{C = [cij] is the product of A and B}
• Average Case: O( 21 22

Complexity of Matrix Multiplication Complexity of Matrix Multiplication

Example: How many additions of integers and Example: How many additions of integers and
multiplications of integers are used by the multiplications of integers are used by the
matrix multiplication algorithm to multiply matrix multiplication algorithm to multiply
two n n matrices. two n n matrices.
Solution: There are n2 Solution: There are n2
n n n n
3 2
n n n n3 n2 n
O(n3). O(n3).

23 24
 Matrix-Chain Multiplication Matrix-Chain Multiplication
• Compute matrix-chain A1A2∙ ∙ ∙An with fewest • Compute matrix-chain A1A2∙ ∙ ∙An with fewest
multiplications, where A1 , A2 , ∙ ∙ ∙ , An are , multiplications, where A1 , A2 , ∙ ∙ ∙ , An are ,
,∙∙∙ integer matrices. Matrix ,∙∙∙ integer matrices. Matrix
multiplication is associative. multiplication is associative.
Example: In which order should the integer matrices Example: In which order should the integer matrices
A1A2A3 - where A1 is 30 20 , A2 20 40, A3 40 10 - be A1A2A3 - where A1 is 30 20 , A2 20 40, A3 40 10 - be
multiplied? Solution: two possible ways for A1A2A3. multiplied? Solution: two possible ways for A1A2A3.
– A1(A2A3): A2A3 takes 20 ∙ 40 ∙ 10 8000 mults.. A1 – A1(A2A3): A2A3 takes 20 ∙ 40 ∙ 10 8000 mults.. A1
by the 20 10 matrix A2A3 takes 30 ∙ 20 ∙ 10 6000 by the 20 10 matrix A2A3 takes 30 ∙ 20 ∙ 10 6000
mults. Total number is 8000 6000 14,000. mults. Total number is 8000 6000 14,000.
– (A1A2)A3: A1A2 takes 30 ∙ 20 ∙ 40 24,000 mults. – (A1A2)A3: A1A2 takes 30 ∙ 20 ∙ 40 24,000 mults.
A1A2 by A3 takes 30 ∙ 40 ∙ 10 12,000 mults. Total A1A2 by A3 takes 30 ∙ 40 ∙ 10 12,000 mults. Total
is 24,000 12,000 36,000. is 24,000 12,000 36,000.
So the first method is best. So the first method is best.
25 26

Understanding the Complexity of Understanding the Complexity of

Algorithms Algorithms

Times of more than 10100 years are indicated

27 with an *. 28
 P Versus NP Problem
Complexity of Problems Stephen Cook
(Born 1939)

• The P versus NP problem asks whether the class P = NP? Are there problems
• Tractable Problem: There exists a polynomial time algorithm to whose solutions can be checked in polynomial time, but can not be solved in
solve this problem. These problems are said to belong to the polynomial time?
Class P. – Note that just because no one has found a polynomial time algorithm is
• Intractable Problem: There does not exist a polynomial time different from showing that the problem can not be solved by a
algorithm to solve this problem polynomial time algorithm.
• Unsolvable Problem : No algorithm exists to solve this problem, • If a polynomial time algorithm for any of the problems in the NP complete
e.g., halting problem. class were found, then that algorithm could be used to obtain a polynomial
• Class NP: Solution can be checked in polynomial time. But no time algorithm for every problem in the NP complete class.
polynomial time algorithm has been found for finding a solution – Satisfiability (in Section 1.3) is an NP complete problem.
to problems in this class. • It is generally believed that P NP since no one has been able to ind a
• NP Complete Class: If you find a polynomial time algorithm for polynomial time algorithm for any of the problems in the NP complete
one member of the class, it can be used to solve all the problems class.
in the class. • The problem of P versus NP remains one of the most famous unsolved
problems in mathematics including theoretical computer science . The
29
Clay Mathematics Institute has offered a prize of $1,000,000 for a 30
solution.