Data Structures and

Algorithms Using Python

Lecture 5: Hard problems

Heikki Peura
[email protected]
 Homework 2

I Start early!
I Please do not import numpy or other libraries
I In the last question, test cases may include up to
twenty letters corresponding to any values from
1-9999
I Timeout 30 seconds
I But most test cases will be small

2 / 18
 Previously

Graphs:
I Breadth-first search in unweighted graphs
I Extra material on weighted graphs and web scraping
on the Hub

Plan for today:

I Lecture: Greedy algorithms and the knapsack
problem
I Workshop
I 11:15: Final project, parting words (10 min)
I Finish workshop

3 / 18
 Designing algorithms is tricky!

No single recipe solves all our problems...

I ...but there are some useful principles

4 / 18
 Designing algorithms is tricky!

No single recipe solves all our problems...

I ...but there are some useful principles

We’ve used the divide-and-conquer paradigm

I Divide into smaller subproblems
I Eg binary search

4 / 18
 Designing algorithms is tricky!

No single recipe solves all our problems...

I ...but there are some useful principles

We’ve used the divide-and-conquer paradigm

I Divide into smaller subproblems
I Eg binary search

But there are others:

I Greedy algorithms (today)
I Dynamic programming (for you to discover...)

4 / 18
 Greedy algorithms

Rough idea: make myopic decisions iteratively

I Make a choice for immediate benefit without worrying about
future consequences
I Attractive but dangerous

5 / 18
 Greedy algorithms

Rough idea: make myopic decisions iteratively

I Make a choice for immediate benefit without worrying about
future consequences
I Attractive but dangerous

Greedy algorithms are:

I Easy to come up with
I Easy to analyse in terms of running time
I Difficult to show to be correct
I DANGER: a greedy choice is often not correct

5 / 18
 Knapsack problem

Problem: fill a bag with the most valuable items available.

Input:
I Set of n items, with values vi and sizes wi (integer)
I Capacity W

6 / 18
 Knapsack problem

Problem: fill a bag with the most valuable items available.

Input:
I Set of n items, with values vi and sizes wi (integer)
I Capacity W

Output: subset S of items that maximizes the sum of values subject

to a capacity constraint:
I max i∈S vi
P

I subject to i∈S wi ≤ W
P

6 / 18
Knapsack problem applications

7 / 18
Knapsack problem applications

7 / 18
 Knapsack problem applications

Many problems with budget constraints are versions of

knapsack
I Selecting portfolios (eg projects to invest in)
I Lots of “operational” problems...

7 / 18
 Two-item example

Items to pack
I Shirt: value 5, weight 5
I Bottle: value 10, weight 5

8 / 18
 Two-item example

Items to pack
I Shirt: value 5, weight 5
I Bottle: value 10, weight 5

Subsets:
I {},{Shirt},{Bottle},{Shirt,Bottle}

8 / 18
 Two-item example

Items to pack
I Shirt: value 5, weight 5
I Bottle: value 10, weight 5

Subsets:
I {},{Shirt},{Bottle},{Shirt,Bottle}

Knapsack size limits feasible solutions

I W < 5: can pick neither
I 5 ≤ W < 10: neither or just one
I W ≥ 10: all subsets feasible

8 / 18
 Go through all possibilities?

Input:
I Set of n items, with values vi and sizes wi (integer)
I Capacity W

Output: subset S of items that maximizes the sum of values subject

to capacity constraint:
I max i∈S vi
P

I subject to i∈S wi ≤ W
P

9 / 18
 Go through all possibilities?

Input:
I Set of n items, with values vi and sizes wi (integer)
I Capacity W

Output: subset S of items that maximizes the sum of values subject

to capacity constraint:
I max i∈S vi
P

I subject to i∈S wi ≤ W
P

Exhaustive (brute-force) search?

I Go through all subsets of {1, 2, 3, ..., n}
I Suppose we have 50 items: O(2n ) subsets...

9 / 18
 Greedy approaches for knapsack problem
Input:
I Set of n items, with values vi and sizes wi
I Capacity W

Output: subset S of items that maximizes the sum of values subject

to capacity constraint:
I max i∈S vi
P

I subject to i∈S wi ≤ W
P

Greedy approaches? — Pick myopically without worrying about

future choices

10 / 18
 Greedy approaches for knapsack problem
Input:
I Set of n items, with values vi and sizes wi
I Capacity W

Output: subset S of items that maximizes the sum of values subject

to capacity constraint:
I max i∈S vi
P

I subject to i∈S wi ≤ W
P

Greedy approaches? — Pick myopically without worrying about

future choices
I Highest-value item first?

10 / 18
 Greedy approaches for knapsack problem
Input:
I Set of n items, with values vi and sizes wi
I Capacity W

Output: subset S of items that maximizes the sum of values subject

to capacity constraint:
I max i∈S vi
P

I subject to i∈S wi ≤ W
P

Greedy approaches? — Pick myopically without worrying about

future choices
I Highest-value item first?
I Lowest-size item first?

10 / 18
 Greedy approaches for knapsack problem
Input:
I Set of n items, with values vi and sizes wi
I Capacity W

Output: subset S of items that maximizes the sum of values subject

to capacity constraint:
I max i∈S vi
P

I subject to i∈S wi ≤ W
P

Greedy approaches? — Pick myopically without worrying about

future choices
I Highest-value item first?
I Lowest-size item first?
I Some easy way of combining value and size?

10 / 18
 Greedy algorithms for knapsack
Greedy approaches?
I Highest-value item first?

11 / 18
 Greedy algorithms for knapsack
Greedy approaches?
I Highest-value item first?

Example: capacity W = 20, three items with values v = [10, 10, 11], weights
w = [10, 10, 11].
I Picking highest value first is bad...

11 / 18
 Greedy algorithms for knapsack
Greedy approaches?
I Highest-value item first?
I Lowest-size item first?

Example: capacity W = 20, three items with values v = [10, 10, 11], weights
w = [10, 10, 11].
I Picking highest value first is bad...

11 / 18
 Greedy algorithms for knapsack
Greedy approaches?
I Highest-value item first?
I Lowest-size item first?

Example: capacity W = 20, three items with values v = [10, 10, 11], weights
w = [10, 10, 11].
I Picking highest value first is bad...

Example: capacity W = 20, three items with values v = [10, 10, 50], weights
w = [10, 10, 11].
I Picking lowest weight first is bad...

11 / 18
 Greedy algorithms for knapsack
Greedy approaches?
I Highest-value item first?
I Lowest-size item first?

Example: capacity W = 20, three items with values v = [10, 10, 11], weights
w = [10, 10, 11].
I Picking highest value first is bad...

Example: capacity W = 20, three items with values v = [10, 10, 50], weights
w = [10, 10, 11].
I Picking lowest weight first is bad...

Some easy way of combining value and size?

I Eg sort items by unit weight vi /wi and pick them in this order

11 / 18
 Greedy algorithm for knapsack

Greedy algorithm:
I Sort items in decreasing bang-for-buck vi /wi
I Pick items until capacity full

12 / 18
 Greedy algorithm for knapsack

Greedy algorithm:
I Sort items in decreasing bang-for-buck vi /wi
I Pick items until capacity full

Running time?
I Sorting? — O(n log n)
I Picking items? — O(n)

12 / 18
 Greedy algorithm for knapsack

Greedy algorithm:
I Sort items in decreasing bang-for-buck vi /wi
I Pick items until capacity full

Running time?
I Sorting? — O(n log n)
I Picking items? — O(n)
I Total O(n log n)

12 / 18
 Is the algorithm correct?

Example: capacity W = 510, three items with values

v = [10, 10, 500], weights w = [10, 10, 501].
I Greedy picking is bad...

13 / 18
 Is the algorithm correct?

Example: capacity W = 510, three items with values

v = [10, 10, 500], weights w = [10, 10, 501].
I Greedy picking is bad...

It would be optimal if we could divide items into fractions

13 / 18
 Is the algorithm correct?

Example: capacity W = 510, three items with values

v = [10, 10, 500], weights w = [10, 10, 501].
I Greedy picking is bad...

It would be optimal if we could divide items into fractions

Here we would need a different approach to find correct solution —

dynamic programming
I Exploit problem structure to loop through all items and possible
capacities
I Correct solution, O(nW ) time (“pseudo-polynomial”)

13 / 18
 Many important problems are intractable
Tractable problem = Solvable in polynomial time O(nk ) for some k

14 / 18
 Many important problems are intractable
Tractable problem = Solvable in polynomial time O(nk ) for some k

Finding the shortest route to a Pikachu nest (Dijkstra’s algorithm) vs.

finding the shortest tour of all Pokemon nests

14 / 18
 Many important problems are intractable
Tractable problem = Solvable in polynomial time O(nk ) for some k

Finding the shortest route to a Pikachu nest (Dijkstra’s algorithm) vs.

finding the shortest tour of all Pokemon nests

Example of intractability: traveling salesperson problem (TSP)

14 / 18
 Many important problems are intractable
Tractable problem = Solvable in polynomial time O(nk ) for some k

Finding the shortest route to a Pikachu nest (Dijkstra’s algorithm) vs.

finding the shortest tour of all Pokemon nests

Example of intractability: traveling salesperson problem (TSP)

I Input: undirected graph with non-negative edge costs

14 / 18
 Many important problems are intractable
Tractable problem = Solvable in polynomial time O(nk ) for some k

Finding the shortest route to a Pikachu nest (Dijkstra’s algorithm) vs.

finding the shortest tour of all Pokemon nests

Example of intractability: traveling salesperson problem (TSP)

I Input: undirected graph with non-negative edge costs
I Goal: find minimum cost tour visiting every node

14 / 18
 Many important problems are intractable
Tractable problem = Solvable in polynomial time O(nk ) for some k

Finding the shortest route to a Pikachu nest (Dijkstra’s algorithm) vs.

finding the shortest tour of all Pokemon nests

Example of intractability: traveling salesperson problem (TSP)

I Input: undirected graph with non-negative edge costs
I Goal: find minimum cost tour visiting every node
I Conjecture: no polynomial-time algorithm

14 / 18
 Many important problems are intractable
Tractable problem = Solvable in polynomial time O(nk ) for some k

Finding the shortest route to a Pikachu nest (Dijkstra’s algorithm) vs.

finding the shortest tour of all Pokemon nests

Example of intractability: traveling salesperson problem (TSP)

I Input: undirected graph with non-negative edge costs
I Goal: find minimum cost tour visiting every node
I Conjecture: no polynomial-time algorithm
I Many other important problems too...
14 / 18
 My problem is intractable!

What can you do?

1. There may be tractable special cases (small
knapsack DP)

15 / 18
 My problem is intractable!

What can you do?

1. There may be tractable special cases (small
knapsack DP)
2. Get an approximate solution using heuristics —
fast but not “correct” (next slide)

15 / 18
 My problem is intractable!

What can you do?

1. There may be tractable special cases (small
knapsack DP)
2. Get an approximate solution using heuristics —
fast but not “correct” (next slide)
3. Solve in exponential time but try to improve on brute
force (large knapsack DP)

15 / 18
 Greedy knapsack heuristic

Greedy knapsack was incorrect but very fast: O(n log n)

Greedy algorithm:
I Sort items in decreasing bang-for-buck vi /wi
I For each item, pick the item until reach total W

16 / 18
 Greedy knapsack heuristic

Greedy knapsack was incorrect but very fast: O(n log n)

Greedy algorithm:
I Sort items in decreasing bang-for-buck vi /wi
I For each item, pick the item until reach total W

How bad is it?

I Example: capacity W =510, three items with values
v = [10, 10, 500], weights w = [10, 10, 501].

16 / 18
 Greedy knapsack heuristic

Greedy knapsack was incorrect but very fast: O(n log n)

Greedy algorithm:
I Sort items in decreasing bang-for-buck vi /wi
I For each item, pick the item until reach total W

How bad is it?

I Example: capacity W =510, three items with values
v = [10, 10, 500], weights w = [10, 10, 501].
I “Worst-case scenario”: leave out (a single) extremely valuable
object that would fit into knapsack

16 / 18
 Better greedy knapsack heuristic
Modified algorithm: pick either the greedy solution or the most
valuable item

17 / 18
 Better greedy knapsack heuristic
Modified algorithm: pick either the greedy solution or the most
valuable item
I Sort items in decreasing vi /wi
I For each item, pick the item until reach total W
I Return either greedy solution or the most valuable item,
whichever is better

17 / 18
 Better greedy knapsack heuristic
Modified algorithm: pick either the greedy solution or the most
valuable item
I Sort items in decreasing vi /wi
I For each item, pick the item until reach total W
I Return either greedy solution or the most valuable item,
whichever is better

Claim: This gets at least 50% of maximum possible value.

17 / 18
 Better greedy knapsack heuristic
Modified algorithm: pick either the greedy solution or the most
valuable item
I Sort items in decreasing vi /wi
I For each item, pick the item until reach total W
I Return either greedy solution or the most valuable item,
whichever is better

Claim: This gets at least 50% of maximum possible value.

I “Worst-case scenario”: greedy leaves out two extremely valuable
objects that would both fit into knapsack
I Still pick the better of them: at least 50%
I Often items are small → the greedy choice cannot leave out
many of them

17 / 18
 Review

Some problems are inherently intractable

I Knapsack, traveling salesperson
I Greedy algorithms, approximations and heuristics can
help

Workshop after the break

I Knapsack and fantasy football

11:15: Parting words

I Final project
I What next?

18 / 18
 Workshop

Workshop zip file on the Hub

I HTML instructions
I At some point, you’ll need the .py-file with skeleton
code (open in Spyder)

11:15: Parting words

I Final project
I What next?