Priority Queues (Heaps)

Cpt S 223. School of EECS, WSU

Motivation

Queues are a standard mechanism for ordering tasks on a first-come, first-served basis However, some tasks may be more important or timely than others (higher priority) Priority queues

Store tasks using a partial ordering based on priority Ensure highest priority task at head of queue

Heaps are the underlying data structure of priority queues

Cpt S 223. School of EECS, WSU

Priority Queues: Specification

Main operations

insert (i.e., enqueue)

Dynamic insert specification of a priority level (0-high, 1,2.. Low)

deleteMin (i.e., dequeue)

Finds the current minimum element (read: highest priority) in the queue, deletes it from the queue, and returns it

Performance goal is for operations to be fast

Cpt S 223. School of EECS, WSU

Using priority queues

4 5
3 13 11 22

10

19

insert()

deleteMin()
2

Dequeues the next element with the highest priority

Cpt S 223. School of EECS, WSU 4

Can we build a data structure better suited to store and retrieve priorities?

Simple Implementations

Unordered linked list

10

O(1) insert O(n) deleteMin

2 3 5

Ordered linked list

10

O(n) insert O(1) deleteMin

2 3 5 10

Ordered array

O(lg n + n) insert O(n) deleteMin O(log2n) insert and deleteMin

Cpt S 223. School of EECS, WSU 5

Balanced BST

Binary Heap
A priority queue data structure

Cpt S 223. School of EECS, WSU

Binary Heap

A binary heap is a binary tree with two properties

Structure property Heap-order property

Cpt S 223. School of EECS, WSU

Structure Property

A binary heap is a complete binary tree

Each level (except possibly the bottom most level) is completely filled The bottom most level may be partially filled (from left to right)

Height of a complete binary tree with N elements is log 2 N

Cpt S 223. School of EECS, WSU 8

Structure property

Binary Heap Example

N=10 Every level (except last) saturated

Array representation:

Cpt S 223. School of EECS, WSU

Heap-order Property

Heap-order property (for a MinHeap)

For every node X, key(parent(X)) key(X) Except root node, which has no parent Alternatively, for a MaxHeap, always keep the maximum key at the root

Thus, minimum key always at root

Insert and deleteMin must maintain heap-order property

Cpt S 223. School of EECS, WSU 10

Heap Order Property

Minimum element

Duplicates are allowed No order implied for elements which do not share ancestor-descendant relationship
Cpt S 223. School of EECS, WSU 11

Implementing Complete Binary Trees as Arrays

Given element at position i in the array

Left child(i) = at position 2i Right child(i) = at position 2i + 1 Parent(i) = at position i / 2

i 2i 2i + 1

i/2
Cpt S 223. School of EECS, WSU 12

Just finds the Min without deleting it insert deleteMin Note: a general delete() function is not as important for heaps but could be implemented

Stores the heap as a vector

Fix heap after deleteMin

Cpt S 223. School of EECS, WSU 13

Heap Insert

Insert new element into the heap at the next available slot (hole)

According to maintaining a complete binary tree

Then, percolate the element up the heap while heap-order property not satisfied
Cpt S 223. School of EECS, WSU 14

Percolating Up

Heap Insert: Example

Insert 14:

14 hole

Cpt S 223. School of EECS, WSU

15

Percolating Up

Heap Insert: Example

Insert 14: (1) 14 vs. 31
14 14 hole

Cpt S 223. School of EECS, WSU

16

Percolating Up

Heap Insert: Example

Insert 14: (1) 14 vs. 31
14 14 hole

(2) 14 vs. 21
14

Cpt S 223. School of EECS, WSU

17

Percolating Up

Heap Insert: Example

Insert 14: (1) 14 vs. 31

14 hole

(2) 14 vs. 21
14

(3) 14 vs. 13

Heap order prop Structure prop

Path of percolation up

Cpt S 223. School of EECS, WSU

18

Heap Insert: Implementation

// assume array implementation void insert( const Comparable &x) { ? }

Cpt S 223. School of EECS, WSU

19

Heap Insert: Implementation

O(log N) time

Cpt S 223. School of EECS, WSU

20

Heap DeleteMin

Minimum element is always at the root Heap decreases by one in size Move last element into hole at root Percolate down while heap-order property not satisfied

Cpt S 223. School of EECS, WSU

21

Percolating down

Heap DeleteMin: Example

Make this position empty

Cpt S 223. School of EECS, WSU

22

Percolating down

Heap DeleteMin: Example

Copy 31 temporarily here and move it down

Make this position empty

Is 31 > min(14,16)? Yes - swap 31 with min(14,16)

Cpt S 223. School of EECS, WSU

23

Percolating down

Heap DeleteMin: Example

31

Is 31 > min(19,21)? Yes - swap 31 with min(19,21)

Cpt S 223. School of EECS, WSU

24

Percolating down

Heap DeleteMin: Example

31

31

Is 31 > min(19,21)? Yes - swap 31 with min(19,21)

Is 31 > min(65,26)? Yes - swap 31 with min(65,26)

Percolating down

Cpt S 223. School of EECS, WSU

25

Percolating down

Heap DeleteMin: Example

31

Percolating down

Cpt S 223. School of EECS, WSU

26

Percolating down

Heap DeleteMin: Example

31

Heap order prop Structure prop

Cpt S 223. School of EECS, WSU 27

Other Heap Operations

decreaseKey(p,v)

increaseKey(p,v)

Lowers the current value of item p to new priority value v Need to percolate up E.g., promote a job
Increases the current value of item p to new priority value v Need to percolate down E.g., demote a job
Run-times for all three functions?

remove(p)

First, decreaseKey(p,-) Then, deleteMin E.g., abort/cancel a job

Cpt S 223. School of EECS, WSU

O(lg n)

30

Improving Heap Insert Time

What if all N elements are all available upfront? To build a heap with N elements:

Default method takes O(N lg N) time We will now see a new method called buildHeap() that will take O(N) time - i.e., optimal

Cpt S 223. School of EECS, WSU

31

Building a Heap

Construct heap from initial set of N items Solution 1

Perform N inserts O(N log2 N) worst-case Randomly populate initial heap with structure property Perform a percolate-down from each internal node (H[size/2] to H[1])

Solution 2 (use buildHeap())

To take care of heap order property

Cpt S 223. School of EECS, WSU 32

BuildHeap Example
Input: { 150, 80, 40, 10, 70, 110, 30, 120, 140, 60, 50, 130, 100, 20, 90 }

Leaves are all valid heaps (implicitly) Arbitrarily assign elements to heap nodes Structure property satisfied Heap order property violated Leaves are all valid heaps (implicit) So, let us look at each internal node, from bottom to top, and fix if necessary
33

Cpt S 223. School of EECS, WSU

BuildHeap Example
Nothing to do Swap with left child

Randomly initialized heap Structure property satisfied Heap order property violated Cpt (implicit) S 223. School of EECS, WSU Leaves are all valid heaps

34

BuildHeap Example
Nothing to do

Swap with right child

Dotted lines show path of percolating down

Cpt S 223. School of EECS, WSU 35

BuildHeap Example
Nothing to do

Swap with right child & then with 60

Dotted lines show path of percolating down

Cpt S 223. School of EECS, WSU 36

BuildHeap Example

Swap path

Final Heap Dotted lines show path of percolating down

Cpt S 223. School of EECS, WSU 37

BuildHeap Implementation

Start with lowest, rightmost internal node

Cpt S 223. School of EECS, WSU 38

BuildHeap() : Run-time Analysis

Run-time = ?

O(sum of the heights of all the internal nodes) because we may have to percolate all the way down to fix every internal node in the worst-case
HOW?

Theorem 6.1

For a perfect binary tree of height h, the sum of heights of all nodes is 2h+1 1 (h + 1)

Since h=lg N, then sum of heights is O(N) Will be slightly better in practice
Implication: Each insertion costs O(1) amortized time
Cpt S 223. School of EECS, WSU 39

Cpt S 223. School of EECS, WSU

40

Binary Heap Operations Worst-case Analysis

Height of heap is log 2 N insert: O(lg N) for each insert

In practice, expect less

buildHeap insert: O(N) for N inserts deleteMin: O(lg N) decreaseKey: O(lg N) increaseKey: O(lg N) remove: O(lg N)
Cpt S 223. School of EECS, WSU 41

Applications

Operating system scheduling

Process jobs by priority

Graph algorithms

Find shortest path

Instead of checking for events at each time click, look up next event to happen
Cpt S 223. School of EECS, WSU 42

Event simulation

An Application: The Selection Problem

Given a list of n elements, find the kth smallest element Algorithm 1:

Sort the list => O(n log n) Pick the kth element => O(1) Use a binary heap (minheap)
Cpt S 223. School of EECS, WSU 43

A better algorithm:

Selection using a MinHeap

1. 2. 3.

Input: n elements Algorithm:

buildHeap(n) Perform k deleteMin() operations Report the kth deleteMin output ==> O(n) ==> O(k log n) ==> O(1)

Total run-time = O(n + k log n) If k = O(n/log n) then the run-time becomes O(n)

Cpt S 223. School of EECS, WSU

44

Other Types of Heaps

Binomial Heaps d-Heaps

Generalization of binary heaps (ie., 2-Heaps)

Leftist Heaps

Supports merging of two heaps in o(m+n) time (ie., sublinear)

Skew Heaps

O(log n) amortized run-time

Fibonacci Heaps
Cpt S 223. School of EECS, WSU 45

Run-time Per Operation

Insert
Binary heap O(log n) worst-case O(1) amortized for buildHeap

DeleteMin
O(log n)

Merge (=H1+H2)
O(n)

Leftist Heap
Skew Heap Binomial Heap

O(log n)
O(log n) O(log n) worst case O(1) amortized for sequence of n inserts

O(log n)
O(log n) O(log n)

O(log n)
O(log n) O(log n)

Fibonacci Heap

O(1)

O(log n)
Cpt S 223. School of EECS, WSU

O(1)
46

Priority Queues in STL

Uses Binary heap Default is MaxHeap Methods

Push, top, pop, empty, clear

#include <priority_queue> int main () { priority_queue<int> Q; Q.push (10); cout << Q.top (); Q.pop (); } Calls DeleteMax()

For MinHeap: declare priority_queue as: priority_queue<int, vector<int>, greater<int>> Q; Refer to Book Chapter 6, Fig 6.57 for an example
Cpt S 223. School of EECS, WSU 47

Binomial Heaps

Cpt S 223. School of EECS, WSU

48

Binomial Heap

A binomial heap is a forest of heap-ordered binomial trees, satisfying:

i) Structure property, and ii) Heap order property

A binomial heap is different from binary heap in that:

Its structure property is totally different Its heap-order property (within each binomial tree) is the same as in a binary heap
Cpt S 223. School of EECS, WSU

49

Note: A binomial tree need not be a binary tree

Definition: A Binomial Tree Bk

A binomial tree of height k is called Bk:

It has 2k nodes k The number of nodes at depth d = ( d )

k (d ) is the form of the co-efficients in binomial theorem

Depth:

B3:

#nodes:
3 (0 ) 3 (1 ) 3 (2 ) 3 (3 )

d=0 d=1 d=2 d=3

Cpt S 223. School of EECS, WSU

50

What will a Binomial Heap with n=31 nodes look like?

We know that:
i) A binomial heap should be a forest of binomial trees ii) Each binomial tree has power of 2 elements

So how many binomial trees do we need?

B4 B3 B2 B1 B0

n = 31 = (1 1 1 1 1)2

Cpt S 223. School of EECS, WSU

51

A Binomial Heap w/ n=31 nodes

B4 B3 B2 B1 B0

n = 31 = (1 1 1 1 1)2
Forest of binomial trees {B0, B1, B2, B3, B4 }

Bi == Bi-1 + Bi-1

B0

B1 B3 B2

Cpt S 223. School of EECS, WSU

52

Binomial Heap Property

Lemma: There exists a binomial heap for every positive value of n Proof:

All values of n can be represented in binary representation

Have one binomial tree for each power of two with co-efficient of 1 Eg., n=10 ==> (1010)2 ==> forest contains {B3, B1}

Cpt S 223. School of EECS, WSU

53

Binomial Heaps: Heap-Order

Property

Each binomial tree should contain the minimum element at the root of every subtree

Just like binary heap, except that the tree here is a binomial tree structure (and not a complete binary tree)

The order of elements across binomial trees is irrelevant

Cpt S 223. School of EECS, WSU 54

Definition: Binomial Heaps

A binomial heap of n nodes is:

(Structure Property) A forest of binomial trees as dictated by the binary representation of n

(Heap-Order Property) Each binomial tree is a min-heap or a max-heap

Cpt S 223. School of EECS, WSU

55

Binomial Heaps: Examples

Two different heaps:

Cpt S 223. School of EECS, WSU

56

Key Properties

Could there be multiple trees of the same height in a binomial heap?

no

What is the upper bound on the number of binomial trees in a binomial heap of n nodes? lg n

Given n, can we tell (for sure) if Bk exists?

Bk exists if and only if: the kth least significant bit is 1 in the binary representation of n
Cpt S 223. School of EECS, WSU 57

An Implementation of a Binomial Heap

Maintain a linked list of tree pointers (for the forest) Example: n=13 == (1101)2

B7

B6

B5

B4

B3

B2

B1

B0

Shown using the left-child, right-sibling pointer method

Analogous to a bit-based representation of a binary number n

Cpt S 223. School of EECS, WSU 58

Binomial Heap: Operations

x <= DeleteMin() Insert(x) Merge(H1, H2)

Cpt S 223. School of EECS, WSU

59

DeleteMin()

Goal: Given a binomial heap, H, find the minimum and delete it Observation: The root of each binomial tree in H contains its minimum element Approach: Therefore, return the minimum of all the roots (minimums) Complexity: O(log n) comparisons (because there are only O(log n) trees)
Cpt S 223. School of EECS, WSU 60

FindMin() & DeleteMin() Example

B0 B2 B3
B0 B1 B2

For DeleteMin(): After delete, how to adjust the heap? New Heap : Merge { B0, B2 } & { B0, B1, B2 }
Cpt S 223. School of EECS, WSU 61

Insert(x) in Binomial Heap

Goal: To insert a new element x into a binomial heap H Observation:

Element x can be viewed as a single element binomial heap

=> Insert (H,x) == Merge(H, {x})

So, if we decide how to do merge we will automatically figure out how to implement both insert() and deleteMin()
Cpt S 223. School of EECS, WSU 62

Merge(H1,H2)

Let n1 be the number of nodes in H1 Let n2 be the number of nodes in H2 Therefore, the new heap is going to have n1 + n2 nodes

Assume n = n1 + n2

Logic:

Merge trees of same height, starting from lowest height trees If only one tree of a given height, then just copy that Otherwise, need to do carryover (just like adding two binary numbers)
Cpt S 223. School of EECS, WSU 63

Idea: merge tree of same heights

Merge: Example

B0
13

B1

B2

Cpt S 223. School of EECS, WSU

64

How to Merge Two Binomial Trees of the Same Height?

B2: B2: B3:

Simply compare the roots

Note: Merge is defined for only binomial trees of with the same height Cpt S 223. School EECS, WSU

65

Merge(H1,H2) example

carryover

13

14 26 16 26

Cpt S 223. School of EECS, WSU

66

How to Merge more than two binomial trees of the same height?

Merging more than 2 binomial trees of the same height could generate carryovers
+
14

+
26

16 26

Merge any two and leave the third as carry-over

Cpt S 223. School of EECS, WSU 67

Input: +

Merge(H1,H2) : Example
Output:

There are two other possible answers Merge cost log(max{n1,n2}) = O(log n) comparisons
Cpt S 223. School of EECS, WSU 68

Run-time Complexities

Merge takes O(log n) comparisons Corollary:

Insert and DeleteMin also take O(log n)

It can be further proved that an uninterrupted sequence of m Insert operations takes only O(m) time per operation, implying O(1) amortize time per insert

Proof Hint:

unaffected affected

10010111 1 -------------10011000

For each insertion, if i is the least significant bit position with a 0, then number of comparisons required to do the next insert is i+1 If you count the #bit flips for each insert, going from insert of the first element to the insert of the last (nth) element, then => amortized run-time of O(1) per insert
Cpt S 223. School of EECS, WSU 69

Binomial Queue Run-time Summary

Insert

O(lg n) worst-case O(1) amortized time if insertion is done in an uninterrupted sequence (i.e., without being intervened by deleteMins) O(lg n) worst-case

DeleteMin, FindMin

Merge

O(lg n) worst-case

Cpt S 223. School of EECS, WSU

70

Run-time Per Operation

Insert
Binary heap O(log n) worst-case O(1) amortized for buildHeap

DeleteMin
O(log n)

Merge (=H1+H2)
O(n)

Leftist Heap
Skew Heap Binomial Heap

O(log n)
O(log n) O(log n) worst case O(1) amortized for sequence of n inserts

O(log n)
O(log n) O(log n)

O(log n)
O(log n) O(log n)

Fibonacci Heap

O(1)

O(log n)
Cpt S 223. School of EECS, WSU

O(1)
71

Summary

Priority queues maintain the minimum or maximum element of a set Support O(log N) operations worst-case

insert, deleteMin, merge

Many applications in support of other algorithms

Cpt S 223. School of EECS, WSU

72