Algorithms

Data Structures: Review

Abstract Data Type
List
Stack
Queue
Set
Dictionary
Priority Queue : Heaps

Imdadullah Khan
Introduction

Algorithms work on data

Data must be represented in a usable way
Data is stored in data structures
Choice of data structure significantly affects the efficiency of the
algorithm
Abstract Data Types (adts)
Just as functions (procedures) extend the notion of operators in a
programming language,
adts extend the notion of data types in a programming language
adts are data types that have associated
set of (valid) values (type of data)
set of operations that can be applied on the set of values
For example the Set adt could be defined as
Sets of Integers
Union, Intersection, set complement, set differences

Concrete implementation of an adt is a Data Structure

Set can be implemented as arrays, linked list, bit vectors

Some Useful adt’s
Some of the frequently used adt’s are
List
Stack
Queue
Set
Dictionary
Priority Queue
List

Sequence of elements of a certain type

Elements can be linearly ordered
Notion of position, next/previous, start/end of list
Associated operations:
insert(L, x, p)
delete(L, x) or delete(L, p)
retrieve(L, p)
search(L, x)
next(L, p)
first(L)
Implemented using an array/linked list
Stacks

Last-In First-Out(LIFO) list

Associated operations:
Push(x)
Pop()
isEmpty()
isFull()
size()
Implemented using an array/linked list
Use: In OS to handle recursive and procedural calls, DFS
Queues

First-In First-Out (FIFO) list

Associated operations:
Enqueue(x)
Dequeue(x)
isEmpty()
isFull()
size()
Implemented using an array/linked list
Use: In processes scheduler, BFS
Dictionary

The Set adt, in addition to Insert, Search and Delete, includes

operations such as Union (A ∪ B), Intersection (A ∩ B) and
Subtraction (A \ B)
The full Set adt is not generally needed
E.g. Students’ record at RO (Zambeel)
E.g: How would you store the quiz scores for a set of students?
We need to maintain a set with insertion, deletion and searching
The scores of a quiz can be represented using a dictionary with roll
numbers as keys and scores as values
scores =
{‘160201020 : 17,
‘110100510 : 84,
‘110500010 : 22,
‘120600090 : 92}
Dictionary

A dictionary maintains a set of elements

Unique elements; elements are known by their “keys”
Elements could be compound (key , value) pairs
Associated operations:
Insert(D, k, v )
Delete(D, k)
Search(D, k)
isEmpty(D)
Size(D)
What if an entry (k, v 0 ) exists in D and Insert(D, k, v ) is called?
Dictionary can be implemented using
an array (sorted or unsorted)
a linked list (sorted or unsorted)
binary search trees (balanced or unbalanced)
hash tables
Dictionary Implementations - Array
Unsorted Array:
Search: Linear search - traverse array sequentially . O(n)
Insert: Insertion at the end of array (first empty slot) . O(1)
Delete: Given a position, shift left remaining elements . O(n)

Sorted Array:
Search: Binary search; repeatedly halve search interval . O(log n)
Insert: Lookup to find position and shift to make space . O(n)
Delete: Given a position, shift left remaining elements . O(n)
Dictionary Implementations - Linked List
Unsorted Linked List:
Lookup: Linear search . O(n)
Insert: Insertion at start or end . O(1)
Delete: Lookup and link previous with next (doubly linked list) . O(n)

Sorted Linked List:

Lookup: Linear search (can’t jump to mid directly) . O(n)
Insert: Lookup to find position and update previous and next . O(n)
Delete: Lookup and link previous with next . O(n)
prev node next node
head ... prev key val next prev key val next ... tail

new node
prev key val next

Insertion to a sorted doubly linked list

Dictionary Implementations - Binary Search Tree
A binary tree has a root node, a left subtree and a right subtree
Each node contains data, left pointer and right pointer
Binary Search Tree is a Binary Tree with additional properties:
Nodes have keys for comparison
Keys in left subtree are smaller than node’s key
Keys in right subtree are larger than node’s key

5 5
3 8 3 6
2 6 7 1 4 8
1 4 2 7
Binary Tree Binary Search Tree
Dictionary Implementations - Binary Search Tree
For a dictionary, the node data is a (key ) pair
Search: Compare with root; recursively lookup in appropriate
subtree
Insert: Lookup for appropriate leaf position to insert node
Delete: - Given key, lookup to find pointer to node. Given pointer
to node, remove and recursively link parent with one of the children
Read Textbook
For a BST of height h, all the above operations take O(h) time
Dictionary Implementations - AVL Tree
An AVL tree is a binary search tree with additional properties:
The height of the left and right subtree of a node differ by at most 1
The left and right subtrees of a node are AVL trees
AVL tree is a balanced BST; it’s height is always O(log (n))

1 BST AVL

4
2
2 5
3
1 3 6
4
5 6
Dictionary Implementations - AVL Tree

Search, Insert and Delete methods are the same as BST but an
AVL tree may become unbalanced after Insert and Delete
An unbalanced tree is rebalanced using rotation; an adjustment to the
tree, around an item, that maintains the required ordering of items
Read Textbook
All operations of AVL Tree have the same time complexity as BST i.e.
O(log n), as rotation takes only constant time
Dictionary Implementations - Hash Tables
Suppose n data elements are to be stored in a dictionary with keys
k ∈ U where universe set U = [1 . . . N]
Let m ∈ Z+ and h : U → [m]
Make a array (or table) T [1, ..., m]
Search: return T [h(k)] . O(1)
Insert: T [h(k)] ← 1 . O(1)
Delete: T [h(k)] ← 0 . O(1)
Dictionary Implementations - Hash Tables

What if h(kx ) = h(ky )? Collision occurs

Collision occurs between k2 and k5

Dictionary Implementations - Hash Tables
Hashing with Chaining:
Make T [1, ..., m] an array of linked lists
Lookup: Lookup in list T [h(k)]
Insert: Insert in list T [h(k)]
Delete: Delete from list T [h(k)]
Runtime of all operations: O(length of longest list in T [k])
ensure not many keys k map to the same index in T under h
Dictionary Implementations - Hash Tables
Uniform hashing:
each key k has an equal chance of being mapped to the an index in T
under h
1
Pr [h(x) = h(y )] =
m
n
Expected length of list of each index = m
Space and time complexity trade-off controlled by size of table m
Example: Linear Congruential Hash Function
Select a prime number p ≥ m
Choose integers a, b randomly s.t. a 6= 0
Then,
ha,b (x) = ((ax + b) mod p) mod m)
Read Number Theory slides from CS 210 Discrete Mathematics
Dictionary Implementations - Summary

Runtimes of dictionary operations: different implementations

Unsor. Sor. Unsor. Sor. Hash
Operation BST AVL
Array Array L-list L-list Function
Search(D,k) O(n) O(log n) O(n) O(n) O(h) O(log n) O(1)
Insert(D,k,v) O(1) O(n) O(1) O(n) O(h) O(log n) O(1)
Delete(D,k) O(n) O(n) O(n) O(n) O(h) O(log n) O(1)
Priority Queue ADT

Data elements have an associated priorities (called keys)

Retrieval is done on the basis of this priority
Operations:
P = Initialize()
Initialize a priority queue with n elements with associated priorities
Insert(P, v , k)
Insert an element v with priority k,
ExtractMin(P)
Returns the element with minimum priority and delete it (also called deleteMin)
One can analgously define ExtractMax(P)
DecreaseKey(P, v , k 0 )
Change the priority of element v to k 0
One can analgously define IncreaseKey(P, v , k 0 )
Priority Queue ADT: Applications
Used in many algorithms
scheduling systems
shortest process first
longest request first
Cache Replacement algorithms (e.g. LRU, LFU)
Hierarchical (Agglomerative) Clustering
Dijkstra’s and Prim’s algorithm
Priority Queue: Implementation
Can be Implemented using
Arrays (sorted or unsorted)
Linked List (sorted or unsorted)
Binary Heaps

Note: The terms Heap and Priority Queue are often used interchangeably as priority
queues are mostly implemented using heaps and they support the same operations
Priority Queue: Implementation
Unsorted Array
Initialize: create array with elements in arbitrary order . O(n)
Insert: insert at the end of the array . O(1)
ExtractMin: FindMin in array by key, return, delete and shift . O(n)
Sorted Array
Initialize: sort array in descending order by key . O(n log n)
Insert: binary search for position and shift elements on right . O(n)
ExtractMin: remove the last element from array . O(1)
Unsorted Linked-List (doubly linked list)
Initialize: create linked list with elements in arbitrary order . O(n)
Insert: insert new node at head of linked list . O(1)
ExtractMin: FindMin in array by key, return, delete and shift . O(n)
Sorted Linked-List
Initialize: sort linked list in ascending order by key . O(n2 )
Insert: lienear search for position and insert new node . O(n)
ExtractMin: remove the element at head of linked-list . O(1)
DecreaseKey is essentially a search, replace and (if needed) reorder
Priority Queue: Heap Implementation
A binary heap can be used to implement priority queues
A rooted binary tree that satisfies the heap property
Min-Heap Property: If u is parent of v , then key (u) ≤ key (v )
Max-Heap Property: If u is parent of v , then key (u) ≥ key (v )
Associated Operations (min-heap):
H ← Initialize() . O(n)
builds a heap given a set of data elements with keys.
Insert(H, v , k) . O(log n)
inserts an element v with key value k in H
Delete(H, v ) . O(log n)
deletes the element x from H given the pointer to x
DecreaseKey(H, v , k 0 ) . O(log n)
decreases the key of element x in H to new value k 0 given pointer to x
v ← extractMin(H) . O(log n)
returns the element v with min key value and deletes it from H
Priority of each element in P is key of the respective element in H.
Min-Heap

A min-heap maintains a set of n elements each with a key and

satisfies the min-heap property.
Heap implementation uses a complete binary tree (binary heap) H
Every node has a key smaller than its both children
Data element with minimum key is at the root
Binary heap can also be represented as an array A of n elements
The minimum value (root) is at A[1]
The left and right child of a node at A[i] are at A[2i] and A[2i + 1],
respectively
The parent of a node at A[i] is at A[b 2i c]
The sequence of vertices visited by level order traversal (BFS) of
the binary tree maps to the order of nodes in the array representation
Min-Heap: Binary Tree and Array Representation

2 3

4 5 6 7

8 9 10 11 12 13 14 15

Point of next insertion

1 2 3 4 5 6 7 8 9 10 12 13 14 15

A binary min-heap and its array representation

Min-Heap Operations: Insert

4 5

7 17 6 21

8 9 28 31 12 13 27

Insert(H, v , 3)
Min-Heap Operations: Insert

4 5

7 17 6 21

8 9 28 31 12 13 3

Insert(H, v , 3)
Insert v to next available position in H
Min-Heap Operations: Insert

4 5

7 17 6 3

8 9 28 31 12 13 21

Insert(H, v , 3)
Sift-up the added node as needed to restore heap property
Each Sift-up moves the node to the level above in the tree
Min-Heap Operations: Insert

4 3

7 17 6 5

8 9 28 31 12 13 21

Insert(H, v , 3)
Min-Heap property restored
At max, as many Sift-Up moves can be made as the height of the tree
Recall: Height of a balanced complete binary tree with n nodes is O(log n)
Therefore, sifting up takes O(log n) time
Pseudocode : Insert, Decrease-Key and Sift-Up

function Insert(H, v , k) . O(log n)

H.Append(v ) . insert v at end, i.e. last available position in H
key (H[v ]) ← k
SiftUp(H, v )

function SiftUp(H,v ) . O(log n)

p ← getParent(v )
if key (H[v ]) < key (H[p]) then
swap(H[v ], H[p])
SiftUp(H, p)

function DecreaseKey(H, v , k) . O(log n)

key (H[v ]) ← k
SiftUp(H, v )
Min-Heap Operations: Extract-Min

4 3

7 17 6 5

8 9 28 31 12 13 21

Extract-Min(H)
Min-Heap Operations: Extract-Min

4 3

7 17 6 5

8 9 28 31 12 13 21

Extract-Min(H)
Extract the root node to be returned
Min-Heap Operations: Extract-Min

21

4 3

7 17 6 5

8 9 28 31 12 13

Extract-Min(H)
Delete the last filled node at its place and move its element to root
Min-Heap Operations: Extract-Min

4 21

7 17 6 5

8 9 28 31 12 13

Extract-Min(H)
Sift-down as needed to restore heap property
Each sift-down moves the node to the level below in the tree
Min-Heap Operations: Extract-Min

4 5

7 17 6 21

8 9 28 31 12 13

Extract-Min(H)
Min-heap property restored
At max, as many Sift-Down moves can be made as the height of tree
Recall: Height of a balanced complete binary tree with n nodes is O(log n)
Therefore, sifting down takes O(log n) time
Pseudocode: Extract-Min and Delete

function Extract-Min(H) . O(log n)

return Delete(H, root)

function Delete(H, node) . O(log n)

key ← H[node] . node is pointer to element to be removed
Swap( H[node], H[lastFilledPosition] )
Remove(H[lastFilledPosition])
SiftDown(node)
return key
Pseudocode: Sift-Down

function SiftDown(H, node) . O(log n)

l ← leftChild(node)
r ← rightChild(node)
if key (H[l]) < key (H[r ]) then
if key (H[node]) > key (H[l]) then
Swap( H[node], H[l] )
SiftDown(H, l)
else
if key (H[node]) > key (H[r ]) then
Swap( H[node], H[r ] )
SiftDown(H, r )