THEORY OF ALGORITHMS

INTRODUCTION (SLIDES + TEXTBOOK + JANE NOTES)

Algorithm:
• Sequence of unambiguous instructions, if unambiguous → can’t be converted into code
• Used for solving well-defined problems → must be well-defined otherwise can’t solve it → else, uncertainty will
carry on to the instructions of the ALG.
• alg’s guaranteed to terminate in input is valid
• Subset of procedures guaranteed to terminate
• Sits above code
• Study of algorithms → algorithmics
• TEXTBOOK: a computer scientist should be able to construct, manipulate, understand, and analyze alg’s.

Halting problem:
• Write an ALG that given any program as input, will tell us whether/not for any input, the program stops?
• No = can’t prove

Algorithm Features:
Feature Def
Finite Terminates after finite no. of steps
Definite Rigorously and unambiguously specified
ALG should only be interpreted in one way with
the same behavior.
Input Valid inputs → clearly specified
Output Can be proved to produce the correct output
given a valid input
Effective Steps are comprehensible → basic & simple

Notion of an Algorithm:

• Diagram: the (well-defined) problem is used to construct/design an algorithm. Then, given a certain input, the
computer uses the algorithm to produce a certain output.
• Again, each step must be unambiguous.
• Range of inputs (for which an ALG works) must be specified carefully.
• Same ALG can be represented in diff ways
• Several ALG’s may be used to solve the same problem → these ALG’s can be based on diff ideas and can solve
the problem at diff speeds.
No point in finding better (or faster) algorithms if
• that part of the sys is not the bottleneck
• time isn’t the issue
• program will only be used a few times

Design Basics:

• Specify alg – using pseudocode

• Use data structures
• Design Principles:
▪ Simplicity
▪ Generality
(inputs/special cases)
▪ Optimality (the best)
• Formal Alg analysis: Efficiency →
time and space

Algorithm design technique (strategy/paradigm): general approach to solving problems agorithmically that is applicable
to a variety of problems from diff areas of computing.

Proving corectness: have to prove that the algorithm yields a required result for every legitimate input in a finite amount
of time. Common tech: mathematical induction bc ALG iterations provide natural sequence of steps needed for such
proofs.

Problem domains:

• Sorting and searching

• String processing
• Graph problems
o Structuring domain as set of nodes with links
o E.g., traveling salesman
• Combinatorial problems
o E.g., List of all permutations of a set
• Geometric problems
o Closest pair, convex hull.
o GPS: least number of roads to get from A → B
• Numerical problems
o Solving equations, randomisation, matrix work
Types of ALGORITHMS:

1. Brute Force Follow def. Try all possibilities.

2. Decrease and conquer Solve larger instances in terms of smaller instance.
3. Divide and conquer Break problem into distinct subproblems.
4. Transformation Convert problem to another
5. Space/Time tradeoff Use additional DS
6. Dynamic Programming Break into overlapping problems
7. Greedy Repeatedly doing what is best now

Fibonacci
Brute Force:

This recursion example below makes a binary tree The recursive solution calculates Fibonacci numbers using repeated calls to fibonacci(n-1) and fibonacci(n-2).
Exponential Time Complexity: O(2^n)

Better Solution 1:

The iterative solution computes Fibonacci numbers using a single loop.

Linear Time Complexity: O(n)

• Forward order – work out all elements up to & including the one you need
• Compute nth fib number, progressing from F0 & F1 → working forward
• Array F: set 1st and 2nd elements to 0 and 1
• Then, use for-loop from 3rd element until nth term
o At each step, lookup previous 2 and add them together then place in current element
• Better → MIGHT HAVE PROBLEMS
 Better Solution 2:

• Use big integer because Fib sequence grows exponentially. This prevents integer overflow.
• Don’t use entire array
o 3-element array (don’t store all prior elements)
o Overwrite data in prev elements
• Good space complexity and solves problem for large data sizes
• Using the fib mathematical formula: gets increasingly worse for higher n; Floating point rounding problems;
Practical implementation problems

Greatest Common Divisor (from pg 4):

Def: given two integers, find the largest number that divides them both → gcd(60, 21) = 12

Euclid’s algorithm

• Created by Euclid of Alexandria

• 3rd century BC
gcd (m, n) = gcd(n, m % n)
- m % n is the remainder between m & n. Until this is equal to 0. Then the last value of m before 0 is the GCD
• Example: gcd (60, 24) = gcd (24, 12) = gcd (12, 0) = 12

Euclid with recursion:

• Tail recursive: descends rapidly in straight line as the last call

is made to the recurive step

Euclid without recursion:

• No more efficient that recursive solution →

recursive solution is tail recursive → c++ compiler will opitimise and
remove recursion.
Second version:

ALGORITHM EFFICIENCY (SLIDES + TEXTBOOK + JANE NOTES)

Need measures of:
• Inputs size (n) – Influences computational eff.
• Unit of measuring time – the basic operation of the algorithm. Look at operations commonly executed in ALG.

Interested in:
• Growth order (asymptotic complexity)
o O(n^2) vs O(n^3)
• Performance
o Best/Average/Worst
• Time and memory efficiency for input size n
o Basic operations (time)
o How many extra memory units are required (mem)
• Computational eff
o How long it takes to run on certain inputs

Linear search: Searching through an array

• Worst case: it must look at all the elements in the array. Let’s say there are ‘n’ number of elements in the array.
WC = O(n)
• Best case: first element looked at is what we want. BC = O(1)
• Average case: n/2 elements will be looked at before finding the right one. But ½ is a constant so we take it away.
AC = O(n)
• Want the tightest bound

Efficiency Classes:
Big-O:

Example 1:

• Algorithm executes: 2n + 10 operations

• Not interests in 2 or 10
• This is O(n)

Example 2:

• Algorithm executes: 3n^2 + 9n + 5 operations

• Not interested in the contants or even 9n.
• This is O(n^2)

Definition:

• A function t(n) ∈ O(g(n)) if (and only if) there is a c and an n0 such that t(n) <= cg(n) for all n>=n0
o T - Function on the no. of operations and ‘n’
o Belongs to class Big O of some function and if can determine some constant ‘c’ and input size n0, so that
this function ‘t’ will always be less than the constant ‘x’ function of Big-O for all input beyond n0.
o Upper bound on ALG performance
•

Big-Omega:

• A function t(n) ∈ Ω(g(n)) if there is a positive constant c and a non-negative integer n0, such that t(n) >= cg(n) for
all n>=n0
• Not widely used – lower bound growth

Big-Theta:

• Tight bound – both upper and lower – on computational complexity

• Sequentially summing an array – eff = O(n)
o But also, by def, O(n^2); O(2^n), etc.
o So: Any more expensive efficiency classes, also subsume the less expensive ones.
o NB: it is not O(log n)
• Summing array’s basic operation executes EXACTLY proportional to ‘n’
o Efficiency class for summing an array is Θ (n)
o It is not: Θ (n^2) or Θ (n log n)
o It is precisely Θ (n)
• A function t(n) ∈ Θ(g(n)) if there is a c1, c2, and n0 such that c2g(n) <= t(n) <= c1g(n) for all n>=n0
o Take upper and lower bound and ‘sandwich’ function between them.
Useful Theorem → can estimate complexity

If t1(n) ∈ O(g(n)) and t2(n) ∈ O(h(n)) then t1(n) + t2(n) ∈ O(max{g(n), h(n)})

Same holds for Θ and Ω

Comparing Big-O, theta, and omega:

BIG-O BIG-Omega BIG-Theta

Running time (RT) as ‘n’ gets larger is RT as ‘n’ gets larger is at least prop to RT as ‘n’ gets larger is exactly
at most proportional to g(n) g(n) proportional to g(n)
Upper bound Lower bound Exact bound

Examples

• Average of O(n^2) – algorithm (runtime) grows at most as fast as n^2 with average case input

o E.g., Bubble sort

• Worst case of O(n3) – algorithm grows at most as fast as n^3 with its worst case

o E.g., brute force matrix multiplication, which is also Θ(n3)

• Best case of Θ(n) – algorithm grows linearly in the best case

o E.g., sum an array

• Worst case of Ω(2n) – algorithm grows at best exponentially in worst case

o E.g., create the power set

Checkpoint solutions to true/false:

Efficiency Derivation (Sequential)

I need help understanding this

S1:
• Standard for-loop → goes from standard counter at index ‘j’ to ‘n’
• Operation inside loop = 1
• Number of occurrences is n-j+1 → upper limit – lower, subtract 1

R2:
• Summation of ‘c x ai’ where ai depends on index ‘I’

R3:
• Can separate out summations

S2:
• Summation where amount of work done depends on the index
• Use sum of geometric sequence → equivalent to n^2
Analysis solution 1:
What does the alg do:
• Finds the largest element in the array → “maxVal”
What is the basic operation
• A[i] > maxVal
• comparison
• This is basically checking if the number is bigger than the current biggest number
How many times is the basic operation executed?

Asymptotic efficiency class?

• Big Theta (n)

Is it a set?
Description:
• Loaded a file of strings into an array
• Want to check if there are any duplicates → each element must appear once

Brute force: 2 loops. Loop through the array twice, compare one value to every other element in the array. As soon as
there’s a matching element = return false.
• Worst case: Completion of both loops
• Core operations: comparisons (if statement)
• How many times is the inner loop executed? Worst case → Θ(n^2)

Decrease and conquer: Recursion. A new index value is passed each time. Still loop through the array until a match is
found. If a match is found, return false because it's not a set.
• Mathematically identical
• Worst case: Θ(n^2)

Transform & conquer: Sort the array and then do a linear search through the array.
• Most efficient sorting algorithms are Θ(n log n)
• The for-loop is linear, so the worst case is Θ(n)
• Sort + For-loop: Θ(max{n log n, n}) = Θ(n log n)

‘Brute force’ vs ‘transform and conquer’: Transform and conquer is faster than divide and conquer.

Recursive Algorithms:
Pattern for determining efficiency:
1. Determine core operation
2. Create equation for number of repetitions
3. Solve
4. Derive efficiency

Equations are recurrence relations and not summations:

• Build similar math toolset for recurrence relations
• Solve by Backward Substitution or Master Theorem
Recurrence Relations
• Recursive mathematical function (see example below) that has a recursive and base case.

• Acts as a tool to analyze recursive algorithms.

• Recurrence relations
o T(n) = aT(n-k) + f(n) // Like M(n) above
o OR
o T(n) = aT(n/b) + f(n)

Analysis exercise 3: Factorial

Basic operation:
• Multiplication

How many multiplications:

• Once for every recursive call
• f(5) → f(4) x 5
• f(4) → f(3) x 4 x 5, etc.

Backward substitution
• Let M(n) be number of multiplications
• Then M(0) = 0
o And
o M(n) = M(n-1) + 1
o M(n) = M(n-1) +1 = [M(n-2) + 1] + 1 = [M(n-3) + 1] + 2 = M(n-3) + 3
• And in general:
o M(n) = M(n-k) + k,
o for k substitutions
o Thus, when k = n (n substitutions) M(n) = M(n-n) + n = M(0) + n = n
o So M(n) ∈ Θ (n) [it is exactly n in this case]
Analysis 5: Fibonacci I need help to understand this one as well

Basic operation
• Addition
Recurrence relation:
• A(n) = A(n-1) + 1 + A(n-2)
• Uses “characteristic equation”
• A(n) ∈ Θ(1.61803n) (!)

Master Theorem Motivation

https://www.youtube.com/watch?v=OynWkEj0S-s
https://www.youtube.com/watch?v=2y0HQGd1-nA

Not a solution to all recurrences - but, shows the asymptotic efficiency class of recurrence relations in the form:
• T(n) = aT(n/b) + f(n) where f(n) ∈ Θ(n^d)
This provides a shortcut that replaces backward substitution.

Example:
Consider: A(n) = A(n / 2) + 1, A(1) = 0
• For pattern: T(n) = aT(n/b) + f(n), f(n) ∈ Θ(n^d) and T(1) = c
• a = 1, b = 2, c = 0, and f(n) ∈ Θ(1) = Θ(n^0)
• Thus, d = 0
Select master theorem case:
• b^d = 1 → (2^0), therefore, a = b^d
• By master theorem: T(n) ∈ Θ(n^d log n)
Since d = 0
• T(n) ∈ Θ( log n)
• n^0 = 1 → don’t need to show the coefficient.
See example 2 → notebook (ok, here’s the plan) and slide deck 2.

BRUTE FORCE ALGORITHMS (3)

Description:
• Straightforward approach directly based on the problem statement.
o Implies lack of sophistication and thought,
o Exhaustive search is a special case
• Sometimes the quickest, easiest, and best solution.
• Good for occasional use and small problem sizes.

String matching:

• A kind of linear search

• Search for the word ‘SEA’ in the sentence or string ‘SEE SHE SEA’
• Basically, check characters in the pattern (SEA) against the 1st, 2nd, and 3rd position.
o 1st check “SEE” against “SEA” - fails, shift one place to the right.
• High level explanation: Align pattern at beg of word. Moving left to right, compare each character of the pattern
to the character above until all characters matches. While text is not found, realign one position to the right
each time.
• ‘N’ characters in text (T) and ‘m’ characters in pattern (P).
o Last viable pos is at n-m

Explanation of code snippet:

• Text (size n) and pattern (size m)

• Iterate through text from 0 to (n-m) as this is the last possible index that the pattern could occur in.
• ‘j’ is a local index that just passes across the pattern.
 • While loop → pattern check
o Increment if we have not reached the end of the pattern (m)

Worst case: the search string matches on every character accept the last one, for iteration of the outer loop.
• The ALG may have to make ‘m’ comparisons before shifting the pattern and this can happen for each of the ‘(n-
m) + 1’ tries
• Therefore, the worst case is m(n-m)+1
o = nm-m2+m – as long as n >> m, can remove the lower order terms (-m2+m)
• Θ(m x n) for ‘m’ much smaller than n (which is what happens in practice)
• Average case on natural language? – Θ(n)

Closest Pair:

Problem:

• Find the 2 points that are closest together in a set of ‘n’ 2D points
• Almost like ‘is this a set’.
o Take a given point & compare it against other points → record the shorter distance each time.
o Then, discard that point, take another and do the same

Explanation of code snippet:

• Set initial distance between points to ∞, this way the first distance is going to be smaller.
• Iterate over poits from 1 to n-1 (end)
o Then iterate from current point + 1 to the end
• Record distance between 2 points using formula.
• Return indices of closest distance

Efficiency: Θ(n2 )

Expensive in practice but can be avoided.

Convex Hull Which algorithm was being used to get the worst case as O(n^3)

Problem:
• Find convex hull enclosing ‘n’ 2D points
• Convex Hull: if ‘S’ is a set of points then the convex hull of ‘S’ is the smallest convex set containing ‘S’
o BASICALLY → smallest enclosing boundary around a set of points
• By definition, an extreme point of a convex set is a point of this set that is not a middle point of any line segment
with endpoints in the set.

Convex Set: set of points in the plane is convex if the line segment between 2 points belongs to the set
• 2 points on the set
• If a straight line is drawn between, there is no point on the line that is outside of the convex set
ALG: For each pair of points p1 and p2, determine whether all other points lie to the same side of the straight line
through p1 and p2. Then they form the convex hull.

Efficiency: Θ(n3)
• Get every single pair of points O(n^2)
• Compare 2 points against other remaining points O(n)

PROS & CONS OF BRUTE FORCE

Pros

• Wide applicability
• Simple
• Yields reasonable ALG for some important problems & std ALG for simple computational tasks
• Yardstick for better algorithms

Cons

• Rarely produces efficient ALG

• Some infeasibly slow

EXHAUSTIVE SEARCH
EXHAUSTIVE SEARCH

Basically there’s a lot of different situations & have to enumerate through all of them to determine which is best.

Def:

• Brute force solution to search for an element with a special property.

• Usually among combinatorial objects such as permutations & subsets (order within subset doesn’t matter)

Method:

1. Construct a way of listing potential solutions to the problem in a systematic manner.

i) Solutions eventually listed

ii) No solution is repeated

2. Evaluate solution one by one - remove infeasible ones.

3. Announce ‘winner’

1. Traveling salesman problem (TSP):

Problem:

• Given ’n’ cities with known distance between each

• Find the shortest tour that passes through all the cities (once) before returning to the starting city —> Start at ‘a’
& then, end at ‘a’.

Alternatively: Find shortest Hamiltonian Circuit in a weighted connected graph.

• Hamilitonian Circuit is defined as a cycle that passes through all the vertices of the graph exactly once
Solution:

• Calculate all routes —> all permutations of route

• Improvements

o Start & end at the same point

o Remove tours that differ only in direction

Efficiency: (n-1)!/2 = Θ(n!)

2. Knapsack Problem:

• Think of yourself as a thief, how many items can you fit in your bag for the most value
• ‘n’ items of known weights ‘w1’, ‘w2’, etc. and values ‘v1’, ‘v2’, etc.
• And weight or capacity of the bag ‘W’.

Problem example:

• So lets say the capacity is W = 16kg, everthing after that is infeasible and cant fit
• We must find what combination of items is gonna give us the most value
• BUT, total weight must be under 16kg.
Efficiency: Ω(2n)

• Number of subsets of an n-element set is 2^n

• Exhaustive search approach is not efficient.

3.Assignment Problem:

Another problem that be solved using exhaustive search.

About:

• ‘n’ people that need to be assigned to execute ‘n’ jobs – 1 person per job.
• Cost accrued if the ith person is assigned to the jth job is a known quantity C[i, j]
• Point is to find the assignment with the lowest cost.

Example of problem:

• Select one element in each row and place in different columns

o E.g., {1, 4, 3, 2} → means P1 gets job 1, P2 gets job 4, P3 gets job 3, and P4 gets job 2.
o List all possible assignments → notice the permutations

Solution:

• Second line: P1 – Job 1; P2 – Job 2; P3 – Job 4; P4 – Job 3

• Generate all permutations of ‘n’ positive integers → find the total cost of each
• Sum the cost of each permutation
• Retain the one with the cheapest cost

Efficiency: Θ(n!)– we can do better

• More efficient: Hungarian Method
Comments on Exhaustive search:
• Run in realistic amount of time → mainly small instances
• Better alt’s
• Parallel solutions = speedup.

DECREASE AND CONQUER (4)

Strategy:

• Solve smaller instances

o Take original problem & shave off some proportion of it
o Usually solve in recursively
• Extend solution of smaller instance to obtain solution to original problem

Called inductive or incremental

3 Variants:

• Decrease by a constant
• Decrease by a constant factor – divide size of problem by a constant (e.g., 2)
• Var size decrease – depending on data, might reduce by more/less each iteration

Divide and conquer is different:

Decrease & Conquer: Throws away half (or constant factor) of work → Binary Search

Divide & conquer: Divides a problem and solves both halves then combine results → Quicksort

Map of Decrease & Conquer Alg:

1) Decrease by a constant (often 1):

→ Insertion sort, graph searching (DFS, BFS), generating permutations, topological sort
2) Decrease by a constant factor (often 2):
→ binary search, fake-coin problem, multiplication a la Russe.
3) Variable-size decrease:
→ euclid’s alg, interpolation search, finding Kth order stat → median
Strengths and weaknesses:

Strengths:

 Implemented either top-down (recursion) or bottom-up

 Efficient (possibly Θ(log n) for decrease by constant factor)
 Powerful form of graph traversal (BFS and DFS)

Weaknesses:

 Less widely applicable (esp decrease by constant factor).

DECREASE BY A CONSTANT:

Insertion sort:

Idea:

• Take element
• Assume already sorted list of (n-1) for ‘n’ elements
• Insert remaining element in the correct position → find position (where the element should be placed) by
searching through the list, of where this element is (that has been taken away).
• Before doing this, recursively perform insertion sort on smaller list.

Recursive idea but better performed iteratively bottom up

Decrease by 1 and conquer

Worst case efficiency: Θ(n2) → partially sorted inputs

Code example of sorting first then inserting the next element:

• Take & set 1st element aside, insert into already sorted list
o List size = 1
• Reducing the size of the unsorted list at each step
o Reach sorted list

Topological sort

Problem:

• In a directed acyclic graph (DAG) → list the vertices in an order such that edge direction is respected.
• Any directed edge the source vertex must appear before the dest in the list.
• Multiple possible orderings
 • Cycles = not solvable
• DFS (depth first search) can be used to solve

Applications:

• Ordering set of courses that have pre-requisites

• Eval formulae with dependencies in a spreadsheet
• Ordering tasks in a complex project

Generating permutations:

Component of exhaustive search

All possible orderings from range of values

• Can be used as indices to other objects

Solution:

• Generate all (n-1)! Permutations of {1,…, n-1}.

• Insert ‘n’ into each possible position → take a smaller list of numbers (1 by 1), generate all of those perm’s and
then, when that set of perm’s is returned, insert you number
• During insert starting from right → left alternatively, satisfies Minimal-Change requirment (next perm obtained
by swapping 2 elements)
o Useful variation
o Useful to apply this alg where there is come cost assoc with doing it
• Decrease by 1 and conquer

NB: original size is 3 and we decrease the size to 1.

Insert next element in 2 possible options

Right to left insertions

Johnson-Trotter Method:

Alternative permutation generator that avoids permutations of smaller lists → more efficient bc it doesn’t require you to
derive intermediate results (no need for previous tree-like structure).

Use arrows to keep track of what permutations comes next:

• An element ‘k’ is mobile if its arrow points to an adajent element smaller than it.

Johnson-Trotter Example:

• Initiak list of numbers & set each 1 to be a left mobile number → move leftwards each time until it can bc left
number is bigger.
• Then, while you still have mobile elements, repeatedly:
o Search & find largest mobile element
o Swop ‘k’ with its immediate neighbour in the direction in which its pointing to.
o Reverse direction of all elements → ‘k’ arrows flip → REVERSE DIRECTION OF ALL ELEMENTS BIGGER
THAN THE VALUE MOVED.
o Print the current state of those lists of numbers as one of the permutations.
• Carry on until there is no more mobility, then return the list of perm

DECREASE BY A CONSTANT FACTOR:

Fake Coin Problem:

Problem: Among ‘n’ coins, one is fake (& weighs less). There’s a balance scale which can compare any 2 sets.

Algorithm:

• Divide into 2 size floor piles → |n/2| (keeping a coin aside if ‘n’ is odd, e.g., n = 3)
• If ‘n’ is odd, and if the 2 coins on the scale weigh the same then the one that’s left out is the fake coin.
• Otherwise proceed recursively with lighter pile
• Efficiency:
o |_ _| => floor

o W(n) = W(|n/2|) + 1 for n > 1

 o W(n) = |log2 n| = Θ(log2 n) – applying the Master Theorem

Fake Coin Problem 2:

Algorithm → 3 pile:

• Divide the 3 piles →

• If the first 2 piles weigh the same, disgard, and continue with the 3rd pile.
• Otherwise, proceed recursively with the lighter of the 2 piles.

Efficiency:

Multiplication A LA Russe:

Convert multiplication into something that involves halving and doubling.

• More efficient in hardware (shift operations)

• For below example, decrease by factor of 2 and conquer ( UNTIL ‘n’ = 1).

n*m:
• (n/2) * (2m) → if ‘n’ is even
• [(n – 1)/2] * (2m) + m → if ‘n’ is uneven

Example: 50 * 20

• (25 * 40)
• (12 * 80) + 40
• (6 * 160) + 40
• (3 * 320) + 40 → (1 * 640) + 320 + 40 = 1000

VARIABLE SIZE DIFFERENCE:

Euclid GCD:

Greatest common divisor of 2 integers ‘m’ and ‘n’ is the largest integer that divideds both

Solution:

• Gcd(m, n) = gcd(n, m % n)
• Gcd(m, 0) = m
• Right-side args are smaller by neither a constant nor factor

Example: gcd(60,24) = gcd(24,12) = gcd(12,0)=12

Finding the kth order statistic:

Problem: finding the kth smallest element in a list. The median is usually k = n/2. But sorting the list is inefficient.
Solution:

• Exploit quicksort → noting that pivot is placed in the correct position

• Partition as usual = p. Everything less than ‘p’ is on the left and everything bigger is on the right.
• Only need to continue with one of the 2 partitions
• (If pivot ends up ≥ k’th position, search the first partition, else search the 2nd partition)

Variable size because quicksort doesn’t partition into equal size sub lists. → Not divide & conquer bc we throw-away the
other half of the list is thrown away

2
Efficiency: average case Θ(n), worst case Θ(n )

DIVIDE AND CONQUER (5)

Situation – continuously half the problem size.

Best known algorithm design strategy:

• Divide instance of a problem into 2/more smaller instances

• Solve smaller instances recursively
• Obtain solution to OG (larger) instance

Recurrence relations (templates) apply:

• Recurrences are of the form: T(n) = a*T(n/b) + f(n), a >= 1, b>= 2

• Multiply cost of smaller part → T(n/b) by a factor of ‘a’
• Then, do some additional work

E.g.: List of numbers that needs to be added together. Divide the list in half, and recursively add. The add the two
answers.
Illustrated:

Merge Sort:

Efficiency: O (n log n)

Algorithm:

• Split array A into halves of array B and C

• Recursively merge sort arrays B and C
• Merge the sorted arrays B and C into A

Merging (REPEAT until no elements left in B/C):

• Compare 1st elements in the rest of B and C

• Copy smaller into A, incrementing index of corresponding array
• Once all elements in B/C are processed, copy the remaining unprocessed elements from the other array into A.

Example:

Diagram description:

• Split until we get single elements

• LHS: 2 < 7, so 2 goes into the array first to become → [2, 7]
• Combine LHS and RHS.
Master Theorem:

• T(n) = a*T(n/b) + f(n), f(n) ∈ Θ(nd)

• T(n) ∈ Θ(nd log n) if a = bd

Recurrence relation:

• C(n) = 2 C(n/2) + Cmerge(n) for n > 1, C (1) = 0

o C (n/2) → cost of dividing list, and this is multiplied by 2 because we’re working on both halves.
o Cost of merging depends on ‘n’
• Cmerge(n) = n-1 in the Worst Case
o Do comparisons between the sum of the number of elements in both lists – except for last one →
doesn’t require comparison → just be appended.
• Thus, a = 2, b = 2, d = 1.

Efficiency (all cases):

• a = bd (2 = 21) in Master Theorem

• Θ (n log n) → for best, worst, and average case.
• Uses extra space Θ(n) → need another array of same sizes as original data

Quick Sort:

Algorithm steps:

• Select a pivot → partitioning element

• Rearrange the list/array into to subarrays
o Elements before the partition are smaller
o Elements after, are bigger.
o Pivoting alg
• Exchange pivot with last element in first subarray → right place

Algorithm:
Example:

Worst-case efficiency:

• All splits are completely skewed.

• Doesn’t perform well on an already sorted list.
o Choose first element as pivot → everything is lopsided.
o ‘n’ work to get the elements in the one array
• Other array is empty → only make one comparison to get pivot in correct position
o Exchange pivot with itself
o N + 1 comparisons
o Quicksort right = Ø, Quicksort left = A [1… n-1]

General efficiency:

• Worst case (first element of sorted list): Θ(n2)

• Best case (split in the middle): Θ (n log n)
• Average case (random split): Θ (n log n)

Improve efficiency:
• Better pivot selection: E.g., take one element from start, middle, and end of list and select median of those 3.
• Insertion sort: Switch to this on small subfiles
• Eliminate recursion: overhead costs.
Geometric Problems:

Closest pair (by divide and conquer):

• Sort according to x-coordinates → right to left

• Split the set of points into to equal sized subsets, by vertical line x= x(median)
• Solve recursively in left and right subsets to get right (dr) and left (dl) minimum → dmin = min (dr, dl)
• Straddle → find minimal distance in the set S of point of width 2d around the vertical line and update dmin if
need be.

Divide and conquer because dividing space into two halves i.t.o. the number of points in each half

Example:

• Split and solve: Recursively find smallest pair in each half

• Straddle: Search for potential new Dmin among points within ‘d’ of dividing line

o Efficiency: a = 2, b = 2, d =1
o A = b^d in Master Theorem
o Θ(n log n) – Same cost as pre-sorting step
o 2 Sorts in algorithm → initial sort from left to right according to ‘x’
▪ Then another in straddle zone → sort vertically in increasing ‘y’
Limits in the straddle zone:

• Difficult to differentiate from (in terms of efficiency) from brute force

o What if S has almost as many points as the non-S part?
o Then must compare each point in ‘S’ to all points by brute force (for loop)
o Efficiency: Θ(n^2)
• Property in S that limits no. of points → limitation depends on
sorting points in straddle zone by increasing ‘y’

• Only compare against next 5 points → inner loop doesn’t iterate

over all remaining points
• Given the restriction that points to left → right of the dividing line,
must be at least d = dmin apart
• Config on the right represents the closest packing
• Not possible for a 6th point to be close to ‘P’ in the sweep direction

QuickHull:

Solve convex hull like quick sort

Solution:

• ID leftmost (min x co-ord) and rightmost (max x co-ord) points → P1 and P2

o ‘draw’ line between P1 and P2
o Points to the left are upper hull and points to right of lower hull
o Division not even → no even partition
• Compute upper hull
o Find point P-max that is farthest away from line
o Quick Hull the points to the left of the line P1→Pmax
o Quick Hull the points to the left of the line Pmax→P2

• Compute lower hull

TRANSFORM AND CONQUER (6)

Transform problem and solve the transformed problem

Diff sub strategies (NB the way you transform the problem):
 • Create simpler e.g., of the same problem → sort array before trying to solve
• Use another kind of data structure to support solution to the problem → heaps, balance-search trees.
• Transform instance of current problem into another → use existing solution to solve

Approaches:

• Instance simplification: more convenient instance of same problem

o Pre-sorting
o Gaussian elimination
• Representation change: diff representation of the same instance
o Balanced search trees
o Heaps and heapsort
o Polynomial evaluation of Horner’s rule
o Binary exponentiation
• Problem reduction: diff problem all together
o Lowest common multiple
o Reductions to graph problems

Instance Simplification
Gaussian elimination:

Solution of a fully determined sys of linear equations

First transform to upper triangular form by Gaussian elimination (simplification) and then solve by backward substitution
(conquer).

• Convert initial matrix of coefficients into something that is upper diagonal

Convert into RHS form → once have zeroes lower diagonals → easier to solve → use backwards substitution.

Pre-sorting:

Solve instance of problem by pre-processing the problem to transform it into another simpler instance of the same
problem

Problems easier when list is sorted:

• Searching
• Finding median
• Finding repeating elements
• Convex hull and closest pair

Efficiency:

• Overhead of Θ (n log n) pre-process

• Sorted problem improves efficiency by one base class:
o (e.g., Θ(n2) → Θ(n))
Is pre-sorting better?

• Sorting is Θ (n log n) → only worthwhile if other algorithms are less efficient

o Closest pair: Θ(n2) → Θ (n log n) → pre-sorting is best
o Checking for unique elements → pre-sort is best
o Finding min and max → Brute force is Θ(n) → pre-sort NOT better
o Searching array → Brute force is Θ(n) → pre-sort NOT better

Notes on the selection problem:

• Pre-sorting based algorithm: Θ (n log n) + Θ (1) = Θ (n log n)

o Θ (1) → constant fetch time of element

Representation Change
Evaluating polynomials:
Fast Fourier Transform (FFT)
→ Brute force polynomial

• For a polynomial of size ‘n’, just the 1st term required ‘n’ multiplications using brute force
o Becomes an Θ(n2) alg
• Improve efficiency by calculating x^n → calculating lower order terms and then gradually building up
• Use Horner’s rule → does better for large polynomials and is easy

→ Horner’s Rule

• Efficient alg for evaluating a polynomial.

 • Representation-change technique
• Factor ‘x’ out as much as possible (factorization).
• Put coefficients into an array
• Then, for given x-value, successively calculate brackets.

→ Pseudocode:

Parameters:

1. Array of coefficients
2. Value ‘x’

Method content:

• Set ‘p’ equal to first coefficient in final position

o Coefficient that responds to inner-most nested part of the polynomial
• Iterate from the (n-1)th polynomial coefficient, down to 0
• Successively multiply ‘x’ by current running total & adding coefficient for the correct position
• Return evaluated

Efficiency:

• Basic operations → multiplication & addition

o Let no. of multiplications = M(n)
o Let no. of additions = A(n)

o
• For the entire polynomial, it takes as many multiplications as the Brute Force method uses for its first
term.

Binary exponentiation:

Problem: Solve a^n using binary representation of ‘n’

Solution:

• Represent ‘n’ as a polynomial p(x), with x=2

• Then, coefficients of polynomials are digits of the binary presentation.

•
Horner for exponentiation:

• Factorize p(x)
• Calculate ‘p’ → get 13, as ‘p’ is a representation of 13.
o When x=2, p(x) should = 13
o ‘P’ is the values of each consecutive bracket
• Calculate a^p → each step of Horner, take current total multiply by ‘x’ and add coefficient in
Algorithm (for exponentiation):

Horner’s Rule (LHS) Implications (RHS)

P (2) and the coefficients are either 0/1 1st will always be a^1

P = 1: Assume highest order coefficient will be 1, because Successively build up final a^n by applying relevant
in any binary representation first digit is never 0. formula.

Apply for-loop → with 2 subbed in for ‘x’. Simplify statement → take power term and split into
halves → power term is added == multiplying (with same
base).

C [i] → coefficient can be either 0/1

If 0 → then 2nd term falls away
If 1 →
General Horner’s rule for exponentiation:

See efficiency slide 22

PROBLEM REDUCTION

STRENGTHS AND WEAKNESSES OF TRANSFORM AND CONQUER:

Strengths:

• Allows powerful data structures to be applied

• Effective in complexity theory

Weaknesses:

• Difficult to derive → reduction

SPACE AND TIME TRADEOFF (7)

Time algorithm takes to run VS the overall amount of mem it consumes

• More space to reduce overall time alg uses

IDEA: Trading space for time

• More mem to reduce computation cost

• Good idea if need to do the same task multiple times using a subset of the inputs (to start with).

Input enhancement (pre-processing)

• Store supportive info about the problem instance which will make subsequent solving it faster
• Problem instance → do preprocessing on it → build up to a table that will help accelerate the problem solving
• E.g., sorting by counting, Horspool and Boyer-Moore’s string matching

Pre-structuring

• Step to make access to the data faster

• E.g., hashing, indexing with B-trees
Dynamic programming fits into this category.

Sorting by counting:
Idea:
1. Sort list whose elements fall in a restricted range of integers [L…U].
a. Can use pre-processing to make actual stage of sorting, linear → general sort that’s
unrestricted
b. Range within lower and upper integer.
2. Frequency table → counts no. of occurrences of each element
3. Distribution table derived from frequencies → tells where to place the elements
a. Convert freq table to distribution
Example:

Frequency table → low = 11 and high = 13

• 3 elements → 11,12,13
• Run through (Θ(n)time)→ calculate no. of occurrences of integers.
• Prefix sum → See the distribution values 1, 4, 6 → shows where the values will occur.
o E.g., 11 will occur in the 1st position. 12 will occur (last) in position 4 (previous distribution value
+ freq of 12).
• Final run-through → grab elements into input array in linear time and place into their final position →
start from last element in OG array → work backwards
o Start with 12 → maps to position 4 (distributional) → put 12 at position (4-1) 3 → then subtract
one from the distributional value
• 1 linear iteration to get frequencies
• 2nd linear iteration to calculate distributional
• 3rd linear iteration through list to sort accordingly
• Eff = Θ(n) → restricted subset

Pseudocode:
 public int[] sortArray(int[] A, int L, int U) {
int n = A.length;
int[] D = new int[U - L + 1];
int[] S = new int[n];
for (int j = 0; j <= U - L; j++) {// Initialize frequency array
D[j] = 0;
}
for (int i = 0; i < n; i++) {// Calculate frequencies
D[A[i] - L]++;
}
for (int j = 1; j <= U - L; j++) {// Calculate distribution
D[j] += D[j - 1];
}
for (int i = n - 1; i >= 0; i--) {// Sort elements into S
int j = A[i] - L;
S[D[j] - 1] = A[i];
D[j]--;
}
return S;
Horspool’s String Matching:
}
Idea:

• Start matching from the end of the pattern and on a mismatch → shift the pattern by more than a single space
• Shift table ‘T’:
o 1 entry per character ‘c’
o Gives no. of places to shift he pattern when text character is aligned with the last character of the
pattern.
o Max shift is ‘m’ → length of pattern
o Mismatch → retrieve T(c) where ‘c’ is char in text aligned to last char in pattern → shift patter right T(c)
positions

Example:

Overall alg:

1. Align left-most char of pattern to left-most char of text

2. Match pattern against corresponding text from right to left → backwards through the pattern
3. If: pattern mismatch → 4 cases
a. Compare last letter of pattern against corresponding letter in text
Case 1 → last letter not in pattern

Case 2 → last letter match but not in rest of pattern

Case 3 → last letter in Text is in the pattern but not at the end of the pattern

Case 4 → last letters match & letter is elsewhere in pattern

Example of Shift table:

E.g., E → find the rightmost index that isn’t at the very end

• So, for G the rightmost index that isn’t at the very end is 4.
• ‘m – 1’ → because ignore the last character
• If there’s a mismatch, T (c = character that caused the mismatch). Then, look in the table to see the
corresponding number that it needs to be shifted by.
• If the letter causing the mismatch is not in the pattern at all → shift the pattern by 8 (length of pattern) to the
right.

See examples below.

E.g., 3: Mismatch between ‘N’ and ‘G’ last characters. But T (N)
is in the pattern → So, go to the table and find the
corresponding value (= 1) → shift the pattern by 1 so that the
characters align.
Boyer-Moore
Idea:

Same 2 ideas as Horspool

• Compare pattern characters to text from right to left

• Given pattern, create a shift table that determines how much to shift the pattern

Except:

• Bad-symbol shift table

• Uses a good-suffix shift table with same idea applied to the no. of matched characters

Efficiency:

• Horspool’s approach is simpler & at least as efficient in the average case

Bad-Symbol shift:

• Bad character rule → rule for calculating shift

o Based off Horspool
o Look up the shift of the rightmost mismatched char NOT the end like Horspool
o See example below. The first or mismatched character from the end is ‘S’, so shift on the ‘S’

• Bad character rule:

o Let m = length of string
o Let T = shift table as in Horspool
o Let k = no. of matched chars from back
o Let c =. Mismatched char in text
o Bad shift = max (T (c) – k, 1)
o Like brute force → always shift at least 1 position
Example:

Tanser and Barber:

• E, R matched → k = 2.
• Mismatch at character (c) = ‘S’
• Check for shift value of ‘S’ in table → not there so T (S) = 6 (length of pattern Barber)

Aeroplane and Barber:

• E, R matched → k =2
• Mismatch at text character = ‘A’
• ‘A’ is in the shift table
o T (A) = 6 - 1 – 1
• Shift = max (T(A) – k, 1) → max (4 -2, 1) = 2
• Less shifts than with Horspool

Pre-processing
Before searching – know everything about the pattern

• Shift table is set up using info from the pattern

Both algorithms aim to extract useful info from the pattern in advance of the search → to maximize the size of the shift
they do each on each mismatch.

Point: Shift table is determined solely by the properties of the pattern, not the text.

Good Suffix Rule:

2nd Shift rule calculated by Boyer Moore

• Boyer Moore selects greater shift between good suffix and bad symbol rule
• Based on no. of characters that were successfully matched before had 1st failure

Shift table G (k) that depends on no. of matches

• Parameter ‘k’ is no. of previously successful matches

Case 1: matching suffix doesn’t occur elsewhere in the pattern

Case 2: matching suffix occurs somewhere else in the pattern

• Same mismatch letter recurs in pattern → shift entire pattern

• Diff mismatch letter in pattern → shift to align the matching
Case 3: part of the matching suffix occurs @ the beginning of the pattern

Boyer Moore Summary:

• Good-suffix shift → if ‘k’ symbols matched before failing, shift the pattern up G(k) positions
• Bad-symbol shift → if ‘k’ symbols matched before failing, shift the pattern up max (T (c)-1,1)
• Use whichever shift is larger between the 2.
• Pre-process problem instance → create an acceleration data structure
• Eff if acceleration structure is used repeatedly

DYNAMIC PROGRAMMING (8)

• Principle: Solve problems by storing optimal solutions to overlapping sub problems
-Each sub problem solved only once and stored for look up
-Thus falls into class of space-time tradeoff
• Solution must be derived from recurrence relation specification
• Must obey principle of optimality
-An overall optimal solution can be derived from optimal solutions to sub instances

Algorithms:

• Introductory algos:
-Fibonacci
-Change making
• Transitive closure: Warshalls algo
• All pairs shortest path: Floyd’s algo
• Knapsack problem
 Change making algo:

• Given a collection of coins, find exact change for amount n such that minimum number of coins is
used
-Each denomination has unlimited number
-d1 =R1
• For typical sub cases there is an efficient greedy algorithm
• Recurrence relation:
-Add one coin that accounts for the largest part of n for different denominations, solve for coins
required to make up remaining amount
-F(n) is minimum number of coins adding up to n
-F(0) = 0
-F(n) = min {F(n-dj)} +1 for n>0

Transitive closure:

• For a directed graph with n vertices

• Transitive closure is the n-by-n Boolean matrix T={tij} indicating reachability
-If there is a 1 in the ith row and jth column then there is a directed path from vertex I to vertex
j
-If there is a 0, there is no path

Warshall’s algorithm principle:

• Idea: if there’s a way to from A to B and also from B to C, then there is a way to go get from A to C
• A path exists between two vertices I, j, if there is:
-An edge from I to j
-Or a path from I to j going through vertex 1
 -Or a path from I to j going through vertex 1 and or 2
-Or a path going from I to j going through vertex 1,2 and or 3
-etc etc
-A path going from I to j through any of the other vertices
• Successively build corresponding transition matrices R(0), R(1)… R(n) = t
• In the kth stage, determine if a path exists between two vertices I, j using just vertices among 1,…, K

Warshall’s algorithm code:

 Greedy Techniques:
• Repeatedly tries to maximize return based on local conditions (with hope that outcome = global problem
solution)
-e.g. Dijkstra single source shortest path, Minimum spanning tree construction, Change Making, Huffman coding
• Optimization problems solved through sequence of choices that are:
-Feasible = Satisfy problem constraints
-Locally optimal = best choice among all feasible options for that step
-Irrevocable = no backing out
• Greedy grab of the best alternative, sometimes approximation is acceptable
• Not all optimizations can be solved this way

Change Making Revisited:

• Problem = give change for a specific amount n, with the least number of coins of the denominations
• E.g. Smallest change for R2.54 using R5, R2, R1, 50c, 20c, 10c, 5c, 2c, 1c
-Fewest coins = R2+50c+2c+2c
• Algo:
-At any step choose the coin of the largest denomination that doesn’t exceed the remaining total, repeat
-Optimal for reasonable sets of coins

Text Encoding:

• Assign bit patterns to an alphabet of characters so that in general texts, fewer bits are used overall (this is called
entropy coding)
• Use a fixed length: -Same
number of bits for each character (e.g. ASCII)
• Use a variable length: -
Number of bits for a character varies according to probability of
occurrence -
Leads to better compression (e.g. morse code encoding uses this as
pictured on the right)

Huffman Trees:

• How to determine which bits belong to which character in a variable length

encoding
• Solved by prefix-free code where no code is the prefix of another character
• Huffman Trees:
-Implement prefix-free code
-Leaves are characters, left edges encode 0 bits, right edges encode 1-bits
-Walk the tree to encode and decode
-Example encoding: ABADC -> 0100011101
-Example decoding: 11000101 -> DAAAC

Huffman Coding algo:

1. Initialize n one-node trees labeled with the characters of the alphabet

2. Record the frequency/weight of the character in the root
3. Repeat until a single tree is obtained:
-Find two trees with the smallest weight
-Make them the left and the right sub-tree of a new tree and record their sum weights in the root
Huffman coding notes:

• Compression ratio(CR):
-Standard measure of compression
-CR = 100 * (y-x) / y where x is compressed and y is uncompressed
-Typically 20-80% in the case of Huffman coding
• Yields optimal compression if:
-Probabilities of character occurrence are independent and are known in advance and in powers of 2
-Not great with bit strings (alphabet = {0,1}
• More sophisticated approaches = adaptive Huffman, which builds frequency tables on the fly

Strengths:

• Intuitively simple and appealing

Weaknesses:

• Only applicable to optimization problems

• Doesn’t always produce optimal solution
• More sophisticated alternatives work better (e.g. arithmetic coding)