Introduction to Algorithms

Advanced Algorithms
MSCS / MSIT - Fall 2021
Department of CS & IT, University of
Sargodha

1
Agenda
• What is Solution?
• Algorithms vs Heuristics
• Optimization Problems
• What is Time Complexity?
• About the Course

Advanced Algorithms – Fall 2017 2

Solution
• What is a solution?

• What is feasible Solution?

• A solution of the optimization problem
that satisfy the problem constraints.

• Let suppose the problem is MSCS degree.

What are the feasible solutions in this case?

Advanced Algorithms – Fall 2017 3

What is Solution
•  Sphere Function

• subject to

• What is solution space?

• Set of all possible solutions of
an optimization problem that satisfy
the problem constraints.
Advanced Algorithms – Fall 2017 4
Good Solution vs Optimal Solution
• An optimal solution is a feasible solution where the objective function
reaches its maximum (or minimum) value – for example, the most
profit or the least cost.
• A globally optimal solution is one where there are no other feasible
solutions with better objective function values. 
• A locally optimal solution is one where there are
no other feasible solutions “in the vicinity”.
• Good Solution: When procedure stop after certain
amount of time with best solution found so far.

Advanced Algorithms – Fall 2017 5

What is an Algorithm?
• Finite Sequence of Instructions
• An input should not take the program in an infinite loop

• Each instruction having a clear meaning

• Very subjective. What is clear to me may not be clear to you

• Each instruction requiring finite amount of effort

• Very subjective. Finite on a supercomputer or 486

• Each instruction requiring finite time to complete

• Very subjective. 1 min, 1 hour, 1 year. Or a lifetime

Advanced Algorithms – Fall 2017 6

What is a Heuristic?
• Heuristic is a technique that quickly produces good but not necessarily
optimal solution

• In search algorithms at each branching step, heuristic rank each branch

to decide which branch to follow.

• Heuristic is like a guess work.

Advanced Algorithms – Fall 2017 7

Heuristic vs Algorithm

• Heuristic: Good Solution in Reasonable Time

• Algorithm: Optimal Solution in Reasonable Time

Advanced Algorithms – Fall 2017 8

 3

Problem: Graph Coloring

1 5 2
• A graph G=(V,E) is a set of vertices V and edges E
4
• A coloring of a graph is an assignment of a color to each vertex of the
graph, so that no two vertices connected by an edge have the same
color.

• This should be colored with minimum (not minimal) number of

colors.

• What is Adjacency Matrix?

Advanced Algorithms – Fall 2017 9
Heuristic: “Greedy Algorithm”
• A reasonable heuristic for graph coloring is the greedy “algorithm”.
• Try to color as many vertices as possible with the first color, and then as many
uncolored vertices with the second color, and so on.

• Select some uncolored vertex, and color with new color.

• Scan the list of uncolored vertices.

• For each uncolored vertex, determine whether it has an edge to any vertex already
colored with the new color.

• If there is no such edge, color the present vertices with the new color.

Advanced Algorithms – Fall 2017 10

Heuristic: “Greedy Algorithm”
3
Three colors used
3
1 5 2

1 5 2
4
Two colors used
(optimum) 44

Advanced Algorithms – Fall 2017 11

What is a Meta-Heuristic?
• If you are given a candidate solution to your problem, you can test it and
access how good it is

• You have very little heuristic information.

• On simplest solution in such situation could be to try random search

• Before you give up, you must start with a random

solution and then make small random modification
Advanced Algorithms – Fall 2017 12
What are Approximate Algorithms?
•  Ideally we would like to have a guarantee on how close to optimal the
solution is.

• An algorithm producing a solution that is guaranteed to be within

some factor of the optimum.

• Approximate algorithms also give Good Solution

• Approximate algorithms are similar to Heuristic?

Advanced Algorithms – Fall 2017 13

Time: Why do we care?
• When faced with a problem in practice, need to know whether or not one can
expect to produce a “fast” algorithm to solve it

• “Intractable” problems occur frequently in practice

• Surface differences between tractable and intractable problems are often hard
to see

Advanced Algorithms – Fall 2017 14

Why do we Care?

Advanced Algorithms – Fall 2017 15

What is Intractable?
N 10 20
O(N2) 100 400
O(2N) 1,024 1,048,576
O(N!) 3,628,800 2,432,902,008,176,640,000
Assume 1 instruction takes 10-6 seconds
O(N2) < 1 sec < 1 sec
O(2N) < 1 sec  1 sec
O(N!) 3.6 sec 771.46 CENTURIES

Advanced Algorithms – Fall 2017 16

Size Does Matter
N log2N 5N N log2N N2 2N
8 3 40 24 64 256
16 4 80 64 256 65536
32 5 160 160 1024 ~109
64 6 320 384 4096 ~1019
128 7 640 896 16384 ~1038
254 8 1280 2048 65536 ~1076

If one execution takes 10-6 seconds, 2256 will take about

1061 centuries
Advanced Algorithms – Fall 2017 17
NP Completeness
• Some problems are intractable i.e. for large input, we are unable to solve
them in reasonable time

• What constitutes reasonable time? Standard working definition:

polynomial time

• For input size n the worst-case running time is O(nk) for some constant k

• Polynomial time: O(1), O(n), O(n lg n), O(n2), O(n3)

• Not in polynomial time: O(2n), O(nn), O(n!)

Advanced Algorithms – Fall 2017 18

NP Complete Problems
• The NP-Complete problems are an interesting class of problems whose
status is unknown
• No polynomial-time algorithm has been discovered for an NP-Complete
problem

• No supra-polynomial lower bound has been proved for any NP-Complete

problem, either

• We call this the P = NP question

• The biggest open problem in CS

Advanced Algorithms – Fall 2017 19

P and NP
• As mentioned, P is set of problems that can be solved in polynomial
time

• NP (nondeterministic polynomial time) is the set of problems that can

be solved in polynomial time by a nondeterministic computer

• Think of a non-deterministic computer as a computer that magically

“guesses” a solution, then has to verify that it is correct

Advanced Algorithms – Fall 2017 20

What is Non Deterministic Computer?
• If a solution exists, computer always guesses it. One way to imagine it: a
parallel computer that can freely spawn an infinite number of processes

• Have one processor work on each possible solution. All processors

attempt to verify that their solution works

• If a processor finds it has a working solution, it notifies

• So: NP = problems verifiable in polynomial time

Advanced Algorithms – Fall 2017 21

NP: Traveling Salesman Person
• TSPs (and variants) have enormous practical importance
• E.g., for shipping and freighting companies
• Lots of research into good approximation algorithms

1 5 Cost(12  3  4  5  6) = 17
5
1 3
Cost(13  2  6  5  4) = 13
3 3
4 4
2
5
Cost(14  5  6  2  3) = 12
3 2
5
2
1 Time complexity of naïve solution?
6

Advanced Algorithms – Fall 2017 22

NP: Crossing Minimization & Clustering

Crossings = 630 NO Clustering

Crossings = 172 03 Clusters

Advanced Algorithms – Fall 2017 23
NP: Hamiltonian Tour
• A graph has a Hamiltonian tour if there is a tour that visits every
vertex exactly once (and returns to its starting point).

• A graph with a Hamiltonian tour is called a Hamiltonian graph.

• Is there a Hamiltonian Tour in the graph?

Advanced Algorithms – Fall 2017 24

Knapsack Problem
• Given a knapsack with maximum capacity W, and a set S consisting of
n items
• Each item i has some weight wi and benefit value bi (all wi and W are
integer values)
• Problem: How to pack the knapsack to achieve maximum total value
of packed items?

Advanced Algorithms – Fall 2017 25

Clique/Complete Graph/Mesh Graph
• Let G=(V,E) be a graph, where V represents vertices and E represents
Edges.
• Finding subset of the vertices each adjacent to other is a clique.
• Maximum Clique problem is to find the largest size graph vertices that
are adjacent to each other.

Advanced Algorithms – Fall 2017 26

Maximum Matching
• A set of courses, a set of faculty and a “can teach course” relationship
between elements of different sets.
• Find a matching that keeps the greatest possible number of faculty and
courses “active”.
• This is called a maximum matching (as opposed to a maximal
matching which cannot become any larger but is not the best possible)

Advanced Algorithms – Fall 2017 27

Maximum Matching
Faculty Course Faculty Course

a maximum matching (in a maximal matching (in red)

red)

Advanced Algorithms – Fall 2017 28

 3 4

Vertex Cover Problem 1 2

5 6
• Model the layout of the museum as follows: vertices are rooms and an
edge between two vertices means that those two rooms are connected.

• Now, assuming that a guard standing in a room has full view of the
adjacent rooms, find where we should place guards so that all rooms
are guarded at the least cost (i.e. fewest number of guards).

• This is known as the Vertex Cover problem

Advanced Algorithms – Fall 2017 29

Bad Neighbors / Min Cut Problem
• Three neighbors who hate one another live in cabins in the woods.
• They must cut paths from each house to each of three utilities so that
no two paths cross. Can this be done?
• The answer is no.
• This is because the following graph that models the problem cannot be
drawn on the plane so that no edges cross. Such a graph is called “non-
planar”. Neighbors

Utilities

Advanced Algorithms – Fall 2017 30

Map Coloring / Graph Coloring
• You are given a map of the United States and are asked to color each state so
that adjacent states have different colors.

• Model the map as follows: Each vertex represents a state and there is an edge
between vertices if the corresponding states share a border.

• Then, the problem becomes: color the vertices of a planar graph so that
adjacent vertices have different colors.

• It has been shown that for a planar graph four colors suffice to color the graph.

Advanced Algorithms – Fall 2017 31

Which is better?
• The running time of a program.

• Program easy to understand?

• Program easy to code and debug?
• Program making efficient use of resources?
• Program running as fast as possible?

Advanced Algorithms – Fall 2017 32

Measuring Efficiency
• Ways of measuring efficiency:
• Run the program and see how long it takes
• Run the program and see how much memory it uses

• Lots of variables to control

• What is the input data?
• What is the hardware platform?
• What is the programming language/compiler?
• Just because one program is faster than another right now, means it will
always be faster?

Advanced Algorithms – Fall 2017 33

Measuring Efficiency
• Want to achieve platform-independence

• Use an abstract machine that uses steps of time and units

of memory, instead of seconds or bytes

• - each elementary operation takes 1 step

• - each elementary instance occupies 1 unit of memory

Advanced Algorithms – Fall 2017 34

A Simple Example
• // Input: int A[N], array of N integers
• // Output: Sum of all numbers in array A

• int Sum(int A[], int N) {

• int s=0;
• for (int i=0; i< N; i++)
• s = s + A[i];
• return s;
•}

Advanced Algorithms – Fall 2017 35

A Simple Example
• Describe the size of the input in terms of one ore more parameters:
• Input to Sum is an array of N ints, so size is N.

• Then, count how many steps are used for an input of that size:
• A step is an elementary operation such as
• +, <, =, A[i]

Advanced Algorithms – Fall 2017 36

Measuring Efficiency
// Input: int A[N], array of N integers
// Output: Sum of all numbers in array A

int Sum(int A[], int N {

int s=0; 1

for (int i=0; i< N; i++) 1,2,8: Once

2 3 4 3,4,5,6,7: Once per each iteration
s = s + A[i];
5 of for loop, N iteration
6 7 Total: 5N + 3
return s;
} The complexity function of the
8 algorithm is : f(N) = 5N +3

Advanced Algorithms – Fall 2017 37

How 5N+3 Grows
• Estimated running time for different values of N:

• N = 10 => 53 steps
• N = 100 => 503 steps
• N = 1,000 => 5003 steps
• N = 1,000,000 => 5,000,003 steps

• As N grows, the number of steps grow in linear proportion to N for this

Sum function.
Advanced Algorithms – Fall 2017 38
Asymptotic Complexity
• The 5N+3 time bound is said to "grow asymptotically" like N

• This gives us an approximation of the complexity of the

algorithm

• Ignores lots of (machine dependent) details, concentrate

on the bigger picture

Advanced Algorithms – Fall 2017 39

What Dominates?
• What about the 5 in 5N+3? What about the +3?
• As N gets large, the +3 becomes insignificant
• 5 is inaccurate, as different operations require varying
amounts of time
• What is fundamental is that the time is linear in N.
• Asymptotic Complexity: As N gets large, concentrate on the
highest order term:
• Drop lower order terms such as +3
• Drop the constant coefficient of the highest order term i.e. N

Advanced Algorithms – Fall 2017 40

Asymptotic Complexity
• The 5N+3 time bound is said to "grow asymptotically" like N

• This gives us an approximation of the complexity of the

algorithm

• Ignores lots of (machine dependent) details, concentrate

on the bigger picture

Advanced Algorithms – Fall 2017 41

Comparing Functions
• Definition: If f(N) and g(N) are two complexity functions, we say

f(N) = O(g(N))

• (read "f(N) as order g(N)", or "f(N) is big-O of g(N)")

• if there are constants c and N0 such that for N  N0, T(N)  cN
f(N) £ c g(N)
• for all sufficiently large N.

Advanced Algorithms – Fall 2017 42

100n2 Vs 5n3, which one is better?
250000

200000

150000

100000

50000

0
1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3
1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4
100n2 10 40 90 16 25 36 49 64 81 10 12 14 16 19 22 25 28 32 36 40 44 48 52 57 62 67 72 78 84 90 96 1E 1E 1E
5n3 5 40 13 32 62 10 17 25 36 50 66 86 10 13 16 20 24 29 34 40 46 53 60 69 78 87 98 1E 1E 1E 1E 2E 2E 2E

Advanced Algorithms – Fall 2017 43

100n2 Vs 5n3, which one is better?
Differenec of functions

20000
10000
0
-10000 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33
-20000
-30000
-40000
-50000
-60000
-70000
-80000
-90000

Advanced Algorithms – Fall 2017 44

Why is this useful?
• As inputs get larger, any algorithm of a smaller order will be more
efficient than an algorithm of a larger order

0.05 N2 = O(N2)
Time (steps)

3N = O(N)

Input (size)
N = 60

Advanced Algorithms – Fall 2017 45

Comparing Functions
• What is the relationship between the following?

What is the polynomial representation for the following?

f1(n)

f2(n)

f3(n)
F(n)

f4(n)

Input size (n)

Advanced Algorithms – Fall 2017 46

Big-O Notation
• Think of f(N) = O(g(N)) as
• " f(N) grows at most like g(N)" or
• " f grows no faster than g“
• (ignoring constant factors, and for large N)

• Important:
• Big-O is not a function!
• Never read = as "equals“
• Examples:
5N + 3 = O(N)
37N5 + 7N2 - 2N + 1 = O(N5)

Advanced Algorithms – Fall 2017 47

Big Numbers
• Suppose a program has run time O(n!) and the run time for n = 10 is 1
second

• For n = 12, the run time is 2 minutes

• For n = 14, the run time is 6 hours
• For n = 16, the run time is 2 months
• For n = 18, the run time is 50 years
• For n = 20, the run time is 200 centuries

Advanced Algorithms – Fall 2017 48

Constant time statements
• Simplest case: O(1) time statements
• Assignment statements of simple data types
int x = y;
• Arithmetic operations:
x = 5 * y + 4 - z;
• Array referencing:
A[j] = 5;
• Array assignment:
 j, A[j] = 5;
• Most conditional tests:
if (x < 12) ...

Advanced Algorithms – Fall 2017 49

Analyzing Loops
• Any loop has two parts:
• How many iterations are performed?
• How many steps per iteration?
int sum = 0,j;
for (j=0; j < N; j++)
sum = sum +j;

• Loop executes N times (0..N-1)

• 4 = O(1) steps per iteration
• Total time is N * O(1) = O(N*1) = O(N)

Advanced Algorithms – Fall 2017 50

Analyzing Loops
• What about this for-loop?
int sum =0, j;
for (j=0; j < 100; j++)
sum = sum +j;

• Loop executes 100 times

• 4 = O(1) steps per iteration
• Total time is 100 * O(1) = O(100 * 1) = O(100) = O(1)

Advanced Algorithms – Fall 2017 51

Nested Loops
int j,k;
for (j=0; j<N; j++)
for (k=N; k>0; k--)
sum += k+j;

• Start with outer loop:

• How many iterations? N
• How much time per iteration? Need to evaluate inner loop
• Inner loop uses O(N) time
• Total time is N * O(N) = O(N*N) = O(N2)

Advanced Algorithms – Fall 2017 52

Nested Loops
• What if the number of iterations of one loop depends on the counter of the other?

int j,k;
for (j=0; j < N; j++)
for (k=0; k < j; k++)
sum += k+j;

• Analyse inner and outer loop together:

• Number of iterations of the inner loop is:
• 0 + 1 + 2 + ... + (N-1) = O(N2) How?

Advanced Algorithms – Fall 2017 53

Course Outline
• Dynamic Programming
• Divide & Conquer
• Greedy Algorithm
• Genetic Algorithms
• Amortized Analysis
• Heaps
• Graph Algorithms
• Hashing

Advanced Algorithms – Fall 2017 54

Course Outline
• String Matching Algorithms
• Approximation Algorithms
• Streaming Algorithms
• Curse of Dimensionality
• Matrix Completion
• MapReduce

Advanced Algorithms – Fall 2017 55