Analysis & Design of Algorithms Module-5

MODULE-5
LIMITATIONS OF ALGORITHMIC POWER
DECISION TREES:
Many important algorithms, especially those for sorting and searching, work by comparing
items of their inputs. We can study the performance of such algorithms with a device called a
decision tree. As an example, Figure 11.1 presents a decision tree of an algorithm for finding a
minimum of three numbers. Each internal node of a binary decision tree represents a key
comparison indicated in the node, e.g., k < kr. The node’s left subtree contains the
information about subsequent comparisons made if k < kr, and its right subtree does the
same for the case of k > kr. (For the sake of simplicity, we assume throughout this section that all
input items are distinct.) Each leaf represents a possible outcome of the algorithm’s run on
some input of size n. Note that the number of leaves can be greater than the number of
outcomes because, for some algorithms, the same outcome can be arrived at through a
different chain of comparisons.

Mrs.Ayisha Khanum Page |1

Dept. Of CSD, Pesitm
Analysis & Design of Algorithms Module-5

Most sorting algorithms are comparison based, i.e., they work by comparing elements in a list to be
sorted. By studying properties of decision trees for such algorithms, we can derive important lower
bounds on their time efficiencies.
We can interpret an outcome of a sorting algorithm as finding a permutation of the element indices
of an input list that puts the list’s elements in ascending order.

Mrs.Ayisha Khanum Page |2

Dept. Of CSD, Pesitm
Analysis & Design of Algorithms Module-5

In this section, we shall see how decision trees can be used for establishing lower bounds on the number of
key comparisons in searching a sorted array of n keys: A[0] < A[1] < . .. < A[n 1]. The principal algorithm
for this problem is binary search. As we saw in Section 4.4, the number of comparisons made by binary
—
search in the worst case, Cbs worst (n), is given by the formula

Mrs.Ayisha Khanum Page |3

Dept. Of CSD, Pesitm
Analysis & Design of Algorithms Module-5

Mrs.Ayisha Khanum Page |4

Dept. Of CSD, Pesitm
Analysis & Design of Algorithms Module-5

P, NP, AND NP-COMPLETE PROBLEMS:

NP stands for Non-deterministic Polynomial time.

Definition: P is a set of all decision problems solvable by a deterministic algorithm in polynomial

time.

Definition: NP is the set of all decision problems solvable by a nondeterministic algorithm in

polynomial time. This also implies P ⊆ NP

Problems known to be in P are trivially in NP — the nondeterministic machine just never troubles
itself to fork another process, and acts just like a deterministic one. One example of a problem not
in P but in NP is Integer Factorization.

But there are some problems which are known to be in NP but don’t know if they’re in P. The
traditional example is the decision-problem version of the Travelling Salesman Problem
(decision-TSP). It’s not known whether decision-TSP is in P: there’s no known poly-time solution,
but there’s no proof such a solution doesn’t exist.

There are problems that are known to be neither in P nor NP; a simple example is to enumerate all
the bit vectors of length n. No matter what, that takes 2n steps.

Now, one more concept: given decision problems P and Q, if an algorithm can transform a
solution for P into a solution for Q in polynomial time, it’s said that Q is poly-time reducible (or
just reducible) to P.

The most famous unsolved problem in computer science is “whether P=NP or P≠NP? ”

Figure: Commonly believed relationship

between P and NP

Figure: Commonly believed relationship between P, NP, NP- Complete and NP-hard problems

Definition: A decision problem D is said to be NP-complete if:

1. it belongs to class NP
2. every problem in NP is polynomially reducible to D

Mrs.Ayisha Khanum Page |5

Dept. Of CSD, Pesitm
Analysis & Design of Algorithms Module-5

The fact that closely related decision problems are polynomially reducible to each other is not
very surprising. For example, Hamiltonian circuit problem is polynomially reducible to the decision
version of the traveling salesman problem.

NP-Complete problems have the property that it can be solved in polynomial time if all other NP-
Complete problems can be solved in polynomial time. i.e if anyone ever finds a poly-time solution
to one NP-complete problem, they’ve automatically got one for all the NP-complete problems;
that will also mean that P=NP.
Example for NP-complete is CNF-satisfiability problem. The CNF-satisfiability problem deals with
boolean expressions. This is given by Cook in 1971. The CNF-satisfiability problem asks whether or
not one can assign values true and false to variables of a given boolean expression in its CNF form
to make the entire expression true.
Over the years many problems in NP have been proved to be in P (like Primality Testing). Still,
there are many problems in NP not proved to be in P. i.e. the question still remains whether
P=NP? NP Complete Problems helps in solving this question. They are a subset of NP problems
with the property that all other NP problems can be reduced to any of them in polynomial time.
So, they are the hardest problems in NP, in terms of running time. If it can be showed that any NP-
Complete problem is in P, then all problems in NP will be in P (because of NP-Complete definition),
and hence P=NP=NPC.
NP Hard Problems - These problems need not have any bound on their running time. If any NP-
Complete Problem is polynomial time reducible to a problem X, that problem X belongs to NP-
Hard class. Hence, all NP-Complete problems are also NP-Hard. In other words if a NP-Hard
problem is non-deterministic polynomial time solvable, it is a NP- Complete problem. Example of a
NP problem that is not NPC is Halting Problem.
If a NP-Hard problem can be solved in polynomial time then all NP-Complete can be solved in
polynomial time.
“All NP-Complete problems are NP-Hard but not all NP-Hard problems are not NP- Complete.”NP-
Complete problems are subclass of NP-Hard
The more conventional optimization version of Traveling Salesman Problem for finding the
shortest route is NP-hard, not strictly NP-complete.

COPING WITH LIMITATIONS OF ALGORITHMIC POWER:

BACKTRACKING (N-QUEENS PROBLEM, SUBSET-SUM PROBLEM):
BACKTRACKING:

Throughout the book (see in particular Sections 3.4 and 11.3), we have encoun- tered problems that require
finding an element with a special property in a domain that grows exponentially fast (or faster) with the
size of the problem’s input: a Hamiltonian circuit among all permutations of a graph’s vertices, the most
valu- able subset of items for an instance of the knapsack problem, and the like. We addressed in Section
11.3 the reasons for believing that many such problems might not be solvable in polynomial time. Also
recall that we discussed in Section 3.4 how such problems can be solved, at least in principle, by exhaustive
search. The exhaustive-search technique suggests generating all candidate solutions and then identifying
the one (or the ones) with a desired property.
Backtracking is a more intelligent variation of this approach. The principal idea is to construct solutions

Mrs.Ayisha Khanum Page |6

Dept. Of CSD, Pesitm
Analysis & Design of Algorithms Module-5

one component at a time and evaluate such partially constructed candidates as follows. If a partially
constructed solution can be de- veloped further without violating the problem’s constraints, it is done
by taking the first remaining legitimate option for the next component. If there is no legiti- mate option
for the next component, no alternatives for any remaining component need to be considered. In this case, the
algorithm backtracks to replace the last component of the partially constructed solution with its next
option.
N-QUEENS PROBLEM :
The problem is to place n queens on an n × n chessboard so that no two queens attack each other
by being in the same row or in the same column or on the same diagonal.
So let us consider the four-queens problem and solve it by the backtracking technique. Since
each of the four queens has to be placed in its own row, all we need to do is to assign a column for
each queen on the board presented in figure.

We start with the empty board and then place queen 1 in the first possible position of its row,
which is in column 1 of row 1. Then we place queen 2, after trying unsuccessfully columns 1 and 2,
in the first acceptable position for it, which is square (2, 3), the square in row 2 and column 3. This
proves to be a dead end because there is no acceptable position for queen 3. So, the algorithm
backtracks and puts queen 2 in the next possible position at (2, 4). Then queen 3 is placed at (3,
2), which proves to be another dead end. The algorithm then backtracks all the way to queen 1
and moves it to (1, 2). Queen 2 then goes to (2, 4), queen 3 to(3, 1), and queen 4 to (4, 3), which is
a solution to the problem. The state-space tree of this search is shown in figure.

Mrs.Ayisha Khanum Page |7

Dept. Of CSD, Pesitm
Analysis & Design of Algorithms Module-5

Figure: State-space tree of solving the four-queens problem by backtracking. ×

denotes an unsuccessful attempt to place a queen in the indicated column. The
numbers above the nodes indicate the order in which the nodes are generated.

If other solutions need to be found, the algorithm can simply resume its operations at the leaf at
which it stopped. Alternatively, we can use the board’s symmetry for this purpose.
Finally, it should be pointed out that a single solution to the n-queens problem for any n ≥ 4 can
be found in linear time.

Note: The algorithm NQueens() is not in the syllabus. It is given here for interested learners.

The algorithm is referred from textbook T2.

Mrs.Ayisha Khanum Page |8

Dept. Of CSD, Pesitm
Analysis & Design of Algorithms Module-5

SUM OF SUBSETS PROBLEM:

Problem definition: Find a subset of a given set A = {a1, . . . , an} of n positive integers whose sum is
equal to a given positive integer d.
For example, for A = {1, 2, 5, 6, 8} and d = 9, there are two solutions: {1, 2, 6} and {1, 8}. Of course,
some instances of this problem may have no solutions.
It is convenient to sort the set’s elements in increasing order. So, we will assume that

a1< a2< . . . < an.

The state-space tree can be constructed as a binary tree like that in Figure shown below for the
instance A = {3, 5, 6, 7} and d = 15.
The number inside a node is the sum of the elements already included in the subsets represented
by the node. The inequality below a leaf indicates the reason for its termination.

Mrs.Ayisha Khanum Page |9

Dept. Of CSD, Pesitm
Analysis & Design of Algorithms Module-5

The root of the tree represents the starting point, with no decisions about the given elements
made as yet. Its left and right children represent, respectively, inclusion and exclusion of a1 in a set
being sought.
Similarly, going to the left from a node of the first level corresponds to inclusion of a 2 while going
to the right corresponds to its exclusion, and so on. Thus, a path from the root to a node on the ith
level of the tree indicates which of the first in numbers have been included in the subsets
represented by that node.
We record the value of s, the sum of these numbers, in the node. If s is equal to d, we have a
solution to the problem. We can either report this result and stop or, if all the solutions need to be
found, continue by backtracking to the node’s parent. If s is not equal to d, we can terminate the
node as non-promising if either of the following two inequalities holds:

Example: Apply backtracking to solve the following instance of the subset sum problem: A

= {1, 3, 4, 5} and d = 11.

BRANCH-AND-BOUND (KNAPSACK PROBLEM):

Recall that the central idea of backtracking, discussed in the previous section, is tocut off a branch
of the problem’s state-space tree as soon as we can deduce that it cannot lead to a solution. This
idea can be strengthened further if we deal with an optimization problem.
An optimization problem seeks to minimize or maximize some objective function (a tour length,
the value of items selected, the cost of an assignment, and the like), usually subject to some
constraints. An optimal solution is a feasible solution with the best value of the objective function
(e.g., the shortest Hamiltonian circuit or the most valuable subset of items that fit the knapsack).
Compared to backtracking, branch-and-bound requires two additional items:

Prof. Anupriya A G P a g e | 10
Dept. Of CSE, CITNC
Analysis & Design of Algorithms Module-5

1. a way to provide, for every node of a state-space tree, a bound on the best value of the
objective function on any solution that can be obtained by adding further components to
the partially constructed solution represented by the node

2. the value of the best solution seen so far

In general, we terminate a search path at the current node in a state-space tree of a branch- and-
bound algorithm for any one of the following three reasons:
1. The value of the node’s bound is not better than the value of the best solution seen so far.
2. The node represents no feasible solutions because the constraints of the problem are
already violated.
3. The subset of feasible solutions represented by the node consists of a single point (and
hence no further choices can be made)—in this case, we compare the value of the objective
function for this feasible solution with that of the best solution seen so far and update the
latter with the former if the new solution is better.
0/1 KNAPSACK PROBLEM
Note: For this topic as per the syllabus both textbooks T1 & T2 are suggested. Here we
discuss the concepts from T1 first and then that of from T2.
Topic form T1 (Levitin)
Let us now discuss how we can apply the branch-and-bound technique to solving the knapsack
problem. Given n items of known weights wi and values vi, i = 1, 2, . . . , n, and a knapsack of
capacity W, find the most valuable subset of the items that fit in the knapsack.
𝑤𝑖𝑥𝑖 𝑊 𝑎𝑛𝑑 𝑝𝑖𝑥 𝑠 𝑚𝑎𝑥𝑖𝑚𝑖𝑧𝑒𝑑, 𝑤ℎ𝑒𝑟𝑒 𝑥𝑖 = 0 𝑜𝑟 1
𝑛 𝑛
It is convenient to order the items of a given instance in descending order by their value-to-
weight ratios.

Each node on the ith level of state space tree, 0 ≤ i ≤ n, represents all the subsets of n items that
include a particular selection made from the first i ordered items. This particular selection is
uniquely determined by the path from the root to the node: a branch going to the left indicates
the inclusion of the next item, and a branch going to the right indicates its exclusion.
We record the total weight w and the total value v of this selection in the node, along with some
upper bound ub on the value of any subset that can be obtained by adding zero or more items to
this selection. A simple way to compute the upper bound ub is to add to v, the total value of the
items already selected, the product of the remaining capacity of the knapsack W – w and the best
per unit payoff among the remaining items, which is vi+1/wi+1:

ub = v + (W − w)(vi+1/wi+1).
Example: Consider the following problem. The items are already ordered in descending order of
their value-to-weight ratios.

Prof. Anupriya A G P a g e | 11
Dept. Of CSE, CITNC
Analysis & Design of Algorithms Module-5

Let us apply the branch-and-bound algorithm. At the root of the state-space tree (see Figure 12.8),
no items have been selected as yet. Hence, both the total weight of the items already selected w
and their total value v are equal to 0. The value of the upper bound is 100.
Node 1, the left child of the root, represents the subsets that include item 1. The total weight and
value of the items already included are 4 and 40, respectively; the value of the upper bound is 40
+ (10 − 4) * 6 = 76.

Node 2 represents the subsets that do not include item 1. Accordingly, w = 0, v = 0, and ub = 0 +
(10 − 0) * 6 = 60. Since node 1 has a larger upper bound than the upper bound of node 2, it is
more promising for this maximization problem, and we branch from node 1 first. Its children—
nodes 3 and 4—represent subsets with item 1 and with and without item 2, respectively. Since the
total weight w of every subset represented by node 3 exceeds the knapsack’s capacity, node 3 can
be terminated immediately.
Node 4 has the same values of w and v as its parent; the upper bound ub is equal to 40 + (10
− 4) * 5 = 70. Selecting node 4 over node 2 for the next branching (Due to better ub), we get nodes
5 and 6 by respectively including and excluding item 3. The total weights and values as well as the
upper bounds for these nodes are computed in the same way as for the preceding nodes.
Branching from node 5 yields node 7, which represents no feasible solutions, and node 8, which
represents just a single subset {1, 3} of value 65. The remaining live nodes 2 and 6 have smaller
upper-bound values than the value of the solution represented by node 8. Hence, both can be
terminated making the subset {1, 3} of node 8 the optimal solution to the problem.

Prof. Anupriya A G P a g e | 12
Dept. Of CSE, CITNC
Analysis & Design of Algorithms Module-5

Solving the knapsack problem by a branch-and-bound algorithm has a rather unusual

characteristic. Typically, internal nodes of a state-space tree do not define a point of the problem’s
search space, because some of the solution’s components remain undefined. (See, for example,
the branch-and-bound tree for the assignment problem discussed in the preceding subsection.)
For the knapsack problem, however, every node of the tree represents a subset of the items given.
We can use this fact to update the information about the best subset seen so far after generating
each new node in the tree. If we had done this for the instance investigated above, we could have
terminated nodes 2 and 6 before node 8 was generated because they both are inferior to the
subset of value 65 of node 5.

Concepts form textbook T2 (Horowitz)

Let us understand some of the terminologies used in backtracking &branch and bound.
→ Live node - a node which has been generated and all of whose children are not yet been
generated.
→ E-node - is a live node whose children are currently being explored. In other words, an E- node
is a node currently being expanded.
→ Dead node - a node that is either not to be expanded further, or for which all of its children
have been generated
→ Bounding Function - will be used to kill live nodes without generating all their children.

Prof. Anupriya A G P a g e | 13
Dept. Of CSE, CITNC
Analysis & Design of Algorithms Module-5

→ Backtracking - is depth first node generation with bounding functions.

→ Branch-And-Bound is a method in which E-node remains E-node until it is dead.

→ Breadth-First-Search: Branch-and Bound with each new node placed in a queue. The front of
the queen becomes the new E-node.
→ Depth-Search (D-Search): New nodes are placed in to a stack. The last node added is the first
to be explored.

0/1 Knapsack problem - Branch and Bound based solution

As the technique discussed here is applicable for minimization problems, let us convert the
knapsack problem (maximizing the profit) into minimization problem by negating the objective
function

Prof. Anupriya A G P a g e | 14
Dept. Of CSE, CITNC
Analysis & Design of Algorithms Module-5

3.1 LC (Least Cost) Branch and Bound solution

Prof. Anupriya A G P a g e | 15
Dept. Of CSE, CITNC
Analysis & Design of Algorithms Module-5

Prof. Anupriya A G P a g e | 16
Dept. Of CSE, CITNC
Analysis & Design of Algorithms Module-5

Prof. Anupriya A G P a g e | 17
Dept. Of CSE, CITNC
Analysis & Design of Algorithms Module-5

Prof. Anupriya A G P a g e | 18
Dept. Of CSE, CITNC
Analysis & Design of Algorithms Module-5

APPROXIMATION ALGORITHMS FOR NP-HARD PROBLEMS (KNAPSACK

PROBLEM):

In this section, we discuss a different approach to handling difficult problems of combinatorial

optimization, such as the traveling salesman problem and the knapsack problem. As we pointed out in
Section 11.3, the decision versions of these problems are NP-complete. Their optimization versions fall
in the class of NP-hard problems—problems that are at least as hard as NP-complete problems.2 Hence, there
are no known polynomial-time algorithms for these problems, and there are serious theoretical reasons
to believe that such algorithms do not exist. What then are our options for handling such problems,
many of which are of significant practical importance?
DEFINITION A polynomial-time approximation algorithm is said to be a c- approximation
algorithm, where c 1, if the accuracy ratio of t≥
he approximation it produces does not exceed c for any
instance of the problem in question:
r(sa) ≤ c. (12.3)

The best (i.e., the smallest) value of c for which inequality (12.3) holds for all instances of the problem
is called the performance ratio of the algorithm and denoted RA.
APPROXIMATION ALGORITHMS FOR THE KNAPSACK PROBLEM:
The knapsack problem, another well-known NP-hard problem, was also intro- duced in Section
3.4: given n items of known weights w1, . . . , wn and values v1, . . . , vn and a knapsack of
weight capacity W, find the most valuable sub- set of the items that fits into the knapsack.
Greedy Algorithms for the Knapsack Problem We can think of several greedy approaches to this
problem. One is to select the items in decreasing order of their weights; however, heavier items may
not be the most valuable in the set. Alternatively, if we pick up the items in decreasing order of their
value, there is no guarantee that the knapsack’s capacity will be used efficiently. Can we find a greedy
strategy that takes into account both the weights and values? Yes, we can, by computing the value-to-
weight ratios vi/wi, i 1, 2 , . . . , n, and selecting the items in decreasing order of these ratios. (In fact, we
already used this approach in designing the branch-and-bound = algorithm for the problem in Section 12.2.)
Here is the algorithm based on this greedy heuristic.

Prof. Anupriya A G P a g e | 19
Dept. Of CSE, CITNC
Analysis & Design of Algorithms Module-5

Prof. Anupriya A G P a g e | 20
Dept. Of CSE, CITNC