Fundamentals of Algorithms

Unit 14

Unit 14

Complexity of Algorithms

Structure 14.1 Introduction Objectives 14.2 Notations for the Growth Rates of Functions 14.3 Classification of Problems 14.4 Reduction, NP-Complete and NP-Hard Problems 14.5 Establishing NP-Completeness of Problems 14.6 Summary 14.7 Terminal Questions 14.8 Answers

14.1 Introduction
In the previous units, we discussed a number of problems which cannot be solved by algorithmic means and also discussed a number of issues about such problems. In this unit, we will discuss the issue of efficiency of computation of an algorithm in terms of the amount of time used in its execution. On the basis of analysis of an algorithm, the amount of time that is estimated to be required in executing an algorithm will be referred to as the time complexity of the algorithm. The time complexity of an algorithm is measured in terms of some (basic) time unit (not second or nano-second). Generally, time taken in executing one move of a TM, is taken as (basic) time unit for the purpose. Or, alternatively, time taken in executing some elementary operation like addition, is taken as one unit. More complex operations like multiplication etc., are assumed to require an integral number of basic units. As mentioned earlier, given many algorithms (solutions) for solving a problem, we would like to choose the most efficient algorithm from amongst the available ones. For comparing efficiencies of algorithms, that solve a particular problem, time complexities of algorithms are considered as functions of the sizes of the problems (to be discussed). The time complexity functions of the algorithms are compared in terms of their growth rates (to be defined), as growth rates are considered important measures of comparative efficiencies.

Sikkim Manipal University

Page No. 250

Fundamentals of Algorithms

Unit 14

Objectives After studying this unit, you should be able to: explain the concepts of time complexity, size of a problem and growth rate of a function define and explain the well-known notations for growth rates of functions, viz., O, , , , explain criteria for classification of problems into undefinable, definable but not solvable, solvable but not feasible, P, NP, NP-hard and NP- Complete etc. define a number of problems which are known to be NP-Complete problems

14.2 Notations for the Growth Rates of Functions

The time required by a solution or an algorithm for solving a (solvable) problem, depends not only on the size of the problem/input and the number of operations that the algorithm/solution uses, but also on the hardware and software used to execute the solution. However, the effect of change/ improvement in hardware and software on the time required may be closely approximated by a constant. Suppose, a supercomputer executes instructions one million times faster than another computer. Then irrespective of the size of a (solvable) problem and the solution used to solve it, the supercomputer solves the problem roughly million times faster than the computer, if the same solution is used on both the machines to solve the problem. Thus, we conclude that the time requirement for execution of a solution, changes roughly by a constant factor on change in hardware, software and environmental factors. The Constant Factor in Complexity Measure An important consequence of the above discussion is that if the time taken by one machine in executing a solution of a problem is a polynomial (or exponential) function in the size of the problem, then time taken by every machine is a polynomial (or exponential) function respectively, in the size of the problem. Thus, functions differing from each other by constant factors, when treated as time complexities should not be treated as different, i.e., should be treated as complexity-wise equivalent.

Sikkim Manipal University

Page No. 251

Fundamentals of Algorithms

Unit 14

Asymptotic Considerations Computers are generally used to solve problems involving complex solutions. The complexity of solutions may be either because of the large number of involved computational steps and / or large size of input data. The plausibility of the claim apparently follows from the fact that, when required computers are used generally, not to find the product of two 2 x 2 matrices but to find the product of two n x n matrices for large n running into hundreds or even thousands. Similarly, computers, when required, are generally used not to find roots of quadratic equations but for finding roots of complex equations including polynomial equations of degrees more than hundreds or sometimes even thousands. The above discussion leads to the conclusion that when considering time complexities f1(n) and f2(n) of (computer) solutions of a problem of size n, we need to consider and compare the behaviours of the two functions only for large values of n. If the relative behaviours of two functions for smaller values conflict with the relative behaviours for larger values, then we may ignore the conflicting behaviour for smaller values. For example, if the earlier considered two functions f1(n) = 1000 n2 and 4 f2(n) = 5n represent time complexities of two solutions of a problem of size n, then despite the fact that f1(n) f2(n) for n 14,

we would still prefer the solution having f1(n) as time complexity because f1 (n) f2(n) for all n 15.

Self Assessment Question 1. Time requirement for execution of a solution, changes roughly by a constant factor on change in and.

14.3 Classification of Problems

The fact of being engaged in solving problems may be the only sure indication of a living entity being alive (though, the distinction between
Sikkim Manipal University Page No. 252

Fundamentals of Algorithms

Unit 14

entities being alive and not being alive is getting fuzzier day by day). The problems, attempted to be solved, may be due to the need for survival in a hostile and Competitive environment or may be because of intellectual curiosity of knowing more and more of the nature. In the previous unit, we studied a number of problems which are not solvable by computational means. We can go still further and categorize the problems, which we encounter or may encounter, into the following broad classes: i) Problems which cannot even be defined formally. By a formal definition of a problem, we mean expressing in terms of mathematical entities like sets, relations and functions etc., the information concerning the problem, in respect of at least a) Possible inputs b) Possible outcomes c) Entities occurring and operations on these entities in the (dynamic) problem domains. In this sense of definition of a problem, how to talk of solving, most of the problems can not even be defined. Think of the following problems. a) Why the economy is not doing well? b) Why there is hunger, illiteracy and suffering despite international efforts to eradicate these? c) Why some people indulge in corrupt practices despite being economically well? These are some of the problems, the definition of each of which require enumeration of potentially infinite parameters, and hence are almost impossible to define. ii) Problems which can be formally defined but can not be solved by computational means. We discussed some of these problems in the previous unit. iii) Problems which, though theoretically can be solved by computational means, yet are infeasible, i.e., these problems require so large amount of computational resources that practically is not feasible to solve these problems by computational means. These problems are called intractable or infeasible problems. The distinguishing feature of the problems is that for each of these problems any solution has time complexity which is exponential, or atleast non-polynomial, function of the problem size.
Sikkim Manipal University Page No. 253

Fundamentals of Algorithms

Unit 14

iv) Problems that are called feasible or theoretically not difficult to solve by computational means. The distinguishing feature of the problems is that for each instance of any of these problems, there exists a Deterministic Turing Machine that solves the problem having timecomplexity as a polynomial function of the size of the problem. The class of problem is denoted by P. v) Last, but probably most interesting class includes large number of problems, for each of which, it is not known whether it is in P or not in P. These problems fall somewhere between class Ill and class IV given above. However, for each of the problems in the class, it is known that it is in NP, i.e., each can be solved by at least one Non-Deterministic Turing Machine, the time complexity of which is a polynomial function of the size of the problem. A problem from the class NP can equivalently but in more intuitive way, be defined as one for which a potential solution, if given, can be verified in polynomial time whether the potential solution is actually a solution or not. The problems in this class are called NP-Complete problems (to be formally defined later). More explicitly, a problem is NP-complete if it is in NP and for which no polynomial-time Deterministic TM solution is known so far. Most interesting aspect of NP-complete problems, is that for each of these problems neither, so far, it has been possible to design a Deterministic polynomial-time TM solving the problem nor it has been possible to show that Deterministic polynomial-time TM solution can not exist. The idea of NP-completeness was introduced by Stephen Cook* in 1971 and the satisfiability problem defined below is the first problem that was proved to be NP-complete, of course, by S. Cook. Next, we enumerate some of the NP-complete problems without justifying why these problems have been placed in the class. Problem: Hamiltonian circuit problem (HCP) given an undirected graph G = (V, E), does G contain a Hamiltonian circuit? Further Explanation: A Hamiltonian circuit of a graph G = (V, E) is a set of edges that connects the nodes into a single cycle, with each node appearing
Sikkim Manipal University Page No. 254

Fundamentals of Algorithms

Unit 14

exactly once. We may note that the number of edges on a Hamiltonian circuit must equal the number of nodes in the graph. Further, it may be noted that HCP is a special case of TSP in which the cost between pairs of nodes is the same, say, 1. Example: Consider the graph

Fig. 14.1

Then, the above graph has one Hamiltonian circuit viz., (1, 2, 4, 3, 1) Problem: The vertex cover problem (VCP) (also known as Node cover problem). Given a graph G = (V, E) and an integer k, is there a vertex cover for G with k vertices? Explanation: A vertex cover for a graph G is a set C of vertices so that each edge of G has an endpoint in G. For example, for the graph shown above, {1, 2, 3} is a vertex cover. It can be easily seen that every superset of a vertex cover of a graph is also a vertex cover of the graph. Problem: K-Colourability Problem: Given a graph G and a positive integer k, is there a k-colouring of G? Explanation: A k-colouring of G is an assignment to each vertex of one of the k colours so that no two adjacent vertices have the same color. It may be recalled that two vertices in a graph are adjacent if there is an edge between the two vertices.

Sikkim Manipal University

Page No. 255

Fundamentals of Algorithms

Unit 14

Fig. 14.2

As the vertices 1, Z 3 are mutually adjacent therefore, we require atleast three colours for k-colouring problem. Problem: The complete sub-graph problem (CSP Complete Sub) or clique Problem: Given a graph G and positive integer k, does G have a complete sub-graph with k vertices? Explanation: For a given graph G = (V, E), two vertices v1 and v2 are said to be adjacent if there is an edge connecting the two vertices in the graph. A sub-graph H = (V1, E1,) of a graph G = (V, E) is a graph such that V1 V and E1 E. In other words, each vertex of the sub-graph is a vertex of the graph and each edge of the sub-graph is an edge of the graph. Complete Sub-graph of a given graph G is a sub-graph in which every pair of vertices is adjacent in the graph. For example, in the above graph, the sub-graph containing the vertices {1, 2, 3} and the edges (1, 2), (1, 3), (2, 3) is a complete sub-graph or a clique of the graph. However, the whole graph is not a clique as there is no edge between vertices 1 and 4. Problem: Independent set problem: Given a graph G = (V, E) and a positive integer k, is there an independent set of vertices with atleast k elements? Explanation: A subset V1 of the set of vertices V of graph G is said to be independent, if no two distinct vertices in V1 are adjacent. For example, in the above graph V1 = {1, 4} is an independent set. Problem: The sub-graph isomorphism problem: Given graph G1 and G2, does G1 contain a copy of G2 as a subgraph?
Sikkim Manipal University Page No. 256

Fundamentals of Algorithms

Unit 14

Explanation: Two graphs H1 = (V1, E1) and H2 = (V2, E2) are said to be isomorphic if we can rename the vertices in V2 in such a manner that after renaming, the graph H1 and H2 look identical (not necessarily pictorially, but as ordered pairs of sets)

Fig. 14.3

are isomorphic graphs because after mapping 1 4 d, the two graphs become identical.

a, 2

b, 3

c and

Problem: Given a graph g and a positive integer k, does G have an edge cover of k edges? Explanation: For a given graph G = (V, E), a subset E1 of the set of edges E of the graph, is said to be an edge cover of G, if every vertex is an end of at least one of the edges in E1. Example: For the graph

Fig. 14.4

The two-edge set {(1, 4), (2, 3)} is an edge cover for the graph.
Sikkim Manipal University Page No. 257

Fundamentals of Algorithms

Unit 14

Example: Let S = {1, 2,, 10} and P = {S1, S2, S3, S4, S5} such that S1 = {1, 3, 5} S2 = {2, 4, 6} S3 = {1, 2, 3, 4} S4 = {5, 6, 7, 9, 10} S5 = {7, 12, 9, 10} Then Q = {S1, S2, S5} is a set cover for S. Self Assessment Questions 2. Given a graph G and a positive integer k, is there a of G. 3. Complete Sub-graph of a given graph G is a sub-graph in which every pair of vertices is in the graph.

14.4 Reduction, NP-Complete and NP-Hard Problems

We know that a problem is called NP-Complete if P has atleast one NonDeterministic Polynomial-time solution. In this section we formally define the concept and then describe a general technique of establishing the NP-Completeness of problems and finally apply the technique to show some of the problems as NP-complete. For the formal definition of NP-completeness, polynomial-time reduction, as defined below, plays a very important role. We have already discussed in previous units the reduction technique to establish some of the problems as undecidable. The method that was used for establishing undecidability of a language using the technique of reduction, may be briefly described as follows: Let P1 be a problem which is already known to be undecidable. We want to check whether a problem P2 is undecidable or not. If we are able to design an algorithm which transforms or constructs an instance of P2 for each instance of P1, then P2 is also undecidable. The process of transformation of the instances of the problem already known to the undecidable to instances of the problem, (the undecidability is to be checked), is called reduction.

Sikkim Manipal University

Page No. 258

Fundamentals of Algorithms

Unit 14

Some-what similar, but, slightly different, rather special, reduction called polynomial-time reduction is used to establish NP-Completeness of problems. A Polynomial-time reduction is a polynomial-time algorithm which constructs the instances of a problem P2 from the instances of some other problems P1. A method of establishing the NP-Completeness (to be formally defined later) of a problem P2 constitutes of designing a Polynomial time reduction that constructs an instance of P2 for each instance of P1, where P1 is already known to be NP-Complete. The direction of the mapping must be clearly understood as shown below.
P1 Polynomial time Reduction P2

(Problem already known to be undecidable)

(Problem whose NPcompleteness is to be established)

Self Assessment Questions 4. is a polynomial-time algorithm which constructs the instances of a problem P2 from the instances of some other problems P1. 5. A problem L is said to be if for any problem L1 in NP, there is a polynomial-time reduction of L1 to L.

14.5 Establishing NP-Completeness of Problems

In general, the process of establishing a problem as NP-Complete is a two step process. The first step, which in most of the cases is quite simple, constitutes of guessing possible solutions of the instances, one instance at a time, of the problem and then verifying whether the guess actually is a solution or not. The second step involves designing a polynomial-time algorithm which reduces instances of an already known NP-Complete problem to instances of the problem, which is intended to be shown as NP-Complete.
Sikkim Manipal University Page No. 259

Fundamentals of Algorithms

Unit 14

However, to begin with, there is a major hurdle in execution of the second step. The above technique of reduction can not be applied unless we already have established atleast one problem as NP-Complete. Therefore, for the first NP-Complete problem, the NP-Completeness has to be established in a different manner. As mentioned earlier, Stephen Cook (1971) established Satisfiability as the first NP-Complete problem. The proof was based on explicit reduction of the language of any non-deterministic, polynomial-time TM to the satisfiability problem. The proof of satisfiability problem as the first NP-Complete problem, is quite lengthy and we skip the proof. Interested readers may consult any of the texts given in the reference. Assuming the satisfiability problem as NP-Complete, the rest of the problems that we establish as NP-Complete, are established by reduction method as explained above. A diagrammatic notation of the form.

Fig. 14.5

The block diagram in 14.5 indicates that , assuming P as already established to be NP-Complete, the NP- Completeness of Q is established by a polynomial-time reduction from P to Q. A scheme for establishing NP-Completeness of some the problems is suggested by means of Figure 14.6 given below:

Sikkim Manipal University

Page No. 260

Fundamentals of Algorithms

Unit 14

Fig. 14.6: A scheme for establishing NP-Completeness of some the problems

Example: Show that the Clique problem is an NP-Complete problem. Proof: The verification of whether every pairs of vertices is connected by an edge in E, is done for different pairs of vertices by a Non-deterministic TM, i.e., in parallel. Hence, it takes only polynomial time because for each of n vertices we need to verify atmost n (n + 1) /2 edges, the maximum number of edges in a graph with n vertices. Next, we show that 3-CNF-SAT problem can be transformed to clique problem in polynomial time. Take an instance of 3-CNF-SAT. An instance of 3CNF-SAT consists of a set of n clauses, each consisting of exactly 3 literal, each being either a variable or negated variable. It is satisfiable if we can choose literals in such a way that: Atleast one literal from each clause is chosen If literal of form x is chosen, no literal of form x is considered.

Sikkim Manipal University

Page No. 261

Fundamentals of Algorithms

Unit 14

Fig. 14.7: Graph

For each of the literals, create a graph node, and connect each node to every node in other clauses, except those with the same variable but different sign. Example: Show that the Vertex cover problem is an NP-Complete. A vertex cover of an undirected graph G = (V, E) is a subset V of the vertices of the graph which contains atleast one of the two endpoints of each edge.

Fig. 14.8: Graph showing vertex cover problem

Sikkim Manipal University

Page No. 262

Fundamentals of Algorithms

Unit 14

The vertex cover problem is the optimization problem of finding a vertex cover of minimum size in a graph. The problem can also be stated decision problem: VERTEX-COV = {<G, k> /graph G has a vertex cover of size k}. A deterministic algorithm to find a vertex cover in a graph is to list all subsets of vertices of size k and check each one to see whether it forms a vertex cover. This algorithm is exponential in k. Proof: To show that Vertex cover problem NP, for a given graph G= (V, E), we take V V and verifies to see if it forms a vertex cover. Verification can be done by checking for each edge (u, v) E whether u V or v V. This verification can be done in polynomial time. Now, we show that clique problem can be transformed to vertex cover problem in polynomial time. This transformation is based on the notion of the complement of a graph G. Given an undirected graph G = (V, E), we define the complement of G as G = (V, E), where E={(u, v)/(u, v) E}. i.e., G is the graph containing exactly those edges that are not in G. The transformation takes a graph G and k of the clique problem. It computes the complement G which can be done in polynomial time. To complete the proof, we can show that this transformation is indeed reduction: the graph has a clique of size k if and only if the graph G has a vertex cover of size |V| k. Suppose that G has a clique V V with |V| = k. We claim that V V is a vertex cover in G Let (u, v) be any edge in E. Then, (u, v) E, which implies that atleast one of u or v does not belong to V, since every pair of vertices in V is connected by an edge of E. Equivalently, at least one of u or v is in V V, which means that edge (u, v) is covered by V V. Since (u, v) was chosen arbitrarily from E, every edge of E is covered by a vertex in V V. Hence, the set V V, which has size |V| - k, forms a vertex cover for G. Conversely, suppose that G has a vertex cover V V, where |V| = |V| k. Then, for all u, v V, if (u, v) E, then u V or v V or both. The contra positive of this implication is that for all u, v V, if u V and v V, then (u, v) E. In other words, V V is a clique, and it has size |V| - |V| = k. For
Sikkim Manipal University Page No. 263

Fundamentals of Algorithms

Unit 14

example, the graph G(V, E) has a clique {A, B, E}. The complement of graph G is given by G and have independent set given by {C, D, F}. This proves that finding the vertex cover is NP-Complete. In short, problems are designated as NP-complete if their solutions can be quickly checked for corrections, and if the same solving algorithm used can solve all other NP problems. They are the most difficult problems in NP in the sense that a deterministic, polynomial-time solution to any NP complete problem would provide a solution to every other problem in NP. Conversely, if any one of them provably lacks a deterministic polynomial-time solution then none of them has one. One example of an NP complete problem is the subset sum problem which is: given a finite set of integers, determine whether any non-empty subset of them sums to zero. A supposed answer is very easy to verify for correctness, but there is no known efficient algorithm to find an answer, that is, all known algorithms are impractically slow for large sets of integers. Self Assessment Questions 6. The process of establishing a problem as is a two step process 7. Example of an NP complete problem is the .

12.6 Summary
P denotes the class of all problems, for each of which there is atleast one known polynomial time Deterministic TM solving it. The notation O provides asymptotic upper bound for a given function. The notation provides an asymptotic lower bound for a given function The Notation provides simultaneously both asymptotic lower bound and asymptotic upper bound for a given function. A Polynomial-time reduction is a polynomial-time algorithm which constructs the instances of a problem P2 from the instances of some other problems P1.

12.7 Terminal Questions

1. Explain the Constant Factor in Complexity Measure 2. Explain the Different Category of Problems
Sikkim Manipal University Page No. 264

Fundamentals of Algorithms

Unit 14

12.8 Answers
Self Assessment Questions 1. Hardware, software, environmental factors 2. k-colouring 3. Adjacent 4. A Polynomial-time reduction 5. NP-Hard 6. NP-Complete 7. Subset sum problem Terminal Questions 1. If the time taken by one machine in executing a solution of a problem is a polynomial (or exponential) function in the size of the problem, then time taken by every machine is a polynomial (or exponential) function respectively, in the size of the problem. Thus, functions differing from each other by constant factors, when treated as time complexities should not be treated as different, i.e., should be treated as complexitywise equivalent 2. Problems which cannot even be defined formally. By a formal definition of a problem, we mean expressing in terms of mathematical entities like sets, relations and functions etc., the information concerning the problem, in respect of at least a) Possible inputs b) Possible outcomes c) Entitles occurring and operations on these entities in the (dynamic) problem domains. (Refer Section 14.3)

Sikkim Manipal University

Page No. 265

Fundamentals of Algorithms

Unit 14

Acknowledgement, References and Suggested Readings 1. Coremen. T. H., Leiserson. C.E. & Stein. C, (2002), Introduction to Algorithms (second edition), Prentice-Hall of India. 2. Deo Narsing, 1986, Graph Theory with Applications to Engineering and Computer Science, Prentice Hall India, New Delhi. 3. Horowitz. E. & Sahni. S., Fundamentals of Computer Algorithms, (Galgotia Publications) 4. Kolman Bernard, Busby. R.C., Ross Sharon, 1999, Discrete Mathematical Structures PHI,. 5. Levis. H.R. & Papadimitriou. C.H., (1981), Elements of the Theory of computation, PHI, 6. www.wikipedia.com.

Sikkim Manipal University

Page No. 266