ELuIsS HOROWITZ SARTAJ SAHNI SANGUTHEVAR RAJASEKARANCOMPUTER ALGORITHMSCOMPUTER SCIENCE PRESS Allred V. Aho, Columbia University Jeffrey D Ullman, Stanford University Foundations of Computer Science: Pascal Edition Foundations of Computer Science: C Edition Michael J. Clancy, University of California at Berkeley Marcia C. Linn, University of California at Berkeley Designing Pascal Solutions: A Case Study Approach Designing Pascal Solutions: Case Studies Using Data Structures ‘A. K. Dewdney, University of Western Ontario The New Turing Omnibus: 66 Excursions in Computer Science Introductory Computer Science: Bits of Theory, Bytes of Practice Robert Floyd, Stanford University Richard Beigel, Yale University The Language of Machines: An Introduction to ‘Computability and Formal Languages Michael R. Garey, Bell Laboratories David S. Johnson, Bell Laboratories Computers and Intractability: A Guide to the Theory of NP-Completeness Judith L. Gersting, University of Hawaii at Hilo Mathematical Structures for Computer Science, Third Edition Visual Basic® Programming: A Laboratory Approach Ellis Horowitz, University of Southern California Sartaj Sahni, University of Florida Fundamentals of Data Structures in Pascal, Fourth Edition Ellis Horowitz, University of Southern California Sartaj Sahni, University of Florida Susan Anderson-Freed, Illinois Wesleyan University Fundamentals of Data Structures in C Ellis Horowitz, University of Southern California Sartaj Sahni, University of Florida Dinesh Mehta, University of Tennessee Space Institute Fundamentals of Data Structures in C++ Ellis Horowitz, University of Southern California Sartaj Sahni, University of Florida Sanguthevar Rajasekaran, University of Florida Computer Algorithms Ellis Horowitz, University of Southern California Sartaj Sahni, University of Florida Sanguthevar Rajasekaran, University of Florida Computer Algorithms/C++ ‘Thomas W. Parsons, Hofstra University Introduction to Compiler Construction Gregory J. E. Rawlins, Indiana University Compared to What?: An Introduction to the Analysis of Algorithms Wei-Min Shen, Microelectronics and Computer ‘Technology Corporation Autonomous Learning from the Environment James A. Storer, Brandeis University Data Compression: Methods and Theory Steven Tanimoto, University of Washington Elements of Artificial Intelligence Using Common Lisp, Second Edition Kim W. Tracy, Bell Labs/Lucent Technologies, Ine, Peter Bouthoorn, Griningen University Object-Oriented Artificial Intelligence Using CH Jeffrey D. Ullman, Stanford University Principles of Database and Knowledge-Base Systems, Vol I: Classical Database Systems Principles of Database and Knowledge-Base Systems, Vol I: The New TechnologiesCOMPUTER ALGORITHMS Ellis Horowitz University of Southern California Sartaj Sahni University of Florida Sanguthevar Rajasekaran University of Florida & Computer Science Press An imprint of W. H. Freeman and Company New YorkAcqisitions Editor: Richard Bonacci Project Editor: Penelope Hull Text Designer: The Authors Text Illustrations: The Authors Cover Designer: A Good Thing Cover Illustration: Tomek Olbinski Production Coordinator: Sheila Anderson Composition: The Authors Manufacturing: R R Donnelley & Sons Company Library of Congress Cataloging-in-Publication Data Horowitz, Ellis. Computer algorithms / Ellis Horowitz, Sartaj Sahni, Sanguthevar Rajasekaran. P. om, Includes bibliographical references and index. ISBN 0-7167-8316-9 1. Computer algorithms. 2. Pseudocode (Computer program language). I. Sahni, Sartaj. II. Rajasekaran, Sanguthevar. III, Title. QA76.9.443H67 1998 005.1NDC21 97-20318 cIp © 1998 by W. H. Freeman and Company. Alll rights reserved. No part of this book may be reproduced by any mechanical, photographic, or electronic process, or in the form of a phonographic recording, nor may it be stored in a retrieval system, transmitted, or otherwise copied for public or private use, without written permission from the publisher. Printed in the United States of America First printing, 1997 Computer Science Press An imprint of W. H. Freeman and Company 41 Madison Avenue, New York, New York 10010 Houndmills, Basingstoke RG21 6XS, EnglandTo my nuclear family, MARYANNE, PIPL, CHAN OCH, and TRA — Ellis Horowitz To, NEETA, AGAM, NEHA, and PARAM ~ Sartaj Sahni KEERAN, KRISHNA, PANDT, and PONNUTHAL ~— Sanguthevar RajasekaranContents PREFACE 1 INTRODUCTION ll ea] 14 15 WHAT IS AN ALGORITHM? .........-.-5 ALGORITHM SPECIFICATION . 1.2.1 Pseudocode Conventions 1.2.2 Recursive Algorithms PERFORMANCE ANALYSIS 1.3.1 Space Complexity 1.3.2. Time Complexity . 1 Asymptotic Notation (O, 2, @) 1.3.4 Practical Complexitie 1.3.5 Performance Measurement . . RANDOMIZED ALGORITHMS . . 1.4.1 Basics of Probability Theory 1.4.2. Randomized Algorithms: An Informal Description . . 1.4.3 Identifying the Repeated Element 1.4.4 Primality Testing . Le 145 Advantages and Disadvantages: REFERENCES AND READINGS 2 ELEMENTARY DATA STRUCTURES 21 2 STACKS AND Queues TREES 221 Terminology 2.2.2 Binary Trees DICTIONARIES . 2.3.1 Binary Search ‘Trees. 2.3.2 Cost Amortization . . xv Be 69 69 76 oe 78 83 89viii CONTENTS 24 PRIORITY QUEUES 91 24.1 Heaps...... 0. 92 Heapsort 99 25 SETS AND DISJOINT SET UNION . 101 Introduction 7 101 . Union and Find Opera - 102 2.6 GRAPHS . . +. 2 2.6.1 Introduction : ee 1D) 2.6.2 Definitions . . . ee 112 Graph Representations na0 eis 2.7. REFERENCES AND READINGS ......... poco hd DIVIDE-AND-CONQUER 127 GENERAL METHOD . . . .. 27 BINARY SEARCH... .. . 131 FINDING THE MAXIMUM AND MINIMUM... 2... 139 MERGE SORT . 145 QUICKSORT . 154 3.5.1 Performance Measurement 3.5.2. Randomized Sorting Algorithms 159 30; SELECTION sete et 165 3.6.1 A Worst-Case _ Algorithm 169 62 Tuplamentation cfSelecta) ere t 72 3.7 STRASSEN’S MATRIX MULTIPLICATION ........- 179 3.8 CONVEX HULL 3.8.1 Some Geometric Primitives... . 3.8.2 The QuickHull Algorithm . . . 185 3.8.3 Graham’s Scan... . 187 4 An O(nlogn) Divide-and-Conquer Algorithm... .. 188 3.9 REFERENCES AND READINGS ... . . 193 3.10 ADDITIONAL EXERCISES 194 THE GREEDY METHOD 197 4.1 THE GENERAL METHOD 4.2 KNAPSACK PROBLEM . tee we 198 4.3 TREE VERTEX SPLITTING. ..........000- 77203 4.4 JOB SEQUENCING WITH DEADLINES... ...... + 208 4.5 MINIMUM-COST SPANNING TREES . : 216 4.5.1 Prim’s Algorithm. .........-. aaa neCONTENTS ix 45.2 Kruskal’s Algorithm... 02.2.0 .000008 . . 220 453 An Optimal Randomized Algorithm («) ........ 225 4.6 OPTIMAL STORAGE ON TAPES . . , . 229 4.7 OPTIMAL MERGE PATTERNS... . . . . 234 4.8 SINGLE-SOURCE SHORTEST PATHS . 241 4.9 REFERENCES AND READINGS . See ee 249 4.10 ADDITIONAL EXERCISES ...........-... - 250 5 DYNAMIC PROGRAMMING 253 5.1 THE GENERAL METHOD................... 253 5.2. MULTISTAGE GRAPHS 257 5.3 ALL PAIRS SHORTEST PATHS . 265 5.4 SINGLE-SOURCE SHORTEST PATHS: . GENERAL WEIGHTS ......... 270 5.5 OPTIMAL BINARY SEARCH TREES (+) 275 5.6 STRING EDITING - + 284 5.7 O/1-KNAPSACK . . . 5 . 287 5.8 RELIABILITY DESIGN. ..............5 sees 295 5.9 THE TRAVELING SALESPERSON PROBLEM . . 298 5.10 FLOW SHOP SCHEDULING . .. 301 5.11 REFERENCES AND READINGS . .. 307 5.12 ADDITIONAL EXERCISES - 308 6 BASIC TRAVERSAL AND SEARCH TECHNIQUES 6.1 TECHNIQUES FOR BINARY TREES 6.2. TECHNIQUES FOR GRAPHS . 6.2.1 Breadth First Search and ‘Trav 6.2.2 Depth First Search and Traversal . : 6.3 CONNECTED COMPONENTS AND SPANNING TREES : 6.4 BICONNECTED COMPONENTS AND DFS 6.5 REFERENCES AND READINGS 7 BACKTRACKING 339 7.1 THE GENERAL METHOD..... 339 7.2. THE 8-QUEENS PROBLEM .. . 353 7.3 SUM OF SUBSETS. 7.4 GRAPH COLORING 7.5 HAMILTONIAN CYCLES 7.6 KNAPSACK PROBLEMx CONTENTS 7.7 REFERENCES AND READINGS . . 7.8 ADDITIONAL EXERCISES - 374 - 375 8 BRANCH-AND-BOUND Ble CHE METHOD) fest es Least Cost (LC) Search... . The 15-puzzle: An Example . . Control Abstractions for LC-Search Bounding 15 FIFO Branch-and-Bound 8.1.6 LC Branch-and-Bound 8.2. 0/1 KNAPSACK PROBLEM 8.2.1 LC Branch-and-Bound Solution 8.2.2 FIFO Branch-and-Bound Solution . . 8.3. TRAVELING SALESPERSON (+) 8.4 EFFICIENCY CONSIDERATIONS 8.5 REFERENCES AND READINGS 9 ALGEBRAIC PROBLEMS 417 9.1 THE GENERAL METHOD 417 9.2 EVALUATION AND INTERPOLATION 420 9.3 THE FAST FOURIER TRANSFORM 430 9.3.1 An In-place Version of the FFT » 435 9.3.2 Some Remaining Points ... . 438 9.4 MODULAR ARITHMETIC . 440 95 EVEN FASTER EVALUATION AND INTERPOLATION. 448 9.6 REFERENCES AND READINGS ............005 456 10 LOWER BOUND THEORY 457 10.1 COMPARISON TREES 458 10.1.1 Ordered Searching . -. 459 10.1.2 Sorting . . 459 O13 Selection ete . 464 10.2 ORACLES AND ADVERSARY ARGUMENTS - 466 10.2.1 Merging 467 10.22 Largest and Second Largest... 0.0. 468 10.2.3 State Space Method ...... 470 10.2.4 Selection 471 10.3 LOWER BOUNDS THROUGH REDUCTIONS 474CONTENTS 10.3.1 Finding the Convex Hull . 10.3.2 s Problem Multiplying ‘Triangular Matrices Lee Inverting a Lower Triangular Matrix... 2... 5. 4 .3.6 Computing the Transitive Closure . . . 10.4 TECHNIQUES FOR ALGEBRAIC PROBLEMS (+) . 10.5 REFERENCES AND READINGS 11. NP-HARD AND NP-COMPLETE PROBLEMS 11.1 BASIC CONCEPTS 11.1.1 Nondeterminii 11.1.2 The classes NP- 11.2 COOK’S THEOREM (« 11.3 NP-HARD GRAPH PROBLEMS . . 11.3.1, Clique Decision Problem (CDP) 11.3.2. Node Cover Decision Problem [gorithm hard and NF ) 11.3.3. Chromatic Number Decision Problem (CNDP) 11.3.4 Directed Hamiltonian Cycle (DHC) (+) 11.3.5 ‘Traveling Salesperson D ao -6 AND/OR Graph Decision Problem (AOG) ..... . 11.4 NP-HARD SCHEDULING PROBLEMS i 11.4.1 Scheduling Identical Processor 12 APPROXIMATION ALGORITHMS 12.1 INTRODUCTION . . 1?.2 ABSOLUTE APPROXIMATIONS ww Shop Scheduling .. . 11.4.3 Job Shop Scheduling... . 11.5 NP-HARD CODE GENERATION PROBLEMS 1.5.1 Code Generation With Common Subexpi 11.5.2. Implementing Parallel Assignment Instruct 11.6 SOME SIMPLIFIED A’P-HARD PROBLEMS. . 11.7 REFERENCES AND READINGS 11.8 ADDITIONAL EXERCISES 12.2.1. Planar Graph Coloring 122.2 Maximum Programs Stored Problem . . . 12.2.3. N’P-hard Absolute Approximations 12.3 «APPROXIMATIONS sion Problem (TSP) ee oar xi . 533 534 .. 536 .. 538 .. 540 - 542 . 546 - 550 - 553 553,13 CONTENTS 12.3.1. Scheduling Independent Tas! 12.3.2. Bin Packing 12.3.3. NP-hard Approximation Problems 12.4 POLYNOMIAL TIME APPROXIMATION SCHEMES 12.4.1 Scheduling Independent Tasks 1242) 0/1 Knepesde eee 12.5 FULLY POLYNOMIAL TIME ...... APPROXIMATION SCHEMES 12.5.1 Rounding . . 12.5.2. Interval Partitioning 12.5.3. Separation 12.6 PROBABILISTICALLY GOOD ALGORITHMS (+) . . 12.7 REFERENCES AND READINGS ...........- 12.8 ADDITIONAL EXERCISES ..........000.0005 PRAM ALGORITHMS 605 13.1 INTRODUCTION ..... 13.2, COMPUTATIONAL MODEL . 13.3 FUNDAMENTAL TECHNIQUES AND ALGORITHMS . . 615 13.3.1, Prefix Computation 615 .3.2 List Ranking... . 618 SELECTION ....... 627 13.4.1 Maximal Selection With n? Pro 627 13.4.2 Finding the Maximum Using n Pro 628 13.4.3. Maximal Selection Among Integers 13.4.4. General Selection Using n? Processors 13.4.5 A Work-Optimal Randomized Algorithm (*) 13.5 MERGING 13.5.1 A Logarithmic 13.5.2. Odd-Even Merge . 13.5.3 A Work-Optimal Algorithm . . . . 640 1 An O(log logm)-Time Algorithm . .. 641 SORTING... 0... ee eee 643 13.6.1 Odd-Even Merge Sort 643 13.6.2 An Alternative Randomized Algorithm 13.6.3 Preparata’s Algorithm»... .... 13.6.4 Reischuk’s Randomized Algorithm (+) GRAPH PROBLEMS . . 13.7.1. An Alternative Algorithm for Trans 13. = 636 636 637 13. & 13. gCONTENTS 13.7.2 All-Pairs Shortest Paths... . . Lee 13.8 COMPUTING THE CONVEX HULL .... . 13.9 LOWER BOUNDS... 20... . 13.9.1 A lower bound on average case sorting - 13.9.2 Finding the maximum... . . 13.10REFERENCES AND READINGS . 13.1LADDITIONAL EXERCISES 14 MESH ALGORITHMS 14.1 COMPUTATIONAL MODEL. . . 14.2 PACKET ROUTING 14.2.1 Packet Routing on a Linear Array .. 2... 14.2.2 A Greedy Algorithm for PPR on a Mesh . 14.2.3 A Randomized Algorithm With Small Queues FUNDAMENTAL ALGORITHMS ...... 14.3.1 Broadcasting 14.3.2. Prefix Computation u SELECTION .. 14.4.1 A Randomized Algorithm for n 14.4.2 Randomized Selection For n > p (+) 14.4.3 A Determini MERGING ......-....0-45 14.5.1 Rank Merge on a Linear Array 14.5.2 Odd-Even Merge on a Linear Array 14.5.3 Odd-Even nae on a Mesh. . SORTING . a 14.6.1 Sorting ¢ on a Linear Array 14.6.2 Sorting on a Mesh . . . . GRAPH PROBLEMS .. . 14.7.1 Ann xn Mesh Algorithm for Transitive 14.7.2 All Pairs Shortest Paths .........-.-- 14.8 COMPUTING THE CONVEX HULL ......- Ms & Ma & la Ee 11.9 REFERENCES AND READINGS ............- 14.10ADDITIONAL EXERCISES ............-- 15 HYPERCUBE ALGORITHMS 15.1 COMPUTATIONAL MODEL.........-00+005 . 699 . . - 708 Closure xiii 655 - 656 . . 659 . 660 oceans . 663 . 665 667 . . 667 - » 669 cee ... 674 .. 676 . 679 681 .» 681 . 685, . + 686 -. 691 .. 691 » » 692 » . 692 . 698 698, 699 701 701 703 710 .7u 713 718 aeCONTENTS 15.1.1 The Hypercube... . . 723 15.1.2. The Butterfly Network... ... 726 15.1.3 Embedding Of Other Networks . - . 727 15.2 PPR ROUTING 15.2.1 A Greedy Algorithm an 15.2.2 A Randomized Algorithm ..... 15.3 FUNDAMENTAL ALGORITHMS .. . . 15.3.1 Broadcasting... 15.3.2 Prefix Computation 15.3.3 Data Concentration... 2... . Sparse Enumeration Sort 15.4 SELECTION .... 1.4.1 A Randomized Algorithm for n = 15.4.2 Randomized Selection For n > p (* 7 15.4.3 A Deterministic Algorithm Forn>p ......... 745 15.5 MERGING .. . a oe 5.5.1 Odd-Even Merge . « 15.5.2. Bitonic Merge 15.6 SORTING . 15.6.1 Odd-Even Merge Sort 15.6.2 Bitonic Sort... . rn 15.7 GRAPH PROBLEMS ........... 753 15.8 COMPUTING THE CONVEX HULL . . 755 15.9 REFERENCES AND READINGS . . ao ves. TET 15.10ADDITIONAL EXERCISES 758 INDEX 761PREFACE If we try to identify those contributions of computer science which will be long lasting, surely one of these will be the refinement of the concept called algorithm. Ever since man invented the idea of a machine which could per- form basic mathematical operations, the study of what can be computed and how it can be done well was launched. This study, inspired by the computer, has led to the discovery of many important algorithms and design methods. The discipline called computer science has embraced the study of algorithms as its own. It is the purpose of this book to organize what is known about them in a coherent fashion so that students and practitioners can learn to devise and analyze new algorithms for themselves. A book which contains every algorithm ever invented would be exceed- ingly large. Traditionally, algorithms books proceeded by examining only a small number of problem areas in depth. For each specific problem the most efficient algorithm for its solution is usually presented and analyzed. This approach has one major flaw. Though the student sees many fast algorithms and may master the tools of analysis, she/he remains unconfident about how to devise good algorithms in the first place. The missing ingredient is a lack of emphasis on design techniques. A knowledge of design will certainly help one to create good algorithms, yet without the tools of analysis there is no way to determine the quality of the result. This observation that design should be taught on a par with analysis led us to a more promising line of approach: namely to organize this book around some fundmental strategies of algorithm design. The number of ba- sign strategies is reasonably small. Moreover all of the algorithms one would typically wish to study can easily be fit into these categories; for exam- ple, mergesort and quicksort are perfect examples of the divide-and-conquer strategy while Kruskal’s minimum spanning tree algorithm and Dijkstra’s single source shortest path algorithm are straight forward examples of the greedy strategy. An understanding of these strategies is an essential first step towards acquiring the skills of design. Though we strongly feel that the emphasis on design as well as analysis is the appropriate way to organize the study of algorithms, a cautionary remark is in order. First, we have not included every known design principle. xvxvi PREFACE One example is linear programming which is one of the most successful techniques, but is often discussed in a course of its own. Secondly, the student should be inhibited from taking a cookbook approach to algorithm design by assuming that each algorithm must derive from only a single technique. This is not so. ‘A major portion of this book, Chapters 3 through 9, deal with the dif- ferent design strategies. First each strategy is described in general terms Typically a “program abstraction” is given which outlines the form that the computation will take if this strategy can be applied. Following this there are a succession of examples which reveal the intricacies and varieties of the general strategy. The examples are somewhat loosely ordered in terms of increasing complexity. The type of complexity may arise in several ways. Usually we begin with a problem which is very simple to understand and requires no data structures other than a one-dimensional array. For this problem it is usually obvious that the design strategy yields a correct solu- tion. Later examples may require a proof that an algorithm based on this design technique does work. Or, the later algorithms may require more so- phisticated data structures (e.g., trees or graphs) and their analyses may be more complex. ‘The major goal of this organization is to emphasize the arts of synthesis and analysis of algorithms. Auxiliary goals are to expose the student to good program structure and to proofs of algorithm correctness. The algorithms in this book are presented in a pseudocode that resem- bles C and Pascal. Section 1.2.1 describes the pseudocode conventions. Ex- ecutable versions (in C++) of many of these algorithms can be found in our home page. Most of the algorithms presented in this book are short and the language constructs used to describe them are simple enough that any one can understand. Chapters 13, 14, and 15 deal with parallel computing. Another special feature of this book is that we cover the area of random- ized algorithms extensively. Many of the algorithms discussed in Chapters 13, 14, and 15 are randomized. Some randomized algorithms are presented in the other chapters as well. An introductory one quarter course on parallel algorithms might cover Chapters 13, 14, and 15 and perhaps some minimal additional material. We have identified certain sections of the text (indicated with (*)) that are more suitable for advanced courses. We view the material presented in this book as ideal for a one semester or two quarter course given to juniors, seniors, or graduate students. It does require prior experience with pro- gramming in a higher level language but everything else is self-contained. Practically speaking, it seems that a course on data structures is helpful, if only for the fact that the students have greater programming maturity. For aschool on the quarter system, the first quarter might cover the basic design techniques as given in Chapters 3 through 9: divide-and-conquer, the greedy method, dynamic programming, search and traversal, backtracking, branch- and-bound, and algebraic methods (see TABLE 1). The second quarter would cover Chapters 10 through 15: lower bound theory, A’P-completeness andPREFACE xvii approximation methods, PRAM algorithms, Mesh algorithms and Hyper- cube algorithms (sce TABLE I1). [Subject [ Reading Introduction T1tol3 Introduction ~ 4 Data structures 2.1, 2.2 7 || Data structures 2.3 to 2.6 4 || Divide-and-conquer Chapter 3 Assignment I due D || The greedy method Chapter 4 Exain I Dynamic programming Chapter 5 Search and traversal techniques || Chapter 6 igument IT due \ ! || Backtracking Chapter 7 Branch-and-bound Chapter 8 ‘Algebraic methods Chapter 9 Assignment IIT due H Exam II TABLE I: FIRST QUARTER For a er schedule where the student has not been exposed to data structures and O-notation, Chapters 1 through 7, 11, and 13 is about the right. amount of material (see TABLE IIT) ‘A more rigorous pace would cover Chapters 1 to 7, 11, 13, and 14 (see TABLE IV). An advanced course, for those who have prior knowledge about data structures and O notation, might consist of Chapters 3 to 11, and 13 to 15 (see TABLE V). Programs for most of the algorithms given in this book are available from the following URL: http://www.cise.ufl.edu/~raj/BOOK.html. Please send your comments to [email protected]. For homework there are numerous exercises at the end of each chapter. The most popular and instructive homework assignment we have found is one which requires the student to execute and time two programs using the same data sets. Since most of the algorithms in this book provide all the implementation details, they can be easily made use of. Translating these algorithms into any programming language should be easy. The problem then reduces to devising suitable data sets and obtaining timing results. ‘The timing results should agree with the asymptotic analysis that was donexviii PREFACE [Week ]| Subject | Reading T___|| Lower bound theory 10.1 to 10. 2 Tower bound theory 10.4 NP-complete and NP-hard problems |) 11.1, 11.2 3 NP-complete and NP-hard problems | 11.3, 11.4 7 NP-complete and NP-hard problems | 11.5, 11.6 Approximation algorithms | 12.1, 12.2 ignment I due |e Approximation algorithms 3 to 12.6 Exam I 6 PRAM algorithms 13.1 to 13.4 [7 PRAM algorithms ~ 13.5 to 13.9 | Assignment II due 8 Mesh algorithms Td] to 14.5 9 Mesh algorithms 14.6 to 14.8 Hypercube algorithms 15.1 to 15.3 10 || Hypercube algorithms 15.4 to 15.8 Assignment III due | Exam II TABLE II: SECOND QUARTER,PREFACE PRAM algorithms ‘TABLE III: SEMESTER ~ Medium pace (no prior exposure) [ Week ][ Subject | Reading T Tatroduction - TI to 13 2 | Introduction 14 Data structures 2.1, 2.2 3 Data structures 23 to 2.6 14 Divide-and-conquer 3.1 to 34 ~ Assignment I due 5 Divide-and-conquer 3.5 to 3.7 Exam I 6 The greedy method FI to 44 7 The greedy method 25 to 4.7 Assignment II due 8 Dynamic programming 5.1 to 5.5 9 | Dynamic programming 5.6 to 5.10 T0 Search and traversal 6.1 to 64 Assignment III due Exam IL TI “Backtracking 71 to 7.3 T2 Backtracking 74 to 7.6 TS NP-complete and NP-hard problems || 11.1 to 11.3 Assignment IV due It__|| WP-complete and NP-hard problems || 11.4 to 11.6 Th PRAM algorithms 13.1 to 15.4 [16 13.5 to 13.9 Assignment V due Exam IITxx PREFACE [ Week T Subject Reading T Tntroduction T1tol3 2 Tntroduction 14 Data structures 2.1, 2.2 3 Data structures 23 to 26 q Divide-and-conquer 3.1 to 3.5 Assignment I due_|} 5 Divide-and-conquer 3.6 to 3.7 The greedy method 4.1 to 4.3 Exam I 6 The greedy method 44 to 47 7 Dynamic programming 5.1 to 5.7 Assignment II due 8 Dynamic programming BB to 5.10 Search and traversal 6.1 to 6.2 9 ‘Search and traversal te 6.3, 64 Backtracking 7.1, 7.2 10 Backtracking 7.3 to 7.6 Assignment III due |} Exam II TI NP-hard and NP-complete problems || 11.1 to 113 12 | WP-hard and NP-complete problems || 11.4 to 11.6 TS PRAM algorithms 13.1 to 13.4 Assignment IV due if PRAM algorithms 13.5 to 13.9 _ 15, Mesh algorithms 14.1 to 143 16 Mesh algorithms 14.4 to 148 Assignment V due Exam IIL TABLE IV: SEMESTER — Rigorous pace (no prior exposure)PREFACE xxi jubject Reading } Divide-and-conquer 3.1 to 3.5 _ Divide-and-conquer 3.6, 3.7 The greedy method 4.1 to 4.3 3 The greedy method 44 04.7 4 || Dynamic programming 2 Chapter 5 | Assignment I due 5 Search and traversal techniques Chapter 6 =| | Exam I [6 Backtracking Chapter 7 7 Branch-and-bound Chapter 8 Assignment II due 8 ‘Algebraic methods Chapter 9 9 Tower bound theory Chapter 10 0 WP-complete and NP-hard problems || 11.1 to 11.3 | Exam IT Assignment IIL WP-complete and NP-hard problems [114 to 11.6 ‘AM algorithms I3.1 to 13.4 PRAM algorithms 13.5 to 13.9 Assignment IV due | 2t_] Mesh algorithms: ___, ~__ | 147 to 145 15 Mesh algorithms 14.6 to 14.8 |__| Hypercube algorithms 15.1 to 15.3 1 Hypercube algorithms 15.4 to 15.8 | Assignment V due Exam TIT TABLE V: SEMESTER - Advanced course (rigorous pace)xxii PREFACE for the algorithm. This is a nontrivial task which can be both educational and fun. Most importantly it emphasizes an aspect of this field that is often neglected, that there is an experimental side to the practice of algorithms Acknowledgements We are grateful to Martin J. Biernat, Jeff Jenness, Saleem Khan, Ming-Yang Kao, Douglas M. Campbell, and Stephen P. Leach for their critical comments which have immensely enhanced our presentation. We are thankful to the students at UF for pointing out mistakes in earlier versions. We are also thankful to Teo Gonzalez, Danny Krizanc, and David Wei who carefully read portions of this book. Ellis Horowitz, Sartaj Sahni Sanguthevar Rajasekaran June, 1997Chapter 1 INTRODUCTION 1.1 WHAT IS AN ALGORITHM? The word algorithm comes from the name of a Persian author, Abu Ja’far Mohammed ibn Musa al Khowarizmi (c. 825 A.D.), who wrote a textbook on mathematics. This word has taken on a special significance in computer science, where “algorithm” has come to refer to a method that can be used a computer for the solution of a problem. ‘This is what makes algorithm cent from words such as process, technique, or method. Definition 1.1 [Algorithm]: An algorithm is a finite set of instructions that, if followed, accomplishes a particular task. In addition, all algorithms must satisfy the following criteria: 1. Input. Zero or more quantities are externally supplied. 2. Output. At least one quantity is produced. 3. Definiteness. Each instruction is clear and unambiguous. 4, Finiteness. If we trace out the instructions of an algorithm, then for all cases, the algorithm terminates after a finite number of steps. Effectiveness. Every instruction must be very basic so that it can be carried ont, in principle, by a person using only pencil and paper. It is not enough that each operation be definite as in criterion 3: it also must be feasible. a eo An algorithm is composed of a finite set of steps, each of which may require one or more operations. The possibility of a computer carrying out these operations necessitates that certain constraints be placed on the type of operations an algorithm can include.2 CHAPTER 1. INTRODUCTION Criteria 1 and 2 require that an algorithm produce one or more outputs and have zero or more inputs that are externally supplied. According to cri- terion 3, each operation must be definite, meaning that it must be perfectly clear what should be done. Directions such as “add 6 or 7 to 2” or “compute 5/0” are not permitted because it is not clear which of the two possibilit should be done or what the result is. The fourth criterion for algorithms we assume in this book terminate after a finite number of operations. A related consideration is that the time for termination should be reasonably short. For example, an algorithm could be devised that decides whether any given position in the is a winning position. The algorithm works by examining all ble moves and countermoves that could be made from the starting po- sition. The difficulty with this algorithm is that even using the most modern computers, it may take billions of years to make the decision. We must be very concerned with analyzing the efficiency of each of our algorithms. Criterion 5 requires that each operation be effective; each step must be such that it can, at least in principle, be done by a person using pencil and paper in a finite amount of time. Performing arithmetic on integers is an example of an effective operation, but arithmetic with real numbers is not, since some values may be expressible only by infinitely long decimal expan- sion. Adding two such numbers would violate the effectiveness property Algorithms that are definite and effective are also called computational procedures. One important example of computational procedures is the op- erating system of a digital computer. This procedure is designed to control the execution of jobs, in such a way that when no jobs are available, it does not terminate but continues in a waiting state until a new job is en- Though computational procedures include important examples such one, we restrict our study to computational procedures that always terminate. To help us achieve the criterion of definiteness, algorithms are written in a programming language. Such languages are designed so that each legitimate sentence has a unique meaning. A’ program is the expression of an algorithm in a programming language. Sometimes words such as procedure, function. and subroutine are used synonymously for program. Most readers of this book have probably already programmed and run some algorithms on a computer. This is desirable because before you study a concept in general, it helps if you had some practical experience with it. Perhaps you had some difficulty getting started in formulating an initial solution to a problem, or perhaps you were unable to decide which of two algorithms was better. The goal of this book is to teach you how to make these decisions. The study of algorithms includes many important and active areas of research. There are four distinct areas of study one can identify: 1. How to devise algorithms — Creating an algorithm is an art which may never be fully automated. A major goal of this book is to study vari- that they11. WHAT IS AN ALGORITHM? ous design techniques that. have proven to be useful in that they have often yielded good algorithms. By mastering these design strategies, it will become easier for you to devise new and useful algorithms. Many of the chapters of this book are organized around what we believe are the major methods of algorithm design. The reader may now wish to glance back at the table of contents to sce what these methods are called. Some of these techniques may already be familiar, and some have been found to be so useful that books have been written about them. Dynamic programming is one such technique. Some of the techniques are especially useful in fields other than computer science such as operations research and electrical engineering. In this book we can only hope to give an introduction to these many approaches to algorithm formulation. All of the approaches we consider have applica- tions in a variety of areas including computer science. But some important design techniques such as linear, nonlinear, and integer programming are not covered here as they are traditionally covered in other courses. 2. How to validate algorithms — Once an algorithm is devised, it is necessary to show that it computes the correct answer for all possible legal inputs. We refer to this process as algorithm validation. The algorithm need not as yet be expressed as a program. It is sufficient to state it in any precise way. The purpose of the validation is to assure us that this algorithm will work correctly independently of the i: ng the programming language it will eventually be written in. Once the validity of the method has been shown, a program can be written and a second phase begins. This phase is referred to as program proving or sometimes as program verification. ‘A proof of correctness requires that the solution be stated in two forms. One form is usually as a program which is annotated by a set of assertions about. the input and output variables of the program. These assertions are often expressed in the predicate calculus. The second form is called a specification, aud this may also be expressed in the predicate calculus. A proof consists of showing that these two forms are equivalent in that for every given legal input, they describe the same output. A complete proof of program correctuess requires that each statement of the programming language be precisely defined and all basic operations be proved correct. All these details may cause a proof to be very much longer than the program. 3. How to analyze algorithms — This field of study is called analysis of algorithms. As an algorithm is executed, it uses the computer's central sing unit (CPU) to perform operatious and its memory (both imme- and auxiliary) to hold the program and data. Analysis of algorithms or performance analysis refers to the task of determining how much com- puting time and storage an algorithm requires. This is a challenging area which sometimes requires great mathematical skill. An important result of this study is that it allows you to make quantitative judgments about the value of one algorithm over another. Another result is that it allows you to predict whether the software will meet any efficiency constraints that exist ues concern4 CHAPTER 1. INTRODUCTION Questions such as how well does an algorithm perform in the best case, in the worst case, or on the average are typical. For each algorithm in the text, an analysis is also given. Analysis is more fully described in Section 1.3. 4. How to test a program — Testing a program consists of two phases: debugging and profiling (or performance measurement). Debugging is the process of executing programs on sample data sets to determine whether faulty results occur and, if so, to correct them. However, as E. Dijkstra has pointed out, “debugging can only point to the presence of errors, but not to their absence.” In cases in which we cannot verify the correctness of output on sample data, the following strategy can be employed: let more than one programmer develop programs for the same problem, and compare the outputs produced by these programs. If the outputs match, then there is a good chance that they are correct. A proof of correctness is much more valuable than a thousand tests (if that proof is correct), since it guarantees that the program will work correctly for all possible inputs. Profiling or performance measurement is the process of executing a correct program on data sets and measuring the time and space it takes to compute the results. These timing figures are useful in that they may confirm a previously done analysis and point out logical places to perform useful optimization. A description of the measurement of timing complexity can be found in Section 1.3.5. For some of the algorithms presented here, we show how to devise a range of data sets that will be useful for debugging and profiling. four categories serve to outline the questions we ask about algo- throughout this book. As we can’t hope to cover all these subjects completely, we content ourselves with concentrating on design and analysis, spending less time on program construction and correctness. EXERCISES 1. Look up the words algorism and algorithm in your dictionary and write down their meanings. 2. The name al-Khowarizmi (algorithm) literally means “from the toy of Khowarazm.” This city is now known as Khiva, and is located in Uzbekistan. Sce if you can find this country in an atlas. 3. Use the WEB to find out more about al-Khowarizmi, e.g., his dates, a picture, or a stamp.1.2. ALGORITHM SPECIFICATION, 5 1.2 ALGORITHM SPECIFICATION 1.2.1 Pseudocode Conventions In computational theory, we distinguish between an algorithm and a pro- gram. ‘The latter does not have to satisfy the finiteness condition. For ex- ample, we can think of an operating system that continues in a “wait” loop until more jobs are entered. Such a program does not terminate unless the system crashes. Since our programs always terminate, we use “algorithm” and “program” interchangeably in this text We can describe an algorithm in many ways. We can use a natural language like English, although if we select this option, we must make sure that the resulting instructions are definite. Graphic representations called flowcharts are another possibility, but they work well only if the algorithm is small and simple. In this text we present most of our algorithms using a pseudocode that resembles C and Pascal. 1. Comments begin with // and continue until the end of line. 2. Blocks are indicated with matching braces: { and }. A compound statement (i.e., a collection of simple statements) can be represented as a block. The body of a procedure also forms a block. Statements are delimited by ; 3. An identifier begins with a letter. The data types of variables are not explicitly declared. ‘The types will be clear from the context Whether a variable is global or local to a procedure will also be evident from the context. We assume simple data types such as integer, float, char, boolean, and so on. Compound data types can be formed with records. Here is an example: node = record { datatype data; datatype_n data.n; node links } In this example, link is a pointer to the record type node. Individual data items of a record can be accessed with + and period. For instance if p points to a record of type node, p + data-1 stands for the value of the first field in the record. On the other hand, if q is a record of type node, q.data.1 will denote its first field.6 CHAPTER 1. INTRODUCTION 4. Assignment of values to variables is done using the assignment state- ment (variable) := (epression); 5. There are two boolean values true and false, In order to produce these values, the logical operators and, or, and not and the relational operators <,<,=,#,>, and > are provi 6. Elements of multidimensional arrays are accessed using { and ]. For example, if A is a two dimensional array, the (i,)th element of the array is denoted as Ali, j]. Array indices start at zero. 7. The following looping statements are employed: for, while, and repeat- until. The while loop takes the following form: while (condition) do { (statement 1) (statement n) } As long as (condition) is true, the statements get executed. When ondition) becomes false, the loop is exited. The value of (condition) is evaluated at the top of the loop. The general form of a for loop is for variable { valuel to value2 step step do (statement 1) (statement n) + Here valuel, value2, and step are arithmetic expressions. A variable of type integer or real or a numerical constant is a simple form of an arithmetic expression. The clause “step step” is optional and taken as +1 if it does not occur. step could either be positive or negative. variable is tested for termination at the start of each iteration. The for loop can be implemented as a while loop as follows: