05758 Computer Algorithms Introduction to Design and Analysis THIRD EDITION . Sara Baase San Diego State University Allen Van Gelder University of California at Santa Cruz ‘yy ADDISON-WESLEY An imprint of Addison Wesley Longman, Inc. Reatling, Massachusetls # Menlo Pack, California * New York Harlow, Enyland nd + Don Mls, Ontario» Sney + wesio City Maclrid ®To Keith—always part of what I do To my parents, who were my first and most important teachers S.B. AVG.Assistant Editor: Jason Miranda . Composition/Art: Paul C. Abognostopoules, Windlall Software Copy Edtuor: Jona Flaherty Proofreader: Brooke Albright Cover ttustration: Ianetmarie Colby aver Design: Lynne Reed Menufacturing Coordinator: Timothy McDonald ‘Access the latest information abou Addison-Wesley titles from our World Wide Web site; wwsvaelonline com Many of the designations used by manufacturers and solters to distinguish their products are elaimed as ‘rademarks. Where those desigontions appear in this book, and Addison-Wesley was aware of a trademark: the designations have been printed in inital eaps or all caps, ‘The programs and applications presented in this book have been inchaded for their instructional value. They have been tested with care but ore not gvaranieed for any purpose. The publisher does net offer any warranties or representations. nos does it accept any liabilities with respect tothe programs or applications ‘This book was typeset in ZX on @ PC. The font fami used were Times, Optima, Lucida Sans, and ‘MathTime. h was printed on New Era Maite, Library of Congress Catsloging-In-Publlcation Data Basse, Sara, Computer algorithms / Sara Race, Allen Van Gelder, — 3rd ed. Pwd ISBN 0-201-61244.5 1. Computer algorithms. 1. Gelder, Alle Von, Ih. Title. QA769.A43B33 2000 519.7 —4e21 9914185, ce Copyright © 2000 by Addison Wesley Longman ‘All cights reserved, No port of this publication may be reproduced, stored in arevieval system, or transmitted, in any form oF by any menns,electionic, mechanical, photocopying. recording, or otherwise, withent the prior ‘writen permission of the publisher. Printed inthe United States of America 123456789 10-Ma—onm2010099Preface Purpose This book is intended for an upper-division or graduate course in algorithms. It has suff ent material to allow several choices of topics. ‘The purpose of the book is threefold, It is intended to teach algarithms for solving real problems that arise frequently in computer upplications, to teach basic principles and techniques of computational complexity (worst-case and average behavior, space usage, and lower bounds on the complexity of a problem), and to introduce the arens of NP- completeness and parallel algorithms. . Another of the book's aims, which is at least as importunt as teaching the subject matter, is to develop in the reader the habit of always responding to a new algorittun with the questions: How good is it? Is there a better way? Therefore, instead of presenting a series of complete, “pulled-out-of-a-hat” algorithms with analysis, the text often discusses ‘a problem first, considers one or more upproaches (o solving it (as a reader who sees the problem for the first time might), and then begins to develop un algorithm, analyzes it, and ‘modifies or rejects it until a satisfactory result is produced. (Altemnative approaches that are ultimately rejected ure also considered in the exercises; it is useful for the reader to know why they were rejected.) Questions such as: How can this be done more efficiently? What data structure would be usefut here? Which operations should we focus on to analyze this algorithm? How ‘must this variable (or data structure) be initialized? appeur frequently throughout the text. ‘Answers generally follow the questions, but we suggest readers pause before reading the ensuing text and think up their own answers, Learning is not a passive process. We hope readers will also leurn to be aware of how an algorithm uctually behaves ‘on various inputs—that is, Which branches are followed? What is the puttern of growth and shrinkage of stacks? How does presenting the input in different ways (e.., listing the vertices or edges of a graph in u diferent order) affect the behavior? Such questions ute raised in some of the exercises, but are not emplusized in the text because they require carefully going through the details of many examples. ‘Most of the algorithms presented ure of prictical use; we have chosen not to empha- size those with good asymptotic behavior that are poor for inputs of useful sizes (though ‘some important ones are included). Specific. algorithms were chosen for a variety of reasonsviti_ Preface including the importance of the problem, illustrating unalysis techniques, ilNistrating tech niques (¢-g., depth-first search) that give rise to numerous algorithms, and illustrating the development and improvement of techniques and algorithons (e.g., Union-Find programs), Prerequisites The book assumes familiarity with data structures such as linked lists, stacks, and trees, and prior exposure to recursion. However, we include a review, with specifications, for the standard data structures and some speciatized ones. We have also added a student-friendly review of recursion, Analysis of algorithms uses simple properties of logarithms and some calculus (dif- ferentiation to determine the asymptotic order of a function and integration to approximate summations), though virtually no calculus is used beyond Chapter 4. We find many stu- dents intimidated when they see the fitst log or integral sign because a year or more has passed since they had a calculus course, Readers will need only a few properties of logs ‘and a few integrals from first-semester calculus. Section 1.3 reviews some of the necessary mathematics, and Section 1,5.4 provides a practical guide. Algorithm Design Techniques Several important algorithm design techniques reappear in many algorithms. These in- Clude dividetind-conquer, greedy methods. depth-first search (for graphs), and dynamic Programming, This edition puts more emphasis on algorithm design techniques than did the second edition. Dynamic progtamming, as before, has its own chapter and depth-first Search is presented with many applications in the chapter on graph traversals (Chapter 7). Most chapters are organized by application area, rather than by design technique, so we provide here a list of places where you will find algorithms using divide-and-conquer and greedy techniques. The divide-and-conquer technique is described in Section 4.3, It is used in Binory Search (Section 1.6), most sorting methods (Chapter 4), median finding and the general selection problem (Section 5.4), binary search trees (Section 6.4), polynomial evaluation (Section 12.2), matrix multiplication (Section 12.3), the Fast Fourier Transform (Sec- tion 12.4), approximate graph coloring (Section 13.7). and, in a stightly different fori, for paralle! computation in Section 14.5. ‘ Greedy algorithms are used for finding minimum spanning trees and shortest paths in Chapter 8, and for various approximation algorithms for NP-hard optimization problems. such as bin packing. knapsack, graph coloring, and traveling salesperson (Sections 13.4 through 13.8), Changes from the Second Edition This edition has three new chapters and many new topics. Throughout the book, numerous sections have been extensively rewritten, A few topics froin the second edition have been moved to different chapters where we think they fit Better, We aclded nore than 100 new exercises, many bibliographic entries, and an appendix with Java examples. Chapters 2, 3, and 6 are virtually all new. LePreface Chapter 2 reviews abstract data types (ADTs) and inclules specifications for severel standard ADTs. The vole of abstiact data types in algorithm design is emphasized through- out the book, Chapter 3 reviews recursion and induction, emphasizing the connection between the {wo and their usefulness in designing and proving correctness of programs, The chapter also develops recursion trees, which provide a visual and intuitive representation of recur rence equations that arise in the analysis of recursive algorithms, Solutions for commonly occurring patterns are summarized so they are available for use in tater chapters. Chapter 6 covers hashing, red-black trees For balanced binary trees, advanced priority queves. and dynamic equivatence relations (Union-Find). The later topic was moved from a different chapter in the second edition. ‘We rewrote all algorithms in a Java-based pscudocode. Familiarity with Java is not Tequired: the algorithms can be read easily by anyone familiar with C or C++. Chapter | has an introduction to the Java-based pseudocovte We significantly expanded the section on mathematical tooks for algorithm analysis in Chapter I to provide a better review and reference for some of the mathemitics used in the book. The discussion of the asymptotic aider of functions in Sectioit £.5 was designed! {0 help students gain a better mastery of the concepis and techniques for dealing with ‘asymptotic order, We added rules, in informal fanguage, thal summarize the most common cases (Section 1.5.4), Chapter 4 contains an accelerated version of Heapsort in which the number of key comparisons is cut nearly in hulf, For Quicksort, we use the Hoare panition algorithm in the main text. Lomuto’s method is introrluced in an exercise. (This is reversed from the second edition.) We sptit the old graph chapter into two chapters, and changed the order of some topics. Chapter 7 concentrates on (linear time) traversal algorithms, ‘The presentation of depth-first search has been thoroughly revised to emphasize the general structure af the technique and show more applications. We added topological sorting and critical path analysis ts applications and because of their intrinsic value and their connection to dynamic Programming. Sharir’s algorithm, rather than Tarjan’s, is presented for strongly connected ‘components, Chapter 8 covers greedy algorithms for graph problems, The presentations of the Prim algorithm for minimuin spanning trees and the Dijkstra algorithm for shortest paths were rewritten to emphasize the roles of priarity quetes and to illustrate how the use of abstract data types can lead the designer to efficient implementations. The asymptotically optimal ‘(eu + 1 log m) implementation is mentioned, but is not covered in depth. We moved Kroskal's algorithin for minimum spanning trees to this chapter. The presentation of dynamic programming (Chapter 10) was substantially revised to emphasize a general approach to finding dynamic programming solutions. We addled a fnew application, a text-formatting problem, to reinforce the point that not all applications call for a two-dimensional array, We moved the approximate string matching application (which wies in this chapter in the second edition) to the sting matching chapter (Sec- tion 11.5). The exercises inclucle some other new application:Preface ‘Our teaching experience has pinpointed particular areus where students bad sitficulties with concepts related to P and NP (Chapter 13), particutasly nondeterministic algorithms and polynomial transformations, We rewrote souve definitions and examples to make the concepls clearer. We added a short section on approximation algorithms for the traveling salesperson problem sud a seetion on DNA computing. Instructors who used the second edition may particularly want to note that we changed some conventions and terminology (usually to conform to conmmon usage). Array indexes now offen begin at 0 instead of I. (ln some cases, where numbering fiom 1 was clearer, we left it that way.) We now ase the term depal rather than Jevel for the depth of a node ina wee. We use height instead of depit for the maximum depth of any node in a tree. In the second euition, a path in a graph was defined to be what is commonly called a simple arly, we use the ore general definition for part ia this edition and define simple path separately. A directed graph may now contain a self-edge. Exercises and Programs Some exercises ure somewhat open-ended. Kor example, one might ask for a good lower bound for the complexity of a problem, ruler shi asking students to show tha a given function is a lower bound, We did this for two reasons. One is to niake the form of the question move realistic; a solution must be discovered as well as verified. The other iy that it may be har for some students to prove the best known tower bound (or find the most” efficiemt algorithm for a problema), but there is still a range of solutions they cun offer to. show their mastery of the techniques studied, Some topics and interesting problems are introduced only in exercises, For example, wependent set problem For a tee is an exercise in Chapter 3, the maxiniuin subsequence sum problem is an exercise in Chapter 4, and the sink finding problem for @ graph is an exercise in Chapter 7. Several NP-complete problems are introduced in exercises in Chapter 13. “The abilities, background, and mathematical sophistication of students at different uni- versilies vary cousiderably, muking it difficult to decide exactly which exercises should be marked ("starred") as “hard.” We starred exercises that use more than minimal matheraat- ies, require substuntiat creativity, or require a long chain of reasoning. A few exercises have two stars, Some starred exercises have hints. The algorithms presented in this book are not prograns; that is, many details not important to the method or the analysis are omitted. Of course, students should know how to implement efficient algorithms in efficient, debugged programs. Many insivuctors may leach this course as a pure “theory” course without programming. For those who want to assign programming projects, most chapters include a list of programming assignments, “These ure brief suggestions that may need amplification by instructors wlio choose t0 use then, Selecting Topics for Your Course Clearly the amount of material and the particular setection of topics to cover depend on the particular nt population. We present sample outlines for two undergraduate aduate course.Preface xi This outline corresponds at San Diego State University proximately (o the senior-level course Sura Baase teaches 15-week semester with 3 hours per week of lecture. Chapter 1: The whole chapter is assigned as reading but I conceotrate on Sections 1.4 and L.5 in class. : Sections 2.1 through 2.4 assigned as reading. ions 3.1 through 3.4, 3.6, and 3.7 assigned as reuding with light cover- sections 4.1 through ’ : Seetions 5.1 through 5.2. 5.6, and some of 5.4. ections 7.1 through 7.4 and either 7.5 oF 7.6 and 7.7. 8.1 through 8.3 and brief mention of 8.4, Chapter 11: Sections 11.4 through 11.4, Chapter 13: Sections 13.1 through 43.5, 13.8, and 13.9. ‘The next outline is the junior-level course Allen Van Gelder teachdS af the University of California, Santa Cruz, in a 10-week quarter with 3.5 hours per week of lecture, Chapter 1: Sections 1.3 and 1.5, and remaining sections us reading. Chapter 2: Sections 2.1 through 2.3, and remaining sections as reading, Chapter 3: All sections are touched on; a lot is left for reading. Chapter 4: Sections 4.1 through 4.9. ly Section 5.4, the average Linear tine algorithm only, Chapter 6: Sections 6.4 through 6.6, Chapter 7: Sections 7.4 through 7.6. _ Chapter 8: The entire chapter. Chapter 9: Sections 9.1 through 9.4. Chapter 10: Possibly Sections 10.1 through 10.3, but usually no time. For the first-year graduate course at the University of California, Santa Cruz (also 10 weeks, 3.5 hours of lecture), the above material is compressed and the following additional topics are covered. Chapter $: The entire chapter. Chapter 6: The remainder of the chapter, with emphasis on amortized analysis. Chapter 10: The entire chapter. Chapter 13: Sections 13.4 through 13.3, and possibly Section 13.9. ‘The primary dependencies among chapters are shown in the following diagram with sofid lines; some secondary dependencies are indicated with dashed lines. A secondary dependency means that only a few topics in the carlier chapter are needed in the later chapter, of that only the more udvanced sections of the later chapter require the earlier one.Preface While material in Chapters 2 and 6 is important to have seen, a lot of it might have been covered in an earlier course. Some sections in Chapter 6 arc important for the more advanced parts of Chapter 8. We like to remind reacers of common themes or techniques, so we often refer back (o earlier sections; many of these references can be ignored if the earlier sections were not covered, Several chapters have a section on lower bounds, which benefits from the ideas and examples in Chapter 5, but the diagram does not show that dependency because many instructors do not cover lower bounds. We marked (“starred”) sections that contain more complicated mathematics or more complex or sophisticated arguments than most others, but only where the material is ot central to the book, We also starred one or two sections that contain optional digressions. We have not starred a few sections that we consider essential to a course for which the book is used, even though they contain a lot of mathematics. For examaple, at least some of the ‘material in Section 1.5 on the asymptotic growth rate of functions and in Section 3.7 on solutions of recurrence equations should be covered. Acknowledgments ‘We are happy to take this opportunity to thank the people who helped in big and small ways in the preparation of the third edition of this book. Sara Baase acknowledges the influence and inspiration of Dick Karp, who made the subject of computationat complexity exciting and beautiful in his superb lectures. Allen ‘Van Gelder acknowledges the insights gained from Bob Floyd, Don Knuth, Est Mayr, Vaughan Pratt, and Jeff Ullman: they all teach more than is “in the book.” Allen also ‘wishes to acknowledge colleagues David Helmbold for many discussions en how to present algorithms effectively and on fine points of many algorithms, and Charlie McDowell for help on many of the aspects of Java that are covered in this book's appendix. We thank Lila Kari for reading an carly draft of the section on DNA computing and answering our questions, OF course, we'd have nothing to write about without the many people who did the original research that provided the material we enjoy learning and passing on to new generations of students, We thank them for their work. In the years since the second edition appeared, several students and instructors who used the book sent in lists of errors, typos, and suggestions for changes. We don't have @ complete list of names, but we appreciate the time and thought that went into their letters The surveys and manuscript reviews obtained by Addison-Wesley were especially helpful. Our thanks to liana Bjorling-Sachs (Lafayette College), Mohammad B. Dadfar «Bowling Green State University}, Daniel Hirschberg (University of California at Irvine),| j ! | ‘ Preface Mitsunoti Ogihara (University of Rochester), R. W. Robinson (University of Georgia), Yaakov L, Varol (University of Nevada, Reno), William W. White (Southern Ilinois Uni- versity at Edwardsville), Dawn Wilkins (University of Mississippi), and Abdou Youssef (George Washington University). ‘We thank our editors at Addison-Wesley, Maite Suarez-Rivas and Karen Wernholm, for their confidence and patience in working with us on this project that often departed from standard production procedures and schedules. We thank Joan Flaherty for her painstak- ingly careful copy editing and valuable suggestions for improving the presentation. Brooke Albright’s caceful proofreading detected many errors that had survived earlier scrutiny: of ‘course, any that remain are the fault of the authors, ‘We thank Keith Mayers for assisting us in various ways. Sara thanks him for not seminding her too often that she broke her wedding vow to work less than seven days # week, Sara Base, San Diego, California fttp://www-rohan,sdsu.edu/facuty/baase Allen Van Gelder, Santa Cruz, California http://www.cse.ucse.edu/persannel/faculty/avg html June, 1999Contents Preface vii Analyzing Algorithms and Problems: : Principles and Examples 1 1.1 Intyoduction 2 1.2 Javaas an Algorithm Language 3 1.3 Mathematical Background 11 14 Analyzing Algorithms and Problems — 30 1.5. Classifying Functions by Their Asymptotic Growth Rates 43 16 Sewrehing an Ordered Array 53. Exercises 61 Notes and References 67 Data Abstraction and Basic Data Structures 69 2.1 Introduction 70 2.2 ADT Specification and Design Techniques 71 23 Elememury ADTs—Lisis and Trees 73 24 Stucks and Queues 86 2.5 ADTs for Dynamic Sets 89 Exercises 95 Notes and References 100 Recursion and Induction 101 3.1 Introduction 102 3.2 Recursive Procedures 102 3.3. What Isa Proof? 108 3.4 Induction Proofs 111 3.5. Proving Correctness of Procedures 118Contents xvii Programs 309. Notes and References 309 7 Graphs and Graph Traversals 313 7.1 Introduction 314 7.2 Definitions and Representations 314 7.3 Traversing Graphs 328 7.4 Deplh-First Search on Directed Graphs 336 7.5. Strongly Connected Components of a Directed Graph 357 7.6 Depth-First Search on Undirected Graphs 364 7.7 Biconnected Components of an Undirected Graph 366 Exercises 375 Programs 384 Notes and References 385 8 Graph Optimization Problems and Greedy Algorithms 387 8.1 Introduction 388 8.2 Prim’s Minimum Spanning Tree Algorithm — 388 83. Single-Source Shortest Paths 403 84° Kruskal’s Minimum Spanning Tree Algorithm — 412 Exercises 416 Programs 421 Notes and References 422 : 9 Transitive Closure, All-Pairs Shortest Paths 425 9.1. Introduction 426 9.2 The Transitive Closure of a Binary Relation 426 9.3 Warshall’s Algorithm for Transitive Closure 430, 9.4 All-Pairs Shortest Paths in Graphs 433 9.5 Computing Transitive Closure by Matrix Operations 436 9.6 Multiplying Bit Mavrices—Kronrod’s Algorithm 439 * Exercises 446 Programs 449 Notes and References 449 10 Dynamic Programming 451 10.1 Introduction 452 10.2 Subproblem Graphs and Their Traversal 453 10.1. Multiplying a Sequence of Matrices 457 | |xvi Contents 3.6 Recurrence Equations 130 3.7 Recursion Trees 134 + Exercises. 141 Notes and References 146, 4 Sorting . 4.t Introduction £50 4.2 Insertion Sort 151 4.3. Divide and Conquer 157 44° Quicksort 159 4.5 Merging Sorted Sequences 171 46 Mergesort 174 4.7 Lower Bounds for Sorting by Comparison of Keys 178 48 Heapson. 182 4.9 Comparison of Four Sorting Algorithms 197 4.10 4u Shetisont 197 Radix Soning 201 Exercises 206 Programs 221 Notes and References 221 5 Selection and Adversary Arguments SA 5.2 53 54 +55 56 Introduction 224 Finding max and min 226 Finding the Second-Largest Key 229 The Selection Problem 233 A Lower Bound for Finding the Median 238 Designing Against an Adversary 240 Exercises 242 Notes and References 246 6 Dynamic Sets and Searching 61 62 63 64 65 66 +67 Introduction 250 Array Doubling 250 Amortized Time Analysis 251 Red-Black Trees 253 Hashing 275 Dyna Priority Queues with a Decrease Key Operation 295 Exercises 302 Equivalence Relations and Union-Find Programs 283 149 223 249Contents 466 . an 10.6 Developing a Dynamic Programming Algorithm — 474 Exercises 475 Programs 481 Noles and References 482 11 String Matching 483 HLA Introduction 484 11.2. A Suaightforward Solution 485 11.3 ‘The Kuuth-Mortis-Pratt Algorithm — 487 11.4 The Boyer-Moore Algorithm — 495 11.5. Approximate String Matching 504 Exercises 508 Programs — $12 Notes und References 512 ° 12 Polynomials and Matrices 515 12.1 Introduction 516 12,2. Evaluating Polynomial Functions 516 12.3 Vector and Matrix Multiplication 522 #124 The Fust Fourier Transform and Convolution 528 Exercises 542 Programs 546 Notes and References 546 13 NP-complete Problems 547 13.) Inweduetion 548 13.2 Pand NP $48 13.3 NP-Complete Problems 559 13.4 Approximation Algorithms 570 135 Bin Packing $72 13.6 The Knapsack and Subset Sum Problems $77 13.7. Graph Coloring $81 13.8 The Traveling Salesperson Problem 589 13.9 Computing with DNA 592 Exercises 600 Notes and References 60814 Parallel Algorithms 14.1 Introduction 612 14.2. Parullclism, the PRAM, and Other Models 14.3. Some Simple PRAM Algorithms 616 144° Handling Write Conflicts 622 145. Merging and Sorting 624 146 . Finding Connected Components 621 14.7 A Lower Bound for Adding » Integers Exercises 643 Notes and References 647 A Java Examples and Techniques A. Introduction 650 A2 AJavaMain Program 651 A3° ASimple Input Library 656 ‘A4 Documenting Java Classes 658 AS Generic Order and the “Comparuble” Interface 641 e 659 AG Subclasses Extend ihe Capability of Their Superclass. A.1 Copy via the “Cloneable” Interface Bibliography Index 667 663 Contents xix 611 649 669 679Analyzing Algorithms and Problems: Principles and Examples + 1.1 Introduction 1.2 Javaas an Algorithm Language 1.3. Mathematical Background 1.4 Analyzing Algorithms and Problems 1.5 Classifying Functions by Their Asymptotic Growth Rates 1.6 Searching an Ordered Array11 Chapter 1 Anafyzing Algorithms and Problems: Principles and Examples Introduction . To say that a problem is solvable algorithmically means, informally, that a computer program can be written that will produce the correct answer for any input éf we Let it run fong enough and allow it as much storage space as il needs. tn the 1930s, before the advent ‘of computers, mathemitickuns worked very actively to formalize and study the notion of , which was then interpreted informally to mean a clearly specified set of simple instructions to be followed to solve a problem or compute a funtion. Various Format models of computation were devised and investigated. Much of the emphasis in the ely work in this field, called compuiability theory, was on describing or churacterizing those problems that could be solved algorithmically and on exhibiting some problems that could not be, One of the inportant negative results, established by Alan Turing, was the proof ‘of the unsolvability of the “halting problem.” The halting problem is to determine whether an arbitrary given algorithm (or computer program) will eventually halt (rather than, say, gel into an infinite loop) white working on a given input. There cannot exist a computer program that solves this problem. Although computability theory has obvious and fundamental implications for com- puter science, the knowledge that @ problem can (heoretieally be solved on a computer is not sufticien|Jo tetl us whether it is practical to do so, For example, a perfect chess-playing program could be written. This would not be a very difficult tisk; there are only a finite umber of ways to arrange the chess pieces on the boatd, and under certain rules a game ust (erminate alter a finite number of moves, The program could consider each of the computer's possible moves, each of its opponemt’s possible responses, each of its possi- ble respouses to those moves, and so on until each sequence of possible moves reaches an cud, Then since it knows the ullimace result of euch move, the computer can choose the best ane, The mumber of distinet arrungemens of pieces on the boaed that itis reasonable to consider (much less the number of sequences of moves) is roughly 10 by sume esti- mates, A program that examined them all would take several thousand years to run. Thus such a program has net been sun. Numerous problems with practical appfications can be solved—that is, programs can be written for them—but the Gine and storage requirements are mach tgo great for these is to be of practical use. Clearly the time and space requirements of a progeam are portance, They have become. therefore, the subject of theoretical study in the area of computer science called comautational complexity. One branch of this study, which is not covered ia this book, is concerned with setting up a formal and somewhat abstract theory of the compleaity of computable functions, (Solving a problem is equivalent to computing a function from the set of inputs to the set of ouiputs.) Axioms for measures of complexity have been formulated: they are basic and genesal enowh so that either the number of instructions executed or the number of slorige bits used by a program can be taken as a complexity measure. Using these axjoms, we can prove the existence of Jy complex problems and of problems for which there is no best program, ch of computational complexity studied in this book is concerned with an- alyzing specific problems and specific algorithms. This book is intended to help readers build a repertoire of classic algorithms to solve common problems, some general design1.2 1.2 Java as an Algorithm Language techniques, tools and principles for analyzing algorithms and problems, and methods of proving correctness. We will present, study, and analyze algorithms to solve a variety of problems for which computer programs are frequently used. We will analyze the amount of time the algorithins take (o execute, and we will also often analyze the amount of space used by the algorithms. In the course of describing algorithms for a variety of problems, ‘we will see that several algorithm design techniques often prove useful. Thus we will pause now and then to talk about some general techniques, such as divide-and-vonquer, greedy algorithms, depth-first search, and dynamic programming, We will also study the com> putational complexity of the problems themselves, that is, the time and space inherently required (0 solve the problem no matter what algorithm is died. We will study the class of NP-complete problens—problems for which uo efficient algorithms are known—and consider some heuristies for getting useful results. We will also describe an approach tor solving these problems using DNA instead of electronic computers, Finally, we will intro- duce the subject of algorithms for pavallel computers. {n the following sections we outline the alzorithim language, review some background and tools that will be used throughout the book, and illustrate the main concepts involved mnalyzing aw algorithm. ° Java as an Algorithm Language We chose Java as the algorithm language for this book by bulancing several criteria. The algorithins should be easy to read, We want 10 focus on the strategy and techniques of an al~ gorithm, aot declarations and syntax details of concern to a compiler. The language should support data abstraction and problem decomposition, to make il easy to express algorithmic ideas cleurly, The language should provide a practical pathway to implementation. Itshould be widely avaitable and provide support for program development. Actually implementing and running algorithms can enhance the student's understanding greatly, and should wot (urn into a frustrating bute with the compiler and debugger. Finally, because this book is teaching algorithms, not 2 programming language, it should be reasonably easy to fate an algorithim to a variety of languages that reuders might wish to use, und special language features should be minimized. ‘Java showed up well by several of our criteria, although we would not clvine it is ideal. 1t supports data abstraction naturally. Itis type-safe, meaning that objects of one type cannot be used in operations intended lora different type; arbitrary type conversions (called casts”) are not permitted, either, There is an explicit bootean type. so if one types (the assigament operator) when “==" (the equality operutor) was intended, the compiler catches it Java does not permit pointer manipulations, which are a frequent source of obscure ‘errors; in fact, pointers are hidden from the programmer and handled automatically bebind the scenes, At run time, Java checks for ou-oF-range array subscripts, and other incon- sistencies that might be other sources of obscure errors. Ht performs “gurbage collection,” which means that it recycles the storage space of ubjects that are no longer referenced; this takes a big burden of space management off the programmer,1.2 Java as an Algorithm Language Int getMin(PriorityQ pq) tells us that getMin takes one parameter of type (or elass) Priority and returns type int. Java has a few primitive types and all remaining types are called classes. The primitive types are logical (boolean) and numerical (byte. char, short, int, tong, float, and double} types. All classes (nonprimitive types) in Java are reference classes, Behind the scenes, variables declared in classes are “pointors”; their values are adresses. Instances of classes are called objects, Declaring a variable does not create an object. Generally, objects are created with a “new” operator, which returns a reference to the new object. The data fields of an object are called instance fields in object-oriented terminology. The binary dot operator is used to access instance fields of an object. Example 1.1. Creating and accessing Java objects For this example, let’s assume that date information has the following nested logical structure: * year + number + isleap + month + day That is, using informal terminology. year is a compound attribute that consists of the boolean attribute isLeap and the integer attribute number, while month and day are simple integer attributes, To reflect this nested stracture, we have to define two classes in Java, one for the whole date and another for the year field. Assume we choose the names Date and Year, respectively, for these classes. Then we would declare number and isLeap as instance fields in the Year class ane declare year, month, and day as instance fictds in the Date class. Moreover, we would most likely define Year as an innes class of Date. ‘The syntax is shown i Figure 1.1 class Date { public Year year; public int month; public int day; public static class Year t public int number; public boolean isLeap; ? Figure 1.1 Java syntax for the Date class with an inner Year chissChapter 1 Analyzing Algor ms and Problems: Principles and Examples ‘On the downside, Java has many of the same terse, cryptic syntax featurewof C. The object structure may force inefficiencies in time and space. Many Java constructs require greater verbosity than other languages, stich as C, for instance. ‘Although Java has many specialized features, the algorithms presented in this book avoid most of them, in the interest of being language-incependent. In fact, some steps within an algorithm may be stated in pseudocode for easier readability. This section de- scribes a small subset of Java that we use for the book, and the pseudocode conventions that we use to improve readability of the algorithms, The Java-specific Appendix A gives some additional implementation details for readers who want (0 get a Java program running, but these dotails are not pertinent to understanding the main text. 1.2.1. A Usable Subset of fava A thorough acquaintance with Java is not important 10 understand the algorithms in this text. This section gives a brief overview of the Java features that do appear, for those readers who wish (o follow the implementation issues closely. In some cases we point out object- oriented features of Java that might be used, but which we avoid so that the text can be fairly language-independent: this is mainly for the benefit of readers who are familiar with some other object-oriented language, such as C++, but who are not completely familiar with Java, A sample Java “main program” appears in Appendix A. Many books are available for in-depth coverage of the Java language, Readers who are well acquainted with Java will undoubtedly notice many instances in which some nice Java feature could have been used. However, the concepts behind the algorithms do net require any special features, and we want these concepts to be easy to grasp and apply in a variety of languages, so we leave it to the readers, once they have Brasped the concepts, to tailor the implementations to th Tanguage. Readers familiar with C syntax will recognize many similarities in Java syntax: Blocks are delimited by curly braces, “{" and °°; square brackets, “f” and “J”, enclose array indexes. As in C and C+. a two-dimensional array is reatly a one-dimensional array ‘whose clements are themselves one-dimensional arrays, so (wo pairs of square brackets are needed to access an element, as in “matrixil[}". Opermors "==", " are the keyboard versions of the mathematical relational operates "= examples use the “#4” and“=~" operators 0 iacemen an deren but never use them embedded in other expressions. There are also the operators "+ adopted from C, For example, ptaq, /+Add qtop. +/ Y= x; // Subtract x from y. As just illustrated, comuments extend from °//™ to end-of-line, or from “/#" to "#/". as in CH. ‘ Funezion headers normally look the same in Java as in C. The function header specifies the parameter ixpe signature in parentheses after the Function name; it specifies the rerun Axpe before the function name, The combination of return type and parameter type signature is called the function's fill type signature, or protonype. ThusChapter 1 Analyzing Algorithms and Problems: Principles and Examples Without the public keyword, the instance fields woutd not be accessibte outside of the Date and Year classes; for simplicity, we make them public here. The reason for declaring the inner class, Year, (o be static is so we can create an instance of Year that is not associated with any particular Date object. All inner classes will be static in this book. Suppose we have created a Date object that is referenced by variable dueDate, To access the instance fietd year in this object, the dot operator is used, us in “dueDate.year.” It the instance field is in a cluss (as opposed to being in a primitive type), then further dot operators access its instance fields, as in “dueDate.year.isLeap” The assignment statement copies only the reference, or address, of an object in a " ure usually written, instead of their keyboard versions, Relational operators are used on types where the meaning is clear, such as String, even though this would be invalid syntax in Java, 6. Keywords, which are either reserved words or stundard parts of Java, are set in this font: int, String. Comments are set in this font. Code statements and program variable names are set in this font. However, pseudocode statements are set in the regular font of the text, like this sentence. Occasional departures from this scheme occur when we are making & specific poimt about the Java language. Mathematical Background We use a variety of mathematical concepts, tools, and techniques in this book. Most should already be familiar to you, although a few might be new. This section collects therm 10 provide a ready reference, as well as a brief review. Proof concepts are covered in greuter depth in Chapter 3. 1.3.1 Sets, Tuples, and Relations is section provides informal definitions and a few elementary properties of se selaied concepts, A set is a collection of distinel elements that we wish to real ds a single object. Usually the elements are of the sume “type” a and have some uddilional common W1.3 Mathematical Background follows: The first element of the sequence can he chosen from any element of S, so there are n choices. Then the second element of the sequence can be chosen fram any remaining element of 5, so there are (11 — 1) choices for this, and so on until k elements are chosen, (if k > 1 it is impossible to make K distinct choices, so the result is 0.) Therefore there are n(n — 1) +++ (n= & + 1) distinet sequences of distinct elements. But we saw that a specific set of & elements can he represented us k! sequences. So the number of distinct subsets of k, drawn from a set of n, is. m\ neko )asabapachs Coys ( Gane Since every subset must have some size from 0 through 7, we arrive at the identity zy () a2. 1.2) =o forn> k20. (1) Tuples and the Cross Product A tuple is a finite sequence whose elements often do not have the same type. For example, in a two-dimensional plane, a point can be represented by the ordered pair (x, y). If itis a geometric plane, x and y are both “length” But if it is a plot of running time vs. problem size, then ¥ might be seconds and x might be an integer. Short tuples have special names: pair, triple, quadruple. quintuple, and so on, In the context of “tuple” these are unclerstood to be ordered; in other contexts “pai” might mean “set of two” instead of “sequence of two," and so on. A k-uple is a mple of & elemen ‘The cmss product of two sets, say S and 7’ is the set of pairs that can be formed by choosing an element of $ as te first element of the tuple and an clement of F as the second, In mathematical notation we have SxTHiG lees yeT}. 0.3) Therefore |S x T] = [S47 1. It often happens that 5 and 7 are the same set. but this is not necessary. We can define the iterated cross product to produce longer tuples. For example, ST x U isthe set of all triples formed hy taking aa element of S, followed by an clement of 7, followed by an clement of U. Relations and Functions A relation is sinaply some subset of a (possibly iterated) cross product. This subset might be finite or infinite, and can be empty or the entire cross product. The most important case is a binary relation, which is simply some subset of a simple cross product. We are all familiar with many examples of binary relations, such as “less than” on the reals, Letting R denote the set of all reals, the “less than” relation can be defined formally as {Oxy} fr ER. y € Ry x < yl). As we see, this is a subset of R x R. As another example. if P isthe set of all people, then P x P is the sct of all pairs of people, We can define “parent of as (x. y) such that x is a parent of y, “ancestor of” as (4, ») stich that x is an ancestor of y, and these are subsets of P x P, . 13Chapter 1 Analyzing Algorithens and Problems: Principles and Examples properties that make it useful to think of them as one object, The notation ¢ ¢ S is read “element ¢ is a member of set 5” of, briefly, “e is in $.” Notice that ¢ and S'are different types in this case. Por example, if ¢ is an integer, S is.a ser of integers, which is different from being an integer A particular sot is defined by listing or describing its elements between a pair of curty braces. Examples of this notation are Si = {a,b,c}, Sa= {1 [x is an integer power of 2}, Spe (hens). ‘The expression for Sp is read “the set of all clements x such that x is an integer power of 2" The |" symbol is read “such that” in this context. Sometimes a coton (*:") is used in this place. The ellipsis“, .." may be used when the implicit elements are clear, Tf all clements of one set, S), are also in another set, S>, then 5; is said to be a subset Of Sz and S3 is said to be a superset of S;. The notations are Si C Sp and Sz D S;. To denote that Sy is a subset of S2 and is not equal to Sp, we write §| C Sz oF $2 > S). Its important not to confuse “e” with "C." The former means “is an element in” and the latter means “is a set of elements contained within.” The enppry set, denoted by @, has no elements, so it is a subset of every set. A set has no inherent order. Thus, in the above examples, 5; could have been defined as {b,¢, a}4ind $3 could have been defined as {é f 1 J and x > 0, log, x (read “log to the base b of x”) is that real number L such that bE = x; that is, log, x is the power to which & must be raised to getx. ‘The following properties of logarithms foltow easily from the definition. Lemma 1. Let.x and y be arbitrary positive real numbers, let a be any real number, and et > Lande > 1 be real numbers. logy is a strictly increasing function, that is, if > y, then tog, x > log, y. log, is a one-to-one function, that is, if log, x = log, y, then x = y, log, 1 =0. log, bf =a. log, (xy) = logy x + log, ». logy (x#) =a log, x. Hat a yl, PIAA ae ‘To convert from one base to another: tog,.x = (log, x)/(logy ¢). ve the log to the base 2 is used most often in computational complexity, there is a special notatis “tgs that is, 1g x = logy x. The natural logarithm (log to the basc ¢) is denoted by “In”; that is, Inx = log. When log() is used without any base being, mentioned, it means the statement is true for any base, Sometimes the logarithm function is applied lo itself. The notation Ig Ig(x) means Ig(le(z)). The notation Ig"??(x) means p applications, so Ig'?"(x) is the same as Ig le(x). Note that 1g)(65536) =: 2, which is quite different from (1g(65536))? = 4096. 151.3 Mathematical Background with “heads” facing up or with “tails” facing up. We let s1 = ‘heads’ and s) = ‘tails’ and assign Pr(s\) = 1/2 and Pr(sy) = 1/2. (If someone objects because the coin could land on its edge, we may let 5 = ‘edge’ and define Pr(sy) = 0. However, with a finite number of events, an event of probability zero can be ignored, so such elemtentary events are not usu- ally defined.) If a six-sided die is thrown, there are six poss 1 = “the die tands with side number { facing up." and Pr(s,) = 1/6, In general, if there are & possible outcomes and each is considered equally likely, then we let Pr(si) = 1/k for each i, Often, there is no reason to assume all outcomes are equally likely; primarily, this is an assumption used in examples or used because there.is no data to support a better assumption, If the experiment involves several objects, then an elementary event must take into 2ocount what is observed about all of them. For example, if two dice, A and B, are thrown, then the event “A lands with side 1 facing up” is not an elementary event because there are several outcomes associated with B. In this case, the elementary events would be 5; = “die A lands with side j facing up and die B lands with side j facing wp.” for | 3.32. The derivative. of In¢x) is 1/x. Using part 8 of Lemma 11, the derivative of Ig(x) is Ig(e)/x. Permutations A permutation of 1 distinct objects is a sequence that contains each object once. Let $= (51,52, +Sn}- Note that the elements of § are ordered by their indexes; that is, 51 is the first element, «2 the second, and so on. A permutation of 5 is a one-to-one function a from the set {1, 2...) onto itself. We think of = as rearranging S by moving the ith element, sifto the 2()th position. We may describe 2 simply by listing its values, that is, (r(1), (2). .... (a). For example, for n = 5, m = (4, 3, 1, 5,2) rearranges the elements of $ as follows: $3. $5, $2.51. 54. The number of permutations of n distinct objects is 1!, To see this, observe that the first element can be moved to any of the m positions; then that position is filled and the second element can be moved to any of the — J remaining positions; the third element can be moved to any of the remaining m — 2 positions, and so on, So the total number of possible rearrangements is n x (n ~ 1) x (1 =2) x... x2 x Laat Probability Suppose that in a given situation an event, or experiment, may have any one, and only one, of & outcomes, 51,52, »St- These outcomes are called elementary events, The set of all elementary events is called the universe and is denoted U. With Each outcome 5; we associate a real number Pr(s;), called the probability of s;, such that O 0, log(x) for x > 0, and ¢. Less familiar monotonic functions are [x] and [x], showing that monotonic functions need not be continuous. An antimonotonic example is 1/x for x > 0. Definition 1.8 Linear interpolation function ‘The linear interpolation of a given function f(x) between two points « and v, u < v, is the function defined by 21.3 Mathematical Background 2. A function f(a) defined on imegers is convex if and onty if, for any n,0 +1, n+2, Slat SEL) + fOr4D). In words, (7 + 1) is at most the average of f(n) and f(a +2). Lemma 1.4 summarizes several useful properties of monotonicity and convexity. It states that functions defined only on the integers can be extended to the reals by linear interpolation, preserving properties of monotonicity and convexily. Also, some properties involving derivatives are stated. The proofs are in Exercises 117 through 1,19. Lemma 14 1. Let f(n) be defined only on integers. Let f*(x) be the extension of f to the reals by linear interpolation between consecutive integers (see Figure 1.3b). a. f (2) is monotonic if and only if f*(x) is monotonic. b. f(2) is convex if and only if f(x) is convex, 2. I the first derivative of (x) exists and is nonnegative, then f(x} is monotonic. iW the first derivative of f(x) exists and is monotonic, then f(x) is convex. 4, If the second derivative of f (x) exists and is nonnegative, then f(x) is convex. (This follows from parts 2 and 3.) 0 e Summations Using Integration Several summations that arise often in the analysis of algorithms can be approximated (or bounded from above or below) using integration, First, let us review some useful integration formulas: fe drs peg [anton 1). , : \ (LIS) ‘ etl nen) = tH f at inte) d= pnt Inn) ay if f(x) is monotonic (or nondecreasing), then b +t S@drsT Os} feeds (1.16) ina Similarly, if f(x) is antimonotonic (or nonincrensing), then aa b b f fdx s Pi) sf fous. (ain 2524 Chapter 1 Analyzing Algorithms and Problems: Principles and Examples fo (a) Linear interpolation {b) Extension of f(1) to f*(x) Figure 1.3 Illustrations for convexity discussion: The function f is different in parts (a) and {b). In part (b), f*(2) is conver. @- Df) +O—wWfe) Lyal) = wow TIW+G = 1 LO=LO = py yn fP=, iy vom uae that is, the straight-line segment joining f(u) and f(v) (see Figure 1.3a), Definition 1.9 Convex functions A function f(x) is said to be convex if forall u mga-alge. Since Ige < 1.443, Dileiz aign — 1.4430 (1.18) as of the previous example, bul with more precise mathermathies, it is possible to derive Stirting's formula giving bounds for nt: (2) vam O0 (1.20) then ASC then A+CS84D then aA sab1.3 Mathematical Background bbe more like the logical form, and then ask yourself if it mewns the same thing in natural language. For example, “Any person must hreathe to live” might be the sentence you start with, Possible rephrasings are "For all peaple x must breathe to live and “For some person x, x must breathe to five.” Which means the same as the original sentence? Negating a Quantified Statement, Counterexamples What is necessary (0 prove that a general statement, say ¥e(A(x) => B(x)), is false? We can use the foregoing identities to clarify the goal. The first thing to realize is that it is nov necessary to prove Wx(A(x) = ~B(x)). This is tod strong a statement, The nogation of ¥r(ACx) => BCD) is ~Cex(ACe) => B(x})), which can be put throngh a series of transformations: A(¥x(A(X) => BLX))) is logically equivalent to 3x-(AC) = BEX) logically equivalent to Ax(—AGr) v BOX) (1.26) logically equivalent to Ax(AGx) A 7BQ))- Tn words, if we can exhibit some object x for which A(x) is true and B (3) is false, then we have proven that ¥x(A(x) => B(x) is false, Sucl an object (x) is called a counterexample. Contrapositives When trying to prove a statement, it is offen convenient to manipttate it into a logically equivalent form. One such form is the comrapositive, The contrapasitive of A => B is (-B) =+ (7A), Equation (1.21) allows us to verity that the eonrapasicie of an implication is true exactly when the implication itself is true: Axe Bis logically equivalent to (4B) => (>A). 27 Sometimes, proving the contrapositive of a statement is called “proof hy contra but “proof by contraposition” is a more acctirate description, The genuine “proof by con- tradiction” is described next. Proof by Contradiction Suppose the goal is to prove a statement of the form A => B.A genuine praf by eantradic- tion adlds an adklitional hypothesis of +B. and then proves B itself. That is, (A AB) => B is the Full statement that is proved. The following identity justifies this method: A= B- islogicully equivatentto (AA7B) = B. 1.28) ‘A genuine proof by contradiction is rare in algorithm analysis, However, Exercise 1.24 calls for onc. Most so-called proofs by contradiction are actually proofs by contraposition. Rules of Inference So far we have seen numerous pairs of logically equivalent statements. oF logical identities: ‘One statement is true if and only if the second statement is true. Klentities are “reversible: 2928 Chapter 1 Analyzing Algorithms and Problems: Principles and Examples 1.3.3 Elements of Logic . Logic is a system for formalizing natural language statements so that we can reason more accurately. The simplest statements are called atomic formulas, More complex statements can be built up through the use of fogieal connectives. Examples of atomic formulas are 4 > 32° "4.2 is am integer,” and “x + 1 > x.” Notice that a logical statement need not be truc. The objective of a proof is to show that a logical statement is true. The mast familiar logicat connectives are “A” (and). "V" (or), and "+" (not), which are also called Boolean operators. The truth value of a complex statement is derived fram the truth values of its atomic formulas, according to rules for the connectives. Let A and B bbe logical statements. Then, 1, AA Bis true if and only if A is true and & is true: 2. AY Bis true if and only if A is true of B is true, or both, 3, 7A is true if and only if A is false. Another important connective for reasoning is called “implies,” which we denote with the symbol “=>”, (The symbol “+” is also seen.) The statement A => B is read os “A implics BY or “if A then B.” (Notice that this statement has no “else” clause.) The “implies” oparator can be represented with a combination of other operators, according to the following identity: A => Bis logically equivalentto. —A vB. (121) This can be verified by checking alt combinations of truth assignments to A and B. Another useful set of identities are called DeMorgan’s laws: (AA B) is logically equivalem to AAV —B, (122) “(AV B) is logically equivalent to A AB. (1.23) rs. Quan Another important kind of logical connective is the qumrifier. The symbol Vs is called the universal quantifier and is read “for all x." while the symbol Ax is called the existential quantifier and is read “there exists x." These connectives can be applied to statements that ‘contain the variable x, The statement ¥x P(x) is true if and only if P(x) is true for ali x. The statement 3x P(x) is tre if and only if P(x) is true for some value of x. Most frequently. a universally quantified statement is conditional: Wx (A(x) =» B(x)), This ean be read "For all x such that A(x) holds, B(x) holds.” ‘Quantified statements obey a variation on DeMorgan's laws: VrA(x) is logically equivalent to -3x(=A(x)), (1.24) 3xA(x) is logically equivalent i “Va(A(x))- (1.25) Sometimes the translation from natural Language into a quantified statement is trouble- some. People don’t speak in the stilted language of logic, usually. We need to realize that “for any 2” usually means “for all x," although “any” and “some” are often interchangeable in normal speech. The best guideline is to try rephrasing a sentence in matural language to30 Chapter 1 Analyzing Algorithms and Problems: Principles and Examples Most proofs ive directed at “irreversible” combinations of statements, however The com plete statement 10 be proved is of the form “if kxpotteses, then conelasion.” The reversal, “it conclusion, then hypotheses” is often of true. Logical identities are not flexible enough o prove such “if-then” statements, In these situations, we need rales of inference. A rule of inference is a generat pattern that allows us to draw some new conclusion from a set of given statements, It can be stated, “If we know By, ..., Be, then we can conclude C2" where Bi... Bas and C ure logical statements in their own right, Heve are a few well-known rules: Mowe know then we can conclude B and BC c (1.29) A+B und BSC Ave (1.30) Bac md BSC OC a3) Some of the these rules are known by their Greek of Latin names, Equation (1.29) is modus is sy/fogisin, and Equation (1.31) is the rule of cases. These rules ‘are not independent; in Exercise 1.21 you will prove the rule of cases using other rules of inference und logical identities. , Analyzing Algorithms and Problems We analyze algorithms with the intention of improving them, if possible, and for choosing mong several available fora problem. We will use the following criteria: Convetness Amount of work done Amount of space used Simplicity, etarity Optimality yRepe We will discuss each of these criteria at length and give several examples of their appli- cation, When considering the optimatity of algorithm uroduce techniques for estublishing lower bounds on the complexity of problems, 1.4.1 Correctness ‘There are three nxajor steps involved in establishing the correctness of an algorithm, First, belive we can even attempt to determine whether an ulgorithm is correct, we must have a clear understanding of what “correct” means, We need a precise statement about the characteristics of the inpuis it is expected to work on (called the preconditions), and what result it is to produce for each input (calted the pastcenditions). Then we can try to prove statements about the relitionsbips between the input and the output, that is, that if the. preconditions are sitisfied, the postconditions will be true when the algorithm terminates. “There are two aspects 40 an algorithm: the sulution method! and the sequence of instructions for carrying it out, that is, its implementation, Establishing the correctness of1.4 Analyzing Algorithms and Problems the method and/or formulas used may be easy or niay require a long sequence of lemmas and theorems about the objects on which the algorithm works (e.g., graphs, permutations, matrices). For example, the vatidity of the Gauss elimination method for solving systems of linear equations depends on a number of theorems in linear algebra. Some of the methods used in algorithms in this book are not obviously correet; they must be justified by theorems. Once the method is established, we implement it in a program. Ifan algorithm is fairly short and straightforward, we generally use some informal means of convincing ourselves that the various parts do what we expect thei to do. We may-check some details carefully (e.g. initial and final values of loop counters), and hand-simulate the algorithm on a few small examples. None of this proves that itis correct, but informal techniques may suffice for small programs, More formal techniques, such as loop invariants, may be used to verify correctness of parts of programs. Section 3.3 expands upon this topic. ‘Most progranis written outside of classes are very large and very complex. To prove, the correctness of a large program, we can ity (0 break the program down into smaller modules; show that, if all of the smaller modules do their jobs properly, then the whole program is correct; and then prove that each of the modules is correct. This task is made easier if (it may be more accurate 10 say, “This task is possible only if”) algorithons and programs are written in modules that are largely independent and can be verified separately. This is one of the many strong arguments for structured, modular programming, Most of the algorithms presented in this book are the small segments from which large programs ‘are built, $0 we will not deal with the difficulties of proving the correctness of very long algorithms or programs. We will not always do formal proofs of correctness in this book, though we will give arguments or explanations to justify complex or tricky parts of algorithms. Correctness can be proved, though indeed for tong and complex programs it is a formidable task. In Chapter 3 we will introduce some techniques to help make proofs more manageable. 1.4.2. Amount of Work Done How shall we measure the amount of work dane by an algorithm? The measure we chaose should aid in comparing two algorithms for the sume problem so that we can determine whether one is more efficient than the other. It would be handy if our measure of work ‘gave some indication of how the actual execution times of the two algorithms compare, but we will not use execution time as a measuce of work for a number of reasons, First, of course, it vasies with the computer used, and we don’t want to develop a theory for ‘one particular computer. We may instead count alt the instructions or statements executed by a program, but this'measure still has several of the other faults of execution tinne. ft is highly dependent on the programming language used and on the programmer's style. It would also require that we spend time and effort writing and debugging programs for each algorithm to be studied, We want a measure of work that tells us something about the eificiency of the method used by an algorithm independent of not only the computer, programming Janguage, and programmer, but also of the many implementation details, overhead (or “bookkeeping” operations) such as incrementing loop indexes, computing 3114 Analyzing Algorithms and Problems proportional to the number of basic operations, just céunting the latter can give us a pretty clear idea of how feasible it is to use the algorithm on large inputs. Finally. this choice of the measure of work allows a great deal of flexibility. Though we will often try to chaose one, or at mast two, specific operations fo count, we cautd include some overhead operations, and, in the extreme, we could choose as the basic operations the set of machine instructions for a particular computer. At the other extreme, we could consider “one pass through a loop” as the basic operation. Thus by varying (he choice of basic operations, we can vary the degree of precision and abstraction in our analysis to fit ‘our needs, - ‘What if we choose a basic operation for a problem and then find that the total mumber ‘of operations performed by an algorithm is not proportional to the number of basic oper- ations? What if it is substantially higher? In the extreme case, we might choose a basic operation for a certain problem and then discover that some algorithms for the probfem use such different methods that they do not do any of the operations we are counting, In such a situation, we have two choices. We could abandon our focus on the particular operation and revert to counting passes through loops. Or. if we are especially interested in the par- ticular operation chosen, we could restrict our study to a particular class of afgarithms, one for which the chosen operation is appropriate. Algorithins that use other techniques for which a different choice of basic operation is appropriate could be stuclied separately. A class of algorithms for a problem is usually defined by specifying the operations that may be performed on the data, (The degree of formality of the specifications will vary; usually informal descriptions will suffice in this book.) Throughout this section, we have often used the phrase “the amount of work done by an algorithm." It could be replaced by the term “the complexity of an algorithin.” Conplexity means the amount of work done, measured by some specified complexity measure, which in many of our examples is the number of specificd basic operations performed. Note that, in this sense, complexity has nothing to do with how complicated or tricky an algorithm is; a very complicated algorithm may have low complexity, We will use the terms “complexity,” “amount of work done,” and “number of basic operations cone” almost interchangeably in this book. 1.4.3. Average and Worst-Case Analysis ‘Now that we have a general approach to analyzing the amount of work done by an algo- rithm, we need a way to present (he results of the analysis concisely. The amount of work done cannot be described by a single number because the number of steps performed is not the same for all inputs, We observe first that the amount of wark clone usually depends on the size of the input, For example, alphabetizing an array of 1000 names usually requires more operations than alphabetizing an array of 100 names, using the same algorithm. Solv- ing a system of 12 linear equations in 12 unknowns generally takes more work than solving a system of 2 linear equations in 2 unknowns. We observe, secondly, that even if we con- sider inputs of only one size, the number of operations performed by an algorithm may depend on the particular input, An algorithm for alphabetizing an array of names may do very little work if only a few of the names are ont of order, bul it may have to do much 3332 Chapter 1 Analyzing Algorithms and Problems: Principles and Examples array indexes, and setting pointers in data structures. Our measure of work'should be both precise enough and general enough to develop a rich theory that is useful for many algorithms and applications. A simple algorithm may consist of some initialization instructions and a loop. The number of passes made through the body of the loop is a fairly good indication of the work done by such an algorithms. Of course, the amount of work done in one pass through a loop may be much more than the amount done in another pass, and one algorithm may have longer loop bodies than another algorithm, but we are narrowing in on a good measure of work, Though some loops may have, say, five steps and some nine, for large inputs the number of passes through the loops will generatly be large compared to the loop sizes, ‘Thus counting the passes through alt the loops in the algorithm is a good idea, In many cases, to analyze an algorithm we can isolate a particular operation funda- ‘mental to the problem under study (or to the types of algorithms being considered), ignore initialization, loop control, and other bookkeeping, and just count the chosen, or bs operations performed by the algorithm. For many algorithms, exactly one of these opera- ions is performed on each pass through the main loops of the algorithm, so this measure is similar to the one described in the previous paragraph. Here are some examples of reasonable choices of basic operations for several prob- lems: Problem Operation Find x in an array of names. Comparison of x with an entry in the array Multiply two matrices with real entries. Multiplication of two real numbers (or multiplication and addition of real numbers) Sort an array of numbers, ‘Comparison of two array entries Traverse a binary tree (see Traversing an edge Section 2.3.3). Any noniterative procedure, including Procedure invocation * recursive So jong as the basic operation(s) are chosen well and the (otal number of operations performed is roughly proportional to the number of basic operations, we have a good measure of the work done by an algorithm and a good criterion for comparing several algorithms. This is the measure we use in this chapter and in several other chapters in this book. You may not yet be entirely convinced that this is a good choice; we will add more Justification for it in the next section. For now, we simply make a few points. First, in some situations, we may be intrinsically interested in the basic operation: 11 might be a very expensive operation compared to the others, or if might be of some theoretical interest. Second, we are often interested in the rate of growth of the time required for the algorithm as the inputs get larger. So long as the total number of operations is roughly34 Chapter? Analyzing Algorithms and Problems: Principles and Examples more work on an array that is very scrambled, Solving a system of 12 linear equations may not require much work if most of the coefficients ave zero. The first observation indicates that we need a measure of the size of the input for a problein. Itis usually eusy to choose a seasunable measure of size. Here are some examples: Problem Size of input Find x in an array of names “The number of nanves in the array ‘Multiply wo matrices. ‘The dimensions of the matrices Sort an array of numbers. ‘The number of entries in the array “Traverse a binary wee. ‘The number of nodes in the tree Solve a system of linear equations. ‘The oumber of equations, or the number of unknowns, or both Solve a problem conceming a graph. ‘The number of nodes in the graph, or the number of edges, or both Even if the input size is fixed at, say, 1, the number of operations performed may depend on the particular input, How, then, are the zesults of the analysis of an atgorithm to be expressed? Most often we describe the behavior of an algorithm by stating its worst-case complexity. Definition 1.10 Worst-case complexity Let D,, be the set of inputs of size n for the problem under consideration, and let # be an clement of Dy. Let r(J) be the number of basic operations performed by the algorithm on input 1. We define the function W by Wn) = max (|e D, ‘The function Wn) is called the worst-case complexity of the algorithm. W(a) is the maximum number of basic operations performed by the algorithm on any input of size 1. : " tis often not very difficult to compute (1), Section 1.5 introduces techniques for ‘cases where an exact computation would be difficult. The worst-case complexity is valuable because il givesan upper bound on the work done by the algorithm, The worst-case analysis ‘could be used (0 help form an estimate for a time limit for a particular implementation of an algorithm, We will do worst-case analysis for most of the algorithms presented in this book. Unless otherwise stated, whenever we refer to the amount of work done by an algorithm, we mean the umount of work done in the worst ease, In may seem that a more useful and natural way to describe the behavior of an algorithin is (0 ell how much work itdoes on the average: that is, (6 compute the number of operations performed for each input of size und then take the average. In pructice some inputs might ‘occur much more frequently than others so a weighted average is more meaningful,1.4 Analyzing Algorithms and Problems Definition 1.11 Average complexity Let Pr(J) be the probability that input occurs. Then the averuge behavior of the algorithm is defined as Aq) = So Pre. 16D ‘We determine *(7) by analyzing the algorithn, but Pr() cannot be computed analyt- ically, The function Pr(Z) is determined from experience and/or special information about the application for which the algorithm is (0 be used, or by making some simplifying as- sumption (e-g,, that all inputs of size 2 are equally likely t0 occur}. If Pr(J) is complicated, the computation of average behavior is difficult. Also, of course, if Pr(Z) depends on a particular application of the algorithm, the function A describes the average behavior of the algorithm for only that application. “The following examples illustrate worst-case and average analysis. Example 1.9 Search in an unordered array Problem: Let € be an array containing 1 entries (called keys), E(O], .... E{n1], in no particular order, Find an index of a specified key KX, if K is in the array; return —1 as the answer if K is not in the array. (The problem in which the array entries are in order is studied in Section 1.6.) Strategy: Compare K to each entry in turn until a match is found or the array is ex- hausted. If K is not in the array, the algorithm returns ~1 as its answer. ‘There is a large class of procedures similar to this one, and we call these procedures generalized searching routines. Often they occur as subroutines of more complex proce- dures, Definition 1.12 Generalized searching routine A generalized searching routine is a procedure that processes an indefinite amount of data until it either exhausts the data or uchieves its goal. {follows this high-level outline: if there is no more data to examine: Fail. else Examine one datum. Jf this dutum is what we want: Succeed, else Keep searching in remaining data. The scheme is called generalized searching because the routine often performs some other simple operations as it searches, such as moving data elements, adding 10 oF deleting from a data structure, and soon, 351.4 Analyzing Algorithms and Problems - Finally. we combine the cases in which K is in the array and is notin the array. Let q be the probability that X is in the array. By the law of conditional expectations (Lemma 1.2): Alst) = Pr(suce) Arg e(u) + Pr(faildA paiitn) =a (feeb) + -anand -datha. that is, # K is always in the array, thon A(n) = (n+ 1)/2, as before. Iq = 1/2, that is, iF there is a 50-50 chance that X is not in the array, then A(n) =3.n/4 + 1/4 roughly three-fourths of the entries are examined. This concludes Example 1.9. Example 1.9 illustrates how we should interpret D,, the set of inputs of size a. Rather than consider all possible arrays of names, numbers. or whatever, that conle occur as inputs, ‘we identify the properties of the inputs that affect the behavior of the algoritinm; in this case, whether X is in the array at all and, if sa, where it appears, An element J in Dy may be thought of as a set (or equivalence class) of all arrays and values for K such that K occurs in the specified place in the array (or not at all). Then #(/) is the number of operations dane for any one of the inputs in 7. ~ Observe also that the input for which an algorithm behaves worst depends on the particular algorithm, not on the problem. For Algorithm £1 a worst case occurs when the only position in the array containing X is the last. For an algorithm that searched the array backwards (i., beginning with index =a ~ 1), a worst case would occur if K appeared only in position 0. (Another worst case would again be when X is notin the array at all.) Finally, Example 1.9 illustrates an assumption we often make when doing average analysis of sorting and searching algorithms: that the elements are distinct, The average amalysis forthe case of distinct elements gives a fair approximation for the average behavior in cases with few duplicates. If there might be many duplicates, it is harcler to make seasonable assumptions aboul the probability that K's first appearance in the array occurs at any particular position, Example 1.10 Matrix multiplication Problem: Let A = (aj) be anm x n matrix and B = (b;,) be ann x p matrix, both with real entries. Compute the product matrix C = AB. (This problem is discussed much more thoroughly in Chapter 12. In many cases we assume the matrices are square, that is, #7 = 98 and p =n.) Strategy: Use the algorithm implied by the definition of the matrix proxluct: mt c= Dos for si cm, OS jf 1, the number of entries Output: Returns max, the largest entry in E. int findMax(E, n) max = £0]; ‘ 2. for (index = 1; index 0 and some nonnegative integer constant sr, fn) seat) foralln> no. Lis offen useful to think of g as some given function, and fas the function we are analyzing. Notice that a function f may be in O(g) even if f() > g(r) for all m, The important point is that f is bounded above by some consiam multiple of g. Also, the relation between f and g for small values of n is not considered. Figuse 1.6 shows the order relations for a few functions. (Note that the functions in Figure 1.6 are drawn ax continuous functions defined on R* or R*. The functions that describe the behavior of most of the algorithms we will study have such natural extensions.) The set O(g) is usually called “big oh of g” or just “oh of g” although the “oh” is acwwally the Greek letter omicron. And, although we have defined O(g) as a set, itis common practice to say “f is oh of g." rather than fis a member of oh of g. ‘There is an alternative technique for showing that f is in O(g): fy ictudi i Lemma L$ A fonction f € O(8) if jim, <7 = <0, cluding the ease in which ato g(a he limit of the ratio of f to g exists and isnot oo, then f grows no faster than g. Ifthe limit is 00, then f does grow faster than g. 45a4 Chapter 1. Analyzing Algorithms and Problems: Principles andl Examples ‘overhead operations. The rate of growth ofa cubic function is so much greater than that of a quadratic fnction that the constant of proportionality doesn't matter when # gets large. As these examples suggest, we want a way to compare or classify functions that ignores constant factors and small inputs. We get just such a classification by studying what is called the asymptotic growth sate, asymptotic order of, simply, the oder of functions, reasonable to ignore constants and small input values? Flere is a completely nor technical, nonmathematical analogy that may help you understand our use of asymptotic order, Suppose you are choosing a city to live in and your main eriterion is that it have a very hot climate, The choices are El Paso, Texas. ant Yuma, Arizona, There's not murch difference in temperature between them, is there? But suppose you are choosing between three cities: El Paso. Yuna, and Anchorage, Alaska. You'd rule out Anchorage immedi- ately. This is analogous (o saying two functions are of the same order, and a third one is of a different order. Knowing the order lets us make broad distinctions: we can eliminate those that are poor by our criterion. Now, how will you choose between El Paso and Yuma (or (wo algorithms whose running time is of the same order)? We could look up temperature records to find out that temperatures in one city average a few degrees higher than the other. This might be analogous 10 Tooking at the constant on two functions of the same order; for algorithms it might mean counting all operations. including overhead to get a more precise estimate of running tie. Another approach would be to consider other criteria, perhaps availability of jobs ane cultural amenities when choosing a city, or the amount of extra space used when choosing an algorithm, Is there ever a day when itis warmer in Anchorage than in El Paso? Sure; there might be a beautiful, unusually warm spring day in Anchorage when a cold front is passing through El Paso. This doesn’t make it wrong to say, in general, that Anchorage is much older than Ei Paso, In the definitions we will give for big ob, big theta, and the other “order sets,” the behavior of the functions being compared is ignored for smail values of n. Ignoring some simali arguments (input sizes, for algorithms) is analogous to ignoring the few days when Anchorage might be warmer than El Paso or Yuma, 1.5.1 De ions and Asymptotic Notation We will use the usual notation for natural numbers and real numbers. Defi 1.13 Notation for natural numbers and reals {0,1 2.3.00.) 3. de ‘The set of naneral numbers is denoted as N ‘The set of positive integers is denoted as Nt ‘The set of real numbers ix denoted as R. The set of positive reals is denoted as R*. ‘The set of nonnegative reals is denoted as R"s yeene Let f and g be functions from N to R*. Figure 1.5 informally describes the sets we use to show the relationships between the orders of functions, Keeping the picture and46 Chapter 1 Analyzing Algorithms and Problems: Principles and Examples Figure 1.6 The orders of functions: fs € O(f4), even though fa(x) > fa(x) for x > 4, since ott ate linear, f and fo are of the same order. They grow faster than the other three functions. sis of the lowest order union the functions shown. Example 1.13 Functions of different asymptotic orders Let f(t) = n3/2 and gn) =37n? + 120 + 17. We will show that REO), bu fe Ol). 2 for n > 78, g(a) mp,” nt 2 yf Men? + 120en + Te. So.1.5 Classifying Functions by Their Asymptotic Growth Rates fesse e+ OE <174e, Since ¢ is a constant and n may be arbitrarily large, itis impossible to have n/2 < 17de for alin 2H ‘The following theorem is useful for computing limits when f and g extend to contin uous, differentiable functions on the reals. ‘Theorem 1.6 (L7Hépital’s Rule) Let f and g be differentiable functions, with derivatives Jf and g?, respectively, such that dim, £2) = lim, gin) = Then tim £0 2 tig £00 we glu) 449 sn)” Example 1.14 Use of L’Hépital’s Rule Let f(n) =n? and g(n) =a lg n. We will show that f ¢ O(g), but g € O(f). Fist, we simplify, FO) ig tim i gen) a ign iS ign Now we note (see Lerma 1.1) that 1g = In(n)/ In(2) in preparation for using L’ Hpital’s Rule: W) _ . = fing, Typ = ig, Mn) = 00. ‘Therefore f ¢f Og). However, g € O(f) since the inverse ratio goes to 0. m= ‘The definition of 92(g), the set of functions that grow at least as fast as g, is the dual of the definition of O(g)? Definition 1.15 The set 2(g) Let g be a function from the nounegative integers into the positive real numbers, Then 2(g) is the set of functions f, also from the nonnegative integers into the positive real sumbers, such that for some real constant ¢ > O-and some nonnegative integer constant zo, PO) 2 cg(n) foralla> ay. ‘The alternative technique for showing that f is in Q(g) is as follows: 2 Readers who plan to consul other books and papers should be aware thatthe definition of &anay vary slighly: ‘The phrase “forall” may be weakened to “for inlnitely many” The definition of © shifts wecordingly. 471.5 Classifying Functions by Their Asymptotic Growth Rates 49 Algorithin 1 2 5 “Time function (microsce.) 33" dorige eBay oF 5 Input size n) Solution time } 180033 sec. 01S sec, SONA sec, OOM see, 008 ste. WOR see, FEE, Nate, Bee, LO ye 1.000, 038 sec. AS see. 1S ec.” 4h 10,000 A3sce. BL see, 22min days 106.000 33sec.“ L3min,—ASdays 108 yr Time allowed ‘Moxinuon solvable input sce Copprox) Vsecond 30.00 -2.000 280 “ R8 minute 1.800.900 82,000 2.200260 Table 1.1 How functions grow 1.5.2. How Important Is Asymptotic Order? Table 1.1" shows the running times for several actual algorithms for the same problem, (The last column does not correspond to an algorithm for the problem; it is included to demonstrate how fast exponential functions grow, and hence how bad exponential algo- rithms are.) Look over the entries in the table to sec how fast the running time increases with input size for the algorithms of higher complexities, One of the important lessons in the tabie is that the high constant factors on the @(n) and @(r log n) algorithms do not make them slower than the other algorithms except for very small inputs. The second part of the table looks at the effect of asymptotic growth rate on the increase in the size of the input that can be handled with mare computer time (or by using a faster computer). Itis not true in general that if we multiply the time (or speed) by 60 we can handle an input 60 times as large; that is true only for algorithms whose complexity is in O(n). The O(n) algorithm, for example, can handle an input only /60 times as large. To further drive home the point that the asymptotic order of the running time of an algorithm is more important than a constant factor (for large inputs), look at Table 1.2, A program for the cubic algorithm from Table 1.1 was written for the Cray-1 supercomputer; it ran in 343 nanoseconds for input of size n. The linear algorithm was programmed on a ‘TRS-80 (an inexpensive 1980s personal computer); it ran in 19,51 milliseconds (which is 19,500, 0001 nanoseconds). Even though the constant on the linear algorithm is 6.5 riilion times as big as the constant on the cubic algorithm, the linear algorithm is faster 3This mble (except the Fst columa) and Table 1.2 are adyted from Programming Pearis by Jon Bentley (Addison-Wesley, Reading, Mast, 1986) and are reproduced ere with permission.48 Chapter t Analyzing Algorithms and Problems: Principles and Examples Lemma 1.7 Function f € 9(@) flim, 2 > 0, inctuding the case in which the limit isco. a Definition 1.16 - The set @(g), asymptotic order of ¢ Let g be a function from the nonnegative integers into the positive real numbers. Then Og) = Og) M Qe), that is, the set of functions that are in both O¢g) and $2(g). The most common way of reading “f € ©(g)" is "f is order g.” We often use the phrase “asymptotic order” for definiteness, and the term “asymptotic complexity” is also seen, a ‘We also have: Lemma 18 Function f ¢@(g) it tim £2 whee gto) ¢ for some constant ¢ such that O 2500, (Whether one considers this a large or small input size would depend on the context of the problem.) Uf we focus on the asymptotic order of functions (thus inctuding, say, » and 1,000,000 1 in the same class), then when we can show that two functions are not of the same order, we awe making a strong statement about the difference between the algorithms described by those functions. if wo functions are of the same order, they may differ by a large constant factor. However, the value of the constant is irrelevant in determining the effect of a faster computer on the maximum input size an algorithm can handle in 2 given amount of time. That is, the value of the constant is irrelevant (o the increase between the last two rows of ‘Table I.1. Let's look a little more closely af the meaning of those numibess, Suppose we fix on a certain amount of time (one second, one minute—the specific choice is unimportant), Let s be the nuaximum input size a particular algorithm can handle within that amount of time. Now suppose we allow ¢ times as much time (ar our computer speed increases by a factor ofr, either because technology has improved, or simply because we went out and bought a more expensive machine). Table 1.3 shows the effect of the speedup for several cumptexities.1.5 Classifying Functions by Their Asymptotic Growth Rates ‘The values in the third column are computed by observing that ‘F Gree) = number of steps after speedup 4+ (number of steps before the speedup) = rf (3) and solving SCaew) =4f(S) for Snew~ “ ‘Now, if we multiply the functions in the first column by sonte constant c, the entries in the third column will not change! This is what we meant by saying that the constant is irrelevant to the effect of increased computer time (oF speed) on the maximum input size an algorithin can handle. 1.5.3 Properties of O, 2, and © ‘The order sets have a number of useful properties. Most of the proofs are left as exercises; they follow easily from the definitions, For all the properties, assume that f, gh :N—> R* ‘That is, the functions map nonnegative integers into nonnegative reals. Lemma 19 If f € O(g) and g € O(h), then f € O(h); that is, O is transitive, Also, 2, ©, 0, and « ave transitive, Proof Let cy and sty be such that f(n) < e1g(n) for ull > a1, and let cz and ng be such that g(t) 2. Then for all > max(ti, ta), £0) < creah(n). So f € O(h). The proofs for 2, ©, 0, and w are similar. oF Lemma 1.10 Lf € O(g)if and only if g € 2¢f) 2. fe Ole), then ge OC/). 3. @ defines an equivalence relation on the functions. (See Section 1.3.1 for what needs 0 be shown.) Each set @( f) is an equivalence class, which we call a complexity class. 4. OCF +g) = Olomaxf, g)). Similar equations hold for 2 and ©. (They are usefut when analyzing complex algorithms, where f and g may describe the work done by different parts of the algorithm.) Since © defines an equivalence relation, we can indicate the complexity class of an algorithm by specifying any function in the class. We usually choose the simplest representative. Thus if the number of steps carried ont by an algorithm is described by the function f(r) = 3/6 +n? + 2 Ign + 12, we say simply that the complexity of the algorithm is in ©(n3). If f € @(n), we say that f is linear, if f € Ol), we say f is st52 Chapler 1 Analyzing Algorithms and Problems: Principles and Examples quadratic; and if f € @Gx"), f is cubic.4 O(1) denotes the set of functions bounded by a constant (for large n). Here are two useful theorems. The proofs use the techniques presented in Sec- tion 1.5.1, especially L' Hopital’ rute; they are left for exercises. Theorem 1.11 _Igirisino(n®) for any e > 0. That is, the bog function grows more slowly than any positive power of » {including fractional powers). 0 ‘Theorem 1.12 nf isin 0(2") for any k > 0. That is, powers of 1 grow more stowly than the exponential function 2”. (In fact, powers of m grow more slowly than any exponential function ¢” where e > 1.) 0 1.5.4 The Asymptotic Order of Commonly Occurring Sums Order notation makes it easy to derive and remember the asymptotic order of many sums that come up over and over in the analysis of algorithms. Some of these summations were defined in Section 1.3.2. ° Theorem 1.13 Let d be a nonnegative constant and let r be a positive constant not equal tol. 1. The sum of a pofynomiat series increases the exponent by 1. Recall that a polynomial series of degree d is a sum of the form i. The ruleis that ist this kind of sum is in © (n4*") 2. The sum of a geometric series in @ of its largest term. é Recall that a geometric series is a sum of the form ) The rule applies whether 0 1, but clearly not whe r= 1. The limits «a and b are not both constants; typically, the upper limit b is some function of and the lower limit a is a constant. 3, The sum of a logarithmic series is in Q{the muraber of terms times the log ofthe largest term), A logarithmic series is a sura of the form) log(i). The rule states that this kind of a sum is in © (1 log()). Recall that, for statements about asymptotic order, the base of the logarithm does not matter. : Note that the terms divear, quadratic, and cubic are used seurewhat more loosely bere than they esually ae used bby mathematicians1.6 1.6 Searching an Ordered Array 53 be Bost oer togt 2 Figure 1.7. Rectangles provide upper and lower bonnds for may kinds of sums, When the areas of both rectangles have the same asymptotic order, that must be the order of the sum, 4, The sum of a palynomial-togarithoic series, which is a sum of the form Yi log(i), is in © (x@*! log(n)). in the theorem ate of the form J £4). Proof Look at Figure 1.7. Since all the seri jet where f (i) is monotonic, itis clear that the targer rectangle, of height fn) and width 2. is ‘an upper bound on the sum. Also, as seen in Figure 1.4(b), the area under the graph of (7) between / = 0 and i =n is a lower bound on the sum. For the cases of polynomial series and logarithmic series, that area can easily be hounded below by the area of the smaller, darkly shaded, rectangle, In the left picture, the area of the larger rectangle is u4*!, while the area of the smaller rectangle is u#*"/2"*!. Since the areas of both sectangles have the same asymptotic order, the sum must have that order also. In the right picture, the 1wo areas are n log n and (n/2)(log n — log 2). Polynomial-logarithmic series are similar, but this technique does not work for geometric series, The rule for the geometric serics fotlows directly from Equation (1.9) in Section 1.3.2. 0 Searching an Ordered Array To illustrate the ideas presented in the previous sections, we will study a familiar problem, Problem LI Ordered array search Given an array E containing n entries sorted in nondcercasing order, and given a value K, find an index for which K = Elindex] or, if K is not in the array, return — as the answer, . Tnpractice, K is usually the key for an entry and the entries are in some class with other instance fields beside the key, so. more precise requirement might be Ki = EfindexL.key. To simplify the discussion, we assume the entire array entry is the key and it is some numeric type.54 Chapter 1 Analyzing Algorithms and Problems: Principles and Examples Let’s pretend for the moment that we don’t know the Binary Search algorithm; we approach the problem as if for the frst time, We will consider various algorithms, analyze worst-case and average behavior, and finally consider Binary Search and show that it is ‘optimal by establishing a lower bound on the number of key comparisons needed, 1.6.1 Some Solutions ‘Observe that the sequential search algorithm (Algorithm 1.1) solves the problem, but it makes no use of the fact thal we now have an array in which the entries are in order. Can. ‘we modify that algorithm so that it uses the adied information and does less work? ‘The first improvement is prompted by the observation that, since the array is in non- decreasing order, us soon as an entry larger thun & is encountered, the algorithm can terminate with the answer ~1. (How should the test un line 4 of that algorithm be changed to avoid doing two comparisons on each pass thvough the loop?) How does this change af- fect the analysis? Clearly, the modified algorithm is better in some cases; it will terminate sooner for some inputs, The worst-case complexity, however, remains unchanged. If K is the fast entry in the array or if K is larger than all the entries, then the algorithm will do » comparisons. For the average analysis of the modilied algorithm, we must know how likely itis that K is berweéh any two utray entries. Suppose we define a gap, gry to be the set of values y such that Eli-1] < y < Efi] fori = 1,...,2 — 1. Also, let go be all values less than E{0] and gq be ull values greater than Efa~1). We will assume, as we did in Example 1.9, that there is a probability q that K isin the armuy. If Kis in E we assume that all positions in the aay are equully likely (L¢., have a conditional probability L/n). If X is not in the array, we assume that all gaps are equally likely (i.e., have a conditional probability 1/( + 1))- For 0

You might also like

• The Subtle Art of Not Giving a F*ck: A Counterintuitive Approach to Living a Good Life

  

  From Everand

  The Subtle Art of Not Giving a F*ck: A Counterintuitive Approach to Living a Good Life

  Mark Manson

  Rating: 4 out of 5 stars

  4/5 (5975)
• The Gifts of Imperfection: Let Go of Who You Think You're Supposed to Be and Embrace Who You Are

  

  From Everand

  The Gifts of Imperfection: Let Go of Who You Think You're Supposed to Be and Embrace Who You Are

  Brené Brown

  Rating: 4 out of 5 stars

  4/5 (1110)
• The Glass Castle: A Memoir

  

  From Everand

  The Glass Castle: A Memoir

  Jeannette Walls

  Rating: 4.5 out of 5 stars

  4.5/5 (1737)
• Hidden Figures: The American Dream and the Untold Story of the Black Women Mathematicians Who Helped Win the Space Race

  

  From Everand

  Hidden Figures: The American Dream and the Untold Story of the Black Women Mathematicians Who Helped Win the Space Race

  Margot Lee Shetterly

  Rating: 4 out of 5 stars

  4/5 (932)
• Elon Musk: Tesla, SpaceX, and the Quest for a Fantastic Future

  

  From Everand

  Elon Musk: Tesla, SpaceX, and the Quest for a Fantastic Future

  Ashlee Vance

  Rating: 4.5 out of 5 stars

  4.5/5 (476)
• Her Body and Other Parties: Stories

  

  From Everand

  Her Body and Other Parties: Stories

  Carmen Maria Machado

  Rating: 4 out of 5 stars

  4/5 (831)
• Bad Feminist: Essays

  

  From Everand

  Bad Feminist: Essays

  Roxane Gay

  Rating: 4 out of 5 stars

  4/5 (1058)
• The Emperor of All Maladies: A Biography of Cancer

  

  From Everand

  The Emperor of All Maladies: A Biography of Cancer

  Siddhartha Mukherjee

  Rating: 4.5 out of 5 stars

  4.5/5 (275)
• The Outsider: A Novel

  

  From Everand

  The Outsider: A Novel

  Stephen King

  Rating: 4 out of 5 stars

  4/5 (1953)
• Angela's Ashes: A Memoir

  

  From Everand

  Angela's Ashes: A Memoir

  Frank McCourt

  Rating: 4.5 out of 5 stars

  4.5/5 (443)
• Brooklyn: A Novel

  

  From Everand

  Brooklyn: A Novel

  Colm Toibin

  Rating: 3.5 out of 5 stars

  3.5/5 (2029)
• The Little Book of Hygge: Danish Secrets to Happy Living

  

  From Everand

  The Little Book of Hygge: Danish Secrets to Happy Living

  Meik Wiking

  Rating: 3.5 out of 5 stars

  3.5/5 (424)
• A Man Called Ove: A Novel

  

  From Everand

  A Man Called Ove: A Novel

  Fredrik Backman

  Rating: 4.5 out of 5 stars

  4.5/5 (4851)
• The Yellow House: A Memoir (2019 National Book Award Winner)

  

  From Everand

  The Yellow House: A Memoir (2019 National Book Award Winner)

  Sarah M. Broom

  Rating: 4 out of 5 stars

  4/5 (99)
• The Art of Racing in the Rain: A Novel

  

  From Everand

  The Art of Racing in the Rain: A Novel

  Garth Stein

  Rating: 4 out of 5 stars

  4/5 (4255)
• Team of Rivals: The Political Genius of Abraham Lincoln

  

  From Everand

  Team of Rivals: The Political Genius of Abraham Lincoln

  Doris Kearns Goodwin

  Rating: 4.5 out of 5 stars

  4.5/5 (235)
• The Woman in Cabin 10

  

  From Everand

  The Woman in Cabin 10

  Ruth Ware

  Rating: 3.5 out of 5 stars

  3.5/5 (2587)
• Fear: Trump in the White House

  

  From Everand

  Fear: Trump in the White House

  Bob Woodward

  Rating: 3.5 out of 5 stars

  3.5/5 (805)
• On Fire: The (Burning) Case for a Green New Deal

  

  From Everand

  On Fire: The (Burning) Case for a Green New Deal

  Naomi Klein

  Rating: 4 out of 5 stars

  4/5 (75)
• Rise of ISIS: A Threat We Can't Ignore

  

  From Everand

  Rise of ISIS: A Threat We Can't Ignore

  Jay Sekulow

  Rating: 3.5 out of 5 stars

  3.5/5 (139)
• John Adams

  

  From Everand

  John Adams

  David McCullough

  Rating: 4.5 out of 5 stars

  4.5/5 (2411)
• Never Split the Difference: Negotiating As If Your Life Depended On It

  

  From Everand

  Never Split the Difference: Negotiating As If Your Life Depended On It

  Chris Voss

  Rating: 4.5 out of 5 stars

  4.5/5 (893)
• Shoe Dog: A Memoir by the Creator of Nike

  

  From Everand

  Shoe Dog: A Memoir by the Creator of Nike

  Phil Knight

  Rating: 4.5 out of 5 stars

  4.5/5 (545)
• Grit: The Power of Passion and Perseverance

  

  From Everand

  Grit: The Power of Passion and Perseverance

  Angela Duckworth

  Rating: 4 out of 5 stars

  4/5 (619)
• Yes Please

  

  From Everand

  Yes Please

  Amy Poehler

  Rating: 4 out of 5 stars

  4/5 (1941)
• A Heartbreaking Work Of Staggering Genius: A Memoir Based on a True Story

  

  From Everand

  A Heartbreaking Work Of Staggering Genius: A Memoir Based on a True Story

  Dave Eggers

  Rating: 3.5 out of 5 stars

  3.5/5 (232)
• The Hard Thing About Hard Things: Building a Business When There Are No Easy Answers

  

  From Everand

  The Hard Thing About Hard Things: Building a Business When There Are No Easy Answers

  Ben Horowitz

  Rating: 4.5 out of 5 stars

  4.5/5 (356)
• Devil in the Grove: Thurgood Marshall, the Groveland Boys, and the Dawn of a New America

  

  From Everand

  Devil in the Grove: Thurgood Marshall, the Groveland Boys, and the Dawn of a New America

  Gilbert King

  Rating: 4.5 out of 5 stars

  4.5/5 (270)
• Principles: Life and Work

  

  From Everand

  Principles: Life and Work

  Ray Dalio

  Rating: 4 out of 5 stars

  4/5 (622)
• The Unwinding: An Inner History of the New America

  

  From Everand

  The Unwinding: An Inner History of the New America

  George Packer

  Rating: 4 out of 5 stars

  4/5 (45)
• Steve Jobs

  

  From Everand

  Steve Jobs

  Walter Isaacson

  Rating: 4.5 out of 5 stars

  4.5/5 (814)
• The World Is Flat 3.0: A Brief History of the Twenty-first Century

  

  From Everand

  The World Is Flat 3.0: A Brief History of the Twenty-first Century

  Thomas L. Friedman

  Rating: 3.5 out of 5 stars

  3.5/5 (2272)
• The Sympathizer: A Novel (Pulitzer Prize for Fiction)

  

  From Everand

  The Sympathizer: A Novel (Pulitzer Prize for Fiction)

  Viet Thanh Nguyen

  Rating: 4.5 out of 5 stars

  4.5/5 (124)
• The Light Between Oceans: A Novel

  

  From Everand

  The Light Between Oceans: A Novel

  M.L. Stedman

  Rating: 4.5 out of 5 stars

  4.5/5 (789)
• Manhattan Beach: A Novel

  

  From Everand

  Manhattan Beach: A Novel

  Jennifer Egan

  Rating: 3.5 out of 5 stars

  3.5/5 (883)
• Wolf Hall: A Novel

  

  From Everand

  Wolf Hall: A Novel

  Hilary Mantel

  Rating: 4 out of 5 stars

  4/5 (4042)
• The Perks of Being a Wallflower

  

  From Everand

  The Perks of Being a Wallflower

  Stephen Chbosky

  Rating: 4.5 out of 5 stars

  4.5/5 (2119)
• A Tree Grows in Brooklyn

  

  From Everand

  A Tree Grows in Brooklyn

  Betty Smith

  Rating: 4.5 out of 5 stars

  4.5/5 (1934)
• Little Women

  

  From Everand

  Little Women

  Louisa May Alcott

  Rating: 4 out of 5 stars

  4/5 (105)
• The Constant Gardener: A Novel

  

  From Everand

  The Constant Gardener: A Novel

  John le Carré

  Rating: 3.5 out of 5 stars

  3.5/5 (108)
• Sing, Unburied, Sing: A Novel

  

  From Everand

  Sing, Unburied, Sing: A Novel

  Jesmyn Ward

  Rating: 4 out of 5 stars

  4/5 (1217)