FUNDAMENTALS DATA STRUCTURES SORTING SEARCHING 4 it Bitpped BH oi ROBERT SEDGEWICKAlgorithms THIRD EDITION inC PARTS 1-4 FUNDAMENTALS DATA STRUCTURES SORTING SEARCHING Robert Sedgewick Princeton University ¥ ADDISON-WESLEY An imprint of Addison Wesley Longman, Ine. Reading, Massachusetts © Harlow, England # Menlo Park, California Berkeley, California Don Mills, Ontario # Sydney ¢ Bonn # Amsterdam Tokyo # Mexico CityPublishing Partner: Peter $. Gordon Associate Editor: Deborah Lafferty Cover Designer: Andre Kuzniarek Production Editor: Amy Willcutt Copy Editor: Lyn Dupré The programs and applications presented in this book have been in- cluded for their instructional value. They have been tested with care, but are not guaranteed for any particular purpose. The publisher neither offers any warranties or representations, nor accepts any lia bilities with respect to the programs or applications. Library of Congress Cataloging-in-Publication Data Sedgewick, Robert, 1946 — Algorithms in C / Robert Sedgewick. — 3d ed. 720 p. 24cm, Includes bibliographical references and index. Contents: v. 1, pts. 1-4. Fundamentals, data structures, sorting, searching. ISBN 0-201-31452-5 1. C (Computer program language) 2. Computer algorithms, I. Title. QA76.73.C15S43 1998 005.13°3—de21 97-23418 cI Reproduced by Addison-Wesley from camera-ready copy supplied by the author. Copyright © 1998 by Addison-Wesley Publishing Company, Inc. Reprinted with corrections, December 1997. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or other- wise, without the prior written permission of the publisher. Printed in the United States of America. 23456789 10~CRW - 010099897Preface HIS BOOK IS intended to survey the most important computer algorithms in use today, and to teach fundamental techniques to the growing number of people in need of knowing them. It can be used as a textbook for a second, third, or fourth course in com- puter science, after students have acquired basic programming skills and familiarity with computer systems, but before they have taken specialized courses in advanced areas of computer science or com- puter applications. The book also may be useful for self-study or as a reference for people engaged in the development of computer systems or applications programs, since it contains implementations of useful algorithms and detailed information on these algorithms? performance characteristics. The broad perspective taken makes the book an appropriate introduction to the field Thave completely rewritten the text for this new edition, and I have added more than a thousand new exercises, more than a hun- dred new figures, and dozens of new programs. I have also added detailed commentary on all the figures and programs. This new ma- terial provides both coverage of new topics and fuller explanations of many of the classic algorithms. A new emphasis on abstract data types throughout the book makes the programs more broadly useful and relevant in modern object-oriented programming environments. People who have read old editions of the book will find a wealth of new information throughout; all readers will find a wealth of pe gogical material that provides effective access to essential concepts. Due to the large amount of new material, we have split the new edition into two volumes (each about the size of the old edition) of which this is the first. This volume covers fundamental concepts, data structures, sorting algorithms, and searching algorithms; the second volume covers advanced algorithms and applications, building on the basic abstractions and methods developed here. rial on fundamentals and data structures in this edition is new. Nearly all the mate-PREFACE This book is not just for programmers and computer-science stu- dents, Nearly everyone who uses a computer wants it to run f ter or to solve larger problems, The algorithms in this book represent a body of knowledge developed over the last 50 years that has be- come indispensible in the efficient use of the computer, for a broad variety of applications. From NV-body simulation problems in physies to genetic-sequencing problems in molecular biology, the basic meth- ods described here have become essential in scientific research; and from database systems to Internet search engines, they have become essential parts of modern software systems. As the scope of computer applications becomes more widespread, so grows the impact of many of the basic methods covered here. The goal of this book is to serve asa resource for students and professionals interested in knowing and making intelligent use of these fundamental algorithms as basic tools for whatever computer application they might undertake. Scope The book contains 16 chapters grouped into four major parts: funda- mentals, data structures, sorting, and searching. The descriptions here are intended to give readers an understanding of the basic properties of as broad a range of fundamental algorithms as possible. Ingenious methods ranging from binomial queues to patricia tries are described, all related to basic paradigms at the heart of computer science. The second volume consists of four additional parts that cover strings, ge- ometry, graphs, and advanced topics. My primary goal in developing these books has been to bring together the fundamental methods from these diverse areas, to provide access to the best methods known for solving problems by computer. ‘You will most appreciate the material in this book if you have had one or two previous courses in computer science or have had equivalent programming experience: one course in programming in a high-level language such as C, Java, or C++, and perhaps another course that teaches fundamental concepts of programming systems. This book is thus intended for anyone conversant with a modern pro- gramming language and with the systems. References that might help to fill in gaps in your background are suggested in the text. asic features of modern computerMost of the mathematical material supporting the analytic re- sults is sell-contained (or is labeled as beyond the scope of this book), so little specific preparation in mathematics is required for the bulk of the book, although mathematical maturity is definitely helpful. Use in the Curriculum There is a great deal of flexibility in how the material here can be taught, depending on the taste of the instructor and the preparation ofthe students. The algorithms described here have found widespread use for years, and represent an essential body of knowledge for both the practicing programmer and the computer-science student. There is sufficient coverage of basic material for the book to be used for a course on data structures, and there is sufficient detail and coverage of advanced material for the book 10 be used for a course on al- gorithms. Some instructors may wish to emphasize implementations and practical concerns; others may wish to emphasize analysis and theoretical concepts A complete set of slide masters for use in lectures, sample pro- gramming assignments, interactive exercises for students, and other course materials may be found via the book's home page. An elementary course on data structures and algorithms might emphasize the basic data structures in Part 2 and their use in the implemeniations in Parts 3 and 4. A course on design and analysis of algorithms might emphasize the fundamental material in Part 1 and Chapter 5, then study the ways in which the algorithms in Parts 3 and 4 achieve good asymptotic performance. A course on software engineering might omit the mathematical and advanced algorithmic material, «ind emphasize how to integrate the implementations given here into large programs or systems. A course on algorithms might take a survey approach and introduce concepts from all these areas. Earlier editions of this book have been used in recent years at scores of colleges and universities around the world as a text for the second or third course in computer science and as supplemental reading for other courses. At Princeton, our experience has been that the breadth of coverage of material in this book provides our majors with an introduction to computer science that can be expanded upon in later courses on analysis of algorithms, systems programming andvi PREFACE theoretical computer science, while providing the growing group of students from other disciplines with a large set of techniques that these people can immediately put to good use, The exercises—most of which are new to this edition—fall into several types. Some are intended to test understanding of material in the text, and simply ask readers to work through an example or to apply concepts described in the text. Others involve implementing, and putting together the algorithms, or running empirical studies to compare variants of the algorithms and to learn their properties. Still others are a repository for important information at a level of detail that is not appropriate for the text. Reading and thinking about the exercises will pay dividends for every reader. Algorithms of Practical Use Anyone wanting to use a computer more effectively can use this book for reference or for self-study. People with programming experience can find information on specific topics throughout the book. To a large extent, you can read the individual chapters in the book inde- pendently of the others, although, in some cases, algorithms in one chapter make use of methods from a previous chapter, The orientation of the book is to study algorithms likely to be of practical use. The book provides information about the tools of the teade to the point that readers can confidently implement, debug, and put to work algorithms to solve a problem or to provide functionality in an application, Full implementations of the methods discussed are included, as are descriptions of the operations of these programs on a consistent set of examples. Because we work with real code, rather than write pseudo-code, the programs can be put to practical use quickly. Program listings are available from the book’s home page. Indeed, one practical application of the algorithms has been to produce the hundreds of figures throughout the book. Many algo- rithms are brought to light on an intuitive level through the visual dimension provided by these figures. Characteristics of the algorithms and of the situations in which they might be useful are discussed in detail. Although not empha- sized, connections to the analysis of algorithms and theoretical com- puter science are developed in context. When appropriate, empiricaland analytic results are presented to illustrate why certain algorithms are preferred. When interesting, the relationship of the practical al- gorithms being discussed to purely theoretical results is described. Specific information on performance characteristics of algorithms and implementations is synthesized, encapsulated, and discussed through- out the book Programming Language The programming language used for all of the implementations is Any particular language has advantages and disadvantages; we use C because it is widely available and provides the features needed for our implementations. The programs can be translated easily ro other modern programming languages, since relatively few constructs are unique to C, We use standard C idioms when appropriate, bur this book is not intended to be a reference work on C programming. There are many new progeams in this edition, and many of the old ones have been reworked, primarily to make them more readily useful as abstract-data-type implementations. Extensive comparative empirical tests on the programs are discussed throughout the text. Previous editions of the book have presented basie programs in Pascal, C++, and Modala-3. This code is available through the book home page on the webs code for new programs and code in new languages such as Java will he added as appropriate. A goal of this book is to present the algorithms in as simple and direct a form as possible. The style is consistent whenever pos- sible, so that programs that are similar look similar. For many of the algorithms in this book, the similarities hold regardless of the language: Quicksort is quicksort (to pick one prominent example}, whether expressed in Algol-60, Basic, Fortran, Smalltalk, Ada, Pascal, C, PostScript, Java, or countless other programming languages and environments where it has proved to be an effective sorting method. We strive for elegant, compact, and portable implementations, bur we take the point of view that efficiency matters, so we try to be aware of the performance characteristics of our code at all stages of development. Chapter 1 constitutes a detailed example of this approach to developing efficient C implementations of our algorithms, and sets the stage for the rest of the book. viivit PREFACE Acknowledgments Many people gave me helpful feedback on earlier versions of this book, In particular, hundreds of students at Princeton and Brown have suffered through preliminary drafts over the years. Special thanks are due to Trina Avery and Tom Freeman for their help in producing the first edition; to Janet Incerpi for her creativity and in genuity in persuading our early and primitive digital computerized typesetting hardware and software to produce the first edition; to Mare Brown for his part in the algorithm visualization research that was the genesis of so many of the figures in the book; and to Dave Hanson for his willingness to answer all of my questions about C. 1 would also like to thank the many readers who have provided me with detailed comments about various editions, including Guy Almes, Jon Bentley, Mare Brown, Jay Gischer, Allan Heydon, Kennedy Lemke, Udi Manber, Dana Richards, John Reif, M. Rosenfeld, Stephen Seid- man, Michael Quinn, and William Ward. To produce this new edition, I have had the pleasure of working with Peter Gordon and Debbie Lafferty at Addison-Wesley, who have patiently shepherded this project as it has evolved from a standard up- dare to a massive rewrite. It has also been my pleasure to work with several other members of the professional staff at Addison-Wesley. The nature of this project made the book a somewhat unusual chal- lenge for many of them, and I much appreciate their forbearance. Lhave gained two new mentors in writing this book, and par- ticularly want to express my appreciation to them. First, Steve Sum- mit carefully checked early versions of the manuscript on a technical level, and provided me with literally thousands of detailed comments, particularly on the programs. Steve clearly understood my goal of providing elegant, efficient, and effective implementations, and his comments not only helped me to provide a measure of consistency across the implementations, but also helped me to improve many of them substantially. Second, Lyn Dupré also provided me with thou- sands of detailed comments on the manuscript, which were invaluable in helping me not only to correct and avoid grammatical errors, but also—more important—to find a consistent and coherent writing style that helps bind together the daunting mass of technical material hereLam extremely grateful for the opportunity to learn from Steve and Lyn—their input was vital in the development of this book Much of what [ have written here [have learned from the teach- ing and writings of Don Knuth, my advisor at Stanford. Although Don had no direct influence on this work, his presence may be felt in the book, for it was he who put the study of algorithms on the scientific footing that makes a work such as this possible. My friend and colleague Philippe Flajolet, who has been a major force in the development of the analysis of algorithms as a mature research area, has had a similar influence on this work. Tan deeply thankful for the support of Princeton University, Brown University, and the Institut National de Recherce en Informa tiue et Automatique (INRIA), where I did most of the work on the hook; and of the Institute for Defense Analyses and the Xerox Palo Alto Research Center, where I did some work on the book while vis- iting. Many parts of the hook are dependent on research that has been generously supported by the National Science Foundation and the Office of Naval Research. Finally, [ thank Bill Bowen, Aaron. Lemonick, and Neil Rudenstine for their support in building an aca demic environment at Princeton in which I was able to prepare this book, despite my numerous other responsibilities. Robert Sedgewick Marly-le-Roi, France, February, 1983 Princeton, New Jersey, January, 1990 Jamestown, Rhode Island, August, 1997To Adam, Andrew, Brett, Robbie, and especially LindaNotes on Exercises Classifying exercises is an activity fraught with peril, because readers of a book such as this come to the material with various levels of knowledge and experience. Nonetheless, guidance is appropriate, so many of the exercises carry one of four annotations, to help you decide how to approach them Exercises that test your understanding of the material are marked with an open triangle, as follows: 9.57. Give the binomial queue that results when the keys EAS Y QUESTION are inserted into an initially empty binomial queue. Most often, such exercises relate directly to examples in the text They should present no special difficulty, but working them might teach you a fact or concept that may have eluded you when you read the text. Exercises that add new and thought-provoking information to the material are marked with an open circle, as follows: (0.14.20 Write a program that inserts N random integers into a table of size N/100 using separate chaining, then finds the length of the shortest and longest lists, for N = 10°, 10", 10°, and 10°, Such exercises encourage you to think about an important concept that is related to the material in the text, or to answer a question that may have occurred to you when you read the text. You may find it worthwhile to read these exercises, even if you do not have the time to work them through Exercises that are intended to challenge you are marked with a black dot, as follows: ©8.46 Suppose that mergesort is implemented to split the file at a randos position, rather than exactly in the middle. How many comparisons are used by such a method to sort V elements, on the average: Such exercises may require a substantial amount of time to complete, depending upon your experience. Generally, the most productive ap- proach is to work on them in a few different sittings, A few exercises that are extremely difficult (by comparison with most others} are marked with (wo black dots, as follows: xi15.29 Prove that the height of a trie built from V random bit- strings is about 2 lg N. These exercises are similar to questions that might be addressed in the research literature, but the material in the book may prepare you to enjoy trying to solve them {and perhaps succeeding) The annotations are intended to be neutral with respect to your programming and mathematical ability. Those exercises that require expertise in programming or in mathematical analysis are self-evident. Alll readers are encouraged to test their understanding of the algo- rithms by implementing them. Still, an exercise such as this one is straightforward for a practicing programmer or a student in a pro- gramming course, but may require substantial work for someone who has not recently programmed: 1.23 Modify Program 1.4 to generate random pairs of integers between 0 and NI instead of reading them from standard input, and to loop until N’~ 1 sion operations have been performed Run your program for V = 10', 10°, 10°, and 10and print out the roral number of edges generated for each value of In a similar vein, all readers are encouraged to strive to appreciate the analytic underpinnings of our knowledge about properties of al- gorithms. Still, an exercise such as this one is straightforward for a scientist ora student ina discrete mathematics course, but may require substantial work for someone who has not recently done mathemat- ical analysis: 1.13 Compute the average distance from a node to the root in a worst-case tree of 2” nodes built by the weighted quick-union algorithm, There are far too many exercises for you to read and assimilate them all; my hope is that there are enough exercises here to stimulate you fo strive to come to a broader understanding on the topics that interest you than you can glean by simply reading the text.Contents Fundamentals Chapter 1. Introduction 3 1.1 Algorithms » 4 1.2 A Sample Problem—Conneetivity - 6 1.3 Union-Find Algorithms + 11 LA Perspective » 22 1.5. Summary of Topies - 23 Chapter 2. Principles of Algorithm Analysis 27 1 Implementation and Empirical Analysis - 28 2.2 Analysis of Algorithms - 33 2.3 Growth of Functions » 36 2.4 Big-Oh notation » 44 2.5 Basic Recurrences » 49 2.0 Examples of Algorithm Analysis - 53 2.7 Guarantees, Predictions, and Limitations - 60TABLE OF CONTENTS Data Structures Chapter 3. Elementary Data Structures 69 3.1 Building Blocks - 70 3.2 Arrays + 83 3.3, Linked Lists - 90 3.4. Elementary List Processing - 96 3.5. Memory Allocation for Lists - 105 3.6 Strings - 108 3.7 Compound Data Structures » 115 Chapter 4. Abstract Data Types 127 4.1 Abstract Objects and Collections of Objects» 131 4.2 Pushdown Stack ADT - 135 4.3. Examples of Stack ADT Clients - 138 4.4. Stack ADT Implementations - 145 4.8 Creation of a New ADT - 149 4.6 FIFO Queues and Generalized Queues - 153 4.7 Duplicate and Index Irems - 161 4.8. First-Class ADTs - 166 4,9 Application-Based ADT Example » 179 4.10 Perspective - 185 Chapter 5. Recursion and Trees 187 5.1 Recursive Algorithms - 188 Divide and Conquer - 196 Dynamic Programming - 209 Trees - 217 Mathematical Properties of Trees » 226 Tree Traversal » 230 Recursive Binary-Tree Algorithms - 236 Graph Traversal » 241 Perspective - 248Sorting Chapter 6. Elementary Sorting Methods 253 6.1 Rules of the Game - 255 6.2 Selection Sort + 261 6.3 Insertion Sort » 262 6.4 Bubble Sort » 265 6.5 Performance Characteristics of Elementary Sorts - 267 6.6 Shellsort - 273 6.7 Sorting Other Types of Data - 282 6.8 Index and Pointer Sorting - 287 6.9 Sorting Linked Lists » 295 6.10. Key-Indexed Counting - 298 Chapter 7. Quicksort 303 7A The Basic Algorithm - 304 7.2. Performance Characteristics of Quicksort - 309 Stack Size - 313 Small Subfiles » 316 Median-of-Three Partitioning » 319 Duplicate Keys - 324 Strings and Vectors » 427 Selection - 329 ass Qeanke Chapter 8. Mergesort 335 8.1. Two-Way Merging » 336 8.2 Abstract In-place Merge - 339 8.3. Top-Down Mergesort - 341 Improvements to the Basic Algorithm - 344 Bottom-Up Mergesort - 347 Performance Characteristics of Mergesort - 351 Linked-List Implementations of Mergesort » 35: Recursion Revisited - 357 4 Chapter 9. Priority Queues and Heapsort 361 9.1 Elementary Implementations - 365 9.2. Heap Data Structure» 369 xvTABLE OF CONTENTS 9.3. Algorithms on Heaps » 377 9.4 Heapsort - 377 9.5. Priority-Queue ADT - 384 9.6 Priority Queues for Index Items « 389 9.7 Binomial Queues - 392 Chapter 10. Radix Sorting 403 10.1. Bits, Bytes, and Words - 405 10.2 Binary Quicksort - 409 10.3 MSD Radix Sort 413 10.4 Three-Way Radix Quicksort » 421 10.5 LSD Radix Sort - 425 10.6 Performance Characteristics of Radix Sorts - 429 10.7 Sublinear-Time Sorts - 433 Chapter 11. Special-Purpose Sorts 439 11.1 Batcher’s Odd-Even Mergesort - 441 11.2 Sorting Networks » 446 11.3: External Sorting 454 11.4 Sort-Merge Implementations - 460 11.5 Parallel Sort/Merge - 467 Searching Chapter 12. Symbol Tables and BSTs 477 12.1 Symbol-Table Abstract Data Type - 479 Key-Indexed Search - 485 Sequential Search - 489 Binary Search » 497 Binary Search Trees (BSTs) » 502 Performance Characteristics of BSTs » 508: Index Implementations with Symbol Tables - 511 Insertion at the Root in BSTs » 516 12.9. BST Implementations of Other ADT Functions - 520Chapter 13. Balanced Trees 13.1, Randomized 13.2 Splay BSTs BSTs 540 13.3. Top-Down 2-3-4 Trees - 54 1 4 Red-Black Trees 13.5. Skip Lists + Sor 13.6 Performance Characteristics Chapter 14. Hashing ns = S74 14.2. Separate Chaining 14.1 Hash Function 14. Linear Probing, 14.4 Double Hashing hb Tables - 399, 14.5. Dynamic Has! 14.0. Perspective 603 ss sss so4 Chapter 15. Radix Search 15.1 Digital Search Trees - 610 15.2 Tries» 614 15.3 Patricia Tries 623 15.4 Multiway Tries and TSTs 15,5 Text String Index Applications Chapter 16. External Searching 16.1 Rul of the Game 657 16.2 Indexed Sequential Access 16.3 B Trees » 462 16.4 Extendible Hashing - 676 16.5 Perspective Index 689 6 632 660 649 573 609 655 693PART ONE. FundamentalsCHAPTER ONE Introduction HE OBJECTIVE OF this book is to study a broad variety of important and useful algorithms: methods for solving problems that are suited for computer implementation, We shall deal with many different areas of application, always concentrating on fun- damental algorithms that are important to know and interesting to study. We shall spend enough time on each algorithm to understand its essential characteristics and to respect its subtleties. Our goal is to learn a large number of the most important algorithms used on computers today, well enough to be able to use and appreciate them. The strategy that we use for understanding the programs pre- sented in this book is to implement and test them, to experiment with their variants, to discuss their operation on small examples, and to try them out on larger examples similar to what we might encounter in practive. We shall use the C programming language to describe the algorithms, thus providing useful implementations at the same time Our programs have a uniform style that is amenable to translation into other modern programming languages, as well. We also pay careful attention to performance characteristics of our algorithms, to help us develop improved versions, compare dif ferent algorithms for the same task, and predict or guarantee per- formance for large problems. Understanding how the algorithms per- form might require experimentation or mathematical analysis or both. ‘We consider detailed information for many of the most important al- gorithms, developing analytic results directly when feasible, or calling on results from the research literature when necessary,guar CHAPTER ONE To illustrate our general approach to developing algorithmic so- lutions, we consider in this chapter a detailed example comprising a number of algorithms that solve a particular problem. The problem that we consider is not a toy problem; it is a fundamental compu- tational task, and the solution that we develop is of use in a variety of applications. We start with a simple solution, then seck to un- derstand that solution’s performance characteristics, which help us to see how to improve the algorithm. After a few iterations of this process, we come to an efficient and useful algorithm for solving the problem. This prototypical example sets the stage for our use of the same general methodology throughout the book ‘We conclude the chapter with a short discussion of the contents of the book, including brief descriptions of what the major parts of the book are and how they relate to one another. 1.1 Algorithms When we write a computer program, we are generally implementing a method that has been devised previously to solve some problem. This method is often independent of the particular computer to be used—it is likely to be equally appropriate for many computers and many computer languages. It is the method, rather than the computer program itself, that we must study to learn how the problem is being attacked, The term algorithm is used in computer science to describe a problem-solving method suitable for implementation as a computer program. Algorithms are the stuff of computer science: They are central objects of study in many, if not most, areas of the field. Most algorithms of interest involve methods of organizing the data involved in the computation. Objects created in this way are called data structures, and they also are central objects of study in computer science. Thus, algorithms and data structures go hand in hand. In this book we take the view that data structures exist as the byproducts or end products of algorithms, and thus that we must study them in order to understand the algorithms. Simple algorithms can give rise to complicated data structures and, conversely, compli- cated algorithms can use simple data structures. We shall study the properties of many data structures in this book; indeed, the book might well have been called Algorithms and Data Structures in C.INTRODUCTION gua ‘When we use a computer to help us solve a problem, we typically are faced with a number of possible different approaches. For small problems, it hardly matters which approach we use, as long as we have one that solves the problem correctly. For huge problems (or applications where we need to solve huge numbers of small problems), however, we quickly hecome motivated to devise methods that use time or space as efficiently as possible The primary reason for us to learn about algorithm design is that this «liscipline gives us the potential to reap huge savings, even to the point of making it possible to do tasks that would otherwise he impossible. In an application where we are processing millions of objects, it is not unusual to be able to make a program millions of times faster by using a well-designed algorithm. We shall see such an example in Section 1.2 and on numerous other occasions throughout the hook. By contrast, investing additional money or time to buy and install a new computer holds the potential for speeding up a program by perhaps a factor of only 10 or 100, Careful algorithm design is an extremely effective part of the process of solving a huge problem, whatever the applications area. When a huge or complex computer program is to be developed, a great deal of effort must go into understanding and defining the problem to be solved, managing its complexity, and decomposing it into smaller subrasks that can be implemented easily. Often, many of the algorithms required after the decomposition are trivial to im- plement. In most cases, however, there are a few algorithms whose choice is critical because most of the system resources will be spent running those algorithms, Those are the types of algorithms on which we concentrate in this book. We shall study a variety of fundamen- tal algorithms that are useful for solving huge problems in a broad variety of applications areas. The sharing of programs in computer systems is becoming more widespreid, so, although we might expect to be using a large fraction of the algorithms in this book, we also might expect to have to imple ‘ment only a smaller fraction of them, However, implementing simple versions of basic algorithms helps us to understand them better and thus to use advanced versions more effectively. More important, the opportunity to reimplement basic algorithms arises frequently. The primary reason to do so is that we are faced, all too often, with com-6 bra CHAPTER ONE pletely new computing environments (hardware and software) with new features that old implementations may not use to best advantage In other words, we often implement basic algorithms tailored to our problem, rather than depending on a system routing, to make our solutions more portable and longer lasting. Another common reason to reimplement basic algorithms is that mechanisms for sharing soft- ware on many computer systems are not always sufficiently powerful to allow us to tailor standard programs to perform effectively an spe- cific tasks (or it may not be convenient to do so), so it is sometimes easier to do a new implementation. Computer programs are often overoptimized. It may not be worthwhile to take pains to ensure that an implementation of a par ticular algorithm is the most efficient possible unless the algorithm is to be used for an enormous task or is to be used many times, Other- wise, a careful, relatively simple implementation will suffice: We can have some confidence that it will work, and it is likely to run perhaps five or 10 times slower at worst than the est possible version, which means that it may run for an extra few seconds. By contrast, the proper choice of algorithm in the first place can make a difference of a factor of 100 or 1000 or more, which might translate to minutes, hours, or even more in running time. In this book, we concentrate on the simplest reasonable implementations of the best algorithms. The choice of the best algorithm for a particular task can be a complicated process, perhaps involving sophisticated mathematical analysis. The branch of computer science that comprises the study ‘of such questions is called analysis of algorithms. Many of the algo- rithms that we study have been shown through analysis to have ex cellent performance; others are simply known to work well through experience. Our primary goal is to learn reasonable algorithms for important tasks, yet we shall also pay careful attention to comparative performance of the methods. We should not use an algorithm with- out having an idea of what resources it might consume, and we strive to be aware of how our algorithms might be expected to perform, 1.2. A Sample Problem: Connectivity Suppose that we are given a sequence of pairs of integers, where integer represents an object of some type and we are to interpret theINTRODUCTION pair p=q as meaning “p is connected to q.” We assume the relation “is connected to” to be associative: If p is connected to q, and q is connected to x, then p is connected to x. Our goal is to write a program to filter out extraneous pairs from the set: When the program inputs a pair p-q, it should output the pair only if the pairs it has seen to thar point do not imply that p is connected to q. If the previous pairs do imply that p is connected to q, then the program should ignore p-q and should proceed to input the next pair. Figure 1.1 gives an example of this process. Our problem is to devise a program that can remember sufficient information about the pairs it has seen to be able to decide whether or not a new pair of objects is connected. Informally, we refer to the task of designing such a method as the connectivity problem. This problem arises in a number of important applications. We briefly consider three examples here to indicate the fundamental nature of the problem. For example, the integers might represent computers in a large network, and the pairs might represent connections in the network Then, our program might be used to determine whether we need to establish a new direct connection for p and q to be able to com- municate, or whether we could use existing connections to set up a communications path. In this kind of application, we might need to process millions of points and billions of connections, or more, As wwe shall see, it would be impossible to solve the problem for such an application without an efficient algorithm, Similarly, the integers might represent contact points in an elec- trical network, and the pairs might represent wires connecting the points. In this case, we could use our program to find a way to connect all the points without any extraneous connections, if that is possible. There is no guarantee that the edges in the list will suffice fo connect alll the points—indeed, we shall soon see that determining whether or not they will could he a prime application of our program. Figure 1.2 illustrates these two types of applications in a larger example, Examination of this figure gives us an appreciation for the difficulty of the connectivity problem: How can we arrange to tell quickly whether any given two points in a such a network are a4 34 9049 8-0 8-0 23 23 56 5-6 2-9 2-3-4-9 5-9 5-9 7-30 7-3 3 48 5-6 0-2 6-1 64 Figure 1.1 Connectivity example Given a sequence of pairs of in- tegers representing connections hetween objects ile), the task of a connectivity algorithm is to output those pairs that provide new con- nections (center). For example, the pair 2-9 is not part of the output because the connection 2-3-4-9 is implied by previous connections {this evidence is shown at right)8 Figure 1.2 A large connectivity example The objects ina connectivity prob- Jem might represent connection points, and the pairs might be con nections between them, as indi cated in this idealized example that might represent wires connect- ing buildings in a city or compo- nents an a computer chip. this graphical representation makes it possible for a human to spot nodes that are not connected, but the al- gorithm has to work with only the pairs of integers that itis given. ‘re the two nodes marked with the lage black dots connected? Sue CHAPTER ONE ert setts yy ESETAlrh Still another example arises in certain programming environ- ments where it is possible to declare two variable names as equiv- alent. ‘The problem is to be able to determine whether two given names are equivalent, after a sequence of such declarations. This ap- plication is an early one that motivated the development of several of the algorithms that we are about to consider. It directly relates our problem to a simple abstraction that provides us with a way to make our algorithms useful for a wide variety of applications, as we shall see. Applications such as the variable-name-equivalence problem de- scribed in the previous paragraph require that we associate an integer with each distinct variable name. This association is also implicit in the network-connection and circuit-connection applications thatINTRODUCTION hae cribed. We shall be Chapters 10 through 16 that can provide this association in an ef- sume in this chapter, without loss we have de considering a host of algorithms in ficient manner. Thus, we can a of generality, that we have \V objects with integer names, from 0 10 N-1 We are asking for a program that does a specific and well- defined task. There are many other related problems that we might want to have solved, as well. One of the first tasks that we face in de- veloping an algorithm is to be sure that we have specified the problem in a reasonable manner. The more we require of an algorithm, the more time and space we may expect it to need to finish the task. It is impossible to quantify this relationship a priori, and we often modify a problem specification on finding that it is difficult or expensive to solve, of, in happy circumstances, on finding that an algorithm can provide information more useful than was called for in the original specification. For example, our connectivity-problem specification requires only that ur program somehow know whether or not any given pair p-q is connected, and not that it be able to demonstrate any or all ways 10 connect that pair, Adding a requirement for such a spec- ification makes the problem more difficult, and would lead us to a different family of algorithms, which we consider briefly in Chapter 5 and in derail in Part 7 ‘The specifications mentioned in the previous paragraph asks us for more information than our original one did; we could also ask « simply want to be able for less information. For example, we mig to answer the question: “Are the A/ connections sufficient to connect ether all NV objects?” This problem illustrates that, to develop to do high-level reasoning about to} efficient algorithms, we off the abstract objects that we are processing. In this case, a fundamental result from graph theory implies that all N objects are connected if n ne and only if the number of pairs output by the connectivity algorithm is precisely N — 1 (see Section 5.4). In other words, a connectivity algorithm will never output more than N — 1 pairs, because, once it has output V—1 pairs, any pair thar it encounters from that point on will be connected, Accordingly, we can get a program that answers the yes-no question just posed by changing a program thar solves the connectivity problem to one that increments a counter, rather thanfre CHAPTER ONE writing out each pair that was not previously connected, answering “yes” when the counter reaches ~1 and “no” if it never does. This question is but one example of a host of questions that we might wish to answer regarding connectivity. The set of pairs in the input is called a graph, and the set of pairs output is called a spanning tree for that graph, which connects all the objects. We consider properties of graphs, spanning trees, and all manner of related algorithms in Part 7. It is worthwhile to try to identify the fundamental operations that we will be performing, and so to make any algorithm that we develop for the connectivity task useful for a variety of similar tasks. Specifically, each time that we get a new pair, we have first to deter mine whether it represents a new connection, then to incorporate the information that the connection has been seen into its understanding, about the connectivity of the objects such that it can check connec- tions 0 be seen in the future. We encapsulate these two tasks as abstract operations by considering the integer input values to repre- sent elements in abstract sets, and then design algorithms and data structures that can # Find the set containing a given item. # Replace the sets containing two given items by their ution. Organizing our algorithms in terms of these abstract operations does not seem to foreclose any options in solving the connectivity problem, and the operations may be useful for solving other problems. Devel- oping ever more powerful layers of abstraction is an essential process in computer science in general and in algorithm design in particu- lar, and we shall turn to it on numerous occasions throughout this book. In this chapter, we use abstract thinking in an informal way to guide us in designing programs to solve the connectivity problen Chapter 4, we shall see how to encapsulate abstractions in C code in The connectivity problem is easily solved in terms of the find and union abstract operations. After reading a new pair p-q from the input, we perform a fird operation for each member of the pair. If the members of the pair are in the same set, we move on to the next pair; if they are not, we do a union operation and write out the pair. The sets represent connected components: subsets of the objects with the property that any two objects in a given component are connected, This approach reduces the development of an algorithmic solution for connectivity to the tasks of defining a data structure representing the