Introduction to Computation and Programming Using Python Spring 2018 Edition John V. GuttagIntroduction to Computation and Programming Using Python (Spring 2013 Edition) John V. Guttag ‘The MIT Press Cambridge, Massachusetts London, England© 2013 Massachusetts Institute of Technology All rights reserved. No part of this book may be reproduced in any form by any electronic or mechanical means (including photocopying, recording, or information storage and retrieval) without permission in writing from the publisher. MIT Press books may be purchased at special quantity discounts for business or sales promotional use. For information, please email [email protected] or write to Special Sales Department, The MIT Press, 55 Hayward Street, Cambridge, MA 02142 Printed and bound in the United States of America. ISBN: 978-0-262-51963-2 Library of Congress Control Number: 2012953229 w987654321 Cover photograph: Antelope Canyon, courtesy of Frédo Durand.To my family: Olga David Andrea Michael Mark AddieCONTENTS PREFACE, SPRING 2018 EDITION ACKNOWLEDGMENTS. 1 GETTING STARTED 2. INTRODUCTION TO PYTHON 2.1 The Basic Element of Python 2.1.1 Objects, expressions, and numerical types 2.1.2 Variables and assignment 2.3 IDLE 2.2 Branching Programs 2.3 Type Strand Input 2.3.1 Input. 2.4 Iteration 3 SOME SIMPLE NUMERICAL PROGRAMS 3.1. Bxhaustive enumeration 8.2 Por Loops 8.3 Approximate Solutions and Bisection Search 3.4. A Rew Words About Using Floats 3.5. Newton-Raphson 4 FUNCTIONS AND ABSTRACTION BY SPECIFICATION 4.1 Punctions and Scoping 4.1.1. Function definitions 4.1.2 Keyword Arguments and Default Values. 4.1.8. Scoping 4.2. Specifications 4.3. Recursion 4.3.1. Fibonacci numbers. 4.8.2. Pelindromes and divide-and-conquer. 4.4. Global Variables 4.5 Modules and Import 46 Piles ee5 STRUCTURED TYPES, MUTABILITY, AND HIGHER-ORDER FUNCTIONS. 5.1 Tuples 5.1.1 Sequences and Multiple Assignment. 5.2 Lists and Mutability. 52.1 Cloning 52,2 List comprehension 5.2 Punetions as Objects 5.4 Strings, tuples, and lists 5.5 Dictionaries 6 TESTING AND DEBUGGING. 6.1 Testing 6.1.1 Black-box testing 6.1.2 Glass-box testing. 6.1.9 Conducting Tests 6.2 Debugging 6.2.1 Learning to Debug 6.22 Desig g the Experiment 6.2.8 When the Going Gets Tough 6.2.4 And When You Have Found “The” Bug. 7 EXCEPTIONS AND ASSERTIONS 7 7.2 Bxceptions as a Control Flow Mechanism 7.3 Assertions & CLASSES AND OBJECT-ORIENTED PROGRAMMING 8.1 Abstract Data Types and Classes 8.11 Desig 1g Programs Using Abstract Data Types. 8.1.2 Using Classes to Keep Track of Students and Faculty 82. Inheritance 8.2.1 Multiple Levels of Inheritance 2.2. The Substitution Principle 8.3. Encapsulation and Information Hiding 8.3.1 User-Defined Iterators 8.4 Mortgages, an Extended Example 62 62 63 o 66 70 70 73 74 76 78 79 al a2 83 83 86 a8 90 90 94 94 96 99 100 101 104 1069A SIMPLISTIC INTRODUCTION TO ALGORITHMIC COMPLEXITY 9.1 Thinking About Computational Complexity. 9.2 Asymptotic Notation. 9.8 Some Important Complexity Classes 9.3.1 Constant complexity. 9.3.2 Logarithmic complexity 9.3.3 Linear Complexity 9.3.4 Log-linear Complexity. 9.3.5 Polynomial Complexity 9.8.6 Exponential Complexity. 9.3.7. Comparisons of Complexity Classes, 10 SOME SIMPLE ALGORITHMS AND DATA STRUCTURES 10.1. Search Algorithms 10.1.1 Linear Search and Using Indirection to Access Elements 10.1.2 Binary Search and Exploiting Assumptions 10.2 Sorting Algorithms. 10.2.1 Merge Sort. 10.2.2 Exploiting Functions as Parameters. 10.2.8 Sorting in Python 10.3 Hash Tables. 11 PLOTTING AND MORE ABOUT CLASSES 11.1 Plotting Using PyLab 11.2 Plotting Mortgages, an Extended Example 12 STOCHASTIC PROGRAMS, PROBABILITY, AND STATISTICS. 12.1 Stochastic Programs 12.2 Inferential Statistics and Simulation 12.3 Distributions 12.3.1 Normal Distributions and Confidence Levels: 12.3.2 Uniform Distributions 12,3. Exponential and Geometric Distributions 12.3.4 Benford’s Distribution 12.4 How Often Does the Better Team Win?. 12.5 Hashing and Collisions 116 116 116 47 1B 18 120 121 123 124 124 126 129 130 133 13313 RANDOM WALKS AND MORE ABOUT DATA VISUALIZATION 13,1. The Drunkard’s Walk, 13.2 Biased Random Walks 13.8 Treacherous Fields, 14 MONTE CARLO SIMULATION 14.1 Paseal’s Problem 14.2 Pass o Don't Pass? 14.3 Using Table Lookup to Improve Performance 14.4 Finding x 14.5 Some Closing Remarks About Simulation Models. 15 UNDERSTANDING EXPERIMENTAL DATA 15.1 The Behavior of Springs 15.1.1 Using Linear Regression to Find a Fit 15.2 The Behavior of Projectiles 15.2.1 Coefficient of Determination 15.2.2 Using a Compuational Model 15. Not All Close Fits are Good Fits. 15.4 Fitting Exponential Distributions 15.5 When Theory is Missing, 16 LIES, DAMNED LIBS, AND STATISTICS 16.1 Garbage In Garbage Out (G1GO) 16.2. Pictures Can Be Deceiving 16.8 Cum Hoe Brgo Propter Hoe 16.4 Statistical Measures Don't Tell the Whole Story 16.5. Sampling Bias. 16.6 Context Matters 16,7 Beware of Extrapolation 10.8 The Texas Sharpshooter Fallacy 16.9 Percentages Can Confuse 16,10 Just Beware 17 KNAPSACK AND GRAPH OPTIMIZATION PROBLEMS. 17.1 Knapsack Problems 17.1.1 Greedy Algorithms17.1.2 Exponential Solution to the 0/1 Knapsack Problem, 17.2 Graph Optimization Problems 17.2.1 Some Classic Graph Theoretic Problems 17.2.2. ‘The Spread of Disease and Min Cut 17.2. Shortest Path, Depth-first search, and Breadth-first Search 18 DYNAMIC PROGRAMMING 18.1 Fibonacci Sequences, Revisited 18.2 Dynamic Programming and the 0/1 Knapsack Problem. 18.3 Dynamic Programming and Divide-and-Conquer PYTHON 2.7 QUICK REFERENCE,PREFACE ‘The book is based on an MIT course that has been offered twice a year since 2006. The course is aimed at students with little or no prior programming experience, but who have a need (or at least a desire} to understand computational approaches to problem solving, Bach year, a few of the students in the class use the course as a stepping stone to more advanced computer science courses. But for most of the students it will be their only computer science course, Because the course will be the only computer science course for most of the ts, we focus on breadth rather than depth. The goal is to provide ts with a brief introduction to many topics, so that they will have an idea of what’s possible when the time comes to think about how to use computation to accomplish a goal. That said, it is not a “computation appreciation” course. It is a challenging and rigorous course in which the students spend a lot of time and effort learning to bend the computer to their will, ‘The main goal of this book is to help you, the reader, become skilful at making productive use of computational techniques. You should leam to apply computational modes of thoughts to frame problems and to guide the process of extracting information from data in a computational manner. The primary knowledge you will take away from this book is the art of computational problem solving, The book is abit eccentric, Part 1 (Chapters 1-8) is an unconventional introduction to programming in Python, We braid together four strands of material: +The basics of programming, + The Python programming language, ‘* Concepts central to understanding computation, and ‘* Computational problem solving techniques, We cover most of Python’s features, but the emphasis is on what one can do with @ programming language, not on the language itself. For example, by the end of Chapter 3 the book has covered only a small fraction of Python, but it has already introduced the notions of exhaustive enumeration, guess and check algorithms, bisection search, and efficient approximation algorithms. We introduce features of Python throughout the book. Similarly, we introduce aspects of programming methods throughout the book. The iclea is to help you learn Python and how to be a good programmer in the context of using computation to solve interesting problems, Part 2 (Chapters 9-16} is primarily about using computation to solve problems, It assumes no knowledge of mathematics beyond high school algebra, but it, does assume that the reader is comfortable with rigorous thinking and not intimidated by mathematical concepts. It covers some of the usual topics foundin an introductory text, €.g., computational complexity and simple algorithms, But the bulk of this part of the book is devoted to topies not found in most introductory texts. Part 3 (Chapters 17-18) looks at two related and slightly advanced topics— optimization problems and dynamic programming. Experience suggests that it is quite comfortable to fit Parts 1 and 2 of this book into a one-semester course. When the material in Part 3 is included, the course hecomes more demanding than is comfortable for some students. ‘The book has two pervasive themes: systematic problem solving and the power of abstraction, When you have finished this book you should have ‘+ Learned a language, Python, for expressing computations, + Learned a systematic approach to organizing, writing and debugging medium-sized programs, + Developed an informal understanding of computational complexity, * Developed some insight into the process of moving from an ambiguous problem statement to @ computational formulation of a method for solving the problem, ‘* Learned a useful set of algorithmic and problem reduction techniques, ‘* Learned how to use randomness and simulations to shed light on problems that don easily succumb to closed-form solutions, and ¢ Learned how to use computational tools, including simple statistical and visualization tools, to model and understand data, Programming is an intrinsically difficult activity. Just as “there is no royal road to geometry,” there is no royal road to programming, — It is possible to deceive students into thinking that they have learned how to program by having them complete a series of highly constrained “fill in the blank” programming problems, However, this does not prepare students for figuring out how to harness computational thinking to solve problems. If you really want to learn the material, reading the book will not be enough, Various versions of the course have been available on MIT's OpenCourseWare (CCW) Web site since 2008. The site includes video recordings of lectures and a complete set of problem sets and exams. Since the fall of 2012, edX and MITx, have offered an online version of this course. Ifyou really want to learn the material, we strongly recommend that you do the problem sets associated with one of these online offerings, This was Euclid’s purported response, circa 300 BC, to King Ptolemy's request for an easier way to learn mathematics.SPRING 2013 EDITION Thave been working on this book for many years, and had originally planned to have it ready for publication by the summer of 2013. However, in June of 2012 Tagreed to team up with Eric Grimson and Chris Terman to offer a Fall 2012 MITx version of the MIT course on which this book is based. That led me to rush to complete a version of the hook that could be made available to the students taking that course, I knew that | could not accomplish in three months everything that I had planned to accomplish in twelve. I therefore made a decision to focus on producing an edition with the content as close to right as possible—at the cost of ignoring some of the other things (c.g., careful copy edliting) that go into producing a high-quality texthook. ‘This version is based on the Fall 2012 version. I fixed a significant number of typos, a few bugs, and made other small improvements. But I resisted the temptation to make major changes. L expect to produce a stable version of the text by summer 2013.ACKNOWLEDGMENTS ‘This book grew out of a set of lecture notes that I prepared while teaching an undergraduate course at MIT. The course, and therefore this book, benefitted from suggestions from faculty colleagues (especially Eric Grimson and Srinivas Devadas), teaching assistants (especially Asfandyar Qureshi who was the teaching assistant the first time I taught the course), and the students who took the course ‘The process of transforming my lecture notes into a book proved far more onerous than I had expected, Fortunately, this misguided optimism lasted long, enough to keep me from giving up, The encouragement of colleagues and family also helped keep me going, As the Pall 2012 version of the book neared completion, Bric Grimson, Chris ‘Terman, and David Guttag provided vital help. Eric, who is MIT's Chancellor, managed to find the time to read almost the entire book with great care, He found numerous errors (including an embarrassing, to me, number of technical errors) and pointed out places where necessary explanations were missing, Chris also read parts of the manuscript and discovered errors. He also helped me battle Microsoft Word, which we eventually persuaded to do most of what we wanted, David overcame his aversion to computer science, and proofread the later chapters. He checked my math and tuned up my verbiage. ‘The Fall 2012 version was used in the MIT course 6.00 and the MITx course 6.00x, A number of students in these courses pointed out errors. One 6,00x student, J.C. Cabrejas, was particularly helpful. He found a large number of typos, and more than a few technical errors. Thank you J.C. Lowe a special debt of gratitude to Julie Sussman, P.P.A. Until I started working with Julie, I had no idea how much difference an editor could make. [had worked with capable copy editors on previous books, and thought that was what I needed for this book. I was wrong. I needed a collaborator who could read the book with the eyes of a student, and tell me what needed to be done, what should be done, and what could be done if | had the time and energy to do it. Julie buried me in “suggestions” that couldn't be ignored. Her combined command of both the English language and programming is quite remarkable, I only wish that I hacl started working with Julie earlier in the process, The errors and awkward sentences that remain in the book are an indication that I did not allow time to take full advantage of Julie’s talent, Frédo Durand took the beautiful photograph of Antelope Canyon on the cover, 1 appreciate his allowing me to use it. Thanks also to Jason Dorfman for taking the photo on the back cover, and to the graduate students in my group for joining the pyramid. Finally, thanks to my wife Olga, for pushing me to finish and for pitching in at critical times. I should also thank her for putting up with my grumpiness during the latter stages of this project.1 GETTING STARTED A computer does two things, and two things only: it performs calculations and it remembers the results of those calculations, But it dees those two things extremely well. The typical computer that sits on a desk or in a briefcase performs a billion or so calculations a second, It’s hard to image how truly fast that is, Think about holding a ball a meter above the floor, and letting it go. By the time it reaches the floor, your computer could have executed over a billion instructions. As for memory, a typical computer might have hundreds of gigabytes of storage. How big is that? Ife byte (the number of bits, typically eight, required to represent one character] weighed one ounce (which it doesn), 100 gigabytes would weigh more than 9,000,000 tons. For comparison, that’s roughly the weight of all the coal produced in a year in the U.S, For most of human history, computation was limited by the speed of calculation of the human brain and the ability to record computational results with the human hand, This meant that only the smallest problems could be attacked computationally. Buen with the speed of modern computers, there are still problems that are beyond modern computational models (¢.g., understanding climate change, but more and more problems are praving amenable to computational solution. It is our hope that by the time you finish this book, you will fcc] comfortable bringing computational thinking to bear on solving many of the problems you encounter during your studies, work, andl even everyday life. What do we mean by computational thinking? All knowledge can be thought of as either declarative or imperative, Declarative knowledge is composed of statements of fact. For example, “the square root of x is a number y such that y-y=x.” This is a statement of fact. Unfortunately it doesnt tell us how to find a square root, Imperative knowledge is “how (o” knowledge, or recipes for deducing information. Heron of Alexandria was the first to document a way to compute the square root of a number.? His method can be summarized as: © Start with @ guess, 9, © If gig is close enough to x, stop and say that g is the answer ‘+ Otherwise create a new guess, by averaging g and x/g, ic. (9 + x/aV2 ‘+ Using this new guess, which we again call g, repeat the process until g-g is close enough to x. 2 Many believe that Heron was not the inventor of this method, and incleed! there is some evidence that it was well known to the ancient Babylonians,