INTRODUCTOR TECHNIQUES for 3-D COMPUTE VISION Emanuele Trucco Alessandro Verri‘nunc agere incipiam tibi, quod vementer ad has res attinel, esse ea quae rerum simulacra vocamus. you shall now see me begin to deal with what is of high importance to the Subject, and to show that there exists what we call images of things) Lucretius, De Rerum Nanura, 424-44 Contents Foreword Preface: About this Book Introduction 1 11 Whatis Computer Vision? 1 12 The Mary Faces of Computer Vision 2 121. Rested Disciplines 2 122. Recrchand Application Areas 3 13. Exploring the Computer Vision World. 4 131. CaxferencesJourals and Books 4 132 Income, 6 133 Some Hints on Math Sofware, 11 14 TheRoad Ahead 11 Digital Snapshots 5 21 Introdudion 16 2.2 Intensitylmages 16 21 Main Concens 16 222 Baie Optics 18 223 Baie Radiomery, 22 224 Gemmaric Image Formation, 26 23 Acquirirg Digital Images 28 2D Basic acs 28 232 Spatial Sampling 31 233 Acquistion Noise and How to Esimat 32 24 — CameraParameters M 24d Denton 34 242. Berisic Prameters, 38Contents 243 Inrnsic Parameters 36 24S Camera Models Rested, 38 25 Range Data and Range Sensors 40 25.1. Representing Range Images 41 252 Range Sensors 41 253 Active Trangulaon, 254 A Simple Sensor $4 26 Summary 47 27 Further Readings 47 28 Review 48 Dealing with Image Noise st 31 Image Noise 51 BLL Gausian Noise 3 B12 Impulsive Nowe 53 32 Noise Filtering 55 4121 Smoothing by Averaging, 56 322 Gaussian Smoodhing 423 Areour Samples Really Gausian’, 60 324 Nonlinear Filtering, 62 33 Summary 64 34 Further Readings 64 35 Review 6 Image Features 67 4.1 What Are Image Features? 68 42 Bdge Detection 69 424 Bases, 9 422. The Canny Edge Decor 71 423 Other Ee Detectors 50 424 Concluding Remarks on Edge Detection 89 43, Point Beatures: Corners. 82 44 Surface Extraction from Range Images 85 44.1. Deine Shape Clases 86 442. Esimasing Local Shape 87 45° Summary 90 46 Further Readings 90 47 Review 91 More Image Features 95 SA Introduction: Line and Curve Detection 96 52 The Hough Transform 97 S21. The Hough Transform for Lines 97 522 The Hough Traform for Curves, 100 Contents vl 523. Concluding Remarks on Hough Transforms, 100 53. Fitting Elipses to Image Data 101 $41 Euclidean Distance Fi 101 532. Algebra Disance Fi 103, ‘533 Robust ing 106 534 Concluding Remarks on Elie Fiting 107 54 Deformable Contours 108 SAL The Energy Petional, 109 542 The Element ofthe Enerty Functional, 109 543 AGreedy Algorithm, 110 55 LineGrouping 114 56 Summary 17 57 Further Readings 117 58 Review 118 Camera Calibration 123 61 Introdtetion 123 62. Direct 2arameter Calibration 125 621. Basie Equations 25 622 eal Length, Aspect Rati, and Eurinsic Parameters 127 (623 Fuimasing the Image Cenen,131 63 Camera Parameters from the Projection Matrix 132 631. imation ofthe Projection Mari, 132 632 Computing Camera Parameters, 134 64 — Conclusing Remarks 136 65 Summary 136 66 Further Readings. 136 67 Review 137 Stereopsis 139 71 Introdvetion 140 ZA Tee Two Problems of Stra, 140 712. ASinple Stereo Sytem, 188 213. Toe Poramaor ofa Str Spon, 144 712 The Correspondence Problem 145 72. Bascs 145 722 Corelation-Based Methods 146 723. Fate ated Methods, 148 724 Conluding Remarks 149 73 Epipoler Geometry 150 73. Notation, 150 732. Bases 151 733. Tre Esental Mais, E182 734 Tae Fundamental Mtr, F156Contents 735. Computing Eand F: The Figh-point gordon, 155 736 Locating the Epipotes ftom Eand F156 737. Rectfcaion, 157 74 3D Reconstruction 161 74.1 Reconsructon by Tangulaton, 122 742. Reconsnction up oa Sele Factor 164 743 Reconstruction up toa Projectve Transformation, 165 78° Summary 171 76 Further Readings 171 77 Review 172 Motion ” 81 Introduction 178, SLL. Tre Importance of Visual Motion, 178 812 The Problems of Modon Analysis 180 82 The Motion Field of Rigid Objects 183 821 Basics 18 {$22 Special Case: Pare Translation, 188 823. Special Cave 2 Moving Plane, 187 824 Motion Parlay 188 325 The Instantaneous Epipole, 191 83 The Notion of Optical Flow 191 831. ‘The Image Brighnes Constancy Equation, 122 832. The Aperure Problem, 192 833. TheValiliy ofthe Constaney Equation: Optical Flow, 166 84 Estimating the Motion Field 195 84D. Differential Techniques. 195 842. Feaure-based Telniqus 198 85 Using the Motion Feld 208 SI. 4D Motion and Sracture rom a Sparse Motion Field, 203 852 ED Motion and Sucre rom a Dense Motion Feld, 208 86 — Motion-based Segmentation 212 87 Summary 215 88 Further Readings 215 89° Review 216 Shape from Single-image Cues a9 9.1 Introduction 220 92 Shape from Shading 221 921. Dhe Refecance Map, 221 922 The Fundamental Equation, 23 93. Finding Albedo and lluminant Direction 226 83.1 Some Necessary Assumptions 226 932A Simple Method for Lamberian Swfoces 277 10 " 94 9s 96 97 98 Contents ‘A Variational Method for Shape from Shading 229 9441 The Functional 1 be Minimized, 229 942 The Euler Lagrange Equations, 230 943. from the Continuous othe Discrete Case, 231 944 The Algor, 231 945. Enforcing Igri, 232 946. Some Necessary Deals 244 Shape ‘tom Texture 235, 95.1. Whatis Texte’, 235 95.2. Using Terre to Infor Shape: Fundamentals 297 953 Surface Orientaion from Stas Texture 239 954 Concluding Remarks, 242 Summary 241 Furthe- Readings 242 Review 242 Recognition 207 101 102 03 104 10s 106 107 108 109 What Does it Mean to Recognize? 248 Interpretation Trees 249 1021 An Example, 251 1022 Wild Cards ond Spurious Features 253 1023 A Feable Algorithm, 253 Invariants. 255 1031 troduction 285 02 Definitions 26 1033 twvariant Based Recognition Algorithm, 259 Appescance Based Identification 252 M41 Images or Features? 22 1042 mage Eigenspaces, 25 Concluding Remarks on Object Identification 270, 3D Object Modelling 270, 1061 Feawre-ased and Appearance-based Modes 271 1062 Objes Versus Vewercenered Representation, 272 1063 Coneluding Remarks 273 Summary 273 Furthe- Readings 274 Review 275 Locating Objects in Space 279 Md m2 Introduction 280 Matehing from Intensity Data. 283, 1121 3D Location from a Pepectv Image, 283 122 ¥D Location froma Weak perspective Image, 288 1123 Pose from Elipes, 292Contents 11.24 Conctding Remarks, 299 113 Matching from Range Data. 294 1131 Esimating Translation Firs, 296 M22. Esimaing Rowton First, 300 1133 Concluding Remarks 301, 114 Summary 302 115 Further Readings 302 116 Review 303, ‘A Appendix 307 AJ Experiments: Good Practice Hints 307 A2 Numerical Differentiation 311 A3.TheSampling Theorem 314 Ad Projective Geometry 316 AS Differential Geometry 320, AG — Singular Value Decomposition 322 AT Robust Estimators and Model Fiting. 326 AS Kalman Filtering 328 AQ Three-dimensional Rotations 332 Index 335 Foreword Until recently, computer vision was regarded asa fleld of research stil in its infancy, rot yet mature and siable enough to be considered part ofa standard curriculum in ‘computer science. Asa consequence, most hooks on computer vision became absolete as soon as they were published. No book thus far has ever managed to provide a ‘comprehensive avervew of theficld since even the good ones focusonanarrow subarea ‘typically the author's research endeavor. With Trucco and Veri, the situation has finally changed. Ther book promises to bo the first true textbook of computer vision, the first to show that computer vision is now a mature discipline with solid foundations. Among connoisseurs, the aulhors are ‘well knowa a carefuland critical experts inthe eld (Tam proud to have figured inthe career of one of therr: Alessandro Verti worked with me at MIT fora short yea, and ‘twas ajoy to work with him) ‘Over the years I have been asked many times by new graduate students or colleagues what to readin order to lear about computer visio. Until now, my answer ‘was that I could not recommend any single book. As.a substitute, | would suggest an ‘ever-changing list of existing books together witha small collection of specific papers. rom now on, however, my answers clear: Introductory Techniques for 3-D Computer Vision i the ext to read, T personaly belive that Jnroductory Techniques for3.D Campute Visi wil he the standard textbook for graduate and undergraduate courses on computer vision in years to come, Its ar almost perfect combination of theory and practice. It provides a complete introduction to computer vison, effectively giving the base background for practitioners and future researchers in the fl “Trucco and Ver have written a textbook that is exemplary in its larity of ex- positon and in its intentions. Despite the intial warning (“Frail dice e il fare c di smezz0 il mare™), the objectives stated in the preface are indeed achieved. The book "penore wor a deere thi Foreword ‘not only places a correctly balanced emphasis on theory and practice but also provides ‘needed material about typically neglected but important topics such as measurements, calibration, SVD, robust estimation, and numerical diferentiaion, Computer vision is just now maturing from an almost esoteric corner of research to.a key discipline in computer science. In the last couple of years, the first billion- dollar computer vision companies have emerged, a phenomenon no dovbt facilitated by the irrational exuberance of the stock market. We will undoubtedly see many more commercial applications of computer vision inthe near future, ranging from industrial inspection and measurements to security database search, surveillance, multimedia and ‘computer interfaces. This is a transition that other fields in engineering, such as signal processing and computer graphics, underwent long ago. Trucco and Vers imely book isthe first to represent the discipline of computer vision ia its new, mature state, asthe industries and applications of computer vision grow and mature as wel. sit reaches ‘adulthood, computer vision is sil far from being a solved problem. The most exciting ‘developments, iscoveries and application ie ahead of us. Though a similar statement can be made about most areas of computer science, iis true for computer vision in ‘much deeper sense than, ay, for databases or graphics. Afterall, understanding the Principles of vision has implications far beyond engineering, since visual perception is ‘one of the key modules of human intelligence, Ulimately, understanding the problem ‘of vision islielyt help us understand the brain. For this reason, Iam sure that along and successful series of new editions will follow this book, with updates most likely to.come in the chapters dedicated to object recognition and in new hot topics such as adaptation and learning. Introductory Techniques for 3-D Computer Vision is much more than a good ‘textbook: I's the fist book to mark the coming of ge of our own discipline, computer ‘Tomaso Poggio ‘Cambridge, MA Brain Sciences Department and Artificial Intelligence Laboratory “Massachussetts Insitute of Technology Preface: About this Book Here tae this book and peruse it wel (Christopher Marlowe, Doctor Fuss, rai die ei fare ce’ di mezzo il mar! Ttlin proves What this Book is and is Not ‘This book le meant:0 be: ‘+ anapplie introduction tothe problems and solutions of modern computer vision ‘+ practical texbook, teaching how to develop and implement algorithms to rep: resentative problems. ‘+a structured, easy-to-follow textbook, in which each chapter concentrates on a specifi problem and solves it building on previous results, and all chapters form logical progression, + a colection ofseleted, well-tested methods (theory and algorithms), aiming to balance difficulty and applicability. ‘+ a starting point to understand and investigate the literature of computer vision, including confxences, journals, and Internet sites a sel-teaching tool for esearch students acad and professional scientists This book is nor meant to be: + an alLembracing book on computer vision and image processing ‘+ a book reporting research results that only specialist ean appreciate: It is meant for taching an exhaustive or historical review of methods and algorithms propose for each problem. ‘The choive of topics has been guided by our feeling of practitioners. There is ‘no implication whatsoever that what is lft out is unimportant. selection has been Between words ndnkthere ithe se.Preface: About this Book imposed by space limits and the intention of expining both theory and algorithms to the level of detail necessary to make implementation really posible What are the Objectives ofthis Book? + ‘Tointroduce the fundamental problems of computer vision, ‘+ Toenable the reader to implement solutions for easonaby complex problems. + To develop two parallel tracks, showing how fundamental problems are solved using both intensity and range images, wo most popular types of images in today's ‘computer vision community. ‘+ To enable the reader to make sense of the literature of computer vision. ‘What isthe Reader Expected to Know? This book hasbeen written for people itrestedin programming solutions tocomputer ‘ison problems The bes way of reading iso ty out the algorithms ona computer. ‘We assume that th eader is able oraslte ou pseudocode into computer programs, and therefore that he o hei familiar with language suitable for numerical compu tations (for instance C, Fortran). We also expect thatthe reader has aces to popular rumercl bares tik the Numerical Rerpes or Meschach orto high-evellanguages, for developing numerical software like MATLAB, Mathematica or Sia The whole book is non-languae specif: We have endeavored to present ll he necessary vision specif information, so thatthe reader anly needs some competence ina programming Language Although some of the mathematics may appear complex at fist glance, the whole book revolves around base calls linear algebra including least squares, eigenvee: tors andsingular value decomposition andthe fundamentals ofanalyticand projective geometry. Who can Benefit from this Book? + Students of university courses on computer vison, typically final-year undergrad uates or postgraduates of degrees like Computer Science, Engineering, Mathe: matics, and Physics. Most of the knowledge required to read this book should be part oftheir normal background, ‘+ Researchers looking for a modem presentation of computer vision, as well asa collection of practical algorithms covering the main problems ofthe discipline. ‘+ Teachers and students of professional training courses. ‘+ Industry scientists and academics interested in learning the fundamentals and the practical aspests of computer vision. 3For information on tia the othe packages msatiaed ere se Chae Preface: About this Book xv How is this Book Organized? Each chapter is opened by a summary fits contents, and concluded by asel-check list, review questions, acancse guide tofurther readings, as wellasexercses and suggestions for computer project. For each problem analyzed, we give 1. problem statement, defising the objective to be achieved 2, atheoretical treatment ofthe problem, 3, one or two algorithms in pseudocode. 4, hints on the practical applicability ofthe algorithms ‘few maberatil concep are cui tothe understanding of solos and signin bt not neces Haown to everybody To make te ook ressoaby Scfeontaned, we lave ince an append ih several brit section reviewing Bachgound tps We ed to gear te append to he velo etl neo 0 Understand te dscsions ofthe main extn tempt ovo jut amere spel ot vague reminder. it iade an fro Keep the fone informal thowghot,hopefly without rsaxing oo mosh te material gor. "he paps ave ben designed oft ick etication of important mater Pelem Stement port defitons and algorithms at ese i fas hin and coments of pace elvan alling codag suggestion spear ina fren point rand are bight a pointer (=). Final we ht nuded in Caper information on he computer von com smn indding ptr Internet von ies (eta image ad dimen) Anda tofthe na pucatons electron newts ndcnferences Suggestions for Instructors “Thematerialinthistextshould be enough for two semesters atthe senior undergraduate level, assuming thre hours per week. Ukimately, this depends onthe students’ back: ‘round, the desired evel of detail the choice of topics, and how much time is allocated ta project work. Insructors may want to review some of the material in the appendix in the first few lectures ofthe course nase ony ore semester i available, we suggest two selections of topics. + Stereo and Motion Chapters 1 to 6 (image acquisition, noise attenuation, feature texttaction andcalibration), then Chapters 7 (stereopsis) and 8 (motion analysis), ‘Object Recognition Chapters | to 6, then Chapters 10 (object recognition) and 11 {object location). [Ideally the students should be assigned projects to implement and test atleast some of the algorittms It is up to th instructor to decide which ones, depending on how the course is stctured, what existing software is available to students, and which parts ofthe book one wants to cover.