Image Processing with Python Expert techniques for advanced image analysis and effective fierce teem tac le ots ce) me (laps sea) Sandipan DeyHands-On Image Processing with Python Expert techniques for advanced image analysis and effective interpretation of image data Sandipan Dey Packt» BIRMINGHAM - MUMBAIHands-On Image Processing with Python Copyright ©2018 Packt Publishing Allrights reserved, No part ofthis book may be reproduced, stored in a retrieval system, of transmitted in any form forby any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in eiticl articles or reviews, Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Nether the author, nor Packt Publishing or its dealers and distributors, wil beheld lable for any damages caused or alleged to hhave been caused direcly oF indirectly by this book. Packt Publishing has endeavored to provide trademark information about al of the companies and products ‘mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantoe the accuracy ‘ofthis information, ‘Commissioning Editor: Pravin Dhandre Acquisition Editor: Devika Batike Content Development Editor: Unnati Guha ‘Technical Editor Dinesh Chaudhary. Copy Editor. Safs Editing Project Coordinator: Manthan Patel Proofreader SafisFlting ndexer Prauk Shirodkar GGraphies: isha Chieayil Production Coordinator: Shraddha Falebhai First published: November 2018 Production reference: 1201118 Published by Packt Publishing Ltd Livery Place 85 Livery Street Birmingham 132P8, UK. ISBN 97841-78934.973-1I dedicate this book to my beloved parents.sar. Mapt apt io Maptis an online digital library that gives you full access to over 5,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website. Why subscribe? * Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals + Improve your learning with Skill Plans built especially for you * Get a free eBook or video every month + Mapt is fully searchable + Copy and paste, print, and bookmark content Packt.com Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at ww packt .con and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at custonercareépacktpub . com for more details. At www.packt.con, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.Contributors About the author Sandipan Dey is a data scientist with a wide range of interests, covering topics such as machine learning, deep learning, image processing, and computer vision. He has worked in numerous data science fields, working with recommender systems, predictive models for the events industry, sensor localization models, sentiment analysis, and device prognostic He earned his master’s degree in computer science from the University of Maryland, Baltimore County, and has published in a few IEEE Data Mining conferences and journals. He has earned certifications from 100* MOOCs on data science, machine learning, deep learning, image processing, and related courses/specializations. He is a regular blogger on his blog (sandipanweb) and is a machine learning education enthusiast. Tam grateful to the excellent online courses provided by top schools across the globe over the last few years. Few of them are: Image processing (@Coursera by Duke, Northwestern), Computer vision and image analysis (@edX by Microsoft), Computational photography (@Coursera by Georgia Tech), Machine learning (@Coursera by Stanford, University of Toronto; @eaX by UCSD), Deep learning (e@Coursera by deeplearning.ai; @Udacity by Google)About the reviewer Nikhil Borkar holds a COF designation and a postgraduate degree in quantitative finance. He also holds the certified financial crime examiner and certified anti-money laundering professional qualifications. He is a registered research analyst with the Securities and Exchange Board of India (SEBI) and has a keen grasp of the Indian regulatory landscape pertaining to securities and investments. He is currently working as an independent FinTech and legal consultant. Prior to this, he worked with Morgan Stanley Capital International (MSCD) as a global RFP project manager. Packt is searching for authors like you If you're interested in becoming an author for Packt, please visit a and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can ‘make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea packtpub-comTable of Contents Preface Chapter 1: 4: Getting Started with Image Processing What is image processing and some applications ‘What is an image and how it is stored on a computer What is image processing? Some applications of image processing The image processing pipeline Setting up different image processing libraries in Python Installing pip Installing some image processing libraries in Python Installing the Anaconda distribution Installing Jupyter Notebook Image I/O and display with Python Reading, saving, and displaying an image using PIL Providing the correct path to the images on the disk Reading, saving, and displaying an image using Matplotlib Intefpolating while displaying with Matplotio imshow) Reading, saving, and displaying an image using scikit-mage Using scikit:mage's astronaut dataset Reading and displaying multiple images at once Reading, saving, and displaying an image using scipy misc Using scipy.misc’s face dataset Dealing with different image types and file formats and performing basic image manipulations Dealing with different image types and file formats File formats Converting fom one fle format to another Image types (modes) Converting rom one image mode into another ‘Some color spaces (channels) Converting from one color space into another Data structures to store images Converting image data structures Basic image manipulations Image Manipulations with numpy array slicing Simple image morphing -a-blending of two images using cross-cissolving Image manipulations with PIL. Cropping an image Resising an image Nogating an image Converting an image into grayscale Some gray-level transformationsTable of Contents Some geometric transformations Changing pixel values ofan image Drawing on an image Drawing text on an mage Greating a thumbnail Computing the basic statistics ofan image Plotting the histograms of peel values forthe RGB channels of an image Separating the RGB channels ofan mage Combining muftpie channels ofan image scblending two images Superimpesing two images ‘sing we images Computing the difference between two images Subtracting two images and superimposing two image negatves Image manipulations with scikitimage inverse warping and geometric ransfrmation using the warp funtion ‘oping the sul transform ‘Adding fandom Gaussian noise to images Computing he cumulative distribution nection of an image Image manipulation with Matpiotib Drawing contour nes for an image Image manipulation with the scipy misc and scipy.ndimage modules Summary Questions Further reading Chapter 2: 2: Sampling, Fourier Transform, and Convolution Image formation - sampling and quantization ‘Sampling Up-sampling Up-sampling and interpolation Down-sampling Down-sampiing and anti-aliasing Quantization ‘Quantizing with PIL Discrete Fourier Transform Why do we need the DFT? The Fast Fourier Transform algorithm to compute the DFT The FFT with the scipy fflpack module Plotting the frequency spectrum The FFT with the numpy.fft module Computing the magnitude and phase of a DFT Understanding convolution Why convolve an image? Convolution with SciPy signal's convolvezd ‘Applying convolution to a grayscale image Convolution modes, pad values, and boundary conditions Applying convolution to a color (RGB) image Convolution with SciPy ndimage.convolve Correlation versus convolution Template matching with cross-correlation between the image and template 38 40 41 2 3 a 44 45 46 48 47 43 49 49 50 51 31 52 52 53 53 53 54 56 58 59 59 60 81 64 66 69 n 2 2B B a 75 B 79 20 80 at a2 a2 a6 86 89 [ii]Table of Contents Summary Questions Further reading Chapter 3: 3: Convolution and Frequency Domain Filtering Convolution theorem and frequency domain Gaussian blur Application of the convolution theorem Frequency domain Gaussian blur fier with numpy ft Gaussian kemelin the frequency domain Frequency domain Gaussian blur fiter wth scipy signal. titconvolve() Comparing the runtimes of SciPy convolve() and fftconvolve() with the Gaussian blur kernel Filtering in the frequency domain (HPF, LPF, BPF, and notch filters) What is a filter? High-Pass Filter (HPF) How SNR changes with frequency cut-off Low-pass filter (LPF) LPF with scipy ndimage and numpy fft LPF with fourier_gaussian LPF with scipy ffipack How SNR changes with frequency cutoff Band-pass fiter (BPF) with DoG Band-stop (notch) filter Using a notch fiter to remove periodic noise from images Image restoration Deconvolution and inverse fitering with FFT Image deconvolution with the Wiener fiter Image denoising with FFT Filter in FFT Reconstructing the final image ‘Summary Questions Further reading Chapter 4: 4: Image Enhancement Point-wise intensity transformations — pixel transformation Log transform Power-law transform Contrast stretching Using PIL as a point operation Using the PIL ImageEnhance module Thresholding With a fixed threshold Half-toning Floyd-Steinberg dithering with error diffusion Histogram processing — histogram equalization and matching Contrast stretching and histogram equalization with scikit-image Histogram matching a a 2 93 93 94 95 96 99 101 103 103, 104 109 110 110 110 112 116 1168 17 118 120 721 125 126 128 129 130 131 131 132 133 134 138 138 139 1a 142 143 145, 148 147 148 153 [iii]Table of Contents Histogram matching for an RGB image Linear noise smoothing Smoothing with PIL ‘Smoothing with ImageFilter. BLUR ‘Smoothing by averaging with the box blur kernel ‘Smoothing with the Gaussian blur fiter Comparing smoothing with box and Gaussian kernels using SciPy ndimage Nonlinear noise smoothing ‘Smoothing with PIL Using the median fiter Using max and min fiter Smoothing (denoising) with scikit-image Using the bilateral fiter Using non-local means Smoothing with scipy ndimage Summary Questions Image derivatives — Gradient and Laplacian Derivatives and gradients Displaying the magnitude and the gradient on the same image Laplacian ‘Some notes about the Laplacian Effects of noise on gradient computation Sharpening and unsharp masking ‘Sharpening with Laplacian Unsharp masking With the SciPy ndimage module Edge detection using derivatives and filters (Sobel, Canny, and so on) ‘With gradient magnitude computed using the partial derivatives The non-maximum suppression algorithm Sobel edge detector with sciki-image Different edge detectors with scikit-mage — Prewitt, Roberts, Sobel, Scharr, and Laplace The Canny edge detector with scikitimage The LoG and DoG filters The LoG filter with the SciPy ndimage module Edge detection with the LoG filter Edge detection with the Marr and Hildreth’ algorithm using the zero-crossing ‘computation Finding and enhancing edges with PIL. Image pyramids (Gaussian and Laplacian) — blending images ‘A Gaussian pyramid with scikit-image transform pyramid module ‘A Laplacian pyramid with scikitimage transform’s pyramid module 155 187 187 187 159 160 161 162 163 163 165 165 168 168 170 1 172 172 173 174 174 7 178 178 180 181 1382 183 183 1986 186 188 189 192 194 196 200 201 202 203, 205 2068 208 liv]Table of Contents Constructing the Gaussian Pyramid Reconstructing an image only from its Laplacian pyramid Blending images with pyramids Summary Questions Further reading Chapter 6: 6: Morphological Image Processing The scikit-image morphology module Binary operations Erosion Dilation Opening and closing Skeletonizing Computing the convex hull Removing small objects White and black top-hats Extracting the boundary Fingerprint cleaning with opening and closing Grayscale operations The scikit-image filter.rank module Morphological contrast enhancement Noise removal with the median filter Computing the local entropy The SciPy ndimage.morphology module Filling holes in binary objects Using opening and closing to remove noise Computing the morphological Beucher gradient Computing the morphological Laplace Summary Questions Further reading Chapter 7: 7: Extracting Image Features and Descriptors Feature detectors versus descriptors Harris Corner Detector With scikit-image With sub-pixel accuracy ‘An application — image matching Robust image matching using the RANSAC algorithm and Harris Comer features Blob detectors with LoG, DoG, and DoH Laplacian of Gaussian (LoG) Difference of Gaussian (DoG) Determinant of Hessian (DoH) Histogram of Oriented Gradients 210 214 216 219 219 220 222 223 223 223, 224 225 27 27 229 231 231 232 234 236 236 237 238 240 240 281 243 244 245, 246, 287 248 248 250 250 251 253, 253 287 258 258 258 260 Iv]Table of Contents Algorithm to compute HOG descriptors Compute HOG descriptors with scikit-image Scale-invariant feature transform ‘Algorithm to compute SIFT descriptors With opency and opencv-contrib Application ~ matching images with BRIEF, SIFT, and ORB Matching images with BRIEF binary descriptors with scikitmage Matching with ORB feature detector and binary descriptor using scikitmage Matching with ORB features using brute-force matching with python-opency Brute-force matching with SIFT descriptors and ratio test with OpenCV Haar-like features Haar-like feature descriptor with scikit-image Application — face detection with Haar-like features Face/eye detection with OpenCV using pre-trained classifiers with Haar- cascade features ‘Summary Questions Further reading Chapter 8: 8: Image Segmentation What is image segmentation? Hough transform — detecting lines and circles Thresholding and Otsu's segmentation Edges-based/region-based segmentation Edge-based segmentation Region-based segmentation “Morphological watershed algorithm Felzenszwalb, SLIC, QuickShift, and Compact Watershed algorithms Felzenszwalb's efficient graph-based image segmentation SLIC RAG merging QuickShift Compact Watershed Region growing with SimpleITK Active contours, morphological snakes, and GrabCut algorithms Active contours Morphological snakes GrabCut with OpenCV Summary Questions Further reading Chapter 9: 9: Classical Mac! Processing ‘Supervised versus unsupervised learning 1e Learning Methods in Image 260 260 261 262 262 264 264 266 268 270 272 272 274 274 276 277 277 279 279 280 284 286 287 289 289 293 293, 297 298 299 301 302 307 307 309 ait 318. 316 316 317 318 [vilTable of Contents Unsupervised machine learning — clustering, PCA, and eigenfaces K:means clustering for image segmentation with color quantization Spectral clustering for image segmentation PCA and eigenfaces Dimension reduction and visualization with PCA '2D projection and visualization Eigenfaces with PCA Eigenfaces Reconstruction Eigen decomposition Supervised machine learning — image classification Downloading the MNIST (handwritten digits) dataset Visualizing the dataset Training KNN, Gaussian Bayes, and SVM models to classify MNIST k-nearest neighbors (KNN) classifier ‘Squared Euclidean distance Computing the nearest neighbors Evaluating the performance of the classifier Bayes classifier (Gaussian generative model) ‘Training the generative model ~ computing the MLE of the Gaussian parameters Computing the posterior probabilities Yo make predcions on test data and model SVM classifier Supervised machine learning — object detection Face detection with Haar-like features and cascade classifiers with AdaBoost — Viola-Jones Face classification using the Haar-like feature descriptor Finding the most important Haar-lke features for face classification withthe random forest ensemble classifier Detecting objects with SVM using HOG features HOG training Classification with the SVM model Computing BoundingBoxes with HOG-SVM Non-max suppression Summary Questions Further reading Chapter 10: 10: Deep Learning in Image Processing - Image Classification Deep learning in image processing ‘What is deep learning? Classical versus deep learning Why deep leaming? CNNs Conv or pooling or FC layers ~ CNN architecture and how it works Convolutional layer Pooling layer Non-linearty ~ ReLU layer 319 319 323 325, 325 328 327 331 33 333 334 335, 338 338 338 338 339 340 341 342 344 346 347 348 352 354 354 355, 356 357 358 359 360 361 362 362 363, 365, 365 365 366 368 368 [vii]Table of Contents FC layer Dropout, Image classification with TensorFlow or Keras Classification with TF Classification with dense FC layers with Keras. Visualizing the network Visualizing the weights inthe intermediate layers CNN for classification with Keras Classifying MNIST Visualizing the intermediate layers Some popular deep CNN VGG-16/19 Classifying cat/idog images with VGG-16 in Keras. ‘Teer (predcion) phase InceptionNet ResNet Summary Questions Further reading Chapter 11: 11: Deep Learning in Image Processing - Object Detection, and more Introducing YOLO v2 Classifying and localizing images and detecting objects Proposing and detecting objects using CNNs Using YOLO v2 Using a pre-trained YOLO model for object detection Deep semantic segmentation with DeepLab V3+ Semantic segmentation DeepLab V3+ DeepLab v3 architecture Steps you must follow to use DeepL.ab V3+ model for semantic segmentation Transfer learning — what it is, and when to use it Transfer learning with Keras Neural style transfers with cv2 using a pre-trained torch model Understanding the NST algorithm Implementation of NST with transfer learning Ensuring NST with content loss. Computing the style cost Computing the overall loss Neural style transfer with Python and OpenCV Summary Questions Further reading Chapter 12: 12: Additional Problems in Image Processing 368 368 369 369 377 379 380 382 382 385, 387 387 389 389 396, 398 399) 401 401 402 403 404 404 408 407 407 415 45 416 416 47 420 421 426 426 «ar 428 428 429 429 432 433 433 434 [viii]Table of Contents Seam carving Content-aware image resizing with seam carving Object removal with seam carving Seamless cloning and Poisson image editing Image inpainting Variational image processing Total Variation Denoising Creating flat-texture cartoonish images with total variation denoising Image quilting Texture synthesis Texture transfer Face morphing Summary Questions Further reading Other Books You May Enjoy Index 434 435 439 440 443 446 447 449 450 450 450 451 453 453 453 455 458 [ix]Preface This book covers how to solve image processing problems using popular Python image processing libraries (such as PIL, scikit-image, python-apenco, scipy nulimage, and SimplelTK), machine learning libraries (scikit-learn), and deep learning libraries (TensorFlow, Keras). It will enable the reader to write code snippets to implement complex image processing algorithms, such as image enhancement, filtering, restoration, segmentation, classification, and object detection. The reader will also be able to use machine learning and deep learning models to solve complex image processing problems. The book will start with the basics and guide the reader to go to an advanced level by providing Python-reproducible implementations throughout the book. The book will start from the classical image processing techniques and explore the journey of evolution of the image processing algorithms all the way through to the recent advances in image processing/computer vision with deep learning. Readers will learn how to use the image processing libraries, such as PIL, scikit-image, and scipy ndimage in Python, which will enable them to write code snippets in Python 3 and quickly implement complex image processing algorithms, such as image enhancement, filtering, segmentation, object detection, and classification. The reader will learn how to use machine learning models using the scikitlearn library and later explore deep CNN such as VGG-19 with TensorFlow/Keras, use the end-to-end deep learning YOLO model for object detection, and DeepLab V3+ for semantic segmentation and neural-style transfer models. The reader will also learn a few advanced problems, such as image inpainting, gradient blending, variational denoising, seam carving, quilting, and morphing. By the end of this book, the reader will learn to implement various algorithms for efficient image processing, This book follows a highly practical approach that will take its readers through a set of image processing concepts/algorithms and help them learn, in detail, how to use leading Python library functions to implement these algorithms. Disclaimer The images used in this book as inputs and the outputs can be found at neepe: //www packtpub.com/sites/default/£iles/downloads/9781789343731_ColorTnages. pat,Preface Who this book is for This book is for engineers/applied researchers, and also for software engineers interested in computer vision, image processing, machine learning, and deep learning, especially for readers who are adept at Python programming and who want to explore various topics on image processing in detail and solve a range of complex problems, starting from concept through to implementation. A math and programming background, along with some basic knowledge of machine learning, are prerequisites What this book covers chapter 1, Getting Started with Image Processing, covers image processing and its applications, different Python libraries, image input/output, data structures, file formats, and basic image manipulations Chapter 2, Sampling, Fourier Transform, and Convolution, covers 2D Fourier transform, sampling, quantization, discrete Fourier transform, ID and 2D convolution and filtering in the frequency domain, and how to implement them with Python using examples. You will learn the simple signal processing tools that are needed in order to understand the following units chapter 2, Convolution and Frequency Domain Filtering, demonstrates how convolution is carried out on images using Python. Topics such as filtering in the frequency domain are also covered, chapter 4, Image Enhancement, covers some of the most basic tools in image processing, such as mean/median filtering and histogram equalization, which are still among the most, powerful. We will describe these and provide a modern interpretation of these basic tools. chapter 5, Image Enhancement using Derivatives, covers further topics associated with image enhancement, in other words, the problem of improving the appearance or usefulness of an image. Topics covered include edge detection with derivatives and Laplacian, sharpening, and pseudo coloring. All the concepts will be described with the help of examples involving Python Chapter 6, Morphological Image Processing, covers binary operations and the use of filter rank module to perform operations such as morphological contrast enhancements, noise removal, and computing local entropy. We will also see how a morphology module is used. chapter 1, Extracting Image Features and Descriptors, describes several techniques for extracting features from images/compute image descriptors, [2]Preface chapter 8, Image Segmentation, outlines the basic techniques for partitioning an image, from a simple threshold to more advanced graph cuts. Chapter 9, Classical Machine Learning Methods in Image Processing, introduces a number of different machine learning techniques for image classification and object detection/recognition. chapter 10, Deep Learning in Image Processing — Image Classification, describes why the image processing/computer vision community gradually transitioned from the classical feature- based machine learning models to deep learning models. chapter 11, Deep Learning in Image Processing - Object Detection, and more, describes a number of remarkable applications of the CNNs for object detection, semantic segmentation, and image style transfer. A few popular models, such as YOLO and object, proposals, will be demonstrated. How to use transfer learning to avoid learning a very deep neural net from scratch will also be outlined chapter 12, Additional Problems in Image Processing, describes a number of additional image processing problems and various algorithms for solving them, Problems include seam carving (for context-aware image resizing), image quilting (for image resizing with non- parametric sampling and texture transfer), poisson (gradient) image editing (blending) to seamlessly blend one image within another, image morphing (to transform one image to another), image inpainting (to restore a degraded image), and some variational image processing techniques (for image denoising, for example) To get the most out of this book 1. A basic knowledge of Python is required to run the codes, along with access to image datasets and the GitFiub link. 2. A basic Math background is also needed to understand the concepts. Download the example code files ‘You can download the example code files for this book from your account at wnve.packt .con. If you purchased this book elsewhere, you can visit wnw.packt .com/supeort and register to have the files emailed directly to you. [3]Preface You can download the code files by following these steps: 1. Log in or register at www pack .com 2. Select the SUPPORT tab. 3, Click on Code Downloads & Errata. 4, Enter the name of the book in the Search box and follow the onscreen instructions. ‘Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of: + WinRAR(/7-Zip for Windows + ZipegiiZip/UnRarX for Mac + 7-Zip/PeaZip for Linux The code bundle for the book is also hosted on GitHub athete: ssing-with case there's an update to the code, it will be updated on the existing GitHub repository. :PubLishing/Hands-on~Inage-Ps: We also have other code bundles from our rich catalog of books and videos available at hetpe://githuh.com/PacktPubliching/. Check them out! Download the color images We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: nt zp: //wxw.packtpub.com/sizes/default/fites, lownloads/9781789343731_colortmages.pat Conventions used ‘There are a number of text conventions used throughout this book. Text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles, Here is an example: "Mount the downloaded fiehst orm-10* .cmg disk image file as another disk in your system.” [4]Preface A block of code is set as follows: viewer = viewer. InageViewer (im) viewer.show(} When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold [sefauit) exten => s exten => s,102, Voicemail (b100) exten => i, 1,Voicenail (s0) Any command-line input or output is written as follows >>> pip install numpy >>> pip install scipy Bold: Indicates a new term, an important word, or words that you sce on screen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: ‘Select System info from the Administration panel Oo Warnings or important notes appear like this. r?) Tips and tricks appear like this. Get in touch Feedback from our readers is always welcome. General feedback: If you have questions about any aspect of this book, mention the book title in the subject of your message and email us at cust [email protected]: [5]Preface Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit ww. st-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details. packt .com/e Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at copyright @packt .con with a link to the material. If you are interested in becoming an author: If there is a topic that you have experti and you are interested in either writing or contributing to a book, piease visit Reviews Please leave a review. Once you have read and used this book, why not leave a review on. the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you! For more information about Packt, please visit pace: [6]