Mastering Machine Learning on AWS: Advanced machine learning in Python using SageMaker, Apache Spark, and TensorFlow

Mastering Machine Learning on AWS

Advanced machine learning in Python using SageMaker, Apache Spark, and TensorFlow

Dr. Saket S.R. Mengle

Maximo Gurmendez

BIRMINGHAM - MUMBAI

Mastering Machine Learning on AWS

Copyright © 2019 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Commissioning Editor: Sunith Shetty

Acquisition Editor: Devika Battike

Content Development Editor: Nathanya Dias

Technical Editor: Utkarsha S. Kadam

Copy Editor: Safis Editing

Project Coordinator: Kirti Pisat

Proofreader: Safis Editing

Indexer: Priyanka Dhadke

Graphics: Jisha Chirayil

Production Coordinator: Shraddha Falebhai

First published: May 2019

Production reference: 1150519

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-78934-979-5

www.packtpub.com

I would like to dedicate this book in memory of my dad. Thanks for being there for me and supporting my dreams. 

                                                                                                             – Dr. Saket S.R. Mengle 

This book is dedicated to Mateo and Paulina, who are my constant source of inspiration, joy and purpose. 

                                                                                                               – Maximo Gurmendez

mapt.io

Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.

Why subscribe?

Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals

Improve your learning with Skill Plans built especially for you

Get a free eBook or video every month

Mapt is fully searchable

Copy and paste, print, and bookmark content

Packt.com

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.packt.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.

At www.packt.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks. 

Contributors

About the authors

Dr. Saket S.R. Mengle holds a PhD in text mining from Illinois Institute of Technology, Chicago. He has worked in a variety of fields, including text classification, information retrieval, large-scale machine learning, and linear optimization. He currently works as senior principal data scientist at dataxu, where he is responsible for developing and maintaining the algorithms that drive dataxu's real-time advertising platform.

I would like to thank my wife, Sharvari, who gives me strength and inspires me to be the best version of myself every day. This book would have not been possible without her love and support. I would also like to thank my parents, Subhash and Rashmi Mengle, who taught me the value of hard work. I would like to express my appreciation to my advisor, Dr. Nazli Goharian, and Dr. Ophir Frieder, who introduced me to the world of machine learning.

Maximo Gurmendez holds a master's degree in computer science/AI from Northeastern University, where he attended as a Fulbright Scholar. Since 2009, he has been working with dataxu as data science engineering lead. He's also the founder of Montevideo Labs (a data science and engineering consultancy). Additionally, Maximo is a computer science professor at the University of Montevideo and is the director of its Data Science for Business program.

I'd like to deeply thank my wife Maggie for her sustained support, encouragement, and patience, especially throughout the long working days and busy weekends that writing this book entailed. Additionally, I'd like to thank my mother, Margarita, who taught me the importance of learning, caring, and hard work through her own example. Finally, I'd like to express my gratitude to the dataxu team, from whom I learned so much in the past ten years.

About the reviewer

Chirag Nayyar helps organizations initiate their digital transformation using the public cloud. He has been actively working on cloud platforms since 2013, providing consultancy to many organizations, ranging from small and mid-size businesses to enterprises. He holds a wide range of certifications from all major public cloud platforms. He also runs a meet-up group and is a regular speaker at various cloud events. He has also reviewed Hands-On Machine Learning on Google Cloud Platform and Google Cloud Platform Cookbook, by Packt Publishing.

Packt is searching for authors like you

If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.

Table of Contents

Title Page

Copyright and Credits

Mastering Machine Learning on AWS

Dedication

About Packt

Why subscribe?

Packt.com

Contributors

About the authors

About the reviewer

Packt is searching for authors like you

Preface

Who this book is for

What this book covers

To get the most out of this book

Download the example code files

Download the color images

Conventions used

Get in touch

Reviews

Section 1: Machine Learning on AWS

Getting Started with Machine Learning for AWS

How AWS empowers data scientists

Using AWS tools for ML

Identifying candidate problems that can be solved using ML

The ML project life cycle

Data gathering

Evaluation metrics

Algorithm selection

Deploying models

Summary

Exercises

Section 2: Implementing Machine Learning Algorithms at Scale on AWS

Classifying Twitter Feeds with Naive Bayes

Classification algorithms

Feature types

Nominal features

Ordinal features

Continuous features

Naive Bayes classifier

Bayes' theorem

Posterior  

Likelihood 

Prior probability 

Evidence 

How the Naive Bayes algorithm works

Classifying text with language models

Collecting the tweets

Preparing the data

Building a Naive Bayes model through SageMaker notebooks

Naïve Bayes model on SageMaker notebooks using Apache Spark

Using SageMaker's BlazingText built-in ML service

Naive Bayes – pros and cons

Summary

Exercises

Predicting House Value with Regression Algorithms

Predicting the price of houses

Understanding linear regression

Linear least squares estimation

Maximum likelihood estimation

Gradient descent 

Evaluating regression models 

Mean absolute error

Mean squared error

Root mean squared error

R-squared

Implementing linear regression through scikit-learn

Implementing linear regression through Apache Spark

Implementing linear regression through SageMaker's Linear Learner

Understanding logistic regression

Logistic regression in Spark

Pros and cons of linear models

Summary

Predicting User Behavior with Tree-Based Methods

Understanding decision trees

Recursive splitting

Types of decision trees

Cost functions 

Gini Impurity

Information gain

Criteria to stop splitting trees

Understanding random forest algorithms

Understanding gradient-boosting algorithms

Predicting clicks on log streams

Introduction to Elastic MapReduce (EMR)

Training with Apache Spark on EMR

Getting the data

Preparing the data

Categorical encoding 

One-hot encoding

Training a model

Evaluating our model

Area under the ROC curve

Area under the precision-recall curve

Training tree ensembles on EMR

Training gradient-boosted trees with the SageMaker services

Preparing the data

Training with SageMaker XGBoost   

Applying and evaluating the model

Summary

Exercises

Customer Segmentation Using Clustering Algorithms

Understanding how clustering algorithms work

k-means clustering 

Euclidean distance

Manhattan distance

Hierarchical clustering

Agglomerative clustering

Divisive clustering

Clustering with Apache Spark on EMR

Clustering with Spark and SageMaker on EMR

Understanding the purpose of the IAM role

Summary

Exercises

Analyzing Visitor Patterns to Make Recommendations

Making theme park attraction recommendations through Flickr data

Collaborative filtering

Memory-based approach

Model-based approach

Matrix factorization

Stochastic gradient descent

Alternating least squares 

Finding recommendations through Apache Spark's ALS

Data gathering and exploration

Training the model

Getting recommendations

Recommending attractions through SageMaker FMs

Preparing the dataset for learning

Training the model

Getting recommendations

Summary

Exercises

Section 3: Deep Learning

Implementing Deep Learning Algorithms

Understanding deep learning

Applications of deep learning

Self-driving cars

Learning to play video games using a deep learning algorithm

Understanding deep learning algorithms

Neural network algorithms

Activation functions

Backpropagation

Introduction to deep neural networks

Understanding convolutional neural networks

Summary

Exercises

Implementing Deep Learning with TensorFlow on AWS

Introducing TensorFlow

TensorFlow as a general machine learning library

Training and serving the TensorFlow model through SageMaker

Creating a custom neural net with TensorFlow 

Summary

Exercises

Image Classification and Detection with SageMaker

Introducing Amazon SageMaker for image classification

Training a deep learning model using Amazon SageMaker

Classifying images using Amazon SageMaker

Summary

Exercises

Section 4: Integrating Ready-Made AWS Machine Learning Services

Working with AWS Comprehend

Introducing Amazon Comprehend

Accessing Amazon Comprehend

Named-entity recognition using Comprehend

Sentiment analysis using Comprehend

Text classification using Comprehend

Summary

Exercises

Using AWS Rekognition

Introducing Amazon Rekognition

Implementing object and scene detection

Implementing facial analysis

Other Rekognition services

Image moderation

Celebrity recognition

Face comparison

Summary

Exercises

Building Conversational Interfaces Using AWS Lex

Introducing Amazon Lex

Building a custom chatbot using Amazon Lex

Summary

Exercises

Section 5: Optimizing and Deploying Models through AWS

Creating Clusters on AWS

Choosing your instance types

On-demand versus spot instance pricing

Reserved pricing

Amazon Machine Images (AMIs)

Deep learning hardware

Distributed deep learning

Model parallelization versus data parallelization

Distributed TensorFlow

Distributed learning through Apache Spark

Data parallelization

Model parallelization

Distributed hyperparameter tuning

Distributed predictions at scale

Parallelization in SageMaker

Summary

Optimizing Models in Spark and SageMaker

The importance of model optimization

Automatic hyperparameter tuning

Hyperparameter tuning in Apache Spark

Hyperparameter tuning in SageMaker

Summary

Exercises

Tuning Clusters for Machine Learning

Introduction to the EMR architecture

Apache Hadoop

Apache Spark

Apache Hive

Presto

Apache HBase

Yet Another Resource Negotiator (YARN)

Tuning EMR for different applications

Configuring application properties

Maximize Resource Allocation

The AWS Glue Catalog

Managing data pipelines with Glue

Creating tables with Glue

Accessing Glue tables in Spark

Summary

Deploying Models Built in AWS

SageMaker model deployment

Apache Spark model deployment

Summary

Exercises

Appendix: Getting Started with AWS

Other Books You May Enjoy

Leave a review - let other readers know what you think

Preface

AWS is constantly driving new innovations that empower data scientists to explore a variety of machine learning cloud services. This book is your comprehensive reference for learning about and implementing advanced machine learning algorithms in AWS.

As you go through this book, you'll gain insights into how these algorithms can be trained, tuned, and deployed in AWS using Apache Spark on Elastic MapReduce, SageMaker, and TensorFlow. While you focus on algorithms such as XGBoost, linear models, Factorization Machines, and deep networks, the book will also provide you with an overview of AWS, as well as detailed practical applications that will help you solve real-world problems. Every practical application includes a series of companion notebooks with all the necessary code to run on AWS. In the latter chapters, you will learn how to use SageMaker and EMR notebooks to perform a range of tasks, from smart analytics and predictive modeling through to sentiment analysis. 

By the end of this book, you will be equipped with the skills you need to effectively handle machine learning projects and implement and evaluate algorithms on AWS.

Who this book is for

This book is for data scientists, machine learning developers, deep learning enthusiasts, and AWS users who want to build advanced models and smart applications on the cloud using AWS and its integration services. Some understanding of machine learning concepts, Python programming, and AWS will be beneficial.

What this book covers

Chapter 1, Getting Started with Machine Learning for AWS, introduces you to machine learning. It explains why it is necessary for data scientists to learn about machine learning and how AWS can help them to solve various real-world problems. We also discuss the AWS services and tools that we will be covered in the book.

Chapter 2, Classifying Twitter Feeds with Naive Bayes, introduces the basics of the Naive Bayes algorithm and presents a text classification problem that will be addressed by the use of this algorithm and language models. We'll provide examples explaining how to apply Naive Bayes using scikit-learn and Apache Spark on SageMaker's BlazingText. Additionally, we'll explore how to use the ideas behind Bayesian reasoning in more complex scenarios. We will use the Twitter API to stream tweets from two different political candidates and predict who wrote them. We will use scikit-learn, Apache Spark, SageMaker, and BlazingText.

Chapter 3, Predicting House Value with Regression Algorithms, introduces the basics of regression algorithms and applies them to predict the price of houses given a number of features. We'll also introduce how to use logistic regression for classification problems. Examples in SageMaker for scikit-learn and Apache Spark will be provided. We'll be using the Boston Housing Price dataset (https://www.kaggle.com/c/boston-housing/) along with scikit-learn, Apache Spark, and SageMaker.

Chapter 4, Predicting User Behavior with Tree-Based Methods, introduces decision trees, random forests, and gradient-boosted trees. We will explore how to use these algorithms to predict when users will click on ads. Additionally, we will explain how to use AWS EMR and Apache Spark to engineer models at a large scale. We will use the Adform click prediction dataset (https://doi.org/10.7910/DVN/TADBY7, Harvard Dataverse, V2). We will use the XGBoost, Apache Spark, SageMaker, and EMR libraries.

Chapter 5, Customer Segmentation Using Clustering Algorithms, introduces the main clustering algorithms by exploring how to apply them for customer segmentation based on consumer patterns. Through AWS SageMaker, we will show how to run these algorithms in skicit-learn and Apache Spark. We will use the e-commerce data from Fabien Daniel (https://www.kaggle.com/fabiendaniel/customer-segmentation/data) and scikit-learn, Apache Spark, and SageMaker.

Chapter 6, Analyzing Visitor Patterns to Make Recommendations, presents the problem of finding similar users based on their navigation patterns in order to recommend custom marketing strategies. Collaborative filtering and distance-based methods will be introduced with examples in scikit-learn and Apache Spark on AWS SageMaker. We will use Kwan Hui Lim's Theme Park Attraction Visits dataset (https://sites.google.com/site/limkwanhui/datacode), Apache Spark, and SageMaker.

Chapter 7, Implementing Deep Learning Algorithms, introduces you to the main concepts behind deep learning and explains why it has become so relevant in today's AI-powered products. The aim of this chapter is to not discuss the theoretical details of deep learning, but to explain the algorithms with examples and provide a high-level conceptual understanding of deep learning algorithms. This will give you a platform to understand what you will be implementing in the next chapters.

Chapter 8, Implementing Deep Learning with TensorFlow on AWS, goes through a series of practical image-recognition problems and explains how to address them with TensorFlow on AWS. TensorFlow is a very popular deep learning framework that can be used to train deep neural networks. This chapter will explain how TensorFlow can be installed and used to train deep learning models using toy datasets. In this chapter, we'll use the MNIST handwritten digits dataset (http://yann.lecun.com/exdb/mnist/), along with TensorFlow and SageMaker.

Chapter 9, Image Classification and Detection with SageMaker, revisits the image classification problem we dealt with in the previous chapters, but using SageMaker's image classification algorithm and object detection algorithm. We'll use the Caltech256 dataset (http://www.vision.caltech.edu/Image_Datasets/Caltech256/) and AWS Sagemaker.

Chapter 10, Working with AWS Comprehend, explains the functionality of an AWS tool called Comprehend, which is a natural language processing tool that performs various useful tasks.

Chapter 11, Using AWS Rekognition, explains how to use Rekognition, which is an image recognition tool that uses deep learning. You will learn an easy way of applying image recognition in your applications.

Chapter 12, Building Conversational Interfaces Using AWS Lex, explains that AWS Lex is a tool that allows programmers to build conversational interfaces. This chapter introduces you to topics such as natural language understanding using deep learning.

Chapter 13, Creating Clusters on AWS, addresses how one of the key problems in deep learning is understanding how to scale and parallelize learning on multiple machines. In this chapter, we'll examine different ways to create clusters of learners. In particular, we'll focus on how to parallelize deep learning pipelines through distributed TensorFlow and Apache Spark.

Chapter 14, Optimizing Models in Spark and SageMaker, explains that the models that are trained on AWS can be further optimized to run smoothly in production environments. In this section, we will discuss various tricks that you can use to improve the performance of your algorithms.

Chapter 15, Tuning Clusters for Machine Learning, explains that many data scientists and machine learning practitioners face the problem of scale when attempting to run machine learning data pipelines at scale. In this chapter, we focus primarily on EMR, which is a very powerful tool for running very large machine learning jobs. There are many ways to configure EMR, and not every setup works for every scenario. We will go through the main configurations of EMR and explain how each configuration works for different objectives. Additionally, we'll present other ways to run big data pipelines through AWS.

Chapter 16, Deploying Models Built on AWS, discusses deployment. At this point, you will have your model built on AWS and would like to ship it to production. There are a variety of different contexts in which models should be deployed. In some cases, it's as easy as generating a CSV of actions that would be fed to some system. Often, we just need to deploy a web service that's capable of making predictions. However, there are many times in which we need to deploy these models to complex, low-latency, or edge systems. We will go through the different ways you can deploy machine learning models to production in this chapter.

To get the most out of this book

This book covers a number of different frameworks, including as Spark and TensorFlow. However, it is not meant to be a comprehensive guide to each framework. Instead, we focus on the way AWS empowers practical machine learning through the use of the different frameworks. We encourage you to refer to other books with framework-specific content when necessary. 

Download the example code files

You can download the example code files for this book from your account at www.packt.com. If you purchased this book elsewhere, you can visit www.packt.com/support and register to have the files emailed directly to you.

You can download the code files by following these steps:

Log in or register at www.packt.com.

Select the SUPPORT tab.

Click on Code Downloads & Errata.

Enter the name of the book in the Search box and follow the onscreen instructions.

Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:

WinRAR/7-Zip for Windows

Zipeg/iZip/UnRarX for Mac

7-Zip/PeaZip for Linux

The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Mastering-Machine-Learning-on-AWS. In case there's an update to the code, it will be updated on the existing GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Download the color images

We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: http://www.packtpub.com/sites/default/files/downloads/9781789349795_ColorImages.pdf.

Conventions used

There are a number of text conventions used throughout this book.

CodeInText: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: "The following screenshot shows the first