Hands-On Deep Learning with Go: A practical guide to building and implementing neural network models using Go

Hands-On Deep Learning

with Go

A practical guide to building and implementing neural network models using Go

Gareth Seneque

Darrell Chua

BIRMINGHAM - MUMBAI

Hands-On Deep Learning with Go

Copyright © 2019 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Commissioning Editor: Pravin Dhandre

Acquisition Editor: Joshua Nadar

Content Development Editor: Roshan Kumar

Senior Editor: Jack Cummings

Technical Editor: Dinesh Chaudhary

Copy Editor: Safis Editing

Project Coordinator: Namrata Swetta

Proofreader: Safis Editing

Indexer: Manju Arasan

Production Designer: Jayalaxmi Raja

First published: August 2019

Production reference: 1060819

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-78934-099-0

www.packtpub.com

Packt.com

Subscribe to our online digital library for full access to over 7,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.

Why subscribe?

Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals

Improve your learning with Skill Plans built especially for you

Get a free eBook or video every month

Fully searchable for easy access to vital information

Copy and paste, print, and bookmark content

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.packt.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.

At www.packt.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks. 

Contributors

About the authors

Gareth Seneque is a machine learning engineer with 11 years' experience of building and deploying systems at scale in the finance and media industries. He became interested in deep learning in 2014 and is currently building a search platform within his organization, using neuro-linguistic programming and other machine learning techniques to generate content metadata and drive recommendations. He has contributed to a number of open source projects, including CoREBench and Gorgonia. He also has extensive experience with modern DevOps practices, using AWS, Docker, and Kubernetes to effectively distribute the processing of machine learning workloads.

Darrell Chua is a senior data scientist with more than 10 years' experience. He has developed models of varying complexity, from building credit scorecards with logistic regression to creating image classification models for trading cards. He has spent the majority of his time working with in fintech companies, trying to bring machine learning technologies into the world of finance. He has been programming in Go for several years and has been working on deep learning models for even longer. Among his achievements is the creation of numerous business intelligence and data science pipelines that enable the delivery of a top-of-the-line automated underwriting system, producing near-instant approval decisions.

About the reviewer

Xuanyi Chew is the primary author of Gorgonia. In his day job, he is the chief data scientist of a rapidly growing local start-up in Sydney. At night, he works on his hobbies of building deep learning AI (using Gorgonia), furthering his hopes of one day building an AGI. He wants to make Go the primary ecosystem for machine learning work and would love your help.

Packt is searching for authors like you

If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.

Table of Contents

Title Page

Copyright and Credits

Hands-On Deep Learning with Go

About Packt

Why subscribe?

Contributors

About the authors

About the reviewer

Packt is searching for authors like you

Preface

Who this book is for

What this book covers

To get the most out of this book

Download the example code files

Download the color images

Conventions used

Get in touch

Reviews

Section 1: Deep Learning in Go, Neural Networks, and How to Train Them

Introduction to Deep Learning in Go

Introducing DL

Why DL?

DL – a history 

DL – hype or breakthrough?

Defining deep learning

Overview of ML in Go 

ML libraries

Word-embeddings in Go 

Naive Bayesian classification and genetic algorithms for Go or Golang

ML for Go 

Machine learning libraries for Golang 

GoBrain

A set of numeric libraries for the Go programming language

Using Gorgonia

The basics of Gorgonia

Simple example – addition

Vectors and matrices

Visualizing the graph

Building more complex expressions

Summary

What Is a Neural Network and How Do I Train One?

A basic neural network

The structure of a neural network

Your first neural network

Activation functions

Step functions

Linear functions

Rectified Linear Units

Leaky ReLU

Sigmoid functions

Tanh

But which one should we use?

Gradient descent and backpropagation

Gradient descent

Backpropagation

Stochastic gradient descent

Advanced gradient descent algorithms

Momentum

Nesterov momentum

RMSprop

Summary

Beyond Basic Neural Networks - Autoencoders and RBMs

Loading data – MNIST

What is MNIST?

Loading MNIST

Building a neural network for handwriting recognition

Introduction to the model structure

Layers

Training

Loss functions

Epochs, iterations, and batch sizes

Testing and validation

Taking a closer look

Exercises

Building an autoencoder – generating MNIST digits

Layers

Training

Loss function

Input and output

Epochs, iterations, and batch sizes

Test and validation

Building an RBM for Netflix-style collaborative filtering

Introduction to RBMs

RBMs for collaborative filtering

Preparing our data – GroupLens movie ratings

Building an RBM in Gorgonia

Summary

Further reading

CUDA - GPU-Accelerated Training

CPUs versus GPUs

Computational workloads and chip design

Memory access in GPUs

Real-world performance

Intel Xeon Phi CPU

NVIDIA Maxwell GPU

Understanding Gorgonia and CUDA

CUDA

Basic Linear Algebra Subprograms

CUDA in Gorgonia

Building a model in Gorgonia with CUDA support

Installing CUDA support for Gorgonia

Linux

Windows

Performance benchmarking of CPU versus GPU models for training and inference

How to use CUDA

CPU results

GPU results

Summary

Section 2: Implementing Deep Neural Network Architectures

Next Word Prediction with Recurrent Neural Networks

Vanilla RNNs

Training RNNs

Backpropagation through time 

Cost function

RNNs and vanishing gradients

Augmenting your RNN with GRU/LSTM units

Long Short-Term Memory units

Gated Recurrent Units

Bias initialization of gates

Building an LSTM in Gorgonia

Representing text data

Importing and processing input

Summary

Further reading

Object Recognition with Convolutional Neural Networks

Introduction to CNNs

What is a CNN?

Normal feedforward versus ConvNet

Layers

Convolutional layer

Pooling layer

Basic structure

Building an example CNN

CIFAR-10

Epochs and batch size

Accuracy

Constructing the layers

Loss function and solver

Test set output

Assessing the results

GPU acceleration

CNN weaknesses

Summary

Further reading

Maze Solving with Deep Q-Networks

What is a DQN?

Q-learning

Optimization and network architecture

Remember, act, and replay!

Solving a maze using a DQN in Gorgonia

Summary 

Further reading

Generative Models with Variational Autoencoders

Introduction to VAEs

Building a VAE on MNIST

Encoding

Sampling

Decoding

Loss or cost function

Assessing the results

Changing the latent dimensions

Summary

Further reading

Section 3: Pipeline, Deployment, and Beyond!

Building a Deep Learning Pipeline

Exploring Pachyderm

Installing and configuring Pachyderm

Getting data into Pachyderm

Integrating our CNN

Creating a Docker image of our CNN

Updating our CNN to save the model

Creating a data pipeline

Interchangeable models

Mapping predictions to models

Using the Pachyderm dashboard

Summary

Scaling Deployment

Lost (and found) in the cloud

Building deployment templates

High-level steps

Creating or pushing Docker images

Preparing your AWS account

Creating or deploying a Kubernetes cluster

Kubernetes

Cluster management scripts

Building and pushing Docker containers

Running a model on a K8s cluster

Summary

Other Books You May Enjoy

Leave a review - let other readers know what you think

Preface

Go is an open source programming language designed by Google to handle huge projects efficiently. It makes building reliable, simple, and efficient software straightforward and easy.

This book immediately jumps into the practicalities of implementing Deep Neural Networks (DNNs) in Go. Simply put, the book's title contains its aim. This means there will be a lot of technical detail, a lot of code, and (not too much) math. By the time you finally close the book or turn off your Kindle, you'll know how (and why) to implement modern, scalable DNNs, and be able to repurpose them for your needs in whatever industry or mad science project you're involved.

Who this book is for

This book is for data scientists, machine learning engineers, and deep learning aspirants who are looking to inject deep learning into their Go applications. Familiarity with machine learning and basic Golang code is expected in order to get the most out of this book.

What this book covers

Chapter 1, Introduction to Deep Learning in Go, introduces the history and applications of deep learning. This chapter also gives an overview of ML with Go.

Chapter 2, What is a Neural Network and How Do I Train One?, covers how to build a simple neural network and how to inspect a graph, as well as many of the commonly used activation functions. This chapter also discusses some of the different options for gradient descent algorithms and optimizations for your neural network.

Chapter 3, Beyond Basic Neural Networks – Autoencoders and RBMs, shows how to build a simple multilayer neural network and an autoencoder. This chapter also explores the design and implementation of a probabilistic graphical model, an RBM, used in an unsupervised manner to create a recommendation engine for films.

Chapter 4, CUDA – GPU-Accelerated Training, looks at the hardware side of deep learning and also at exactly how CPUs and GPUs serve our computational needs.

Chapter 5, Next Word Prediction with Recurrent Neural Networks, goes into what a basic RNN is and how to train one. You will also get a clear idea of the RNN architecture, including GRU/LSTM networks.

Chapter 6, Object Recognition with Convolutional Neural Networks, shows you how to build a CNN and how to tune some of the hyperparameters (such as the number of epochs and batch sizes) in order to get the desired result and get it running smoothly on different computers.

Chapter 7, Maze Solving with Deep Q-Networks, gives an introduction to reinforcement learning and Q-learning and how to build a DQN and solve a maze. 

Chapter 8, Generative Models with Variational Autoencoders, shows how to build a VAE and looks at the advantages of a VAE over a standard autoencoder. This chapter also shows how to understand the effect of varying latent space dimensions on a network.

Chapter 9, Building a Deep Learning Pipeline, looks at what data pipelines are and why we use Pachyderm to build or manage them.

Chapter 10, Scaling Deployment, looks at a number of the technologies that sit underneath Pachyderm, including Docker and Kubernetes, and also examines how we can deploy stacks to cloud infrastructure using these tools .

To get the most out of this book

This book primarily uses Go, the Gorgonia package for Go, the Cu package for Go, CUDA (plus drivers) from NVIDIA, and an NVIDIA GPU that supports CUDA. Docker is also needed for Section 3, Pipeline, Deployment, and Beyond!

Download the example code files

You can download the example code files for this book from your account at www.packt.com. If you purchased this book elsewhere, you can visit www.packt.com/support and register to have the files emailed directly to you.

You can download the code files by following these steps:

Log in or register at www.packt.com.

Select the SUPPORT tab.

Click on Code Downloads & Errata.

Enter the name of the book in the Search box and follow the onscreen instructions.

Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:

WinRAR/7-Zip for Windows

Zipeg/iZip/UnRarX for Mac

7-Zip/PeaZip for Linux

The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Hands-On-Deep-Learning-with-Go. In case there's an update to the code, it will be updated on the existing GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/Hands-On-Deep-Learning-with-Go. Check them out!

Download the color images

We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: http://www.packtpub.com/sites/default/files/downloads/9781789340990_ColorImages.pdf.

Conventions used

There are a number of text conventions used throughout this book.

CodeInText: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: Mount the downloaded WebStorm-10*.dmg disk image file as another disk in your system.

A block of code is set as follows:

type nn struct {

    g *ExprGraph

    w0, w1 *Node

    pred *Node

}

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

intercept Ctrl+C

    sigChan := make(chan os.Signal, 1)

signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)

    doneChan := make(chan bool, 1)

Any command-line input or output is written as follows:

sudo apt install nvidia-390 nvidia-cuda-toolkit libcupti-dev

Bold: Indicates a new term, an important word, or words that you see on screen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: Select System info from the Administration panel.

Warnings or important notes appear like this.

Tips and tricks appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, mention the book title in the subject of your message and email us at [email protected].

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packt.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.

Piracy: If you come across any illegal copies of our works in any form on the