EXPERT INSIGHT

Deep Learning with

TensorFlow 2and Keras
Regression, ConvNets, GANs, RNNs,
NLP. and more with TensorFlow 2
and the Keras API

Second Edition

Antonio Gulli, Amita Kapoor, Sujit Pal

Packt
 Deep Learning with
TensorFlow 2and Keras
Second Edition

Regression, ConvNets, GANs, RNNs, NLP

2
and more with TensorFlow and the Keras API

Antonio Gulli

Amita Kapoor

Sujit Pal

02 07
02349

Packt
BIRMINGHAM -MUMBAI
Deep Learning with TensorFlow 2 and Keras
Second Edition

Copyright 2019 Packt Publishing

All No
part of this book may
be reproduced, stored in a retrieval
rights reserved.
or transmitted in any form or by any means, without the prior written
system,
of the publisher, case of brief quotations embedded in
permission except in the

critical articles or reviews.

the accuracy
Every effort has been made in the preparation of this book to ensure
of the information presented.However, in this
the informationcontained book is
sold without warranty, either express or implied.Neither the authors, nor Packt
caused
Publishing or its dealers and distributors, will be held liablefor any damages
or have been caused directly or indirectly by this book.
allegedto
Packt Publishing all
has endeavored to provide trademark information about of the
book the use
companies and products mentioned in this by appropriate of capitals.

However, Packt Publishing cannot guarantee the accuracy of this information.

Commissioning Editor: Amey Varangaonkar

Acquisition Editors: Yogesh Deokar, Ben Renow-Clarke
Acquisition Editor Peer
Reviews: Suresh Jain

Content Development Editor: lan Hough

Technical Editor: Gaurav Gavas
Project Editor: Janice Gonsalves
Proofreader:Safis Editing
Indexer: Rekha Nair
Presentation Designer: Sandip Tadge

2017
First published: April
Second edition: December 2019

Production reference:2130320

Published by Packt Publishing Ltd.

Livery Place
35 Livery Street
Birmingham B3 2PB, UK.

ISBN 978-1-83882-341-2

www.packt.com
 Packt2
packt.com

Subscribe to our online digital library for full access to over 7,000 booksand videos,
well as industry tools and
as leading to help you plan your personal development
advance your career. For more information,please visit our website.

Why subscribe?
time learning and more time coding with practical eBooks and
Spend less
Videos from over 4,000 industry professionals
Learn better with Skill Plans built especially for you
Get a free eBook or video every month

Fully searchable for easy access to vital information

Copy and paste, print, and bookmark content

eBook of book published, with PDF

Did you know thatPackt offers versions every
ePub You can version www. Packt. .com
and files available? upgrade to the eBook at
as book customer, you are entitled to a on the eBook copy. Get
and a print discount
in touch with us at [email protected] for more details.

At www. Packt.com, you can also reada collection of free technical articles, sign up
newsletters,and receive exclusive discounts and offers on Packt
for a range of free
books and eBooks.
 Table of Contents
Preface
Chapter 1: Neural Network Foundations with TensorFlow 2.0
What is TensorFlow (TF)?
What is Keras?
What are the most important changes in TensorFlow 2.07
Introduction to neural networks
Perceptron
A first example of TensorFlow 2.0 code
Multi-layer perceptron our first example of a network
Problems in training the perceptron and their solutions
Activation function sigmoid 10
Activation function tanh 10
Activation function RelLU 11

Two additional activation functions- ELU and LeakyReLU 2

Activation functions
In short whatare neural networks after all? 13
A real example recognizing handwritten digits 14
One-hot encoding (OHE) 4
Defining a simple neural network in TensorFlow 2.0 15
Running a simple TensorFlow 2.0 net and establishing a baseline 20
Improving the simple net in TensorFlow 2.0 with hidden layers 21
Further improving the simple net in TensorFlow with Dropout 24
Testing different optimizers in TensorFlow 2.0 26
Increasing the number of epochs 32
Controlling the optimizer learning rate 33
Increasing the number of internal hidden neurons
34
Tableof Contents

batch 35
Increasing the size of computation
Summarizing experiments run for recognizing handwritten charts 36
Regularization
36
Adopting regularization to avoid overfitting
36
Understanding BatchNormalization
38
Playing
with Google Colab CPUs, GPUs, and TPUs 39
Sentiment analysis 42
and AutoML
Hyperparameter tuning
Predicting output
45
A practical overview of backpropagation 46
What have we learned so far? 48
Towards a deep learning approach 48
References 49

Chapter 2:TensorFlow 1.x and 2.x 51

Understanding TensorFlow 1.x 51
TensorFlow 1.x computational graph program structure 51
Computational graphs
Working with constants, variables,and placeholders 54
Examples of operations 55
55
Constants
Sequences
Random tensors
Variables

An example of TensorFIow 1.x in TensorFlow 2.x

Understanding TensorFlow 2.x 50
Eager execution 60
AutoGraph 51
Keras APls three programming models
- 53
Sequential API
Functional API 4
Model subclassing 6
Callbacks 57
Saving a model and weights 68
Training from tf.data.datasets 59
tf.keras or Estimators? 72
Ragged tensors 74
Custom training 74
Distributed training in TensorFlow 2.x 76
Multiple GPUS
MultiWorkerMirroredStrategy 78
TPUStrategy
ParameterServerStrategy 8
Changes in namespaces 9
 Table of Contents

Converting from 1.x to 2.x 80

Using TensorFlow 2.x effectively 80
The TensorFlow 2.x 81
ecosystemn
Language bindings 82
Keras or tf.keras? 83
Summary 84
Chapter 3: Regression 87
What is regression? 87
Prediction using linear regression 88
Simple linear regression 89
Multiple linear regression 93
Multivariatelinear regression 93
TensorFlow Estimators 94
Feature columns 94
Input functions 95
MNIST using TensorFlow Estimator API 95
Predicting house price using linear regression 97
Classification tasks and decision boundaries 101
Logistic regression 102
Logistic regression on the MNIST dataset 103
Summary 107
References 108

Chapter 4: Convolutional Neural Networks 109

Deep Convolutional Neural Network (DCNN) 110
Local receptive fields 110
Shared weights and bias 111
A mathematical example 111
Convnets in TensorFlow 2.x 112
Pooling layers 113
Max pooling 113
113
Average pooling
ConvNets summary 113
An example of DCNN
-LeNet 114
LeNet code in 2.0
TensorFlow 114
Understanding the power of deep learning 121

Recognizing CIFAR-10 images with deep learning 122

Improving the CIFAR-10 performance with a deeper network 125
Improving the CIFAR-10 performance with data augmentation 128
Predicting with CIFAR-10 130
Very deep convolutional networks for large-scale image recognition 132
Recognizing cats with a VGG16 Net 134
Table of Contents
135
Utilizing tf.keras built-in VGG16 Net module 136
models for extracting features
ECYcling
prebuilt deep learning 137
Summary 138
References
Networks 139
Neural
Chapter 5: Advanced Convolutional 139
Computer vision 139
tasks
Composing CNNs for complex 140
and localization
141
assihcation
Semantic segmentation 142
Object detection 145
Instance segmentation
estimator model 147
a tf.keras
Classifying Fashion-MNIST with GPUs 150
tf.keras estimator model on
Run Fashion-MNIST the 151
Deep Inception-v3 Net used for transfer learning

horses and humans 154

for classifying
Transfer learning Hub 157
Zoos with tf keras and TensorFlow
Application 158
Keras applications 158
TensorFlow Hub
159
Other CNN architectures
159
AlexNet
Residual networks
159
160
HighwayNets and DenseNets
160
Xception
(VQA) 162
Answering questions about images
165
Style transfer 166
Content distance
167
Style distance
168
Creating a DeepDream network
172
Inspecting what a network has learned
Video 173
Classifying videos with pretrained nets in six different ways 173
Textual documents 174
Using a CNN forsentiment analysis 175
Audio and music 178
Dilated ConvNets, WaveNet, and NSynth 178
A summary of convolution operations 183
Basic convolutional neural networks (CNN or ConvNet) 183
Dilated convolution 184
Transposed convolution 184
Separable convolution 184
Depthwise convolution 185
Depthwise separable convolution 185
Capsule networks 185

[w]-
 Table of Contents

So what is the problem with CNNs? 185

So what is new with Capsule networks? 186
Summary 188
References 188
Chapter 6:Generative Adversarial Networks 191
What is a GAN? 191
MNIST using GAN in TensorFlow 193
Deep convolutional GAN (DCGAN) 198
DCGAN for MNIST digits 200
Some interesting GAN architectures 209
SRGAN 209
CycleGAN 210
InfoGAN 212
Cool applications of GANs 214
CycleGAN in TensorFlow 2.0 218
Summary 228
References 228
Chapter 7: Word Embeddings 231
Word embedding origins and fundamentals 231
Distributed representations 233
Static embeddings 234
Word2Vec 235
GloVe 238
Creating your own embedding using gensim 239
Exploring the embedding space with gensim 240
Using word embeddings for spam detection 243
Getting the data 244
Making the data ready for use 245
Building the embedding matrix 247
Define the spam classifier 248
Train and evaluate the model 250
Running the spam detector 251
Neural embeddings not just for words 252
Item2Vec 253
node2vec 253
Character and subword embeddings 259
Dynamic embeddings 260
Sentenceand paragraph embeddings 262
Language model-based embeddings 264
Using BERT as a feature extractor 267

[v]
Table of Contents
269
Fine-tuning BERT 270
Classifying
with BERT command line 271
Using BERT as part of your own network 275
Summay 275
References
279
Networks
Chapter 8:RecurrentNeural 280
The basic RNN cell 283
time (BPTT)
Backpropagation through 284
Vanishing and exploding gradients 285
RNN cell variants 285
short-term memory (LSTM)
Long 288
Gated recurrent unit (GRU) 288
Peephole LSTM 289
RNN variants 289
Bidirectional RNNs 290
Stateful RNNs 291
RNN topologies
to text 292
learning generate
Example- One-to-Many 300
Example - Many-to-One Sentiment Analysis
307
Example- Many-to-Many - POS tagging
316
Encoder-Decoder architecture seq2seq
without attention for machine translation 318
Example-seq2seq 328
Attention mechanism
for machine translation 330
Example- seq2seq with attention

Transformer architecture 336

340
Summary
References 340

Chapter 9:Autoencoders 345

Introduction to autoencoders 345
Vanilla autoencoders 347
TensorFlow Keras layers defining custom layers 348
Reconstructing handwritten digits using an autoencoder 350
354
Sparseautoencoder
Denoising autoencoders 356
Clearing images using a Denoising autoencoder 357
Stacked autoencoder 360
Convolutional autoencoder for removing noise from images 360
Keras autoencoder example sentence vectors 365
Summaryy 373
Reference 374

[vi] -
 Table of Contents

Chapter 10:UnsupervisedLearning 375

Principal component analysis 375
PCA on the MNIST dataset 376
TensorFlow Embedding API 379
K-means clustering 380
K-means in TensorFlow 2.0 381
Variations in k-means 384
Self-organizing maps 384
Colour mapping using SOM 387
Restricted Boltzmann machines 392
Reconstructing images using RBM 393
Deep belief networks 397

Variational
Autoencoders 399
Summary 404
References 405
Chapter 11: Reinforcement Learning 407
Introduction 407
RL lingo 409
Deep reinforcement learning algorithms 411
Reinforcement success in recent years 414
Introduction to OpenAI Gym 415
Random agent playing Breakout 418
Deep Q-Networks 420
DQN for CartPole 422
DQN to play a game of Atari 427
DQN variants 430
Double DQN 430
Dueling DQN 431
Rainbow 434
Deep deterministic policy gradient 434
Summary 436
References 436
Chapter 12:TensorFlow and Cloud 439
Deep learning on cloud 439
Microsoft Azure 440
Amazon Web Services (AWS) 442
Google Cloud Platform (GCP) 444
BM Cloud 447
Virtual machines on cloud 447
EC2 on Amazon 448
Compute Instance on GCP 450
vil
Table of Contents

Virtual
machine on Microsoft Azure 451
Jupyter Notebooks on cloud 451
SageMaker 452
Google Colaboratoryy 452
Microsoft Azure Notebooks
TensorFlow Extended for production 455
TFX Pipelines
456
TFX pipeline components 457
TFX libraries 458
TensorFlow Enterprise 459
Summary 459
References 460
Chapter13: TensorFlow for Mobile and loT and TensorFlow.js 461
TensorFlow Mobile 461
TensorFlow Lite 462
Quantization 462
FlatBuffers 463
Mobile converter 463
Mobile optimized interpreter 463
Supported platforms 464
Architecture 464
TensorFlow Lite 464
Using
A generic example of application 465
Using GPUs and accelerators 466
An example of application 466
Pretrained models in TensorFlow Lite
468
Image classification 470
Object detection 471
Pose estimation 471
Smart reply 471
Segmentation 471
Style transfer 471
Text classification
472
Question and answering 472
A note about using mobile GPUs 473
An overview of federated learning at the edge 474
TensorFlow FLAPIs 476
TensorFlow.js 478
Vanilla
TensorFlow js 478
Converting models 485
 Table of Contemts

Pretrained models 485

Nodejs 488
Summary 489
References 489
Chapter 14: An introduction to AutoML 491
What is AutoML? 491
Achieving AutoML 492
Automatic data preparation 493
Automatic feature engineering 493
Automatic model generation 494
AutoKeras 497
Google Cloud AutoML 498
Using Cloud AutoML- Tables solution 499
Using Cloud AutoML- Vision solution 513
Using Cloud AutoML -Text Classification solution 524
Using Cloud AutoML
-Translation solution 528
Using Cloud AutoML
-Video Intelligence Classification solution 533
Cost 539
Bringing Google AutoML to Kaggle 540
Summary 541
References 542
Chapter 15:The Math Behind Deep Learning 543
History 543
Some mathematical tools 544
Derivatives and gradients everywhere 544
Gradient descent 546
Chain rule 547
A few differentiation rules 547
Matrix operations 547
Activation functionss 548
Derivative of the sigmoid 548
Derivative of tanh 549
Derivative of RelLU 550
Backpropagation 550
Forward step 553
Backstep 554
Case 1- From hidden layer
to output layer 556
Case 2-From hidden layer to hidden layer 557
Limit of backpropagation 561
Cross entropy and its derivative 561

[ix]
Table of Contents

and mini-batch 563

stochastic gradient descent,
Batch gradient descent, 564
atch Gradient Descent(BGD) 584
(SGD)
Stochastic Gradient Descent 564
Gradient Descent
(MBGO)
Mini-Batch 564
Thinking about backpropagation and convnets 565
Thinking about backpropagation and RNNS 568
A note on TensorFlow and automatic differentiation 568
Summary 570
References

Chapter 16:Tensor Processing Unit 571

CIGIT processing units 571
CPUs and GPUs 571
TPUs 572
TPUs and Edge TPU 573
Three generations of
TPU 574
First-generation
Second-generation TPU
576
Third-generation TPU 577
Edge TPUU 578
TPU performance 578
How to use TPUs with Colab 580
Checking whether TPUs are available 580
Loading data with tf data 581
Building a model and loading it into the TPU 581
Using pretrained TPU models 584
Using
TensorFlow 2.1 and nightly build 587
Summary 588
References 589
Other Books You May Enjoy 591
Index 597