Abstract

This chapter aims to acquaint the users with key parts of TensorFlow and some basic ideas about deep
learning. In particular, users will figure out how to perform fundamental calculations in TensorFlow
and implementation of deep learning using TensorFlow. This chapter intends to gives a
straightforward manual for the complexities of Google's TensorFlow framework that is easy to
understand. The basic steps for the installation and setup of TensorFlow will also be discussed.
Starting with a simple “Hello World” example, a practical implementation of deep learning problem
to identify the handwritten digits will be discussed using MNIST dataset. It is only possible to
understand deep learning through substantial practical examples. For that reason, the authors have
included practical implementation of deep learning problems that motivates the readers to plunge
deeply into these examples and to get their hands grimy trying different things with their own ideas
using TensorFlow because it is never adequate to perceive algorithms only theoretically.

DEEP LEARNING
Deep learning is a subset of machine learning in Artificial Intelligence (AI) that has networks capable
of learning in a supervised, unsupervised or semi-supervised manner given in Figure 1. It is also
known as deep structured learning and hierarchical learning (Jason, 2019). The core component of
deep learning is a neural network. Most of the deep learning models are predicated on Artificial
Neural Networks. Conventional neural networks consist of a single input layer, a single hidden layer,
and one output layer. Deep learning networks are different from these conventional neural networks
comprise more hidden layers as given in Figure 2. They consist of more depths, that is the reason to
known as deep networks. These kinds of networks are proficient in exploring hidden structures from
unlabeled and unstructured data.

Figure 1. Relation between Artificial Intelligence, Machine Learning, and Deep Learning
 a) (b)
Figure 2. a) Conventional Neural Network, b) Deep Neural Network

TENSORFLOW
This chapter, entitled “Deep learning with TensorFlow,” emphasizes TensorFlow. It is one of the best
open-source software library that is employed for the implementation of deep learning developed by
Google. This framework is used for numerical computations using data flow graphs that have nodes
and edges. TensorFlow is nothing; it is like numpy with a twist. In the event that you have worked
with numpy, working with TensorFlow will be simple. The huge distinction among numpy and
TensorFlow is that TensorFlow pursues a lazy programming paradigm. It can do all that typically
would do in numpy! It’s suitably called “numpy on steroids.”

WHY TENSORFLOW?
Several libraries are available for the implementation of deep learning, so why TensorFlow?
These are the following features that TensorFlow offers:
a) It provides both python and C++ API’s that makes it convenient to work.
b) It supports both CPUs and GPUs computing devices.
c) It has a faster compilation time than other deep learning libraries like Keras and Torch.
In Figure 3 (Chollet, 2015), it is clearly mentioned that both industry and research community has the
highest adoption of the TensorFlow library with 96.77 % of the power score in 2018.
 Figure 3. Deep learning frameworks (Chollet, 2015)

TENSOR
Tensor is a generalization of vectors and matrices of potentially higher dimensions. It is the standard
way of representing data in deep learning. In simple terms, tensor is a matrix or an array of data with
different dimensions and ranks that are supplied as input to the neural network. Figures 4 and 5
represent different dimensions and ranks of tensors respectively.

Tensor of dimension 1 Tensor of dimension 2 Tensor of dimension 3

Figure 4. Different dimensions of Tensors
Rank of tensors: It is defined as a quantity of dimensions in which the tensors live.
# Rank 0 is equivalent to scalar
# Rank 1 is equivalent to vector
# Rank 2 is equivalent to 2D matrix
# Rank 3 is equivalent to 3D matrix

Rank 0 Rank 1 Rank 2 Rank 3

 Figure 5. Rank of Tensors

There are two sections in TensorFlow library:

1) Tensor: It is used to convert the data in terms of numerical value representation for
computation.
2) Flow: It describes the flow of operations.

DATAFLOW GRAPH
A computational graph or dataflow graph is a sequence of TensorFlow operations arranged into a
graph of nodes and edges given in Figure 6 (TensorFlow, 2016). It consists of two types of objects:

 tf.Operation: Nodes in the computational graph represent the operations. It defines

calculations that consume and generate tensors.
 tf.Tensor: Edges in the computational graph represent the values (tensors) that will pass
through the graph on which the operations are performed.

Figure 6: Dataflow graph (TensorFlow, 2016)

BASICS OF CODING IN TENSORFLOW
The main object of the TensorFlow program is the tf.Tensor that manipulate and pass around. A
tf.Tensor has the following characteristics:

 A data type (for example int32, float32, or string)

 A shape
The data type of each element in the tensor should be the same and is always known. The shape (i.e.,
the number of dimensions and the size of each dimension) might be partially known. Some special
kinds of tensors are as follows that will be discussed in the next section of this chapter:

 tf.Variable
 tf.constant
 tf.placeholder

TensorFlow core program consists of two distinct sections

 Building the computational graph

 Running the computational graph
Building the computational graph (using tf.Graph): TensorFlow uses a dataflow graph or
computational graph to denote the computation in terms of association between individual operations.
In this step, tf.Operation (node) and tf.Tensor (edge) objects are constructed using TensorFlow API
functions and add them to tf.Graph instance.
For example: tf.constant (32.0) creates a tf.Operation that produces the value 32.0 and added to the
default graph, and returns a tf.Tensor that represents the value of the defined constant.
Let’s understand with the help of python code. Here, a and b are two floating-point constants:

The output of the above code is:

Note that the above tensors do not print the values 3.0, 4.0, and 7.0 that you might expect. It only
builds the computational graph. A unique name is assigned to each operation in a graph. This name is
independent of the names of the objects that assigned to Python. Tensors are named after the
operation that produces them followed by an output index, as in "add:0" above.
Running the computational graph (using tf.Session): In TensorFlow, everything has to be run
within a session. We have to explicitly tell the program to run the command or object within the
TensorFlow session. We can run the computational graph with the help of session in two ways:

Method 1:
Method II:

If we are specifying the session by this particular fashion, then we have to close the session at the end
explicitly. If we are not closing the session, it may create a problem at runtime to provide the results.

TYPES OF DATA IN TENSORFLOW

There are three types of data in TensorFlow:
1) Constant
2) Placeholder
3) Variable
Constant: If the data object is specified as a constant, then its value will remain the same across the
program. We use tf.constant() command to define a constant.
Example: Multiply two constants

Placeholder: Placeholders are used to assign the data to a TensorFlow graph. The value of
the data object is assigned later in a placeholder. We use tf.placeholder() command to define
a placeholder. Final values of placeholder objects will always be assigned using feed_dict
(feed dictionary).
Example: Addition of two placeholders
Variable: It allows us to add trainable parameters to a computational graph. We use tf.Variable()
command to define a variable. In this type of data, any value is assigned initially, but during the
program, the values are changed as per the requirements. Whenever we are assigning the value to the
TensorFlow variable, we need to initialize the variable using tf.global_variables_initializer()
function.

INSTALLATION OF TENSORFLOW IN ANACONDA DISTRIBUTION

General considerations
1) Two variations of TensorFlow can be installed on whether you would like to run TensorFlow on
your CPU or GPU (If you have your own Nvidia graphics card).
2) It is a suggestion to create a different (virtual) environment if you wish to install both TensorFlow
variations on your machine. Without making virtual environments, it will either end up failing or
when you later start running code, there will consistently be incertitude about which variation is being
utilized to execute your code.
Installation steps
Step 1: The first step is to install Anaconda Python 3.7. It is optional, any python framework can be
used but the authors would suggest using Anaconda because of its instinctive way of managing
packages and setting up new virtual environments. Anaconda distribution can be downloaded from
the given link:
https://www.anaconda.com/download/
Step 2: Installation of TensorFlow CPU
It directly runs on the CPU of your machine. In some cases, you will feel the degradation of
performance in comparison to TensorFlow GPU. It can be done in 3 simple steps:
2. 1) Create a new Conda virtual environment (Optional)
a) Open Anaconda/ Prompt window

b) Type the following command given below:

The above command will create a new virtual environment with name tensorflow_cpu
c) Activate the created virtual environment using the following command:

d) After activating the virtual environment, the name of the environment will be displayed at the
beginning of the cmd path specifier within brackets:

2. 2) Install TensorFlow CPU for Python

a) Open Anaconda/ Prompt window and activate the tensorflow_cpu environment (If you have not
done earlier).
b) Type the following command on cmd and wait for the installation to finish.

2. 3) Testing of installation
a) Start python interpreter by typing the command
b) Run below code to check whether TensorFlow is successfully installed or not:

c) For the sake of the completion of testing the installation, check the output by typing ‘Hello
TensorFlow’

TENSORBOARD
TensorBoard is a utility provided by the TensorFlow. It helps to visualize the computational graph. In
other terms, it helps to visualize the training parameters, metrics, hyperparameters, or any statistics of
neural networks. It provides a set of web applications that help us to understand the flow of the
computational graph. It provides five types of visualizations:

 Scalars
 Images
 Audio
 Histograms
 Graphs

The visualization of the TensorBoard window is given in Figure 7 (Aryan, 2018).

 Figure 7. TensorBoard Window (Aryan, 2018)
The first step to visualize the graph with TensorBorad is to write the logs of the program. For this, we
need to create a writer for those logs using below line of code:

Where logdir is the directory where we want to store the log file, the second argument is the graph of
the program on which we will work. There are two methods to get the graph:

 Creating the writer out of the session using tf.get_default_graph() function, which returns the
default graph of the program.
 Creating the writer inside the session using sess.graph, which returns the graph of the session.
Python code to visualize the graph using TensorBoard is given below:

After the execution of the above code, it creates a directory with the name ‘graphs1’ in the current
working directory that contains the event file, as you can see below.
Now, open anaconda prompt and make sure that you are in a current working directory where the
python code is running. If not, change the working directory using the cd command.

Then run below command to display the graph with TensorBoard.

You can replace the name of the directory “./graphs1” if you choose another name. It will generate the
link on the command prompt (http://Sanwarul-ltp:6006/), copy this link and open it on a web browser.
The link will redirect you to the TensorBoard page that looks similar to the below image.
TensorBoard uses a web browser to display the visualization of the code. The above graph represents
the different parts of the used model. Const and const_1 represent a and b respectively and the node
Add represents c.

IMPLEMENTATION
Simple hello world code in python using TensorFlow. Either you can use Jupyter notebook, Spyder,
Pycharm framework for writing the code, or you can open a python interpreter in cmd.

The output of the above code snippet is b’hello world!’, here b represents byte strings rather than
Unicode strings. You can use Print(sess.run(hello).decode()) to remove b from the output.

DATASETS AVAILABLE WITH TENSORFLOW

In TensorFlow, there are 29 well-known datasets available such as MNIST; the Large Movie Reviews
Dataset, the 1 Billion Word Language Model Benchmark and Street View House Numbers, etc.
(Introducing TensorFlow Datasets, 2019).
You can check the detailed explanation of available datasets with TensorFlow under different
categories in given link:
https://www.tensorflow.org/datasets/catalog/overview
You can install all the datasets using the below command:

Command to check the list of available datasets:

Once datasets have been installed, import tensorflow_datasets as tfds

After importing the datasets, you can check the list of available datasets using tfds.list_builders()

A PRACTICAL IMPLEMENTATION OF DEEP LEARNING PROBLEM TO

IDENTIFY THE HANDWRITTEN DIGITS USING MNIST DATASET
In this practical implementation of the Deep Learning problem in Python, the authors are using the
MNIST dataset (LeCun, Cortes & Burges, 1998). This dataset consists of images of handwritten
digits of 28*28 pixels in size as given in Figure 8.
 Figure 8. MNIST Dataset (LeCun, Cortes & Burges, 1998)
Step 1: Importing the dataset

First, import the TensorFlow library and then add the above code to import the MNIST dataset that
stores the image data in mnist variable. Here, one-hot-encoding is used to depict the labels. It uses a
vector of binary values to render numeric or categorical values. The vector includes ten values, one
for each possible digit as we have numbers from 0-9. To represent any particular digit, the value of
the vector at that index will be 1 and the rest will be 0. For example, the digit 4 is represented using
the vector [0,0,0,0,1,0,0,0,0,0]. Now, the images of 28*28 pixels are converted into a 1D vector of
784 pixels in size. Each 784 pixels contain the value between 0 and 255 because we have a grayscale
image. In black and white images, 255 represents a black pixel and white pixel is represented by 0.

Step 2: Division of dataset for training and testing

mnist variable is used to check the size of the divided datasets for training and testing. Using the
below lines of code, it can be determined that the dataset has been split into 55000 images for
training, 5000 for validation and 10000 for testing.

Now, the dataset has been imported, it is a time to define the architecture of Neural Network.
Step 3: Defining the architecture of Neural Network
As the authors have discussed in the above section that deep neural network consists of a single input
layer, multiple hidden layers and a single output layer. The architecture of the human brain highly
inspires the architecture of a neural network. In the human brain, neurons are responsible for passing
signals around the brain. Similarly, units are responsible for taking the inputs, perform computations
and then passing the values to the other units in neural networks.
Add below lines of code to define the number of units per layer. The authors have taken 784 units in
the input layer to represent each pixel of the 1D vector as discussed above. Ten units of the output
layer represents the digits from 0 to 9.

Figure 9 represents the visualization of the defined architecture of the neural network.

Figure 9. The architecture of Deep Neural Network

The hyperparameters need to be defined here, whose values will remain constant throughout the
implementation.

The learning_rate variable defines how much the parameters will amend at each pass of the learning
process. After each pass of the learning process, we tune the weights to reduce the loss. Here,
n_iterations defines the number of iterations that represent how many times we go through the
training step. Batch size represents how many images are processing at each training step. The
dropout variable is used to represent the threshold to eliminate some units at random. Dropout will be
used in the final hidden layer to give a 50% chance to each unit from being eliminated at every
training step. This prevents overfitting. The above-discussed hyperparameters affect the learning
process. Now, the next step is to build the TensorFlow graph.
Step 4: Building the TensorFlow graph
The first step for the execution in TensorFlow is to build the computational graph. Tensor is the core
concept of TensorFlow, a data structure similar to an array or list. Below lines of code defines three
tensors as placeholders whose values will be assigned later. The dropout rate will be controlled by
Keep_prob tensor.

The parameters that will be updated in the network during the training process are weights and bias.
Bias should be initialized with small constant value to activate the tensors in the initial stages.

Now, the next step is to set up the layers in the network by defining the operations to manipulate the
tensors.

The final step in building the computational graph is to define the loss function that we want to
optimize. Cross_entropy is the popular loss function in TensorFlow, which is also known as log-loss.
There is a need for an optimization algorithm to minimize the loss function. The gradient descent
optimization is a popular method to find the minimum of a function.

The next step is to feed the input data to the computational graph for the training process.
Step 5: Training and Testing
Before starting the training process, define the method to evaluate the accuracy of mini-batches of
data during the training. Then calculate the total accuracy score for the model.
Now, initialize the session to run the graph.

The main objective of the training process in deep learning is to optimize the loss function. Here, the
authors are trying to minimize the difference between the predicted labels and the actual labels of the
images. At each training step, parameters are adjusted in such a way that the loss is reduced for the
next step.

The output of the above lines of code is:

Once the training is done, run the session on the test images. Now, the keep_prob dropout rate is 1.0
to ensure that all the units are active during the testing process.
You can see that the accuracy of the above model is approximately equal to 92%. The proposed
model can be tested on your hand-drawn number or any image of digit. One image can be
downloaded from the given command.

The above command will download the test image in the desired mentioned location.
Now, test the input image using below lines of code:

The output of the above code accurately determines that the given input image contains the image of
digit 2.

ADVANTAGES AND DISADVANTAGES OF TENSORFLOW

Advantages
1) It is an open-source library so it is easily available free of cost.
2) It provides graph visualizations using TensorBoard for a better understanding of the model.
3) It supports both CPU and GPU.
4) It has excellent community support
5) It provides regular updates and frequent releases with new features.
6) Scalability

Disadvantages
1) It does not support windows operating system. If you are a window user, you can install it
through anaconda or python package library.
2) Extra steps are required to run even a simple hello world program; we need to run the
program using a session.
3) It supports only NVIDIA GPUs and has only full language support of python.
4) It has a unique structure so it is difficult to debug the TensorFlow program.
REFERENCES
1. Jason, B. (2019), What is Deep Learning? Retrieved 10 January 2020, from
https://machinelearningmastery.com/what-is-deep-learning/
2. Chollet, F. (2015). Keras documentation. keras. io.
3. TensorFlow (2016) Retrieved 4 September 2019, from https://www.tensorflow.org/guide/graphs
4. Aryan, M. (2018), How to Use TensorBorad? Retrieved 12 September 2019, from
https://itnext.io/how-to-use-tensorboard-5d82f8654496
5. Introducing TensorFlow Datasets (2019) Retrieved 8 September 2019, from
https://medium.com/tensorflow/introducing-tensorflow-datasets-c7f01f7e19f3
6. LeCun, Y., Cortes, C., & Burges, C. J. (1998). The MNIST database of handwritten digits,
1998. URL http://yann. lecun. com/exdb/mnist, 10, 34.

ADDITIONAL READING
1) Ramsundar, B., & Zadeh, R. B. (2018). TensorFlow for deep learning: from linear regression to
reinforcement learning. " O'Reilly Media, Inc.".

2) Zaccone, G., Karim, M. R., & Menshawy, A. (2017). Deep Learning with TensorFlow. Packt
Publishing Ltd.

3) Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., ... & Ghemawat, S. (2016).
Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint
arXiv:1603.04467.