Deep Learning Algorithm

Lecture: 06
  What is Deep Learning?
 Defining Neural Networks
 How Deep Learning Algorithms Work?
Agenda
 Types of Algorithms used in Deep Learning
  Deep learning uses artificial neural networks to perform sophisticated
computations on large amounts of data. It is a type of machine
learning that works based on the structure and function of the human
What is Deep brain.
Learning?  Deep learning algorithms train machines by learning from examples.
Industries such as health care, eCommerce, entertainment, and
advertising commonly use deep learning.
  A neural network is structured like the human brain and consists of
artificial neurons, also known as nodes. These nodes are stacked next to
each other in three layers:
• The input layer
 Defining Neural
Networks • The hidden layer(s)
• The output layer
Neural Networks

Data provides each node with information in the form of

inputs. The node multiplies the inputs with random
weights, calculates them, and adds a bias. Finally,
nonlinear functions, also known as activation functions,
are applied to determine which neuron to fire.
  While deep learning algorithms feature self-learning representations,
they depend upon ANNs that mirror the way the brain computes
information. During the training process, algorithms use unknown
elements in the input distribution to extract features, group objects,
and discover useful data patterns. Much like training machines for
How Deep Learning self-learning, this occurs at multiple levels, using the algorithms to
Algorithms Work? build the models.
 Deep learning models make use of several algorithms. While no one
network is considered perfect, some algorithms are better suited to
perform specific tasks. To choose the right ones, it’s good to gain a
solid understanding of all primary algorithms.
  Here is the list of top 10 most popular deep learning algorithms:
1.Convolutional Neural Networks (CNNs)
2.Long Short Term Memory Networks (LSTMs)
3.Recurrent Neural Networks (RNNs)
Types of Algorithms
4.Generative Adversarial Networks (GANs)
used in Deep
Learning 5.Radial Basis Function Networks (RBFNs)
6.Multilayer Perceptrons (MLPs)
7.Self Organizing Maps (SOMs)
8.Deep Belief Networks (DBNs)
9.Restricted Boltzmann Machines( RBMs)
10.Autoencoders
  CNN’s, also known as ConvNets, consist of multiple layers and are
mainly used for image processing and object detection. Yann LeCun
developed the first CNN in 1988 when it was called LeNet. It was used
Convolutional Neural for recognizing characters like ZIP codes and digits. CNN's are widely
Networks (CNNs) used to identify satellite images, process medical images, forecast time
series, and detect anomalies.
  How Do CNNs Work?
 CNN's have multiple layers that process and extract features from data:
Convolution Layer
• CNN has a convolution layer that has several filters to perform the
convolution operation.
Convolutional Neural Rectified Linear Unit (ReLU)
Networks (CNNs) • CNN's have a ReLU layer to perform operations on elements. The output is
a rectified feature map.
Pooling Layer
• The rectified feature map next feeds into a pooling layer. Pooling is a
down-sampling operation that reduces the dimensions of the feature map.
• The pooling layer then converts the resulting two-dimensional arrays from
the pooled feature map into a single, long, continuous, linear vector by
flattening it.
 Fully Connected Layer
• A fully connected layer forms when the flattened matrix from the
pooling layer is fed as an input, which classifies and identifies the
images.
 Below is an example of an image processed via CNN.
Convolutional Neural
Networks (CNNs)
  LSTMs are a type of Recurrent Neural Network (RNN) that can learn
and memorize long-term dependencies. Recalling past information for
long periods is the default behavior.
Long Short Term LSTMs retain information over time. They are useful in time-series
Memory Networks prediction because they remember previous inputs. LSTMs have a
chain-like structure where four interacting layers communicate in a
(LSTMs) unique way. Besides time-series predictions, LSTMs are typically used
for speech recognition, music composition, and pharmaceutical
development.
  How Do LSTMs Work?
• First, they forget irrelevant parts of the previous state
• Next, they selectively update the cell-state values
• Finally, the output of certain parts of the cell state
Long Short Term  Below is a diagram of how LSTMs operate:
Memory Networks
(LSTMs)
  RNNs have connections that form directed cycles, which allow the outputs
from the LSTM to be fed as inputs to the current phase.
 The output from the LSTM becomes an input to the current phase and can
memorize previous inputs due to its internal memory. RNNs are commonly
used for image captioning, time-series analysis natural-language
Recurrent Neural processing, handwriting recognition, and machine translation.
Networks (RNNs)  An unfolded RNN looks like this:
  How Do RNNs work?
• The output at time t-1 feeds into the input at time t.
• Similarly, the output at time t feeds into the input at time t+1.
Recurrent Neural
Networks (RNNs) • RNNs can process inputs of any length.
• The computation accounts for historical information, and the model
size does not increase with the input size.
  Here is an example of how Google’s autocompleting feature works:

Recurrent Neural
Networks (RNNs)
  GANs are generative deep learning algorithms that create new data
instances that resemble the training data. GAN has two components: a
generator, which learns to generate fake data, and a discriminator,
Generative which learns from that false information.
Adversarial Networks  The usage of GANs has increased over a period of time. They can be
(GANs) used to improve astronomical images and simulate gravitational lensing
for dark-matter research. Video game developers use GANs to upscale
low-resolution, 2D textures in old video games by recreating them in
4K or higher resolutions via image training.
 GANs help generate realistic images and cartoon characters, create
photographs of human faces, and render 3D objects.
  How Do GANs work?
• The discriminator learns to distinguish between the generator’s fake
data and the real sample data.
Generative
Adversarial Networks • During the initial training, the generator produces fake data, and the
(GANs) discriminator quickly learns to tell that it's false.
• The GAN sends the results to the generator and the discriminator to
update the model.
  Below is a diagram of how GANs operate:

Generative
Adversarial Networks
(GANs)
  RBFNs are special types of feedforward neural networks that use radial
basis functions as activation functions. They have an input layer, a hidden
layer, and an output layer and are mostly used for classification,
regression, and time-series prediction.
How Do RBFNs Work?
• RBFNs perform classification by measuring the input's similarity to
Radial Basis Function examples from the training set.
Networks (RBFNs) • RBFNs have an input vector that feeds to the input layer. They have a
layer of RBF neurons.
• The function finds the weighted sum of the inputs, and the output layer
has one node per category or class of data.
• The neurons in the hidden layer contain the Gaussian transfer functions,
which have outputs that are inversely proportional to the distance from the
neuron's center.
• The network's output is a linear combination of the input’s radial-basis
functions and the neuron’s parameters.
  See this example of an RBFN:

Radial Basis Function

Networks (RBFNs)
  MLPs are an excellent place to start learning about deep learning
technology.
Multilayer  MLPs belong to the class of feedforward neural networks with multiple
Perceptrons (MLPs) layers of perceptrons that have activation functions. MLPs consist of an
input layer and an output layer that are fully connected. They have the
same number of input and output layers but may have multiple hidden
layers and can be used to build speech-recognition, image-recognition,
and machine-translation software.
 How Do MLPs Work?
• MLPs feed the data to the input layer of the network. The layers of
neurons connect in a graph so that the signal passes in one direction.
• MLPs compute the input with the weights that exist between the input
Multilayer layer and the hidden layers.
Perceptrons (MLPs) • MLPs use activation functions to determine which nodes to fire.
Activation functions include ReLUs, sigmoid functions, and tanh.
• MLPs train the model to understand the correlation and learn the
dependencies between the independent and the target variables from a
training data set.
  Below is an example of an MLP. The diagram computes weights and
bias and applies suitable activation functions to classify images of cats
and dogs.

Multilayer
Perceptrons (MLPs)
  Professor Teuvo Kohonen invented SOMs, which enable data
visualization to reduce the dimensions of data through self-organizing
artificial neural networks.
 Data visualization attempts to solve the problem that humans cannot
Self Organizing Maps easily visualize high-dimensional data. SOMs are created to help users
(SOMs) understand this high-dimensional information.
 How Do SOMs Work?
• SOMs initialize weights for each node and choose a vector at random
from the training data.
• SOMs examine every node to find which weights are the most likely
input vector. The winning node is called the Best Matching Unit
(BMU).
Self Organizing Maps
(SOMs) • SOMs discover the BMU’s neighborhood, and the amount of neighbors
lessens over time.
• SOMs award a winning weight to the sample vector. The closer a node
is to a BMU, the more its weight changes..
• The further the neighbor is from the BMU, the less it learns. SOMs
repeat step two for N iterations.
  Below, see a diagram of an input vector of different colors. This data
feeds to a SOM, which then converts the data into 2D RGB values.
Finally, it separates and categorizes the different colors.

Self Organizing Maps

(SOMs)
  DBNs are generative models that consist of multiple layers of
stochastic, latent variables. The latent variables have binary values and
are often called hidden units.
Deep Belief Networks  DBNs are a stack of Boltzmann Machines with connections between
(DBNs) the layers, and each RBM layer communicates with both the previous
and subsequent layers. Deep Belief Networks (DBNs) are used for
image-recognition, video-recognition, and motion-capture data.
 How Do DBNs Work?
• Greedy learning algorithms train DBNs. The greedy learning algorithm
uses a layer-by-layer approach for learning the top-down, generative
weights.
• DBNs run the steps of Gibbs sampling on the top two hidden layers.
Deep Belief Networks This stage draws a sample from the RBM defined by the top two
(DBNs) hidden layers.
• DBNs draw a sample from the visible units using a single pass of
ancestral sampling through the rest of the model.
• DBNs learn that the values of the latent variables in every layer can be
inferred by a single, bottom-up pass.
  Below is an example of DBN architecture:

Deep Belief Networks

(DBNs)
  Developed by Geoffrey Hinton, RBMs are stochastic neural networks
that can learn from a probability distribution over a set of inputs.
 This deep learning algorithm is used for dimensionality reduction,
classification, regression, collaborative filtering, feature learning, and
topic modeling. RBMs constitute the building blocks of DBNs.
Restricted Boltzmann RBMs consist of two layers:
Machines (RBMs) • Visible units
• Hidden units
 Each visible unit is connected to all hidden units. RBMs have a bias
unit that is connected to all the visible units and the hidden units, and
they have no output nodes.
 How Do RBMs Work?
 RBMs have two phases: forward pass and backward pass.
• RBMs accept the inputs and translate them into a set of numbers that
encodes the inputs in the forward pass.
• RBMs combine every input with individual weight and one overall
Restricted Boltzmann bias. The algorithm passes the output to the hidden layer.
Machines (RBMs) • In the backward pass, RBMs take that set of numbers and translate
them to form the reconstructed inputs.
• RBMs combine each activation with individual weight and overall bias
and pass the output to the visible layer for reconstruction.
• At the visible layer, the RBM compares the reconstruction with the
original input to analyze the quality of the result
  Below is a diagram of how RBMs function:

Restricted Boltzmann
Machines (RBMs)
  Autoencoders are a specific type of feedforward neural network in
which the input and output are identical. Geoffrey Hinton designed
autoencoders in the 1980s to solve unsupervised learning problems.
They are trained neural networks that replicate the data from the input
Autoencoders layer to the output layer. Autoencoders are used for purposes such as
pharmaceutical discovery, popularity prediction, and image processing.
 How Do Autoencoders Work?
 An autoencoder consists of three main components: the encoder, the
code, and the decoder.
• Autoencoders are structured to receive an input and transform it into a
different representation. They then attempt to reconstruct the original
input as accurately as possible.
Autoencoders
• When an image of a digit is not clearly visible, it feeds to an
autoencoder neural network.
• Autoencoders first encode the image, then reduce the size of the input
into a smaller representation.
• Finally, the autoencoder decodes the image to generate the
reconstructed image.
  The following image demonstrates how autoencoders operate:

Autoencoders
  https://www.simplilearn.com/tutorials/deep-learning-tutorial/deep-l
earning-algorithm
 https://www.simplilearn.com/tutorials/deep-learning-
 tutorial/deep-learning-frameworks
Supporting Material
 https://www.simplilearn.com/tutorials/deep-learning-tutorial/introd
uction-to-deep-learning#how_do_neural_networks_work
 https://www.educba.com/deep-learning-software
Q/A Thanks