To you I will cry, O Lord you are my rock, do not be silent to me, lest

if you are silent to me, I become like those who go down to the pit.

DEEP LEARNING

Unit -3

Definitions and Background:

Since 2006, deep structured learning, or more commonly called deep

learning or hierarchical learning, has emerged as a new area of

machine learning research.

Definition 1: A class of machine learning techniques that exploit many

layers of non-linear information processing for supervised or

unsupervised feature extraction and transformation, and for pattern

analysis and classification.

Definition 4: “Deep learning is a set of algorithms in machine learning

that attempt to learn in multiple levels, corresponding to different

levels of abstraction. It typically uses artificial neural networks.

Definition 5: “Deep Learning is a new area of Machine Learning

research, which has been introduced with the objective of moving

1
Machine Learning closer to one of its original goals: Artificial

Intelligence.

Following is a neuron of human brain (Source : Wiki Media) . Billion

and Billions of these basic units along with some other materials

constitute our brain.

But the basic intuition is that, the general idea of a human brain

learning something is simplified down to what input(visual, audio,

touch, smell) is fed to brain how neurons from one layer are

connected to neurons in other layer, how the signal is transformed

within the neuron, and how strong the connections are in between

2
them. Dendrites fetch the input signal, nucleus or cell body transforms

the input signal, axon takes the modified signal to the other neurons.

Following is the modelling of neuron used in artificial neural

networks :

What is Deep Learning?

THREE CLASSES OF DEEP LEARNING NETWORKS

Any Deep neural network will consist of three types of layers:

 The Input Layer

 The Hidden Layer

In the above diagram, the first layer is the input layer which receives

all the inputs and the last layer is the output layer which provides the

desired output.

All the layers in between these layers are called hidden layers. There

can be n number of hidden layers thanks to the high end resources

available these days.

The number of hidden layers and the number of perceptrons in each

layer will entirely depend on the use-case you are trying to solve.

Here, we are passing the high dimensional data to the input layer. To

match the dimensionality of the input data, the input layer will contain

4
multiple sub-layers of perceptrons so that it can consume the entire

input.

The output received from the input layer will contain patterns and will

only be able to identify the edges of the images based on the contrast

levels.

This output will be fed to the Hidden layer 1 where it will be able to

identify various face features like eyes, nose, ears etc.

Now, this will be fed to the hidden layer 2 where it will able to form

the entire faces. Then, the output of layer 2 is sent to the output layer.

Apply Deep networks on a MNIST Data-set.

Deep Networks for Unsupervised or Generative Learning

 The Mnist data-set consists of 60,000 training samples and

10,000 testing samples of handwritten digit images. The task

present on the image.

 To solve this use-case a Deep network will be created with

multiple hidden layers to process all the 60,000 images pixel by

pixel and finally we will receive an output layer.

 The output layer will be an array of index 0 to 9, where each

index corresponds to the respective digit. Index 0 contains the

probability for 0 being the digit present on the input image.

 Similarly, index 2 which has a value of 0.1, actually represents

the probability of 2 being the digit present on the input image.

So, if we see the highest probability in this array is 0.8 which is

present at index 7 of the array. Hence the number present on the

image is 7.

A three-way categorization – Deep networks for unsupervised or

generative learning

Unsupervised learning is a type of machine learning algorithm used to

draw inferences from datasets consisting of input data without labeled

responses. The most common unsupervised learning method is

6
 cluster analysis, which is used for exploratory data analysis to find

hidden patterns or grouping in data.

Common clustering algorithms include:

 Hierarchical clustering: builds a multilevel hierarchy of clusters by

creating a cluster tree

 k-Means clustering: partitions data into k distinct clusters based on

distance to the centroid of a cluster

 Gaussian mixture models: models clusters as a mixture of

multivariate normal density components

 Self-organizing maps: uses neural networks that learn the topology

and distribution of the data

 Hidden Markov models: uses observed data to recover the sequence

of states

Unsupervised learning methods are used in bioinformatics for

sequence analysis and genetic clustering; in data mining for sequence

and pattern mining; in medical imaging for image segmentation; and

in computer vision for object recognition.

7
A Generative Model is a powerful way of learning any kind of data

distribution using unsupervised learning and it has achieved

tremendous success in just few years. All types of generative models

aim at learning the true data distribution of the training set so as to

generate new data points with some variations.

For this, we can leverage the power of neural networks to learn a

function which can approximate the model distribution to the true

distribution.

Two of the most commonly used and efficient approaches are

Variational Autoencoders (VAE) and Generative Adversarial

Networks (GAN). VAE aims at maximizing the lower bound of the

data log-likelihood and GAN aims at achieving an equilibrium

between Generator and Discriminator.

Deep belief network (DBN): probabilistic generative models

composed of multiple layers of stochastic, hidden variables. The top

two layers have undirected, symmetric connections between them.

8
The lower layers receive top-down, directed connections from the

layer above.

Boltzmann machine (BM): a network of symmetrically connected,

neuron-like units that make stochastic decisions about whether to be

on or off.

Restricted Boltzmann machine (RBM): a special type of BM

consisting of a layer of visible units and a layer of hidden units with

no visible-visible or hidden-hidden connections.

Deep neural network (DNN): a multilayer perceptron with many

hidden layers, whose weights are fully connected and are often

initialized using either an unsupervised or a supervised pretraining

technique. (In the literature prior to 2012, a DBN was often used

incorrectly to mean a DNN.)

Deep autoencoder: a “discriminative” DNN whose output targets are

the data input itself rather than class labels; hence an unsupervised

learning model. When trained with a denoising criterion, a deep

autoencoder is also a generative model and can be sampled from.

9
10
Deep networks for supervised learning

Supervised learning is the machine learning task of inferring a

function from labeled training data. The training data consist of a set

of training examples. In supervised learning, each example is a pair

11
 consisting of an input object (typically a vector) and a desired output

value (also called the supervisory signal).

Supervised learning is where you have input variables (x) and an

output variable (Y) and you use an algorithm to learn the mapping

function from the input to the output.

Y = f(X)

The goal is to approximate the mapping function so well that when

you have new input data (x) that you can predict the output variables

(Y) for that data.

Supervised learning problems can be further grouped into regression

and classification problems.

 Classification: A classification problem is when the output variable is

a category, such as “red” or “blue” or “disease” and “no disease”.

 Regression: A regression problem is when the output variable is a

real value, such as “dollars” or “weight”.

Some popular examples of supervised machine learning algorithms

are:

12
 Linear regression for regression problems.

 Random forest for classification and regression problems.

 Support vector machines for classification problems.

13
In deep learning, a convolutional neural network (CNN, or

ConvNet) is a class of deep neural networks, most commonly applied

to analyzing visual imagery. They are also known as shift invariant

or space invariant artificial neural networks (SIANN), based on

their shared-weights architecture and translation invariance

characteristics.[1][2] They have applications in image and video

recognition, recommender systems,[3] image classification, medical

image analysis, and natural language processing.[4]

14
15