An Ingression into Deep Learning

Prelude
Have you ever wondered
How google translates an entire web page to a different language in a matter of
seconds? Or, how your phone gallery groups images into different folders based on
their locations?

All these are innovations of Deep Learning! But, what exactly is Deep Learning?
Explore to know more!

The Big Picture!

Deep Learning is a subset of Machine Learning, which in turn, is a subset of Artificial

Intelligence.
 1. Artificial Intelligence

Artificial Intelligence (AI) is a technique that helps machines to mimic human behavior.

2. Machine Learning

Machine Learning is an application of AI that allows the system to learn and improve
from experience automatically.

3. Deep Learning

Deep Learning is a type of Machine Learning that is inspired by the structure of the
brain. It is also known as Artificial Neural Network (ANN). It uses complex algorithms
and deep neural networks to train models.

What is Deep Learning?

Definition
Deep Learning involves networks which are capable of learning from data and functions
similar to the human brain.

Why Deep Learning?

Let us explore the reasons which make Deep Learning shine in the industry.

 Processes massive amount of data

Deep Learning can process an enormous amount of

both Structured and Unstructured data.

 Performs Complex Operations

Deep Learning algorithms are capable enough to perform complex operations when

compared to the Machine Learning algorithms.

Why Deep Learning?

 Achieves Best Performance
o As the amount of data increases, the performance of Machine Learning
algorithms decreases.
 o On the other hand, Deep Learning maintains the performance of the
model.
 Feature Extraction
o Machine Learning algorithms extract patterns from labeled sample data,
while Deep Learning algorithms take large volumes of data as input,
analyze them to extract the features on its own.

Deep Learning vs. Machine Learning

Say, for instance, we develop a machine that differentiates between cherries and
tomatoes.

Machine Learning

If done through Machine Learning, we need to specify the features based on which

the two can be differentiated like size and stem, in this case.

Deep Learning

In Deep Learning, the features are picked by the Neural Network without any human
intervention. But, that kind of independence can be achieved by a higher volume of data
in training the machine.

A Curtain Raiser
Watch this video to know more about Deep Learning.

Neural Networks
 The human brain contains billions of cells called Neurons. The structure of a
neuron is depicted in the above image.
 Neural Networks is a set of algorithms designed to learn the way our brain
works.
 The biological neurons inspire the structure and the functions of the neural
networks.
Biological and Artificial Neurons -
Terminologies
Biological
Artificial Neuron
Neuron

Dendrites Inputs

Nucleus Nodes

Synapse Weights

Axon Output

Did You Know?

A Node is also called a Neuron or Perceptron.

Structure of an Artificial Neural Network

The basic structure of an Artificial Neural Network (ANN) consists of artificial neuron
that are grouped into 3 different layers namely:

There are three different layers in a neural network, namely:

1. Input Layer
2. Hidden Layer
3. Output Layer

Input Layer

 The input layer communicates with the external environment and presents a

pattern to the neural network.
 Every neuron in the input layer represents an independent variable that
influences the output.
Structure of an Artificial Neural Network
Hidden Layer

 A Neural Network consists of several hidden layers, each consisting of

a collection of neurons.
 The hidden layer is an intermediate layer found between the input layer and
the output layer.
 This layer is responsible forextracting the features required from the input.
 There is no exact formula for calculating the number of the hidden layers as well
as the number of neurons in each hidden layer.

Output Layer

 The output layer of the neural network collects and transmits information in

the desired format.

Structure of an Artificial Neural Network

Every single layer is made up of one or more number of neurons. A simple model of an
Artificial neuron is shown in the figure.

Parameters represented are:

1. X_1,X_2,...,X_mX1,X2,...,Xm are the input signals.

2. W_1,W_2,...W_mW1,W2,...Wm are the weights.
3. \sum∑ is the Summation function
4. \sigmaσ is the Activation function.
5. b_kbk represents the bias value

Artificial Neural Network - Process Flow

The process flow depicted in the image is explained as follows:
  A node is where the computations happen
 A node combines the Input data X_1,X_2,...,X_mX1,X2,...,Xm with a set
of Weights W_1,W_2,...,W_mW1,W2,...,Wm which in turn aids
in amplifying or diminishing the input with respect to the polarity attached to it.
 On multiplying the inputs with weights, a specific significance is assigned to the
inputs with regard to the task the algorithm is learning. It helps in answering the
question Which input is the most helpful in learning data without error?
 The input-weight products are summed up using the Summing function, and the
output yielded is passed to the Activation function to decide whether and to
what extent the signal should progress to have an effect in the final output.

Summation and Activation Functions

Summation Function

 Various inputs are multiplied with their respective connection weights and
summed up together with the bias value.

\sum_{j=1}^{N}X_jW_j+b_k∑j=1NXjWj+bk
where:
X_jXj represents the inputs
W_jWj represents the weights

Summation and Activation Functions

Activation Function

 Activation Function aids in deriving the output.

 It is also known as the Transfer Function.
 It maps the resulting values between 0 to 1 or -1 to 1.
 There are two types of Activation function, namely:
1. Linear Activation Function
2. Non-Linear Activation Function
 Non-Linear Activation Function is the most commonly used Activation function
in Neural Networks.

Non-Linear Activation Functions

Watch this video to know some of the most commonly used Activation Functions in
Neural Networks.

Fact!

The default Activation Function is ReLU.

Working of a Neural Network

Watch this video to know the working of a Neural Network.

Fact!

In January 2018, Google's CEO Sundar Pichai

claimed that Artificial intelligence (AI) would be
more transformative to humanity than electricity.

Prelude
Curious about how weight and bias values are decided? This section will help you in
understanding the concepts of the learning process in Neural Network.

Learning Process of a Neural Network

The learning process of a Neural Network includes updating the network
architecture and connecting the weights for the network to perform efficiently.

Designing a Learning Process involves the following:

1. Learning Paradigm

Having a model from the environment in which Neural Network works.

2. Learning Rules

Figuring out the rules that aid in updating the weights.

 3. Learning Algorithms

Identifying the procedure to update the weights according to the learning rules.

Learning Paradigm
The following are the various Learning Paradigms in Neural Networks:

 Supervised
 Unsupervised
 Reinforcement

Learning Rules
The four basic types of Learning Rules in Neural Network are:

 Error Correction Rules

 Hebbian
 Boltzmann
 Competitive Learning

Learning Algorithms
Following are a few Deep Learning algorithms:

 Adaptive Resonance Theory

 Kochen Self Organization Map
 ADALINE
 Perceptron
 Backpropagation

DRC-Hubo
View the following video on DRC-Hubo developed by Team KAIST from South Korea
which impresses us by remarkably precise and deliberate motion.

Principles
A Neural Network works based on two principles, namely:

1. Forward Propagation
2. Backward Propagation
Forward Propagation in Neural Network
Explore this video to know about the Forward Propagation.

Backward Propagation in Neural Network

Explore this video to know about the Backward Propagation.

Prelude
In this section, let us discover the Network Architecture that prevails in Neural Network.

Neural Network Architecture

The architecture of a Neural Network can be broadly classified into two, namely:
1. Feed Forward Artificial Neural Network
2. Recurrent Neural Network

Feed Forward Artificial Neural Network

 

The information must flow from input to output only in one direction.
 No Feedback loops must be present.

A few Feed Forward Artificial Neural Networks are:

 Single-layer Feed Forward Network

 Multi-layer Feed Forward Network

Recurrent Neural Network

 

Information can be transmitted in both directions.

 Feedback loops are allowed.

A few Recurrent Neural Networks are:

 Hopfield Network
 Elman Network
 Jordan Network

Explore More!
The following video will help you know more about the Feed Forward Networks and
Recurrent Neural Networks in mathematical terms.
Fact!

MELTANT-\alphaα is an avatar robot with the

world’s most powerful and flexible robotic hand which
directly imitates human hand movements.

Prelude
Neural Networks are employed in various fields. Following are a few types of Neural
Networks that we will explore in this section:

 Radial Basis Function

 Long/Short Term Memory
 Gated Recurrent Unit
 Autoencoder
 Convolutional Neural Networks

Radial Basis Function

Radial Basis Function is similar to Feed Forward Network but uses Logistic function as
the Activation Function.

Long/Short Term Memory

 Long/Short Term Memory (LSTM) networks are used to classify, process, and
make predictions based on time series data.
 LSTM networks can predict the action in a specific video frame by keeping in
mind the action that occurred in the earlier frames.
 The applications of LSTM include writing, speech recognition, and so on.

Gated Recurrent Unit

 Gated Recurrent Unit (GRU) is LSTM with a forget gate.
 It is used in sound, speech synthesis, and so on.

More on LSTM and GRU

Watch the following video to know more about LSTM and GRU Neural Networks.

Autoencoder
 Autoencoder (AE) is used for classification, clustering, and feature compression.
 It is trained without supervision.

Convolutional Neural Networks

The following video will help you understand the basic concepts of Convolutional Neural
Networks (CNN).

Image Classification
 Image classification is one of the common applications of deep learning.
 A convolutional neural network can be used to recognize images and label
them automatically.

Object Recognition
 Deepnets (an optimized version of Deep Neural Networks) can be trained to
recognize different objects within the same image.

Video Recognition
 Deepnets can also be trained for real-time object recognition with respect to
time and space.

 Some of the famous use cases are driverless cars, robots, and theft detection.

Sentiment Analysis
 Using sentiment analysis, the underlying intent of the text can be extracted.
 With social media channels, it is possible to automate and measure the feelings
of the public on a given news story, topic, brand, or product.
 Positive sentiment can be identified, thereby allowing the marketing of a product,
or understanding which elements of a business strategy are working.

Medical Applications
  Deepnets can be trained to detect cancerous cells, benign and malignant tumors
from MRI and CT scans.
 They are also applied in drug discovery by training nets with molecular structure
and chemical compositions.

Deepnet Platform
 A deepnet platform is a service that allows you to incorporate deepnet in your
applications without building one from scratch.
 This platform provides a set of tools and interfaces to create a custom deepnet.

Deepnet platforms are of two types, namely:

 Software platform: This platform is available as downloadable packages that

need to be deployed on your hardware.
 Full platform: It is available as an online interactive UI to build and deploy
models without any coding experience.

More on Deepnet Platform

The following video explains what exactly is a deepnet platform.

Deepnet Platforms - Tools

The following are the tools offered by the deepnet platforms:

 Deepnet capability
 Data Munging
 UI/Model Management
 Infrastructure

H2O.ai
 H2O.ai is an open-source machine learning platform.
 Along with many machine learning algorithms, this platform currently provides a
deepnet capability known as Multilayer Perceptron (MLP).

Features
  H_2O.aiH2O.ai supports MLP for a deep learning model, which uses the L-
BFGS algorithm for backpropagation.
 H_2O.aiH2O.ai offers built-in integration tools for platforms like HDFS, Amazon
S3, SQL, and NoSQL.
 Its intuitive UI can be accessed by programming environments like R, Python,
and JSON.
 You can model and analyze data with Tableau, Microsoft Excel, and RStudio.
 Since the package needs to be downloaded and deployed in the
hardware, H_2O.aiH2O.ai comes with an in-memory map-reduce, and
columnar compression to accelerate the training of models.

Turi - A Graph-based Tool

Turi is a software platform that provides two deepnet capabilities (convolutional network
and MLP) along with machine learning and graph algorithms.

Features

 Turi provides built-in support for integration of Amazon S3, SQL DB, HDFS,
Spark RDD, and Pandas data frames.
 It offers a UI interface for model management and also includes a visualization
tool called GraphLab Canvas for visualizing the model results.
 To work with large data, Turi comes with built-in storage support like SFrame,
SArray, and SGraph.

Course Summary
While we've seen great strides in robotic capability recently, there's still a vast gap
between human and robot motor skills. Even at simple motor skills like gripping,
machine still have a very long way to go to suit human ability.
By linking learning with continuous feedback and control, however, we may begin to
bridge that gap, enabling robots to manage the complexities of the real world
intelligently and reliably.

Replicating the human brain at entirety will not just be a Science fiction episode
anymore. The future indeed will be full of surprises. Keep learning!

Turi
Quiz 1 (3 corrects)
Watch this video to know more about the Turi platform.

Process of improving the accuracy of a Neural Network is

called _______________.

Random Walk

Cross Validation

Forward Propagation

Training

In a Neural Network, all the edges and nodes have the same Weight and Bias
values.

False

True

Neural Networks Algorithms are inspired from the structure and functioning of
the Human Biological Neuron.

False
True

Name the component of a Neural Network where the true value of the input is
not observed.

Gradient Descent

Activation Function

Output Layer

Hidden Layer

Gradient at a given layer is the product of all gradients at the previous layers.

False

True

QUIZ 2 (5 CORRECTS)
All the Visible Layers in a Restricted Boltzmannn Machine are connected to
each other.

True

False

What is the method to overcome the Decay of Information through time in

RNN known as?

Gating

A Deep Belief Network is a stack of Restricted Boltzmann Machines.

TRUE

The measure of Difference between two probability distributions is know

as ________________________.

KL Divergence

RELU stands for ______________________________.

Rectified Linear Unit

QUESTIONS (19 CORRECTS)

All the Visible Layers in a Restricted Boltzmann Machine are connected to

each other.

False

True

What does LSTM stand for?

Long Short Term Memory

What is the difference between the actual output and generated output known
as? Cost

_____________ is a recommended Model for Pattern Recognition in Unlabeled

Data.

Autoencoders

GPU stands for __________.

Graphics Processing Unit

Prediction Accuracy of a Neural Network depends

on _______________ and ______________.

Weight and Bias

A Deep Belief Network is a stack of Restricted Boltzmann Machines.

True

RELU stands for ___________________________.

Rectified Linear Unit

Gradient at a given layer is the product of all gradients at the previous layers.
TRUE

What is the best Neural Network Model for Temporal Data?

Recurrent Neural Network

Data Collected from Survey results is an example of _________________.

Structured Data

Recurrent Network can input Sequence of Data Points and Produce a Sequence of
Output.

TRUE

Support Vector Machines, Naive Bayes and Logistic Regression are used for
solving _______________________ problems.

Classification

________________ works best for Image Data.

Convolution Networks

A Shallow Neural Network has only one hidden layer between Input and Output
layers.

TRUE

Process of improving the accuracy of a Neural Network is called _______________.

TRAINING

________________ models are best suited for Recursive Data.

Recursive Neural Networks

Recurrent Neural Networks are best suited for Text Processing.

True

Autoencoders are trained using _____________________.

Back Propagation

Neural Networks Algorithms are inspired from the structure and functioning of
the Human Biological Neuron.

True