ARTIFICIAL NEURAL NETWORK
Instructor: Mr Nasolwa Edson
Email:
[email protected]Office: AB14 (Administration Block)
Introduction
What is ANN?
Why ANN?
Introduction
Neural Computing
Neural Computers mimic certain processing capabilities
of the human brain. Neural Computing is an information
processing paradigm, inspired by biological system,
composed of a large number of highly interconnected
processing elements (neurons) working in unison to solve
specific problems
Introduction
Neural Computing
A neural net is an artificial representation of the human
brain that tries to simulate its learning process. The
term artificial means that neural nets are implemented in
computer programs that are able to handle the large
number of necessary calculations during the learning
process
Introduction
Neural Computing
Artificial Neural Networks (ANNs), like people, learn by
example.
An ANN is configured for a specific application, such as
pattern recognition or data classification, through a
learning process.
Learning in biological systems involves adjustments to the
synaptic connections that exist between the neurons. This
is true for ANNs as well.
Introduction
Biological Model
The human brain consists of a large number (more than a
billion) of neural cells that process information. Each cell
works like a simple processor. The massive interaction
between all cells and their parallel processing, makes the
brain's abilities possible. The structure of neuron is
shown below
Introduction
Biological model
7
Introduction
Biological model
Dendrites are the branching fibers extending from the cell
body or soma.
Soma or cell body of a neuron contains the nucleus and
other structures, support chemical processing and production
of neurotransmitters.
Axon is a singular fiber carries information away
from the soma to the synaptic sites of other neurons
(dendrites and somas), muscles, or glands. 8
Introduction
Biological model
Axon hillock is the site of summation for incoming
information. At any moment, the collective influence
of all neurons, that conduct as impulses to a given
neuron, will determine whether or not an action
potential will be initiated at the axon hillock and
propagated along the axon
9
Introduction
Biological model
Synapse is the point of connection between two neurons or a
neuron and a muscle or a gland. Electrochemical
communication between neurons takes place at these
junctions.
Terminal Buttons of a neuron are the small knobs at the end
of an axon that release chemicals called neurotransmitters.
10
Introduction
Information flow in a Neural Cell
11
Introduction
Information flow in a Neural Cell
Dendrites receive activation from other neurons.
Soma processes the incoming activations and converts them
into output activations.
Axons act as transmission lines to send activation to other
neurons.
Synapses the junctions allow signal transmission between the
axons and dendrites.
The process of transmission is by diffusion of chemicals
called neuro-transmitters.
Advantages of ANN
Nonlinearity
Input-output mapping
Adaptivity
Fault Tolerance
VLSI implementability
Introduction
Question
What is the neuron processing speed?
What is the silicon chip processing speed?
Challenges of ANN
What are the things affecting ANN?
Basic Elements of an Artificial Neuron
Consider the diagram below
16
Basic Elements of an Artificial Neuron
Weighting Factors
The values W1 , W2 , . . . Wn are weights to
determine the strength of input row vector
X = [x1 , x2 , . . . , xn]T. Each input is
multiplied by the associated weight of the neuron
connection XT W. The +ve weight excites and the -ve
weight inhibits the node output
17
Basic Elements of an Artificial Neuron
Threshold
It affects the activation of the node output y as
18
Basic Elements of an Artificial Neuron
Activation Function
An activation function performs a mathematical
operation on the signal output. The most common
activation functions are, Linear Function, Threshold
Function, Piecewise Linear Function, Sigmoidal (S
shaped) function, Tangent hyperbolic function and are
chose depending upon the type of problem to be solved by
the network.
19
Model of an Artificial Neuron
Introduction
20
Model of an Artificial Neuron
Binary Thresholding function
21
Model of an Artificial Neuron
Signum function
22
Model of an Artificial Neuron
Sigmoidal function
23
Model of an Artificial Neuron
Hyperbolic tangent function
24
Model of an Artificial Neuron
Bipolar Sigmoidal function
25
Example 1
A neural network consists four inputs with the weights as
shown.
Calculate output R of the network, if R is given by, and
find the output if binary activation function has
Been used 26
Example 2
Draw the diagram representing the following phenomena and
the calculate the output given that the inputs are
(0.5,0.8,0.6,0.4) and the weights with the bias are [w1..w4,b],
(0.25,0.2,0.1,-0.3,0.36). Using the following activation
function
1. Binary Sigmoid
2. Bipolar Sigmodal
27
Example 3
Consider the neural network as shown below, if the output of
the system is below 0.5 the system triggers flow of water
otherwise no flow of water. Evaluate what will happen
assume that the sigmoid function has been used
28
Neural Network Architectures
Is possible to use a single neuron to solve real life
problem(s)?
Neural Network Architectures
Single Layer Feedforward Network
Neural Network Architectures
Example:
Evaluate the response of the following network given that,
inputs are (0.8,0.5) the weights (2, -4) and the bias is 1
Neural Network Architectures
Multilayer Feedforward Network
Besides possessing an input and output layer also have one
or more intermediary layers
Neural Network Architectures
Example
If the weights are 0.1 and 0.6 for the first layer and 0.3 and
0.9 for the second layer calculate the response of the
system assuming sigmoid activation function
Learning in ANN
What is learning?
What are the different types of Learning in ANN?
Learning Process
Stimulation
Changes in the free parameters of the neural network
Respond in a new way
Types of Learning in ANN
Supervised Learning
Unsupervised Learning
Neural Network Architectures
Back Propagation
What is back propagation?
Why do we need back propagation?
Neural Network Architectures
Advantages of Back Propagation
• Backpropagation is fast, simple and easy to program
• It has no parameters to tune apart from the numbers of
input
• It is a flexible method as it does not require prior
knowledge about the network
• It is a standard method that generally works well
• It does not need any special mention of the features of the
function to be learned.
Neural Network Architectures
Back Propagation
In the backpropagation neural network, input signals are
propagated on a layer by-layer basis from input to output
layer for forward propagation and output to input for
backward propagation. The hidden layer is stacked in
between inputs and outputs, which allows neural networks
to learn more complicated features. The learning in multi-
layer neural networks processed the same way as in
perceptron.
Neural Network Architectures
Back Propagation
At first, a training set of input patterns is presented to the
network and then the network computes its output, if
there is a difference between the desired output and actual
output (an error) then the weights are adjusted to reduce
the difference.
Neural Network Architectures
Recurrent Networks
These networks differ from feedforward network
architecture in the sense that there is at least one feedback
loop
Neural Network Architectures
Backpropagation Algorithm steps
1. Step – 1: Forward Propagation
2. Step – 2: Backward Propagation
3. Step – 3: Putting all the values together and
calculating the updated weight value
Steps for Backward Propagation
Evaluate the total error introduced:
Steps for Backward Propagation
Calculate the error correspond to each weight with the help
of a total error.
The error on weight w is calculated by differentiating total
error with respect to w.
Steps for Backward Propagation
Calculate the updated weight w with the help of the
following formula
Steps for Backward Propagation
Calculate the updated weight for all other remaining w
Steps for Backward Propagation
Calculate all the updated weight for the hidden layer if
present
Steps for Backward Propagation
Replace all the old values of weights with the new values
and check what is the new value of error
Example 1
Consider the backpropagation neural network as shown
below, assume that the neurons have a sigmoid
activation function, do the following:
a) Perform a forward pass on the network.
b) b) Perform a reverse pass (training) once (target = 0.5,
∝=𝟏).
c) c) Perform a further forward pass and comment on the
result.
Example 1
Consider the diagram below
