Artificial Neural Networks

Dr. Md. Aminul Haque Akhand

Dept. of CSE, KUET

2
Contents

1. Introduction

2. Biological Neural Networks

3. Artificial Neural Networks

4. Neural Networks for Classification Tasks

5. Performance Evaluation and Benchmark Problems

3
Why Artificial Neural Networks?
Human brain has many desirable characteristics not presented in modern
computers:
massive parallelism,
distributed representation and computation,
learning ability,
generalization ability,
adaptability,
inherent contextual information processing,
fault tolerance, and
low energy consumption.

It is hope that devices based on biological neural networks will process some of
these characteristics.
4
Why Artificial Neural Networks?

Modern digital computers outperform humans in the domain of numeric

computation and the related symbolic manipulation.

However, humans can solve complex perceptual problems effortlessly,

and more efficiently than the world's fastest computer.

Human performs these remarkable benefits due to the characteristics of

his/her neural system whose (neural system) working procedure is completely
different from a conventional computer system.
Comparison of Brains and Traditional Computers

 200 billion neurons, 32  1 billion bytes RAM but trillions

trillion synapses of bytes on disk
 Element size: 10-6 m  Element size: 10-9 m
 Energy use: 25W  Energy watt: 30-90W (CPU)
 Processing speed: 100 Hz  Processing speed: 109 Hz
 Parallel, Distributed  Serial, Centralized
 Fault Tolerant  Generally not Fault Tolerant
 Learns: Yes  Learns: Some
 Intelligent/Conscious:  Intelligent/Conscious: Generally
Usually No
6
von Neumann Computer vs. Biological Neural System
von Neumann Computer Biological Neural System
complex simple
processor high-speed low-speed
one (or a few) a large number
separate integrated in the neuron
memory localized distributed
noncontent addressable content addressable
centralized distributed
computing sequential parallel
strored-programs self-learning
reliability vulnerable robust
numerical perceptual problems
expertise
symbolic manipulations
poorly defined
environment well-defined
unconstrained
7
The Goal

ANN is designed with the goal of

building intelligent machines to
solve complex perceptual
problems, such as pattern
recognition and optimization, by
mimicking the networks of real
neurons in the human brain.

Tasks handled by ANNs

8
Artificial Neural Networks(ANNs) Motivations
ANNs are inspired by biological neural networks
 Biological Neuron
9
10
Synapses
11
A Real Neural Network
12
ANN’s Relationship with Other Disciplines
13
Computational Model
 Ability of Single Neuron
x1 x 0 Bias (x =1,always)
0

w
1 w
0

x2

w
Inputs

Output
2

 n
1 if  wi xi  0
x3
w
  i 0

3

0 else
x w
4
4

x w
5 5
Logical Operators x2
x 0
0 + +
-0.5
x 1

OR
1
n
1 if  wi xi  0 OR
  i 0
0 else
x - + x1
2 1
x2
x 0
-1.5
x AND - +
1
 n
1 1 if  wi xi  0
  i 0
0 else AND
x 2
1
x - - x1
0
0.0
 n
1 if  wi xi  0
NOT
x 1
  i 0

-1.0 0 else

 Nonlinear Problem
x2
x 0
?
+ -
x 1

?
n
1 if  wi xi  0 XOR
  i 0
XOR
0 else
x 2 ? - + x1

No arrangement work, not linearly separable. Require two boundary lines.

17
XOR solution with HN

AND

OR

XOR

XOR logic through manipulation by hidden neurons

18
XOR solution with HN

0.5
-2
1 1 1.5

1 1

An additional dimension (input) converts the problem to a linearly separable.

19
Learning
The ability to learn is a fundamental trait (characteristic) of intelligence.

ANN-> Updating NN architecture and connection weights to perform a

specific task efficiently.

ANN ability to automatically learning from examples makes them

attractive.

To understand or design a learning process need: a model of environment

or information is available to NN and learning rules.

A learning algorithm -> procedure for adjusting weights

Three main learning paradigms-> Supervised (learning with a teacher),

unsupervised and hybrid

Reinforcement learning is variant of supervised learning receives critique

on the correctness of network output instead of correct answer.
20
Learning
Learning theory must address three fundamental and practical
issues:

1.Capacity – how many patterns can be stored , what functions and

decision boundaries a NN can perform

2.Sample Complexity – Number training patterns needed to train

3.Computational Complexity – Time required for a learning algorithm to

estimate a solution

Four basic types of Learning rules: Error-Correction, Boltzman, Hebbian,

and Competitive.
21
Network Architecture / Topology
Based on Connection pattern (architecture): feed forward and recurrent (feedback)

Different architectures require appropriate learning algorithms.

 23 Well Known Learning Algorithms

Paradigm Learning rule Architecture Learning algorithm Task

Supervised Error-correction Single- or multilayer Perceptron learning algorithms Pattern classification
    perceptron Back-propagation Function approximation
      Adaline and Madaline Prediction, control
  Boltzmann Recurrent Boltzmann learning algorithm Pattern classification
  Hebbian Multilayer feed-forward Linear discriminant analysis Data analysis
        Pattern classification
  Competitive Competitive Learning vector quantization Within-class categorization
        Data compression
    ART network ARTMap Pattern classification
        Within-class categorization
Unsupervised Error-correction Multilayer feed-forward Sammon's projection Data analysis
Feed-forward or
  Hebbian Principal component analysis Data analysis
competitive
        Data compression
    Hopfield Network Associative memory learning Associative memory
  Competitive Competitive Vector quantization Categorization
        Data compression
    Kohonen's SOM Kohonen's SOM Categorization
        Data analysis
    ART networks ART1, ART2 Categorization
Error-correction
Hybrid RBF network RBF learning algorithm Pattern classification
and competitive
      Function approximation
        Prediction, control
24
Learning Rule: Error-Correction

 error signal = desired response –

output signal
 ek(n) = dk(n) –yk(n)

 ek(n) actuates a control

mechanism to make the output
signal yk(n) come closer to the
desired response dk(n) in step by
step manner
 Learning Rule: Boltzmann
 The primary goal of Boltzmann learning is to produce a
neural network that correctly models input patterns
according to a Boltzmann distribution.

 The Boltzmann machine consists of stochastic neurons.

A stochastic neuron resides in one of two possible
states (± 1) in a probabilistic manner.
 The use of symmetric synaptic connections between
neurons.

 The stochastic neurons partition into two functional

groups: visible and hidden.

 During the training phase of the network , the visible

neurons are all clamped onto specific states determined
by the environment.

 The hidden neurons always operate freely; they are

used to explain underlying constraints contained in the
environmental input vectors.
 Learning Rule: Hebbian
1. If two neurons on either side of synapse (connection) are
activated simultaneously, then the strength of that synapse
is selectively increased.
2. If two neurons on either side of a synapse are activated
asynchronously, then that synapse is selectively
weakened or eliminated.

A Hebbian synapse increases its strength with

positively correlated presynaptic and postsynaptic
signals, and decreases its strength when signals
are either uncorrelated or negatively correlated.
 Learning Rule: Competitive

 The output neurons of a neural network

compete among themselves to become
active.
 - a set of neurons that are all the same
(excepts for synaptic weights)
 - a limit imposed on the strength of each
neuron
 - a mechanism that permits the neurons
to compete -> a winner-takes-all

 The standard competitive learning rule

 wkj = (xj-wkj) if neuron k wins the competition = 0 if
neuron k loses the competition
 Learning Paradigms : Learning with a Teacher
Learning with a Teacher (=supervised learning)
The teacher has knowledge of the environment
 Learning Paradigms: Learning without a Teacher

Learning without a Teacher: no labeled examples available

of the function to be learned.
1) Reinforcement learning
2) Unsupervised learning
 Learning Paradigms: Reinforcement

Reinforcement learning:
The learning of input-output mapping
is performed through continued
interaction with the environment
in oder to minimize a scalar index
of performance.
 Learning Paradigms: Unsupervised

 Unsupervised Learning: There is no external teacher or critic to

oversee the learning process.

 The provision is made for a task independent measure of the quality

of representation that the network is required to learn.
32

Training of Multilayer Feed-Forward

Neural Networks : Back-propagtaion
33
Multilayer Feed-Forward Neural Networks
 Feed -forward Networks
 A connection is allowed from a node in layer i only to nodes in layer i + 1.
 Most widely used architecture.

Conceptually, nodes
at higher levels
successively
abstract features
from preceding
layers
34
Geometric interpolation of the role of HN
 How to get appropriate weight set?

Back-propagation is the famous algorithm for training feed-forward networks

http://en.wikipedia.org/wiki/Backpropagation

Backpropagation, or propagation of error, is a common method of teaching

artificial neural networks how to perform a given task.
It was first described by Paul Werbos in 1974, but it wasn't until 1986, through the work of
David E. Rumelhart, Geoffrey E. Hinton and Ronald J. Williams, that it gained recognition,
and it led to a “renaissance” in the field of artificial neural network research.
36
 Back-propagation(BP)

Error =
Input Output Desired Output
- Actual Output

(Input Layer) (Hidden Layer) (Output Layer)

Forward Pass: Error is calculated based on desired and actual output.

Backward Pass: Synaptic weights are adjusted based on calculated error.

Back-propagation

yj

(Input Layer) (Hidden Layer) (Output Layer)

 Back-propagation (Without HN)
39

yj

(Input Layer) (Output Layer)

Activation
Function

∑ Forward Pass:

Backward Pass:
40

Activation
Function

Depends on Activation Function

41
Activation
Function

∑
 Weight Update Sequence in BP

i=0 w10
bias

y1
i=1 w1m
wj0
yj ej = dj- yj

wjm

i=m

(Output Layer)
(Input Layer)
 BP for Multilayer (With HN)
Input
yj
yk
Output
BP for Multilayer (With HN)
BP for Multilayer (With HN)
BP for Multilayer (With HN)
wji(t+1)=wji(t)+wji

δk (n)=ek(n)φ’(vk(n)) for k output neuron

 Weight Update Sequence in BP

i=0 w10
bias
y1

i=1 w1m
wj0
yj e j = d j- y j

wjm

i=m
 Matter of Activation Function(AF) f’(x)= β f(x)(1-f(x))
f(x)=1/(1+exp(-βx)

Max. 0.25β
at f(x)=0.5

k & j are output

δk (n)=ek(n)φ’(vk(n)) & hidden layer neuron

Differentiability is the only

condition for AF

an example of continuously
differentiable nonlinear AF is
sigmoidal nonlinearity; its
two forms are :
49
BP at a Glance

w    x
The local gradient of output unit (δo ) and hidden unit (δh) are defined by:
e f o f h
o    h    o wo
f o xo o xh
eo (n)
1
e(n)  (d (n)  f o (n)) 2 ,   d ( n)  f o ( n ) 
2 f o (n)
f o (n)
f o (n)  1 / 1  exp  xo ( n)    f o ( n ) 1  f o ( n ) 
For logistic sigmoid activation function xo (n)

Now the local gradient of output unit (δo) becomes  o   d ( n )  f o ( n )  f o ( n ) 1  f o ( n ) 

For the same sigmoid activation function the local  h  f h (n)1  f h (n)    o wo
gradient of hidden unit (δh) becomes o
 Operational flowchart of a NN with
50
Single Hidden Layer
Hidden layer

H Output layer

Actual Output
Wo
Wh
Input
-1
∑❑
  Error

1
Desired Output

∆  Wh  𝛿 h  o 𝛿
  0
 Operational flowchart of a NN with
51
Two Hidden Layer

Hidden Layer 1 Hidden Layer 2

Output layer
H1 H2
W2
W1
W3

Actual Output
-1
Input ∑❑
  Error

Desired Output

∆  W 1 𝛿  h 1 ∆  W 2 𝛿  h 2 ∆  W 3  𝛿 0
52
A MLP Simulator for Simple Application
53
Hossein Khosravi : The Developer of the Simulator
54

Lets run the simulator

55

Practical Aspects of BP
56
Practical Considerations

 Training Data: Sufficient / proper training data is require for proper input-
output mapping.
 Network Size: Complex problem require more hidden neurons and may
perform better with multiple hidden layers.
 Weight Initialization: NN works on numeric data. Before training all the
synaptic weights are initialized with small random values, e.g., -0.5 to +0.5.
 Learning Rate () and Momentum Constant (): A small  results slower
convergence and its larger value gives oscillation. Momentum also speed up
training but larger  with large  arise oscillation.
 Stopping Training: Training should stop at better generalization position.
57
Effect of Learning Rate () and Momentum Constant () values
58
Learning and Generalization
Learning and generalization are the most important topics in NN Available
research. Learning is the ability to approximate the training data while Data
generalization is the ability to predict well beyond the training data.

 Generalization is more desirable because the common use of a Training Set

NN is to make good prediction on new or unknown objects. (Use for learning)

 It measures on the testing set that is reserved from available

data and not use in the training.
 Testing error rate (TER), i.e., rate of wrong classification on Testing Set
testing set, is widely acceptable quantifying measure, which (Reserve
value minimum is good. to measure
generalization)
Total testing set misclassified patterns
TER = Total testing set patterns
59
Learning and Overfitting

Available
Data

Training Set
(Use for learning)

Testing Set
(Reserve
to measure
generalization)

Where we find benchmark data for testing NNs or machine learning?

60
Benchmark Problems for Evaluation
A benchmark is a point of reference by which something can be measured.

 For NN or machine learning, the most popular benchmark dataset collection is

the University of California, Irvine (UCI) Machine Learning Repository
(http://archive.ics.uci.edu/ml/).

 UCI contains raw data that require preprocessing to use in NN. Some
preprocessed data is also available at Proben1 (ftp://ftp.ira.uka.de/pub/neuron/).

 Various persons or groups also maintain different benchmark datasets for

specific purpose: Delve (www.cs.toronto.edu/~delve/data/datasets.html), Orange
(www.ailab.si/orange/datasets.asp), etc.
61
62
Benchmark Problems

Problems Related to Human Life

Problem Task
Breast Cancer Predicts whether a tumor is benign (not dangerous to health) or malignant
Wisconsin (dangerous) based on a sample tissue taken from a patient’s breast.
BUPA Liver Identify lever disorders based on blood tests along with other related
Disorder information such as alcohol consumption.
Diabetes Investigate whether the patient shows or not the signs of diabetes.
Heart Disease Predicting whether at least one of four major heart vessels is reduced in
Cleveland diameter by more than 50%.
Hepatitis Anticipate status (i.e., live or die) of hepatitis patient.
Lymphography Predict the situation of lymph nodes and lymphatic vessels.
Lungcancer Identify types of pathological lung cancers.
Postoperative Determine place to send patients for postoperative recovery.
63
Benchmark Problems
Problems Related to Finance
Problem Task
Australian Credit Classify people as good or bad credit risks depend on applicants’
Card particulars.
Car Evaluate cars based on price and facilities.
Labor Negotiations Identify a worker as good or bad i.e., contract with him beneficial or
not.
German Credit Like Australian Card, this problem also concerns to predict the
Card approval or non-approval of a credit card to a customer.

Problems Related to Plants

Problem Task
Iris Plants Classify iris plant types.
Mushroom Identify whether a mushroom is edible or not based on a description of
the mushroom’s shape, color, odor, and habitat.
Soybean Recognize 19 different diseases of soybeans.
64
Benchmark Problems and NN Architecture
After Preprocessing
Depends on Problem
Total Input Features NN Architecture
Abbr. Problem Examp Cont. Discr. Inputs Class Hidd.
Node Input Features
of Diabetes
ACC Australian Credit Card 690 6 9 51 2 10
BLN Balance 625 - 4 20 3 10 1. Number of times
BCW Breast Cancer Wisconsin 699 9 - 9 2 5 pregnant
CAR Car 1728 - 6 21 4 10
DBT Diabetes 768 8 - 8 2 5 2. Plasma glucose
GCC German Credit Card 1000 7 13 63 2 10 concentration
HDC Heart Disease Cleveland 303 6 7 35 2 5 3. Diastolic blood
HPT Hepatitis (HPT) 155 6 13 19 2 5
HTR Hypothyroid 7200 6 15 21 3 5
pressure
HSV House Vote 435 - 16 16 2 5 4. Triceps skin fold
INS Ionosphere 351 34 - 34 2 10 thickness (mm)
KRP King+Rook vs King+Pawn 3196 - 36 74 2 10
LMP Lymphography 148 - 18 18 4 10 5. 2-Hour serum
PST Postoperative 90 1 7 19 3 5 insulin (mu U/ml)
SBN Soybean 683 - 35 82 19 25 6. Body mass index
SNR Sonar 208 60 - 60 2 10
SPL Splice Junction 3175 - 60 60 3 10 7. Diabetes
WIN Wine 178 13 - 13 3 5 pedigree function
WVF Waveform 5000 21 - 21 3 10
8. Age
ZOO Zoo 101 15 1 16 7 10
 Problems show variations in number of examples, input features and classes.
65

Radial Basis Function (RBF) Networks

 Radial Basis Function (RBF) Networks
64

x1

y1
x2

y2
x3
output layer
input layer
(linear weighted sum)
(fan-out) For pattern classification sigmoid
function could be used.
hidden layer
(weights correspond to cluster centre,
output function usually Gaussian)
65
Clustering
 The unique feature of the RBF network is the process
performed in the hidden layer.
 The idea is that the patterns in the input space form
clusters.
 If the centres of these clusters are known, then the
distance from the cluster centre can be measured.
 Furthermore, this distance measure is made non-linear, so
that if a pattern is in an area that is close to a cluster
centre it gives a value close to 1.
 Beyond this area, the value drops dramatically.
 The notion is that this area is radially symmetrical around
the cluster centre, so that the non-linear function becomes
known as the radial-basis function.
66
Gaussian function
1
0.9
0.8
0.7
 0.6
0.5
0.4
0.3
0.2
0.1
0
0

1
0.2

0.4

0.6

0.8
-1

-0.8

-0.2
-0.6

-0.4

x
 The most commonly used radial-basis function is a
Gaussian function
 In a RBF network, r is the distance from the cluster centre.
67
Distance Measure
 The distance measured from the cluster centre is usually
the Euclidean distance.
 For each neuron in the hidden layer, the weights represent
the co-ordinates of the centre of the cluster.
 Therefore, when that neuron receives an input pattern, X,
the distance is:

n
rj   (x  w )
i 1
i ij
2
68
Width of hidden unit basis function
n

 (x i  wij ) 2

(hidden _ unit ) j  exp( i 1

)
2 2

 The variable sigma, , defines the width or radius of

the bell-shape and is something that has to be
determined empirically.
 When the distance from the centre of the Gaussian
reaches , the output drops from 1 to 0.6.
69
XOR Problem Solution
G(|| x - ti||) = exp (- || x – ti ||2) , i = 1,2
1=exp(- || x – t1 ||2) cen. t1 = [1,1]T
1=exp(- || x – t2 ||2) cen. t2 = [0,0]T
The value of radius  is chosen such
that 22 = 1

x1 x2 r1 r2 1 2
0 0 0 2 1 0.1
0 1 1 1 0.4 0.4
1 0 1 1 0.4 0.4
1 1 2 0 0.1 1
Distance calculated using Hamming distance.
 XOR Problem Solution
70 1
1
x2

+ -
0.5
XOR XX
- + x1

0 0.5 1 2
 This is an ideal solution - the centres were chosen
carefully to show this result.
 The learning of an RBF network finds those centre
values
71
Training hidden layer
 The hidden layer in a RBF network has units which have
weights that correspond to the vector representation of the
centre of a cluster.
 These weights are found either using a traditional clustering
algorithm such as the k-means algorithm, or adaptively
using essentially the Kohonen algorithm.
 In either case, the training is unsupervised but the number
of clusters that you expect, k, is set in advance. The
algorithms then find the best fit to these clusters.
72
k -means algorithm
 Initially k points in the pattern space are randomly set.
 Then for each item of data in the training set, the distances are found
from all of the k centres.
 The closest centre is chosen for each item of data - this is the initial
classification, so all items of data will be assigned a class from 1 to k.
 Then, for all data which has been found to be class 1, the average or
mean values are found for each of co-ordinates.
 These become the new values for the centre corresponding to class 1.
 Repeated for all data found to be in class 2, then class 3 and so on
until class k is dealt with - we now have k new centres.
 Process of measuring the distance between the centres and each
item of data and re-classifying the data is repeated until there is no
further change – i.e. the sum of the distances monitored and training
halts when the total distance no longer falls.
75
k -means algorithm

1. Ask user how many 2. Randomly guess k

clusters they’d like. cluster Center locations
(e.g. k=5)
76
k -means algorithm

3. Each data point finds 4. Each Center finds

out which Center it’s the centroid of the
closest to. points it owns
77
k -means algorithm

5. …and jumps there

6. …Repeat until
terminated!
 Adaptive k-means
 The alternative is to use an adaptive k -means
algorithm which similar to Kohonen learning.
 Input patterns are presented to all of the cluster
centres one at a time, and the cluster centres
adjusted after each one. The cluster centre that is
nearest to the input data wins, and is shifted slightly
towards the new data.
 This has the advantage that you don’t have to store
all of the training data so can be done on-line.
 Finding radius of Gaussians
 Having found the cluster centres, the next step is
determining the radius of the Gaussian curves.
 This is usually done using the P-nearest neighbour
algorithm.
 A number P is chosen (a typical value for P is 2), and for
each centre, the P nearest centres are found.
 The root-mean squared distance between the current
cluster centre and its P nearest neighbours is calculated,
and this is the value chosen for .
 So, if the current cluster centre is cj, the value is:

1 P
j  
P i 1
( c k  ci ) 2
 Training output layer

x1

y1
x2

y2
x3
output layer
input layer
(linear weighted sum)
(fan-out)

hidden layer
(weights correspond to cluster centre,
output function usually Gaussian)

 Having trained the hidden layer with some unsupervised

learning, the final step is to train the output layer using a
standard gradient descent technique such as the Least
Mean Squares algorithm.
 RBF vs. MLP
 An RBF network (in its most basic form) has a single hidden
layer, whereas an MLP may have one or more hidden layers
 RBF trains faster than a MLP
 In RBF network the hidden layer is easier to interpret than the
hidden layer in an MLP.
 Although the RBF is quick to train, when training is finished and
it is being used it is slower than a MLP, so where speed is a
factor a MLP may be more appropriate.
82

An Application of Feed Forward Neural Network

Optical Character Recognition(OCR)
83 Optical Character Recognition(OCR)