Artificial Neural

Networks
Introduction
 Artificial Neural Networks (ANN)
 Information processing paradigm inspired by
biological nervous systems
 ANN is composed of a system of neurons
connected by synapses
 ANN learn by example
 Adjust synaptic connections between neurons
History
 1943: McCulloch and Pitts model neural
networks based on their understanding of
neurology.
 Neurons embed simple logic functions:
 a or b
 a and b

 1950s:
 Farley and Clark
 IBM group that tries to model biological behavior
 Consult neuro-scientists at McGill, whenever stuck

 Rochester, Holland, Haibit and Duda

History
 Perceptron (Rosenblatt 1958)
 Three layer system:
 Input nodes
 Output node
 Association layer

 Can learn to connect or associate a given input to a

random output unit
 Minsky and Papert
 Showed that a single layer perceptron cannot learn
the XOR of two binary inputs
 Lead to loss of interest (and funding) in the field
History
 Back-propagation learning method (Werbos
1974)
 Three layers of neurons
 Input, Output, Hidden

 Better
learning rule for generic three layer networks
 Regenerates interest in the 1980s
 Successful applications in medicine, marketing,
risk management, … (1990)
 In need for another breakthrough.
ANN
 Promises
 Combine speed of silicon with proven success
of carbon  artificial brains
Neuron Model
 Natural neurons
Neuron Model
 Neuron collects signals from dendrites
 Sends out spikes of electrical activity through an
axon, which splits into thousands of branches.
 At end of each brand, a synapses converts
activity into either exciting or inhibiting activity of
a dendrite at another neuron.
 Neuron fires when exciting activity surpasses
inhibitory activity
 Learning changes the effectiveness of the
synapses
Neuron Model
 Natural neurons
Neuron Model
 Abstract neuron model:
ANN Forward Propagation
ANN Forward Propagation
 Bias Nodes
 Add one node to each layer that has constant
output
 Forward propagation
 Calculatefrom input layer to output layer
 For each neuron:
 Calculate weighted average of input
 Calculate activation function
Neuron Model
 Firing Rules:
 Threshold rules:
 Calculate weighted average of input
 Fire if larger than threshold
 Perceptron rule
 Calculate weighted average of input input
 Output activation level is
 1
 1  
2

 1
 ( )   0   
 2
0  0


Neuron Model
 Firing Rules: Sigmoid functions:
 Hyperbolic tangent function
1  exp(  )
    tanh( / 2) 
1  exp(  )

 Logistic activation function

   
1
1  exp  
ANN Forward Propagation
ANN Forward Propagation
 Density Plot of
Output
ANN Forward Propagation
ANN Forward Propagation
 Network can learn a non-linearly
separated set of outputs.
 Need to map output (real value) into binary
values.
ANN Training
 Weights are determined by training
 Back-propagation:
 On given input, compare actual output to desired
output.
 Adjust weights to output nodes.

 Work backwards through the various layers

 Start out with initial random weights

 Best to keep weights close to zero (<<10)
ANN Training
 Weights are determined by training
 Need a training set
 Should be representative of the problem
 During each training epoch:
 Submit training set element as input
 Calculate the error for the output neurons

 Calculate average error during epoch

 Adjust weights
ANN Training
 Error is the mean square of differences in
output layer
 1 K   2
E ( x )   ( y k ( x )  t k ( x ))
2 k 1
y – observed output
t – target output
 ANN Training Example
x1 x2 y Error
0 -0,5 2 0.1 0 0 0.69 0.472448
-0.5
4 0 1 0.67 0.110583
1
1 3 -0.5 1 0 0.70 0.0911618
1
-1 -0.5 1 1 1 0.68 0.457959
Bias

Average Error is 0.283038

ANN Training Example
 Calculate the derivative of the error with
respect to the weights and bias into the
output layer neurons
 ANN Training Example
New weights going into node 4
0 -0,5 2 0.1
-0.5 We do this for all training inputs, then
4 average out the changes
1
1 3 -0.5 net4 is the weighted sum of input going
1
-0.5 into neuron 4:
-1 1
net4(0,0)= 0.787754
Bias
net4(0,1)= 0.696717
net4(1,0)= 0.838124
net4(1,1)= 0.73877
ANN Training
 ANN Back-propagation is an empirical
algorithm
ANN Training
 XOR is too simple an example, since
quality of ANN is measured on a finite sets
of inputs.
 More relevant are ANN that are trained on
a training set and unleashed on real data
ANN Training
 Need to measure effectiveness of training
 Need training sets
 Need test sets.
 There can be no interaction between test sets
and training sets.
 Example of a Mistake:
 Train ANN on training set.
 Test ANN on test set.
 Results are poor.
 Go back to training ANN.
 After this, there is no assurance that ANN will work well in
practice.
 In a subtle way, the test set has become part of the training set.
ANN Training
 Convergence
 ANN back propagation uses gradient decent.
 Naïve implementations can
 overcorrect weights
 undercorrect weights
 In either case, convergence can be poor
 Stuck in the wrong place
 ANN starts with random weights and improves them
 If improvement stops, we stop algorithm
 No guarantee that we found the best set of weights
 Could be stuck in a local minimum
ANN Training
 Overtraining
 An ANN can be made to work too well on a
training set
 But loose performance on test sets
Training set
Performance

Test set

Training time
ANN Training
 Improving Convergence
 Many Operations Research Tools apply
 Simulated annealing
 Sophisticated gradient descent
ANN Design
 ANN is a largely empirical study
 “Seems to work in almost all cases that we
know about”
 Known to be statistical pattern analysis
ANN Design
 Number of layers
 Apparently, three layers is almost always good
enough and better than four layers.
 Also: fewer layers are faster in execution and training
 How many hidden nodes?
 Many hidden nodes allow to learn more complicated
patterns
 Because of overtraining, almost always best to set the
number of hidden nodes too low and then increase
their numbers.
ANN Design
 Interpreting Output
 ANN’s output neurons do not give binary
values.
 Good or bad
 Need to define what is an accept.

 Can indicate n degrees of certainty with n-1

output neurons.
 Number of firing output neurons is degree of
certainty
ANN Applications
 Pattern recognition
 Network attacks
 Breast cancer
 …
 handwriting recognition
 Pattern completion
 Auto-association
 ANN trained to reproduce input as output
 Noise reduction
 Compression
 Finding anomalies
 Time Series Completion
Pseudo-Code
 phi – activation function
 phid – derivative of activation function
Pseudo-Code
 Forward Propagation:
 Input nodes i, given input xi:
foreach inputnode i
outputi = xi
 Hidden layer nodes j
foreach hiddenneuron j
outputj = i phi(wjioutputi)
 Output layer neurons k
foreach outputneuron k
outputk = k phi(wkjoutputj)
Pseudo-Code
ActivateLayer(input,output)
foreach i inputneuron
calculate outputi
foreach j hiddenneuron
calculate outputj
foreach k hiddenneuron
calculate outputk
output = {outputk}
Pseudo-Code
 Output Error
Error() {
foreach input in InputSet
Errorinput = k output neuron (targetk-outputk)2
return Average(Errorinput,InputSet)
Pseudo-Code
For each output neuron k calculate:

 k   ' (net k )  ( target k  output k )

For each output neuron k calculate and hidden
layer neuron j calculate:

E
 output j   k
Wkj
Pseudo-Code
For each hidden neuron j calculate:

 j   ' (net j )  k  kWkj 

For each hidden neuron j and each input neuron i

calculate:
E
 output i   j
W ji
Pseudo-Code
 These calculations were done for a single
input.
 Now calculate the average gradient over
all inputs (and for all weights).
 You also need to calculate the gradients
for the bias weights and average them.
ANN Training Example 2
 Start out with random, small weights

1 Node 0: x0
0 2 0.3
0 Node 1: x1
4 Node 2: o2 = (x0 + x1 -0.5)
1
1 3 -0.7 Node 3: o3 = (0.5 x1 -1)
0.5
Node 4: o4 = (0.3 o2 – 0.7 o3 + 1)
-1 -0.5 1

Bias
ANN Training Example 2
 Calculate outputs

1 x1 x2 y=o4
0 2 0.3
0 0 0 0.7160
4
1 0 1 0.7155
1 3 -0.7
0.5 1 0 0.7308
-1 -0.5 1 1 1 0.7273

Bias
ANN Future
 ANNs can do some things really well
 They lack in structure found in most
natural neural networks