Neural Networks

Outline

- Neural Network paradigm

• Bilogical Neurons vs Artificial Neurons
• Model of a Neuron
- First Neural Networks
- Multi layer Networks
- Classification by Backpropagation
- Recap

1
 Neural Network Paradigm

Neural networks paradigm

 Neural network: information processing

paradigm inspired by biological nervous
systems, such as our brain
 Structure: large number of highly
interconnected processing elements (neurons)
working together
 Like people, they learn from experience (by
example)

2
 Definition of ANN

“Data processing system consisting of a large number of

simple, highly interconnected processing elements
(artificial neurons) in an architecture inspired by the
structure of the cerebral cortex of the brain”

(Tsoukalas & Uhrig, 1997).

Inspiration from Neurobiology

Human Biological Neuron

3
 Biological Neural Networks

• A biological neuron
has three types of
main components;
dendrites, soma (or
cell body) and axon.
• Dendrites receives
signals from other
neurons.
• The soma, sums the incoming signals. When
sufficient input is received, the cell fires; that is it
transmit a signal over its axon to other cells.

Artificial Neurons

• ANN is an information processing system that has

certain performance characteristics in common with
biological nets.
• Several key features of the processing elements of
ANN are suggested by the properties of biological
neurons:

1. The processing element receives many signals.

2. Signals may be modified by a weight at the receiving
synapse.
3. The processing element sums the weighted inputs.
4. Under appropriate circumstances (sufficient input), the
neuron transmits a single output.
5. The output from a particular neuron may go to many other
8
neurons.

4
 Artificial Neurons
A physical neuron
• From experience:
examples / training
data
• Strength of connection
between the neurons
is stored as a weight-
value for the specific
connection.
• Learning the solution
to a problem =
changing the
connection weights

An artificial neuron 9

Artificial Neurons

• ANNs have been developed as generalizations of

mathematical models of neural biology, based on the
assumptions that:

1. Information processing occurs at many simple

elements called neurons.
2. Signals are passed between neurons over
connection links.
3. Each connection link has an associated weight,
which, in typical neural net, multiplies the signal
transmitted.
4. Each neuron applies an activation function to its
net input to determine its output signal.
10

5
 Artificial Neurons

Biological Neuron Artificial Neuron

Four basic components of a human biological The components of a basic artificial neuron
neuron

11

Model Of A Neuron

Wa
X1

X2 Wb  f() Y

X3 Wc

Input units Connection Summing

computation
weights function

(dendrite) (synapse) (axon)

(soma)
12

6
 Model Of A Neuron
 A neural net consists of a large number of simple
processing elements called neurons, units, cells
or nodes.

 Each neuron is connected to other neurons by

means of directed communication links, each
with associated weight.

 The weight represent information being used by

the net to solve a problem.

13

Model Of A Neuron
 Each neuron has an internal state, called its
activation or activity level, which is a function of
the inputs it has received. Typically, a neuron
sends its activation as a signal to several other
neurons.

 It is important to note that a neuron can send only

one signal at a time, although that signal is
broadcast to several other neurons.

14

7
 Model Of A Neuron

 Neural networks are configured for a specific

application, such as pattern recognition or data
classification, through a learning process
 In a biological system, learning involves adjustments
to the synaptic connections between neurons
 same for artificial neural networks (ANNs)

15

Artificial Neural Network

Synapse Nukleus

x1 w1  
y Axon

x2 w2 Activation Function:
yin = x1w1 + x2w2 (y-in) = 1 if y-in >= 
and (y-in) = 0

Dendrite
-A neuron receives input, determines the strength or the weight of the
input, calculates the total weighted input, and compares the total
weighted with a value (threshold)

- If the total weighted input greater than or equal the threshold value,
the neuron will produce the output, and if the total weighted input less
than the threshold value, no output will be produced 16

8
 First Neural Networks

17

The First Neural Neural Networks

X1
2

X2 2
Y

-1

X3

The activation of a neuron is binary. That is,

the neuron either fires (activation of one) or
does not fire (activation of zero).
18

9
 The First Neural Neural Networks
X1
2

X2 2
Y

-1

X3 For the network shown here the activation

function for unit Y is

f(y_in) = 1, if y_in >= θ else 0

where y_in is the total input signal received

θ is the threshold for Y
19

The First Neural Neural Networks

X1
• Neurons in a McCulloch-Pitts network are
2 connected by directed, weighted paths

X2 2
Y

• If the weight on a path is positive

-1
the path is excitatory, otherwise it is
X3 inhibitory

• Each neuron has a fixed • The threshold is set such that any
threshold. If the net input into non-zero inhibitory input will
the neuron is greater than the prevent the neuron from firing
20
threshold, the neuron fires

10
The First Neural Neural Networks:
Examples

1
AND
X1

X1 X2 Y
Y

1 1 1
X2 1
1 0 0
AND Function
0 1 0
0 0 0

Threshold(Y) = 2

21

The First Neural Neural Networks:

Examples

OR
2
X1
X1 X2 Y
Y
1 1 1
X2 2
1 0 1
0 1 1
AND Function
OR Function
0 0 0

Threshold(Y) = 2

22

11
The First Neural Neural Networks:
Examples

AND
X1 2 NOT
Y X1 X2 Y
X2
1 1 0
-1
1 0 1
AND NOT Function
0 1 0
0 0 0

Threshold(Y) = 2

23

Bias of a Neuron
• We need the bias value to be added to the
weighted sum ∑wjxj so that we can transform it
from the origin.
v = ∑wjxj + b, here b is the bias

x1-x2= -1
x2
x1-x2=0

x1-x2= 1

x1

24

12
 Bias as extra input
x0 = +1 w0

Activation
x1 W1

Input v function
Attribute
values x2 w2  f ( )
Output
  Summing function class
y
xm wm weights
m
v wx
j0
j j

w0  b

25

Neuron with Activation

• The neuron is the basic information processing unit of a NN.
It consists of:

1 A set of links, describing the neuron inputs, with weights

W1, W2, …, Wm

2 An adder function (linear combiner) for computing the

weighted sum of the inputs (real numbers):
m
u   w jx j
j 1

3 Activation function : for limiting the amplitude of the

neuron output.

y  f (u  b)
26

13
Examples of Activation Functions

* Stept(x) = 1 if x >= t, else 0

* Sign(x) = +1 if x >= 0, else –1
* Sigmoid(x) = 1/(1+e-x)
* Identity Function 27

Examples of Activation Functions

(d) Tanh (e) ReLU

* ReLU(x)= max(0,x)
* Tanh(x) = (ex-e-x)/(ex+e-x)

28

14
 Multilayer Networks

29

Why We Need MultiLayer ?

• Linear Separable:

x y x y
• Linear inseparable:
x y
• Solution?
30

15
 A Multilayer Feed-Forward Neural
Network

Output Class
Ok
Output nodes
w jk
Oj
Hidden nodes

wij - weights

Input nodes
Network is fully connected
Input Record : xi
31

Neural Network Learning

• The inputs are fed simultaneously into the input layer.

• The weighted outputs of these units are fed into hidden

layer.

• The weighted outputs of the last hidden layer are inputs to

units making up the output layer.

32

16
A Multilayer Feed Forward Network

• The units in the hidden layers and output layer are

sometimes referred to as neurodes, due to their
symbolic biological basis, or as output units.

• A network containing two hidden layers is called a

three-layer neural network, and so on.

• The network is feed-forward in that none of the

weights cycles back to an input unit or to an output unit
of a previous layer.

33

A Multilayered Feed – Forward Network

• INPUT: records without class attribute with
normalized attributes values.

• INPUT VECTOR: X = { x1, x2, …. xn}

where n is the number of (non class) attributes.

• INPUT LAYER – there are as many nodes as non-

class attributes i.e. as the length of the input vector.

• HIDDEN LAYER – the number of nodes in the hidden

layer and the number of hidden layers depends on
implementation.

34

17
 A Multilayered Feed–Forward
Network
• OUTPUT LAYER – corresponds to the class attribute.
• There are as many nodes as classes (values of the
class attribute).

Ok k= 1, 2,.. #classes
• Network is fully connected, i.e. each unit provides input
to each unit in the next forward layer.

35

A Multilayered Feed–Forward
Network: XOR function
2
2
X1 -1 Z1
XOR
Y X1 X2 Y
-1
1 1 0
Z2
X2
2
1 0 1
2
0 1 1
XOR Function
0 0 0

X1 XOR X2 = (X1 AND NOT X2) OR (X2 AND NOT X1)

36

18
 Example: combining logistic
models
Inputs
.4 Output
-1 .5744
-.5 S
0.6179
.1 .5
2 -.2 .6454
.3
S
S
.3
5 .1

Dependent
Independent Weights HiddenL Weights variable
variables ayer
Prediction
Sigmoid(x) = 1/(1+e-x) 37

Example: combining logistic

models
Inputs
.4 Output
-1
.5 0.6179
.1
2 S
.3 .3
5

Dependent
Independent Weights HiddenL Weights variable
variables ayer
Prediction
Sigmoid(x) = 1/(1+e-x) 38

19
 Example: combining logistic
models
Inputs
Output
-1
-.5 .5
0.6179
2 -.2
S
.3
5 .1

Dependent
Independent Weights HiddenL Weights variable
variables ayer
Prediction
Sigmoid(x) = 1/(1+e-x) 39

Example: combining logistic

models
Inputs
.4 Output
-1
-.5 .5
.1 0.6179
2 -.2
S
.3 .3
5 .1

Dependent
Independent Weights HiddenL Weights variable
variables ayer
Prediction
Sigmoid(x) = 1/(1+e-x) 40

20
 Example: combining logistic
models

-1 .4 .5744
-.5 S
.1 .5 0.6179
2 -.2 .6456
.3
S
S
.3
5 .1

Dependent
Independent Weights HiddenL Weights variable
variables ayer
Prediction
no target for hidden units... 41

Classification by back
propagation

42

21
Classification by Back propagation
A dataset
Fields class
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0
etc …

43

Classification by Back propagation

Training the neural network

Fields class
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0
etc …

44

22
 Classification by Back propagation

Training data Initialise with random weights

Fields class
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0
etc …

45

Classification by Back propagation

Training data Present a training pattern

Fields class
1.4 2.7 1.9 0
3.8 3.4 3.2 0 1.4
6.4 2.8 1.7 1
4.1 0.1 0.2 0 2.7
etc …
1.9

46

23
Classification by Back propagation

Training data Feed it through to get output

Fields class
1.4 2.7 1.9 0
3.8 3.4 3.2 0 1.4
6.4 2.8 1.7 1
4.1 0.1 0.2 0 2.7 0.8
etc …
1.9

47

Classification by Back propagation

Compare with target output

Training data
Fields class
1.4 2.7 1.9 0 1.4
3.8 3.4 3.2 0
6.4 2.8 1.7 1
2.7 0.8
4.1 0.1 0.2 0
etc … 0
1.9 error 0.8

48

24
Classification by Back propagation
Adjust weights based on error
Training data
Fields class
1.4 2.7 1.9 0 1.4
3.8 3.4 3.2 0
6.4 2.8 1.7 1 2.7 0.8
4.1 0.1 0.2 0
0
etc …
1.9 error 0.8

49

Classification by Back propagation

Training data
Present a training pattern
Fields class
1.4 2.7 1.9 0
3.8 3.4 3.2 0 6.4
6.4 2.8 1.7 1
4.1 0.1 0.2 0 2.8
etc …
1.7

50

25
Classification by Back propagation
Training data Feed it through to get output
Fields class
1.4 2.7 1.9 0
3.8 3.4 3.2 0 6.4
6.4 2.8 1.7 1
4.1 0.1 0.2 0 2.8 0.9
etc …
1.7

51

Classification by Back propagation

Training data Compare with target output

Fields class
1.4 2.7 1.9 0
3.8 3.4 3.2 0 6.4
6.4 2.8 1.7 1
4.1 0.1 0.2 0 2.8 0.9
etc … 1
1.7 error -0.1

52

26
Classification by Back propagation

Training data Adjust weights based on error

Fields class
1.4 2.7 1.9 0
3.8 3.4 3.2 0 6.4
6.4 2.8 1.7 1
4.1 0.1 0.2 0 2.8 0.9
etc … 1
1.7 error -0.1

53

Classification by Back propagation

Training data
And so on ….
Fields class
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1 6.4
4.1 0.1 0.2 0
etc … 2.8 0.9
1
1.7 error -0.1

Repeat this thousands, maybe millions of times – each time

taking a random training instance, and making slight
weight adjustments
Algorithms for weight adjustment are designed to make
changes that will reduce the error 54

27
 The decision boundary perspective
Initial random weights

55

The decision boundary perspective

Present a training instance / adjust the weights

56

28
 The decision boundary perspective
Present a training instance / adjust the weights

57

The decision boundary perspective

Present a training instance / adjust the weights

58

29
 The decision boundary perspective
Present a training instance / adjust the weights

59

The decision boundary perspective

Eventually ….

60

30
 Recap

61

Recap

• weight-learning algorithms for NNs are very simple

• they work by making thousands and thousands of tiny
adjustments, each making the network do better at the
most recent pattern, but perhaps a little worse on many
others
• but, by dumb luck, eventually this tends to be good
enough to learn effective classifiers for many real
applications

62

31
 Neural Network Architectures

Even for a basic Neural Network, there are many design

decisions to make:
1. # of hidden layers (depth)
2. # of units per hidden layer (width)
3. Type of activation function (nonlinearity)
4. Form of objective function

63

Topologies of Neural Networks

completely
connected feedforward recurrent
(directed, a-cyclic) (feedback connections)

64

32
 Networks types

 Feedforward versus recurrent networks

Feedforward: No loops, input  hidden layers  output
Recurrent: Use feedback (positive or negative)

 Supervised versus unsupervised learning

Supervised networks use a “teacher”
The desired output for each input is provided by user.
Unsupervised networks find hidden statistical patterns in
input data Clustering, principal component analysis

65

33