CHAPTER 2

Neuron Model
and Network
Architecture
Ming-Feng Yeh 1
 Objectives
Introduce the simplified mathematical
model of the neuron
Explain how these artificial neurons can
be interconnected to form a variety of
network architectures
Illustrate the basic operation of these
neural networks

Ming-Feng Yeh 2
 Notation
Scalars: small italic letters
e.g., a, b, c
Vectors: small bold nonitalic letters
e.g., a, b, c
Matrices: capital BOLD nonitalic letters
e.g., A, B, C
Other notations are given in Appendix B

Ming-Feng Yeh 3
 Single-Input Neuron
w n
p  f a

b Scalar input: p
1 Scalar weight: w
n = wpsynapse
+b (synapse)
p aw= f(n) = f(wp + b)axon Bias: b
Net input: n
w=3, p=2 f(·) b=a 1.5
f and a
Transfer function : f
1 bn = 321.5=4.5
a = f(4.5) cell body
 dendrites (cell body)
Output: a
Ming-Feng Yeh 4
 Bias and Weight
The bias b is much like a weight w, except
that it has a constant input of 1. It can be
omitted if NOT necessary.
Bias b and weight w are both adjustable
scalar parameters of the neuron. They can be
adjusted by some learning rule so that the
neuron input/output relationship meets
some special goal.

Ming-Feng Yeh 5
 Transfer Functions
The transfer function f may be a linear
or nonlinear function of net input n
Three of the most commonly used func.
 Hard limit transfer function
 Linear limit transfer function
 Log-sigmoid transfer function

Ming-Feng Yeh 6
Hard Limit Transfer Func.
a a

1 1

n p
0 b w 0

1 1

a=hardlim(n) a=hardlim(wp+
b)
a = 0, if n  0
a = 1, if n  1
MATLAB function: hardlim

Ming-Feng Yeh 7
 Linear Transfer Function
a a

1
b
b w
n p
0 0

1

a=purelin(n) a=purelin(wp+b)

a=n
MATLAB function: purelin

Ming-Feng Yeh 8
Log-Sigmoid Transfer Func.
a a

1 1

n p
0 b w 0

1 1

a=logsig(n) a=logsig(wp+b)

a = 1/[1+exp(n)]
MATLAB function: logsig
Other transfer functions see Pages 2-6 & 2-17

Ming-Feng Yeh 9
 Multiple-Input Neuron
A neuron (node) with R inputs, p1, p2,…, pR
The weight matrix W, w11, w12,…,w1R
The neuron has a bias b
Net input: n = w11 p1 + w12 p2+…+ w1RpR + b = Wp + b
Neuron output: a = f(Wp + b)

p1 w1,1
p2
p3 n
 f a
pR w1, R b
1

Ming-Feng Yeh 10
 Single-Layer Network
• R: number of input
• S: number of neuron (node) in
a layer (R  S)
• Input vector p is a vector of
length R
• Bias vector b and output
vector a are vectors of length S
• Weight matrix W is an SR
matrix w

w
11  w 
22

1R

w w22  w2 R
W   21 
    
 
 wS 1 wS 2  wSR 
Ming-Feng Yeh 11
 Multiple-Layer Network
Input Layer Hidden Layer Output Layer

Input Layer Hidden Layer Output Layer

Ming-Feng Yeh 12
 Multiple-Layer Network
Input Layer Hidden Layer Output Layer

a1= f1(W1p+ b1) a2= f2(W2p+ b2) a3= f3(W3p+ b3)

Layer Superscript: the number of the layer
R inputs, S n neurons (nodes) in the nth layer
Different layers can have different numbers of neurons
The outputs of layer k are the inputs of layer (k+1)
Weight matrix W j between layer i and j is an S iS j matrix
Ming-Feng Yeh 13
 Network Architectures
Models of neural networks are specified by the three
basic entities: models of the processing elements
(neurons), models of inter-connections and
structures (network topology), and the learning
rules (the ways information is stored in the network).
The weights may be positive (excitatory) or
negative (inhibitory).
Information is stored in the connection weights.

Ming-Feng Yeh 14
 Network Structures
The layer that receives inputs is called the
input layer.
The outputs of the network are generated
from the output layer.
Any layer between the input and the output
layers is called a hidden layer.
There may be from zero to several hidden
layers in a neural network.

Ming-Feng Yeh 15
 Network Structures
When no node output is an input to a node in
the same layer or preceding layer, the
network is a feedforward network.
When outputs are directed back as inputs to
same- or preceding-layer nodes, the network
is a feedback network.
Feedback networks that have closed loops
are called recurrent networks.

Ming-Feng Yeh 16
 Delay Block & Integrator
a (0) a (0)

u (t ) a (t ) u (t ) a (t )
D
t
a(t )  u (t  1) a (t )   u ( )d  a (0)
0

Ming-Feng Yeh 17
 Recurrent Network
Initial Recurrent Layer
Condition
p
w
S 1 n(t  1) a(t  1) a(t )
SS  D
S 1 S 1 S 1
1 b
S S 1 S

a(0)  p
a(1)  satlins Wa (0)  b  , a(2)  satlins  Wa (1)  b 
a(t  1)  satlins  Wa (t )  b 

Ming-Feng Yeh 18
 Learning Scheme
Two kinds of learning in neural networks:
parameter learning, which concerns the
updating the connection weights in a neural
network, and structure learning, which
focuses on the change in the network
structure, including the number of nodes and
their connection types.
Each kind of learning can be further classified
into three categories: supervised learning,
reinforcement learning, and unsupervised
learning.
Ming-Feng Yeh 19
 Learning Scheme

Ming-Feng Yeh 20
How to Pick an Architecture
Problem specifications help define the network
in the following ways:
1. Number of network inputs = number of
problem inputs
2. Number of neurons in output layer =
number of problem outputs
3. Output layer transfer function choice at
least partly determined by problem
specification of the outputs.

Ming-Feng Yeh 21