Artificial Neural Networks (18CSE388T)

Session 7, 8: Components of Artificial Neural Networks

Dr. Saad Y. Sait

[email protected]
Research Associate Professor, CSE

SRM Institute of Science and Technology

September 3, 2020

Saad Y. Sait (SRMIST) Components September 3, 2020 1 / 30

Concept of Time in Neural Networks

Current time step t, previous t − 1, next t + 1

Some neural networks use a time component
Biological neural networks - not plausible
one neuron does not wait for the next neuron

Saad Y. Sait (SRMIST) Components September 3, 2020 2 / 30

Components of Neural Networks
Definition of Neural Network

Sorted triple (N, V , w )

N is set of neurons
V is set of connections

V = {(i, j)|i, j ∈ N}

w are the weights; w : V → R

weight of connection between neuron i and neuron j written as wij
w may be implemented as matrix

Saad Y. Sait (SRMIST) Components September 3, 2020 3 / 30

Components of Neural Networks

1
Figure: Data Processing of a Neuron

1
Kriesel, D.
Saad Y. Sait (SRMIST) Components September 3, 2020 4 / 30
Artificial Neuron

2
Figure: Artificial Neuron

2
Ahire, J.B, https://medium.com/@jayeshbahire/the-artificial-neural-networks-handbook-part-4-d2087d1f583e
Saad Y. Sait (SRMIST) Components September 3, 2020 5 / 30
Components of Neural Network

3
Figure: Feedforward Neural Network

3
deepai.org
Saad Y. Sait (SRMIST) Components September 3, 2020 6 / 30
Components Of a Neural Network

If I = {i1 , i2 , . . . in } are the neurons connected to neuron j, then the

network input of j, netj is calculated using the propagation function fprop
as:

netj = fprop (oi1 , . . . oin , wi1 ,j , . . . , win ,j )

Typically X
netj = (oi · wi,j )
i∈I

Saad Y. Sait (SRMIST) Components September 3, 2020 7 / 30

Components of Neural Network
Activation State / Activation

Activation State
Indicates the extent of the neuron’s activity and results from the activation
function. Represented as aj .

Saad Y. Sait (SRMIST) Components September 3, 2020 8 / 30

Components of Neural Network
Threshold

Near the threshold value, the activation function of a neuron is

particularly sensitive
From biological point, the threshold represents the value at which
neuron starts firing

Threshold
The threshold Θj is uniquely assigned to j and marks the position of the
maximum gradient value of the activation function.

Saad Y. Sait (SRMIST) Components September 3, 2020 9 / 30

Components of Neural Network
Activation Function

Activation function is defined as

aj (t) = fact (netj , aj (t − 1), Θj )

Activation function defined globally for all neurons or atleast a group

of neurons
Threshold values can be changed using a learning procedure
Activation function is also called transfer function

Saad Y. Sait (SRMIST) Components September 3, 2020 10 / 30

Components Of Neural Networks
Common Activation Functions

4
Figure: Heaviside Step Function

4
en.wikipedia.org
Saad Y. Sait (SRMIST) Components September 3, 2020 11 / 30
Components Of Neural Networks
Common Activation Functions

5
Figure: Logistic and Hyperbolic Tangent Activation Functions

5
https://arxiv.org/pdf/1811.03378.pdf
Saad Y. Sait (SRMIST) Components September 3, 2020 12 / 30
Components Of Neural Networks
Common Activation Functions

6
Figure: Fermi function with Temperature parameter

Saad Y. Sait (SRMIST) Components September 3, 2020 13 / 30

Components Of Neural Networks
Common Activation Functions

Fermi Function or Logistic Function :

1
1 + e −x
Fermi Function with temperature parameter :
1
x
1 + e− T

Saad Y. Sait (SRMIST) Components September 3, 2020 14 / 30

Components Of Neural Networks
Common Activation Functions

Stochastic activation function

not explicitly defined but depend on the input according to a random
distribution

Saad Y. Sait (SRMIST) Components September 3, 2020 15 / 30

Components Of Neural Networks
Output Functions

fout (aj ) = oj

fout applied to activation to obtain output

In this subject, we only consider identity output function

Saad Y. Sait (SRMIST) Components September 3, 2020 16 / 30

Components of Neural Networks
Learning Strategy Definition

Learning Strategy
The learning strategy is an algorithm that can be used to change and
thereby train the neural network, so that the network produces a desired
output for a given input.

Saad Y. Sait (SRMIST) Components September 3, 2020 17 / 30

Network Topologies
Feedforward Network

Figure: Feedforward Network - each neuron has connections only to neurons of

the next layer 8

7
deepai.org
8
deepai.org
Saad Y. Sait (SRMIST) Components September 3, 2020 18 / 30
Network Topologies
Feedforward Network

Feedforward Network
The neuron layers of a feedforward network are clearly separated: One
input layer, one output layer and one or more processing layers which are
invisible from the outside (also called hidden layers). Connections are only
permitted to neurons of the following layer.

Completely linked
every neuron is connected to all neurons of the next layer

Saad Y. Sait (SRMIST) Components September 3, 2020 19 / 30

Network Topologies
Feedforward Network

Feedforward network with shortcut connections

Similar to the feedforward network, but the connections may not only be
directed towards the next layer but also towards any other subsequent
layer.

Saad Y. Sait (SRMIST) Components September 3, 2020 20 / 30

Network Topologies
Recurrent Network

Recurrence - neuron influencing itself by any means

Used to model sequences e.g. text, speech
Direct recurrence
Neurons connected to themselves
Indirect recurrence
Connections between neurons and preceding layer allowed

Saad Y. Sait (SRMIST) Components September 3, 2020 21 / 30

Network Topologies
Recurrent Network

9
Figure: Jordan and Elman Networks

9
developer.ibm.com
Saad Y. Sait (SRMIST) Components September 3, 2020 22 / 30
Network Topologies
Recurrent Network

Lateral recurrences
Connections between neurons within one layer
each neuron often inhibits the other neurons of the layer and
strengthens itself
winner takes all scheme

Saad Y. Sait (SRMIST) Components September 3, 2020 23 / 30

Bias Neuron

Bias Neuron
A bias neuron is a neuron whose output value is always 1. It is used to
represent neuron biases as connection weights, which enables any weight
training algorithm to train the biases at the same time.

XN
fact ( θji ai − b)
i=1

can be written as
N
X
fact ( θji ai )
i=0

where weight from bias unit θj0 = b and a0 = 1; θj0 a0 = b.

Saad Y. Sait (SRMIST) Components September 3, 2020 24 / 30

Bias Neuron

Figure: Bias Neuron; output always 1, connection weight represents the threshold
(bias) 11

10
missinglink.ai
11
missinglink.ai
Saad Y. Sait (SRMIST) Components September 3, 2020 25 / 30
Order of Activations

Synchronous Activation
All neurons of a network calculate network inputs at the same time by
means of the propagation function, activation by means of the activation
function and output by means of the output function. After that the
activation cycle is complete. E.g. recurrent neural networks.

Asynchronous Activation
The neurons do not change their values simultaneously but at different
points of time.

Saad Y. Sait (SRMIST) Components September 3, 2020 26 / 30

Order of Activation
Asynchronous Activation

Random order
A neuron is chosen at random, inputs accumulated and activation
function applied
all neurons may not get their turn in a cycle
Random permutation
each neuron is chosen exactly once, but in random order, during one
cycle.
at the beginning of a cycle, a permutation of neurons is calculated
randomly; each neuron activated in this order.
Both the above used in Hopfield network (associative memory)

Saad Y. Sait (SRMIST) Components September 3, 2020 27 / 30

Order of Activation
Asynchronous Activation

Topological order
With topological order of activation the neurons are updated during one
cycle and according to a fixed order. The order is defined by the network
topology.

E.g. feedforward neural network

input neurons, inner (hidden neurons), output neurons

Saad Y. Sait (SRMIST) Components September 3, 2020 28 / 30

Input and Output of Data

Size of input = number of input neurons

Size of output = number of output neurons

Saad Y. Sait (SRMIST) Components September 3, 2020 29 / 30

 Thank You!

Saad Y. Sait (SRMIST) Components September 3, 2020 30 / 30