Scribd
Upload
25 views

Lect 5-6activation Function

Uploaded by

A Yadav
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
25 views

Lect 5-6activation Function

Uploaded by

A Yadav
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 15

Neural Network Architecture

and Activation functions


Network Architecture

Multiple layer
Single layer fully connected

Unit delay
operator

outputs

inputs Recurrent network


Recurrent network
without hidden units
with hidden units
Network Structure
1. Perceptron
This consists of a single neuron with multiple inputs and a single output. It
has restricted information processing capability. The information
processing is done through a transfer function which is either linear or
non-linear.
2.Multilayered network
It has a layered architecture consisting of input, hidden and output layers.
Each layer consists of a number of Neurons. The output of each layer is
transmitted to the input of nodes in other layers through weighted links.
Usually, this transmission is done only to nodes of the next layer,
leading to what are known as feed forward networks

3. Recurrent network
RNN topology involves backward links from output to the input and hidden
layers. The notion of time is encoded in the RNN information
processing scheme. They are thus used in applications like speech
processing where inputs are time sequences data.
• Feed forward networks (SLP & MLP ):

– Information only flows one way


– One input pattern produces one output
– No sense of time (or memory of previous state)

• Recurrency
– Nodes connect back to other nodes or themselves
– Information flow is multidirectional
– Sense of time and memory of previous state(s)

• Biological nervous systems show high levels of


recurrency
Linear and Non Linear Activation Functions

To make the work more efficient and to obtain the exact output some force
and activation may be given. This activation helps in achieving the exact
output. The activation function is applied over the net input to calculate the
output of an ANN
Information Processing consist of two parts
•Input
•Output
The integration function is associated with the input of a processing
element. This function serves to combine activation, information into a net
input to the processing element
The nonlinear activation function is used to ensure that actual response of
neuron is bounded i.e. conditioned
Activation functions for the hidden units are needed to introduce
nonlinearity into the network

Without nonlinearity, hidden units would not make nets more


powerful than just plain Perceptron (which do not have any hidden
units, just input and output units). The reason is that a linear
function of linear functions is again a linear function.

Functions such as tanh or arc-tan that produce both positive and


negative values tend to yield faster training than functions that
produce only positive values such as logistic, because of better
numerical conditioning

With sigmoid units, a small change in the weights will usually


produce a change in the outputs, which makes it possible to tell
whether that change in the weights is good or bad. With threshold
units, a small change in the weights will often produce no change in
the outputs.
The activation function acts as a squashing function, such
that the output of a neuron in a neural network is
between certain values (usually 0 and 1, or -1 and 1). In
general, there are two types of activation functions,
denoted by Φ(.)

Linear Activation functions

1. Binary Threshold Function - There is the Threshold


Function which takes on a value of 0 if the summed input
is less than a certain threshold value (v), and the value 1
if the summed input is greater than or equal to the
threshold value.
Yk
1

0
-∞ +∞
Vk

If Vk >= θ then Yk = 1
Otherwise 0
Piecewise-linear Function

There is the Piecewise-Linear function. This function again can


take on the values of 0 or 1, but can also take on values
between that depending on the amplification factor in a certain
region of linear operation

Oj

+1
(v) = 1 v >= +1/2

t ini =v v > -1/2


Piecewise-linear
Function =0 v <= -1/2
Signum/Bipolar step Activation Function

This function is represented as

If Vk >= 0 then Yk = +1
Otherwise -1

Where θ represents the threshold value. This function is


also used in single layer nets to convert the net input to an
output that is bipolar ( +1 or -1)
Signum / Bipolar-step Activation Function

Yk
+1

-∞ +∞
Vk
-1
Nonlinear Activation Function

They are aspect of stochastic neural network, when non-


linear approximation is required in n dimensional space
Sigmoidal functions

The sigmoid function, whose graph is s-shaped, is by


far the most common form of non-linear activation
function. It is represented as

(.) = 1 / 1+ exp (-av)


Here v is the linear summation of input units and if v moves
towards -∞ then  is 0 and if it is towards + ∞ then  is 1.
And parameter a determines slope of this curve
Sigmoid Function

Oj

+1

ini
Hyperbolic tangent Function

In some NN logistic functions


are replaced by the hyperbolic +1
tangent form. It scales the signal
range and a slope factor. This is
an example of a bipolar signal vi
function since the signal values
can become both +ve and –ve,
though continously.

(.) = tanh (av)


Stochastic Model of a Neuron
• Deterministic vs stochastic
• stochastic : stay at a state with probability
+ 1 with probability P(v)
x=
− 1 with probability 1 − P(v)
– x: state of neuron
v: induced local field 1
P(v) probability of firing
P(v ) = v
1 + exp( − )
T
where T is pseudo temperature
T → 0, reduced to deterministic form

You might also like