Scribd
Upload
13 views

Notes-1-Activation Functions

Uploaded by

kapoornavodit
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
13 views

Notes-1-Activation Functions

Uploaded by

kapoornavodit
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 2

Activation Functions

Chapter – 6, Textbook
Introduction:

The output from a processing unit is termed as its activation. Activation of a processing unit is a
function of the net input to the processing unit. Th e function that maps the net input value to the
output signal value, i.e., the activation, is known as the activation function of the unit.

Some common activation functions are presented below.

1. Identity Function:
The simplest activation function is the identity function that passes on the incoming signal as the
outgoing signal without any change. Therefore, the identity activation function g(x) is defi ned as

g(x) = x

2. Step Function:
Another frequently used activation function is the step function. Th e basic step function produces a 1
or 0 depending on whether the net input is greater than 0 or otherwise. The step function is also known
as the heaviside function. Mathematically the step function is defined as follows.

The shape of the basic step function is represented graphically below. Occasionally, instead of 0 a non-
zero threshold value θ is used. This is known as the threshold function and is defined as follows.

Occasionally, it is more convenient to work with bipolar data, −1 and +1, than the binary data. If a signal
of value 0 is sent through a weighted path, the information contained in the interconnection weight is
lost as it is multiplied by 0. To overcome this problem, the binary input is converted to bipolar form and
Activation Functions
Chapter – 6, Textbook
then a suitable bipolar activation function is employed. Accordingly, binary step functions have their
bipolar versions. The output of a bipolar step function is −1, or +1, not 0, or 1.

3. The Sigmoid Function


As the step function is not continuous it is not differentiable. Some ANN training algorithm requires
that the activation function be continuous and differentiable. The step function is not suitable for
such cases. Sigmoid functions have the nice property that they can approximate the step function to
the desired extent without losing its differentiability. Binary sigmoid, also referred to as the logistic
sigmoid, is defined as below.

The parameter σ is known as the steepness parameter. Depending on the requirement, the binary
sigmoid function can be scaled to any range of values appropriate for a given application. Th e most
widely used range is from −1 to +1, and the corresponding sigmoid function is referred to as the
bipolar sigmoid function. Th e formulae for the bipolar sigmoid function and its first derivative are
given below as Equations 6.38 and 6.39 respectively. Fig. 6.33 presents its form graphically.

4. Hyperbolic Tangent Function


Another bipolar activation function that is widely employed in ANN applications is the hyperbolic
tangent function. Th e function, as well as its first derivative, are expressed below.

You might also like