Scribd
Upload
36 views17 pages

Activation Functions - Sigmoid - Tanh - ReLU - Softmax - Risk Minimization - Loss Function

The document discusses activation functions in neural networks, explaining their role in computing pre- and post-activation values. It outlines various types of activation functions, including sign, sigmoid, tanh, ReLU, and hard tanh, emphasizing their importance in neural network design and performance. The document highlights the shift towards ReLU and hard tanh due to their computational efficiency and effectiveness in training multilayered networks.

Uploaded by

devanand272003
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
36 views17 pages

Activation Functions - Sigmoid - Tanh - ReLU - Softmax - Risk Minimization - Loss Function

The document discusses activation functions in neural networks, explaining their role in computing pre- and post-activation values. It outlines various types of activation functions, including sign, sigmoid, tanh, ReLU, and hard tanh, emphasizing their importance in neural network design and performance. The document highlights the shift towards ReLU and hard tanh due to their computational efficiency and effectiveness in training multilayered networks.

Uploaded by

devanand272003
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 17

Activation Functions

Mr. Sivadasan E T
Associate Professor
Vidya Academy of Science and Technology, Thrissur
Activation Functions

A neuron really computes two functions within


the node:
The summation as well as
The activation within a neuron.

We use the notation to denote the activation


function:
Activation Functions
Activation Functions

The value computed before applying the


activation function will be referred to as the
pre-activation value,
whereas the value computed after applying
the activation function is referred to as the
post-activation value.
Types of Activation functions
The classical activation functions that were
used early in the development of neural
networks were the sign, sigmoid, and the
hyperbolic tangent functions:
Sign Activation Functions

The choice of activation function is a critical


part of neural network design.
Sign activation Function
In the case of the perceptron, the choice of
the sign activation function is motivated by the
fact that a binary class label needs to be
predicted.
Sigmoid Activation Functions

If it is desirable to predict a probability of a


binary class, it makes sense to use a sigmoid
function for activating the output node.
Sigmoid Activation Functions

The sigmoid activation outputs a value in (0, 1),


which is helpful in performing computations
that should be interpreted as probabilities.
tanh activation function
The tanh and sigmoid functions are related as
follows.

The tanh function is preferable to the sigmoid when


the outputs of the computations are desired to be
both positive and negative.
tanh activation functions

The tanh function has a shape similar to that of the


sigmoid function, except that it is horizontally re-
scaled and vertically translated/re-scaled to [−1, 1].
ReLU activation function

the ReLU function is simple and it consists of no


heavy computation as there is no complicated
math.

The model can, therefore, take less time to train


or run.
ReLU activation function
One more important property that we consider the
advantage of using ReLU activation function is
sparsity.
Hard tanh (tangent hyperbolic)

It is a cheaper and more computationally


efficient version of the tanh activation.
Hard tanh
Hard tanh (tangent hyperbolic)

The ReLU and hard tanh activation functions


have largely replaced the sigmoid and soft tanh
in modern neural networks .

Because of the ease in training multilayered


neural networks with these activation functions.
Hard tanh

Many of these functions are referred to as


squashing functions, as they map the outputs from
an arbitrary range to bounded outputs.
The use of a nonlinear activation plays a
fundamental role in increasing the modeling power
of a network.
If a network used only linear activations, it would
not provide better modeling power than a single-
layer linear network.
Thank You!

You might also like