Scribd
Upload
137 views20 pages

Performance Analysis of Various Activation Functions in Neural

The document discusses various activation functions used in neural networks and compares their performance. It describes commonly used activation functions like sigmoid, tanh, and radial basis functions. Graphs show the performance of these functions over 100 iterations with 10 and 40 hidden neurons. The conclusion is that the tanh activation function performed best based on recognition accuracy compared to other functions like sigmoid and radial basis when used for both hidden and output layers.

Uploaded by

uslg
Copyright
© Attribution Non-Commercial (BY-NC)
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
137 views20 pages

Performance Analysis of Various Activation Functions in Neural

The document discusses various activation functions used in neural networks and compares their performance. It describes commonly used activation functions like sigmoid, tanh, and radial basis functions. Graphs show the performance of these functions over 100 iterations with 10 and 40 hidden neurons. The conclusion is that the tanh activation function performed best based on recognition accuracy compared to other functions like sigmoid and radial basis when used for both hidden and output layers.

Uploaded by

uslg
Copyright
© Attribution Non-Commercial (BY-NC)
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 20

Performance Analysis of Various Activation Functions In Neural Network

Presented by R.Dinesh Kumar

Introduction
One of the most attractive properties of ANNs is the possibility to adapt their behavior to the changing characteristics of the modeled system. Last decades, many researchers have investigated a variety of methods to improve ANN performance by optimizing training methods, learn parameters, or network structure, comparably few works is done towards using activation functions.

ACTIVATION FUNCTION TYPES


The most important unit in neural network structure is their net inputs by using a scalar-to-scalar function called the activation function or threshold function or transfer function output a result value called the unit's activation. An activation function for limiting the amplitude of the output of a neuron. Enabling in a limited range of functions is usually called squashing functions. It squashes the permissible amplitude range of the output signal to some finite value. Some of the most commonly used activation functions are to solve non-linear problems. These functions are: 1.Uni-polar sigmoid, 2.Bi-polar sigmoid, 3.Tanh, 4.Conic Section, and 5.Radial Bases Function (RBF). We will not analyse some activation functions such as identity function, step function or binary step functions as they are not used to solve linear problems.

Uni-Polar Sigmoid Function


Activation function of Uni-polar sigmoid function is given as follows:

This function is especially advantageous to use in neural networks trained by backpropagation algorithms. Because it is easy to distinguish, and this can interestingly minimize the computation capacity for training. The term sigmoid means Sshaped, and logistic form of the sigmoid maps

Bipolar Sigmoid Function


Activation function of Bi-polar sigmoid function is given by

This function is similar to the sigmoid function. For this type of activation function described in below figure, it goes well for applications that produce output values in the range of [-1, 1].

Hyperbolic Tangent Function


This function is easily defined as the ratio between the hyperbolic sine and the cosine functions or expanded as the ratio of the halfdifference and halfsum of two exponential functions in the points x and x as follows :

Hyperbolic Tangent Function is similar to sigmoid function. Its range outputs between -1 and 1 as seen in below figure

Radial Basis Function


Radial basis function (RBF) is based on Gaussian Curve. It takes a parameter that determines the center (mean) value of the function used as a desired value. A radial basis function (RBF) is a real-valued function whose value depends only on the distance from the origin, so that g(x) = g(ll x ll) or alternatively on the distance from some other point c, called a center, so that g(x, c) = g(ll x cll ) (5) Sums of radial basis functions are typically used to approximate given functions. This approximation process can also be interpreted as a simple type of neural network

RBF are typically used to build up function approximations of the form y (x) =wi g(ll x cll) i=1->N where the approximating function y(x) is represented as a sum of N radial basis functions, each associated with a different center ci, and weighted by an appropriate coefficient wi. The weights wi can be estimated using the matrix methods of linear least squares, because the approximating function is linear in the weights. Figure below shows that two unnormalized Gaussian radial basis functions in one input dimension. The basis function centers are located at c1=0.75 and c2=3.25

Conic Section Function


Conic section function (CSF) is based on a section of a cone as the name implies. CSF takes a parameter that determines the angle value of the function as seen in Fig

The equation of CSF can be defined as follows:

where ai is input coefficient, ci is the center, wi is the weight in Multi Layered Perceptron (MLP), 2w is an opening angle which can be any value in the range of [-/2, /2] and determines the different forms of the decision borders.

Summary of activation functions


The hidden units of neural network need activation functions to introduce non-linearity into the networks. Non-linearity makes multi-layer networks so effective. The sigmoid functions are the most widely used functions. Activation functions should be chosen to be suited to the distribution of the target values for the output units. For binary outputs the tangent hyperbolic and sigmoid functions are effective choices. If the target values are positive but have no known upper bound, an exponential output activation function can be used.

COMPARISON WITH DIFFERENT ACTIVATION FUNCTIONS


In this study, different activation functions depend on different number of iterations for comparing their performances by using the same data. For all the activation functions, we used the number of nodes in the hidden layer as follows; firstly 10 nodes, secondly 40 nodes (numbers of iterations are the same for both of them). Following are the graphs for different parameters

Graph 1: 100 Iterations - 10 Hidden Neurons - Bi-Polar Sigmoid

Graph 2: 100 Iterations - 10 Hidden Neurons - Uni-Polar Sigmoid

Graph 3: 100 Iterations - 10 Hidden Neurons - Tangent Hyperbolic

Graph 4: 100 Iterations - 10 Hidden Neurons - Conic Section

Graph 5: 100 Iterations - 10 Hidden Neurons RBF

CONCLUSION
In this study, we have used five conventional differentiable and monotonic activation functions for the evolution of MLP architecture along with Generalized Delta rule learning. These proposed well-known and effective activation functions are Bi-polar sigmoid, Uni-polar sigmoid, Tanh, Conic Section, and Radial Bases Function (RBF). Having compared their performances, simulation results show that Tanh (hyperbolic tangent) function performs better recognition accuracy than those of the other functions. In other words, the neural network computed good results when Tanh-Tanh combination of activation functions was used for both neurons (or nodes) of hidden and output layers.

You might also like