Scribd Logo
Upload

Welcome to Scribd!

  • 95 views

    Activation Functions in Neural Networks: What Is Activation Function?

    Uploaded by

    Navneet Lalwani
    Activation functions are used in neural networks to determine the output of a node. There are two main types: linear and non-linear. Non-linear functions like sigmoid, tanh, and ReLU are most commonly used as they allow the network to learn complex patterns in data. Sigmoid and tanh squash outputs between 0-1 and -1-1, while ReLU is now most widely used as it trains faster but can cause dying ReLU issues if inputs become zero. Leaky ReLU was developed to address this by introducing a small slope for negative inputs. Derivatives of activation functions are important for backpropagation to update weights in the correct direction during training.

    Copyright:

    © All Rights Reserved
    Download as DOCX, PDF, TXT or read online from Scribd

    Activation Functions in Neural Networks: What Is Activation Function?

    Uploaded by

    Navneet Lalwani
    95 views11 pages
    Activation functions are used in neural networks to determine the output of a node. There are two main types: linear and non-linear. Non-linear functions like sigmoid, tanh, and ReLU are most commonly used as they allow the network to learn complex patterns in data. Sigmoid and tanh squash outputs between 0-1 and -1-1, while ReLU is now most widely used as it trains faster but can cause dying ReLU issues if inputs become zero. Leaky ReLU was developed to address this by introducing a small slope for negative inputs. Derivatives of activation functions are important for backpropagation to update weights in the correct direction during training.

    Original Title

    activation functions

    Copyright

    © © All Rights Reserved

    Available Formats

    DOCX, PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Activation functions are used in neural networks to determine the output of a node. There are two main types: linear and non-linear. Non-linear functions like sigmoid, tanh, and ReLU are most commonly used as they allow the network to learn complex patterns in data. Sigmoid and tanh squash outputs between 0-1 and -1-1, while ReLU is now most widely used as it trains faster but can cause dying ReLU issues if inputs become zero. Leaky ReLU was developed to address this by introducing a small slope for negative inputs. Derivatives of activation functions are important for backpropagation to update weights in the correct direction during training.

    Copyright:

    © All Rights Reserved
    Download as DOCX, PDF, TXT or read online from Scribd
    Download as docx, pdf, or txt
    95 views11 pages

    Activation Functions in Neural Networks: What Is Activation Function?

    Uploaded by

    Navneet Lalwani
    Activation functions are used in neural networks to determine the output of a node. There are two main types: linear and non-linear. Non-linear functions like sigmoid, tanh, and ReLU are most commonly used as they allow the network to learn complex patterns in data. Sigmoid and tanh squash outputs between 0-1 and -1-1, while ReLU is now most widely used as it trains faster but can cause dying ReLU issues if inputs become zero. Leaky ReLU was developed to address this by introducing a small slope for negative inputs. Derivatives of activation functions are important for backpropagation to update weights in the correct direction during training.

    Copyright:

    © All Rights Reserved
    Download as DOCX, PDF, TXT or read online from Scribd
    Download as docx, pdf, or txt
    of 11

    Activation Functions in Neural Networks

    What is Activation Function?

    It’s just a thing function that you use to get the output of node. It
    is also known as Transfer Function.

    Why we use Activation functions with Neural Networks?

    It is used to determine the output of neural network like yes or


    no. It maps the resulting values in between 0 to 1 or -1 to 1 etc.
    (depending upon the function).

    The Activation Functions can be basically divided into 2 types-

    1. Linear Activation Function

    2. Non-linear Activation Functions

    FYI: The Cheat sheet is given below.

    Linear or Identity Activation Function

    As you can see the function is a line or linear. Therefore, the output of
    the functions will not be confined between any range.
    Fig: Linear Activation Function

    Equation : f(x) = x

    Range : (-infinity to infinity)

    It doesn’t help with the complexity or various parameters of usual data


    that is fed to the neural networks.
    Non-linear Activation Function

    The Nonlinear Activation Functions are the most used activation


    functions. Nonlinearity helps to makes the graph look something like
    this

    Fig: Non-linear Activation Function

    It makes it easy for the model to generalize or adapt with variety of


    data and to differentiate between the output.

    The main terminologies needed to understand for nonlinear functions


    are:

    Derivative or Differential: Change in y-axis w.r.t. change in


    x-axis.It is also known as slope.
    Monotonic function: A function which is either entirely non-
    increasing or non-decreasing.

    The Nonlinear Activation Functions are mainly divided on the basis of


    their range or curves-

    1. Sigmoid or Logistic Activation Function

    The Sigmoid Function curve looks like a S-shape.

    Fig: Sigmoid Function

    The main reason why we use sigmoid function is because it exists


    between (0 to 1). Therefore, it is especially used for models where we
    have to predict the probability as an output.Since probability of
    anything exists only between the range of 0 and 1, sigmoid is the right
    choice.
    The function is differentiable.That means, we can find the slope of
    the sigmoid curve at any two points.

    The function is monotonic but function’s derivative is not.

    The logistic sigmoid function can cause a neural network to get stuck at
    the training time.

    The softmax function is a more generalized logistic activation


    function which is used for multiclass classification.

    2. Tanh or hyperbolic tangent Activation Function

    tanh is also like logistic sigmoid but better. The range of the tanh
    function is from (-1 to 1). tanh is also sigmoidal (s - shaped).
    Fig: tanh v/s Logistic Sigmoid

    The advantage is that the negative inputs will be mapped strongly


    negative and the zero inputs will be mapped near zero in the tanh
    graph.

    The function is differentiable.

    The function is monotonic while its derivative is not monotonic.

    The tanh function is mainly used classification between two classes.


    Both tanh and logistic sigmoid activation functions are used in
    feed-forward nets.

    3. ReLU (Rectified Linear Unit) Activation Function

    The ReLU is the most used activation function in the world right
    now.Since, it is used in almost all the convolutional neural networks or
    deep learning.

    Fig: ReLU v/s Logistic Sigmoid

    As you can see, the ReLU is half rectified (from bottom). f(z) is zero
    when z is less than zero and f(z) is equal to z when z is above or equal
    to zero.

    Range: [ 0 to infinity)

    The function and its derivative both are monotonic.


    But the issue is that all the negative values become zero immediately
    which decreases the ability of the model to fit or train from the data
    properly. That means any negative input given to the ReLU activation
    function turns the value into zero immediately in the graph, which in
    turns affects the resulting graph by not mapping the negative values
    appropriately.

    4. Leaky ReLU

    It is an attempt to solve the dying ReLU problem

    Fig : ReLU v/s Leaky ReLU

    Can you see the Leak? 😆

    The leak helps to increase the range of the ReLU function. Usually, the
    value of a is 0.01 or so.

    When a is not 0.01 then it is called Randomized ReLU.


    Therefore the range of the Leaky ReLU is (-infinity to infinity).

    Both Leaky and Randomized ReLU functions are monotonic in nature.


    Also, their derivatives also monotonic in nature.

    Why derivative/differentiation is used ?

    When updating the curve, to know in which


    direction and how much to change or update the
    curve depending upon the slope.That is why we
    use differentiation in almost every part of Machine
    Learning and Deep Learning.
    Fig: Activation Function Cheetsheet
    Fig: Derivative of Activation Functions

    You might also like