Scribd
Upload
9 views19 pages

3-Activation Function, Loss Function-24-07-2024

Uploaded by

gupta
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
9 views19 pages

3-Activation Function, Loss Function-24-07-2024

Uploaded by

gupta
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 19

Activation Functions

Dr. Kakelli Anil Kumar


SCOPE, VIT, Vellore
• Activation functions are a critical component of neural
networks in deep learning.

• They introduce non-linearity into the network,


allowing it to model complex relationships in the data.

• Without activation functions, the neural network


would behave like a simple linear model, regardless of
its depth.
Sigmoid Activation Function
Hyperbolic Tangent (Tanh) Activation Function
• The sigmoid (or logistic) function, which ranges from 0 to 1, is
particularly useful in the output layer of binary classification models,
representing the probability of a binary event. However, it can suffer
from the vanishing gradients problem during backpropagation.
• The hyperbolic tangent (or 'tanh') function, which ranges from -1 to 1,
provides a zero-centered output designed to make learning for the
next layer easier. Yet, like sigmoid, it also faces the vanishing gradients
issue.
• The Rectified Linear Unit (or 'ReLU') function is a popular choice in
hidden layers due to its efficiency. It activates a node if its input is
positive, otherwise, it outputs zero. This simplicity reduces
computational cost and mitigates the vanishing gradients problem,
but it can lead to dead neurons where some neurons never activate.
Why Activation Functions are Needed

1.Introduce Non-Linearity:
• Without non-linearity, a neural network would be
equivalent to a single-layer linear model, regardless of
the number of layers. Non-linear activation functions
enable the network to learn complex patterns and
representations.
Enable Complex Mappings:

• With non-linear activation functions, neural


networks can approximate any continuous
function. This ability is crucial for tasks like image
recognition, natural language processing, and
other complex AI applications.
Control the Flow of Information:

• Activation functions control which neurons get


activated and how signals propagate through the
network. This control is vital for learning
hierarchical feature representations.
Summary
Neural networks are powerful tools for modeling complex patterns in
data. Understanding the basic components, structure, and learning
process of neural networks is crucial for building and training deep
learning models.
The combination of multiple layers, non-linear activation functions, and
optimization algorithms enables neural networks to approximate
intricate functions and solve a wide range of problems in areas such as
image recognition, natural language processing, and more.

You might also like