Implementing the activation

function layer
By,
Dr. TINTU GEORGE
Assistant Professor,
Dep of Computer Science with Data Analytics
Sri Ramakrishna College of Arts & Science
 Implementing the activation function layer
•The notion of a computational graph is applied to a neural network.
• A computational graph is a graphical representation of mathematical
expressions involving variables and operations.

• In the context of neural networks, it represents how data flows through the
network, including operations like matrix multiplications, activations, and
backpropagation.

•Here, the layers which form a neural network are used for activation
functions in one class utilizing the ReLU and sigmoid layers.
ReLU function

• An activation function of a rectified linear unit (ReLU) is expressed in

the following equation
•This function sets all negative values in the input to zero.
•It is defined as: ReLU(x)=max (0,x)
•It means that for any input value, if it is negative, the output will be 0;
if positive, the value is unchanged.
• ReLU activation function behaves in the context of a computational
graph during both forward and backward propagation in a neural
network.

1.Forward Propagation with ReLU:

1. During forward propagation in a neural network, input x passes through the
ReLU activation function.
If x>0, the output ReLU(x) is equal to x itself.

If x≤0, the output ReLU(x) is 0.

2. Computational Graph Representation:
• A computational graph visually represents the flow of data and computations
in a neural network.
• Nodes in the graph represent variables or operations, and edges represent
dependencies or flows between them.

3. Backward Propagation (Backpropagation) with ReLU:

• During backpropagation, gradients (derivatives of the loss function with
respect to the network's parameters) are computed and propagated backward
through the network to adjust the parameters (weights).
• As shown in equation (5.8), if x is greater than 0 in forward propagation, the backpropagation passes
upstream value downstream without any alterations.

• However, if x is equal to or less than 0, the signal ceases in backwards propagation. Represent this in a
computational graph as follows:
Sigmoid Activation:
•Sigmoid Activation: This function maps input values to a range between 0
and 1.
• It is defined as:

•​ Here, the exponential function ensures that large positive values result in
outputs near 1 and large negative values result in outputs near 0.
• Sigmoid Formula

• The following diagram shows the computational graph for the sigmoid
function in forward propagation.
• The image shows a computational graph for the sigmoid function during
forward propagation. Here's a step-by-step explanation of the graph:
1.Input x:
The input to the sigmoid function is x.
2.Multiplication by −1:
x is multiplied by −1 to get −x.
3.Exponentiation:
The value −x is passed through an exponentiation operation to calculate exp⁡(-x).
4.Addition:
The value 1 is added to exp⁡(- x) to get 1+exp⁡(-x).
5.Division:
Implementing the affine and soft max layers
Affine layer
• An affine layer, also known as a fully connected layer or a dense layer, is a
fundamental building block used in neural networks. .

• It's a type of layer where each input is connected to each output by a learnable
weight.

• Affine layers are commonly used in both traditional neural networks and deep
learning models to transform input features into outputs that the network can use
for prediction or classification tasks.
• Mathematically, the output of an affine layer can be described by the equation:

• Y=np.dot(X,W) + B

where:

• X is the matrix.

• W is the weight matrix.

• B is the bias

• It consists of a linear transformation (the matrix multiplication) and a

translation (the bias addition) hence it is called as affine layer.
import numpy as np

X = np.random.rand(2) # Input values

W = np.random.rand(2,3) # Weights

B = np.random.rand(3) # Biases

X.shape # (2,)

W.shape # (2, 3)

B.shape # (3,)
Program for batch affine layer

• The biases are applied to each piece of data for• In this example, we initialize loss function of two
forward propagation. data items (N = 2). Y represent the o/p as 2x3

• Therefore, the values of each piece of data in array.

backpropagation must be included in the elements• The np.sum function sums the elements of dY
of biases when backpropagation takes place. along the specified axis.
Softmax Layer
• A softmax layer is often used in neural networks, particularly in the final
layer of a classification model. It converts the logits (raw prediction values)
of a neural network into probabilities.
1. Input Layer: The network starts with an input layer, represented by the number 2. This could mean the

second input instance, or it could signify that the input data has 2 dimensions/features.

2. Affine (Fully Connected) Layers: These layers perform linear transformations on the input data. An

affine transformation involves multiplying the input by a weight matrix and then adding a bias vector.

Mathematically, it is represented as y=W.X+B

3. ReLU Activation Functions: After each affine layer, there is a ReLU (Rectified Linear Unit) activation

function applied.

4. Multiple Affine and ReLU Layers: The network contains multiple alternating affine and ReLU layers.

5. Final Affine Layer: The last affine layer produces the raw output scores for each class. These are often

called logits.

6. Scores: The scores represent the raw outputs from the final affine layer. Each score corresponds to a
• Softmax Layer: The softmax function is applied to the scores to convert them into probabilities.

• Probabilities: The output of the softmax function is a probability distribution over the classes. Each
value represents the probability of the input belonging to a particular class. For instance, in the
diagram, the probabilities are 0.008,0.00005,0.991,0.00004,…

• Calculated by exp(affine)/ Sum (exp(affine)) ,axis

Summary of the Process

• Input: The network receives an input.

• Affine Transformation + ReLU Activation: The input is transformed through multiple affine layers
with ReLU activations in between.

• Final Affine Layer: The last affine layer produces scores for each class.

• Softmax Transformation: The scores are converted into probabilities using the softmax function.
Program to implement affine and softmax layers