Scribd
Upload
109 views14 pages

Neural Networks MATH Explained

This document provides a guide to building a basic neural network from scratch to solve the XOR problem. It defines the necessary functions like sigmoid, sigmoid derivative and mean squared error. It then implements a two layer neural network with one hidden layer of 4 nodes and a single output node. The network weights are initialized randomly and trained over 1500 iterations to minimize the loss function through forward and backpropagation. This results in the network learning the XOR function as indicated by the predicted outputs.

Uploaded by

Gautam Kumar
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
109 views14 pages

Neural Networks MATH Explained

This document provides a guide to building a basic neural network from scratch to solve the XOR problem. It defines the necessary functions like sigmoid, sigmoid derivative and mean squared error. It then implements a two layer neural network with one hidden layer of 4 nodes and a single output node. The network weights are initialized randomly and trained over 1500 iterations to minimize the loss function through forward and backpropagation. This results in the network learning the XOR function as indicated by the predicted outputs.

Uploaded by

Gautam Kumar
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 14

Don’t Worry About it !

A Dummies Guide to Build a Neural Network from Scratch

Dr. Anish Roychowdhury


Problem Statement
Develop a Simple Neural Network for solving the XOR Table Hidden Layer

Weights - W_hy
bias b_y
Input Layer

X1 Output Layer

Y 1

0
Output
X2 Layer

Weights - W_xh
bias b_h
The Math
Forward propagation
The Math contd.
Loss Function

Derivatives wrt Weights – outer Where..


layer
The Math contd.
Derivatives wrt Weights – inner layer

Where
The Math contd.
Input Outer Weight Inner weight
matrix matrix

Output
The Math contd.
Weight Derivatives Bias Terms
The Math contd.
Forward propagation
The Math contd.

Back propagation : outer


Weights
The Math contd.
Back propagation : Inner Weights
The Math contd.
Updating Weights

Notes: In this Example We are


setting the Bias terms to Zero
05/06/2021 Deep_Learning_1_ANN_From_Scratch

ANN From Scratch


In [9]: # Imports
import numpy as np
import matplotlib.pyplot as plt

In [2]: # define transfer function and derivative


def sigmoid(x):
return 1.0/(1 + np.exp(-x))

def sigmoid_derivative(x):
return x*(1.0 - x)

# define loss function


def mean_squared_error(actual,predicted):
sum_square_error = 0.0
for i in range(len(actual)):
sum_square_error += (actual[i] - predicted[i])**2.0
mean_square_error = 1.0 / len(actual) * sum_square_error
return mean_square_error

# define plot loss curve


def plot_loss(loss):
plt.plot(loss)
plt.xlabel("Iteration")
plt.ylabel("Mean Squared Error")
plt.title("Loss curve for training")
plt.show()

Implement a basic two layer NN with Single Output : Hidden layer with 4 nodes

Input to Hidden layer Weights1 = [Input X dim, 4] Hidden to output layer Weights2 = [4,1]

Note: Weights are random init : bias = zero for simplicity

file:///Users/anishroychowdhury/Downloads/Deep_Learning_1_ANN_From_Scratch.html 1/3
05/06/2021 Deep_Learning_1_ANN_From_Scratch

In [3]: # define Neural Network Class

class NeuralNetwork:
def __init__(self,x,y):
self.input = x
self.weights1 = np.random.randn(self.input.shape[1],4)
self.weights2 = np.random.randn(4,1)
self.y = y
self.output = np.zeros(self.y.shape)

def forwardprop(self):
self.layer1 = sigmoid(np.dot(self.input,self.weights1))
self.output = sigmoid(np.dot(self.layer1,self.weights2))

def backprop(self):
# Chain rule applied for Loss func der wrt W[hy] and W[xh] weigh
t matrices
# define intermediate computes for d_weights2 computation
delta2 = 2*(self.y-self.output)*sigmoid_derivative(self.output)
# compute derivatives for weights2 matrix
d_weights2 = np.dot(self.layer1.T,delta2)
delta1 = np.dot(delta2,self.weights2.T)*sigmoid_derivative(self.
layer1)
d_weights1 = np.dot(self.input.T,delta1)
# update weights
self.weights1 += d_weights1
self.weights2 += d_weights2

In [4]: ## Define Test data


# XOR Table
X = np.array([[0,0,1],[0,1,1],[1,0,1],[1,1,1]])
y = np.array([[0],[1],[1],[0]])

# Init call
nn = NeuralNetwork(X,y)

In [5]: # Train through 1500 iterations


loss = []
for i in range(1500):
nn.forwardprop()
nn.backprop()
loss.append(mean_squared_error(nn.y,nn.output))

file:///Users/anishroychowdhury/Downloads/Deep_Learning_1_ANN_From_Scratch.html 2/3
05/06/2021 Deep_Learning_1_ANN_From_Scratch

In [6]: ### Check output predicted


print(nn.output)

[[0.01431983]
[0.97929455]
[0.97575642]
[0.02558122]]

In [10]: ### Plot loss


plot_loss(loss)

In [ ]:

file:///Users/anishroychowdhury/Downloads/Deep_Learning_1_ANN_From_Scratch.html 3/3

You might also like