TRIBHUVAN UNIVERSITY

INSTITUTE OF ENGINEERING
PULCHOWK CAMPUS

Lab - 4
Artificial Intelligence

Adaline and Backpropagation

Implementation

Submitted By: Submitted To

Name: Nishant Luitel Department of Electronics
Roll No: 076BCT041 and Computer Engineering
Group: B
 Adaptive Linear Neuron (Adaline):

Theory
The Adaptive Linear Neuron (Adaline) is a single-layer neural network that uses a linear activation function and
gradient descent algorithm for learningIt is a type of artificial neural network that is used for supervised learning.
It is an extension of the perceptron algorithm and was developed by Bernard Widrow and Ted Hoff in 1960. The
Adaline algorithm is used for classification and regression problems.

The basic structure of Adaline consists of an input layer, a single output neuron, and a set of weights that are
multiplied by the input values. The output of the network is the weighted sum of the inputs, which is then passed
through the activation function. Unlike the step function used in the perceptron algorithm, Adaline uses a linear
activation function, which allows the output to take on any real value.

The goal of Adaline is to minimize the difference between the network's output and the true output, which is
represented by a target value. This is done by adjusting the weights of the network using the gradient descent
algorithm. The gradient descent algorithm calculates the gradient of the cost function with respect to the weights
and updates the weights in the direction of the negative gradient, which reduces the cost function.The Adaline
algorithm works by updating the weights of the input features based on the difference between the predicted
output and the actual output. The weights are updated using a gradient descent algorithm to minimize the cost
function. The cost function is defined as the sum of the squared errors between the predicted output and the actual
output.

The algorithm can be summarized in the following steps:

1. Initialize the weights: Set the initial weights to small random values.
2. Provide inputs and desired outputs: For each training example, provide the input values and the desired
output value.
3. Calculate the weighted sum: Multiply each input value by its corresponding weight, and calculate the
sum of these products.
4. Apply the activation function: The activation function for Adaline is the identity function, so the output
of the neuron is simply the weighted sum from step 3.
5. Calculate the error: Subtract the predicted output from the desired output to get the error.
6. Update the weights: Use the Widrow-Hoff learning rule to update the weights, which involves
multiplying the error by the input values and the learning rate, and then adding the result to the current
weights.
7. Repeat: Repeat steps 3-6 for all training examples, and then repeat the entire process for a fixed number
of epochs or until the error falls below a certain threshold.
8. Test the model: Once the weights have been learned, apply the model to new input data to make
predictions.
Implementation:
Implementation of adaline to perform AND operation:
The code implements the Adaline algorithm for the AND function using a simple artificial neural network with
one layer of two nodes. It trains the weights and biases of the network to produce the correct output for the AND
function using a learning rate and a number of epochs. The output of the function is the predicted value for a
given input.

Backpropagation

Theory:
Backpropagation is an algorithm used to train artificial neural networks. It involves computing the gradient of the
cost function with respect to the weights of the network, and then using this gradient to update the weights in a
way that minimizes the cost function. It works by first forward propagating an input through a neural network to
compute its output. The output is then compared to the desired output (i.e., the target), and the difference between
the two is used to compute a cost function. The cost function measures how well the network is performing, and
the goal of backpropagation is to minimize this cost function.

To do this, backpropagation computes the gradient of the cost function with respect to the weights of the network.
This gradient represents the direction in which the weights should be updated to reduce the cost function. The
weights are then updated using gradient descent, which involves taking a small step in the direction of the
negative gradient.
The computation of the gradient involves propagating the error back through the network, hence the name
"backpropagation." Specifically, the error is first computed at the output layer, and then propagated backwards
through the network using the chain rule of calculus. This involves computing the derivative of the activation
function at each layer, as well as the derivative of the weight matrix. The final result is a gradient that can be used
to update the weights and improve the performance of the network.

Example

To illustrate backpropagation, let's consider a simple example. Suppose we have a neural network with one input,
one hidden layer with two neurons, and one output. The network looks like this:

Input -> Hidden1 -> Hidden2 -> Output

The activation function used at each neuron is the sigmoid function, and the output of the network is a real
number between 0 and 1. We want to train the network to output 1 when the input is 1, and 0 when the input is 0.

To train the network, we first forward propagate an input through the network to compute its output. For example,
if the input is 1, we have:

]
Hidden1: sigmoid(1*w11 + b1)
Hidden2: sigmoid(1*w21 + b2)
Output: sigmoid(h1*w31 + h2*w32 + b3)

where w11, w21, w31, w32, b1, b2, and b3 are the weights and biases of the network, and h1 and h2 are the
outputs of the hidden layer neurons.

We can then compare the output to the target (i.e., 1), and use the difference to compute a cost function. One
commonly used cost function is the mean squared error, which is given by:

E = (1/2) * (target - output)^2

To minimize this cost function, we need to compute the gradient of the cost function with respect to the weights
of the network. The gradient is given by:

dE/dw31 = (output - target) * output * (1 - output) * h1

dE/dw32 = (output - target) * output * (1 - output) * h2
dE/dw11 = (output - target) * output * (1 - output) * w31 * h1 * (1 - h1) * x
dE/dw21 = (output -
target) * output * (1 - output) * w32 * h2 * (1 - h2) * x
dE/db3 = (output - target) * output * (1 - output)
dE/db1 = (output - target) * output * (1 - output) * w31 * h1 * (1 - h1)
dE/db2 = (output - target) * output * (1 - output) * w32 * h2 * (1 - h2)

where `x` is the input. These gradients can be computed using the chain rule of calculus, as mentioned earlier.

Once we have the gradients, we can update the weights using gradient descent. Specifically, we update the
weights as follows:

w31 = w31 - learning_rate * dE/dw31

w32 = w32 - learning_rate * dE/dw32
w11 = w11 - learning_rate * dE/dw11
w21 = w21 - learning_rate * dE/dw21
b3 = b3 - learning_rate * dE/db3
b1 = b1 - learning_rate * dE/db1
b2 = b2 - learning_rate * dE/db2

Conclusion:
In this lab, we created an implementation of the Adaline algorithm using Python. We showed that Adaline is a
useful tool for solving simple classification and regression problems, and it is easy to use. Backpropagation is
another technique we used to help Adaline learn from data by adjusting weights and biases based on predicted
outputs and target outputs. Although backpropagation can be computationally intensive and prone to overfitting, it
is a powerful tool for solving a wide range of machine learning problems. By utilizing libraries like NumPy,
implementing backpropagation in Python is straightforward and can be applied to real-world datasets. Overall,
this lab focused on demonstrating the implementation of Adaline and backpropagation in Python.