Detailed Explanation of Module 4 Lab 1: Perceptron and Gradient Descent
(Fully Updated and Structured for Beginners, with All Your Queries Addressed)
Section 1: What is a Perceptron?
The perceptron is the simplest type of artificial neuron and a fundamental building block for
neural networks.
It is a binary linear classifier: it learns to separate data into two classes (e.g., -1 and +1)
using a straight line (or hyperplane in higher dimensions) [1] [2] .
Invented by Frank Rosenblatt, the perceptron is also the basis for more complex neural
network architectures.
Section 2: Structure and Components of a Perceptron
A perceptron consists of:
Input nodes: Each receives a feature value.
Weights: Each input is multiplied by a weight, which the model learns.
Bias: An extra parameter that shifts the decision boundary.
Activation function: Usually the sign function (outputs +1 or -1).
Mathematical formula:
Section 3: Perceptron Learning Algorithm (How It Works)
A. Training Process
1. Initialize weights and bias to zero (or small random values).
2. For each training sample:
Compute the activation:
Predict the label:
If prediction is wrong:
Update weights:
Update bias:
is the learning rate (often set to 1).
3. Repeat for several epochs (passes through the data), or until all samples are correctly
classified or a maximum number of epochs is reached [3] [4] [2] .
B. Example: Toy Dataset
X = np.array([
[-2, 4, -1],
[4, 1, -1],
[1, 6, -1],
[2, 4, -1],
[6, 2, -1],
])
y = np.array([-1, -1, 1, 1, 1])
Here, each row is a data point with two features and a bias term (-1).
The labels are -1 or +1.
Plotting the data shows it is linearly separable (can be split by a straight line).
Section 4: Coding the Perceptron Algorithm
A. Perceptron Training Function
def perceptron_algo(X, Y):
w = np.zeros(len(X[^0]))
eta = 1
epochs = 10
for epoch in range(epochs):
for i, x in enumerate(X):
if (np.dot(X[i], w) * Y[i]) <= 0:
w = w + eta * X[i] * Y[i]
return w
The weights are updated only when a mistake is made (a "mistake-driven" algorithm) [4] .
B. Plotting Training Loss
You can plot the total error at each epoch to see how quickly the perceptron learns:
def perceptron_algo_plot(X, Y):
w = np.zeros(len(X[^0]))
eta = 1
n = 30
errors = []
for t in range(n):
total_error = 0
for i, x in enumerate(X):
if (np.dot(X[i], w) * Y[i]) <= 0:
total_error += (np.dot(X[i], w) * Y[i])
w = w + eta * X[i] * Y[i]
errors.append(total_error * -1)
plt.plot(errors)
plt.xlabel('Epoch')
plt.ylabel('Total Loss')
return w
The error should decrease and reach zero if the data is linearly separable.
Section 5: Visualizing the Decision Boundary
The decision boundary is the line (or hyperplane) where the perceptron predicts the class
changes.
Equation:
For 2D:
Plotting:
Use matplotlib to plot the data points and the decision boundary line.
Points on one side are classified as +1, on the other as -1 [5] .
Section 6: Perceptron on Non-Linearly Separable Data
If the data cannot be separated by a straight line (e.g., XOR function), the perceptron will
not converge—it will keep updating weights forever or repeat the same mistakes.
Observation:
For non-linearly separable data, the perceptron cannot find a perfect solution.
Section 7: Effect of Learning Rate
Learning rate ( ): Controls how big the weight updates are.
Try different values:
Small : Slow convergence, but stable.
Large : May not converge, weights can oscillate or diverge.
Experiment:
Plot the error for several learning rates to see the effect.
Section 8: Perceptron and Boolean Functions
A. AND Function Example
The AND function is linearly separable, so a perceptron can learn it.
Update steps:
Start with weights $$.
- For each input, update weights if the prediction is wrong.
- After a few iterations, the perceptron finds weights that perfectly separate the
classes.
B. Which Boolean Functions Are Learnable?
For 2-bit input, there are 16 possible Boolean functions.
Linearly separable functions (learnable): Constant functions, AND, OR, NAND, NOR, etc.
Not linearly separable (not learnable): XOR, XNOR.
Why?
XOR cannot be separated by a straight line; no single hyperplane can split the classes
correctly.
Section 9: Perceptron Algorithm in Practice
Applications: Simple binary classification tasks, linearly separable data.
Limitations: Cannot solve problems where the classes are not linearly separable (e.g.,
XOR).
Modern use: Forms the basis for more complex neural networks (multi-layer perceptrons,
deep learning).
Section 10: Key Takeaways
The perceptron is a simple, foundational algorithm for binary classification.
It learns by updating weights when it makes mistakes, moving the decision boundary to
better separate the classes.
Works only for linearly separable data; for more complex patterns, multi-layer networks are
required.
Visualization of the decision boundary and error plot helps understand learning and
convergence.
Learning rate and data separability are critical for perceptron performance.
If you need more detail on any step, code examples, or further explanation of perceptron
theory or visualization, just ask!
⁂
1. https://www.vinsys.com/blog/what-is-perceptron-study-guide
2. https://www.scaler.com/topics/machine-learning/perceptron-learning-algorithm/
3. https://www.w3schools.com/ai/ai_perceptrons.asp
4. https://users.cs.utah.edu/~zhe/pdf/lec-10-perceptron-upload.pdf
5. https://stats.stackexchange.com/questions/71335/decision-boundary-plot-for-a-perceptron
