COSC 480/592

Embedded AI

Instructor: Dr. Ourania Spantidi

Winter 2025
 Recap
x
1
w1
Neuro
INPUTS

x w2 n
Activation
Function
Prediction
f
y
2
Σ
w3
x
y = f (Σi(wi * xi) + b)
3
b
WEIGHTS
(bias)
Why do we have this bias value?
• Imagine a single neuron that classifies whether a
student passes or fails a test based on two inputs
1. X1: How much time they studied
2. X2: How much they slept before the test
• Assume the following 4 students
1. X1 = 1, X2 = 2 - fails
2. X1 = 2, X2 = 1 - fails
3. X1 = 3, X2 = 3 - passes
4. X1 = 4, X2 = 4 - passes
Why do we have this bias value?
• Assume the following 4 students
1. X1 = 1, X2 = 2 - fails
2. X1 = 2, X2 = 1 - fails
3. X1 = 3, X2 = 3 - passes
4. X1 = 4, X2 = 4 - passes
• We create two neurons with pre-defined weights and f =
u
• One without bias: weights = [1, 0], bias = 0
• One with bias: weights = [1, 0], bias = -2.5
Why do we have this bias value?
What is the
output
• Assume the following 4 students equation of
1. X1 = 1, X2 = 2 - fails the neuron?
2. X1 = 2, X2 = 1 - fails
3. X1 = 3, X2 = 3 - passes [w1 w2]⋅]
4. X1 = 4, X2 = 4 - passes
• We create two neurons with pre-defined weights and f =
u
• One without bias: weights = [1, 0], bias = 0 + ≥0
• One with bias: weights = [1, 0], bias = -2.5
Why do we have this bias value?
What is the
output
• Assume the following 4 students equation of
1. X1 = 1, X2 = 2 - fails the neuron?
2. X1 = 2, X2 = 1 - fails
3. X1 = 3, X2 = 3 - passes [w1 w2]⋅]
4. X1 = 4, X2 = 4 - passes
• We create two neurons with pre-defined weights and f =
u
• One without bias: weights = [1, 0], bias = 0 + ≥0
• One with bias: weights = [1, 0], bias = -2.5
The decision boundary has to go through the
origin!
Why do we have this bias value?
What is the
output
• Assume the following 4 students equation of
1. X1 = 1, X2 = 2 - fails the neuron?
2. X1 = 2, X2 = 1 - fails
3. X1 = 3, X2 = 3 - passes [w1 w2]⋅]+b
4. X1 = 4, X2 = 4 - passes
• We create two neurons with pre-defined weights and f =
u
• One without bias: weights = [1, 0], bias = 0
• One with bias: weights = [1, 0], bias = -2.5
+ - 2.5 ≥ 0
The decision boundary can shift!
 Why do we have this bias value?
• Imagine a single neuron that classifies whether a
student passes or fails a test based on two inputs
1. How much time they studied
2. How much they slept before the test

Without Bias
With Bias
• The neuron can only create a
• The neuron can shift its decision
decision boundary that passes
boundary away from the origin
through the origin
• Can correctly classify all students
• This can result in incorrect
based on their study time
classifications for some students
How will the neuron learn?
• Ideas?
1. Start with random guesses
2. For each student, predict if they pass/fail
3. Compare prediction to correct answer
4. Tweak weights and bias if prediction was off
5. Rinse and repeat from step 2
• A pass through all the examples is called an epoch
6. Call it a day and stop after a predetermined number of
epochs
The perceptron learning rule
1. Prediction (forward pass)
• F(w ⋅ + b)
2. Training Objective
• Correctly classify data points, tweak weights and biases
accordingly
3. Update Rule
• If the model prediction is different than what we expected:
wi = wi + r(ytrue – ypred)xi
b = b + r(ytrue – ypred)
4. Iterate for multiple epochs
The perceptron learning rule
1. Prediction (forward pass)
• F(w ⋅ + b)
2. Training Objective
• Correctly classify data points, tweak weights and biases
accordingly
3. Update Rule
• If the model prediction is different than what we expected:
wi = wi + r(ytrue – ypred)xi
b = b + r(ytrue – ypred)
Error
4. Iterate for multiple epochs
The perceptron learning rule
• What is r?
• Learning rate
• A small number that controls how much the weights and
bias of the perceptron are updated each time the model
makes a mistake during training
• The learning rate is like how big of steps the neuron
takes when it is learning
• A small learning rate (0.01) means small, careful steps
• A big learning rate (0.5) means big, bold steps
Surprise code-along!
• Problem Description
• Goal: Train a perceptron to classify input data according to the
OR and AND logical gates
• OR gate truth table?
• Outputs 1 if either input is 1, otherwise 0
• AND gate truth table?
• Outputs 1 only if both inputs are 1, otherwise 0
 Formatted
Step-by-Step Breakdown string literal
import numpy as np
• Subproblem 1: Define the perceptron structure and
def initialize_perceptron():
initialize the weights and bias
• We need weights =anp.zeros(2)
to create perceptron that takes two inputs
bias =them
and classifies 0 according to the OR gate
• Createreturn
a Pythonweights, bias
function initialize_perceptron that
initializes the weights and bias for two input features
and sets them to zeros (hint: numpy might have
weights, bias =
something)
initialize_perceptron()
print(f"Weights: {weights}, Bias:
{bias}")
 Step-by-Step Breakdown
def step_function(z):
• return
Subproblem
1 if2:z Implement
>= 0 elsethe
0 forward pass using the
weighted sum and step function
• Create the forward pass, which calculates the weighted
def forward_pass(X, weights, bias):
sum and passes it through the step activation function
z = np.dot(X, weights) + bias
• Write a Python function forward_pass that takes inputs,
return
weights, step_function(z)
and bias, and computes the perceptron output
using the step activation function
X = np.array([0, 1])
print(f"Output: {forward_pass(X, weights,
bias)}")
def train_perceptron(X, y, weights, bias,
learning_rate=0.1, epochs=10):
for epoch in range(epochs):
Step-by-Step Breakdown
for i in range(len(X)):
y_pred = forward_pass(X[i], weights, bias)
• Subproblem
if y_pred3:!=
Implement
y[i]: the training algorithm for the
OR gateerror = y[i] - y_pred
• Train the perceptron
weights using the perceptron
+= learning_rate * errorlearning
* X[i] rule,
where bias
we adjust the weights and
+= learning_rate bias when the
* error
perceptron
return makes
weights, incorrect predictions
bias
• Write a train_perceptron function that updates the
weights
X_train and bias based
= np.array([[0, 0],on prediction
[0, 1], [1, 0], errors for the OR
[1, 1]])
gate= np.array([0, 1, 1, 1])
y_train
• Reminder:
wi = wi + r(ytrue – ypred)xi
weights, bias = train_perceptron(X_train, y_train, weights,
bias) b = b + r(ytrue – ypred)
 Step-by-Step Breakdown
• Subproblem 4: Test the trained perceptron
def test_perceptron(X_test, weights, bias):
• Write
for X in aX_test:
test_perceptron function that checks if the
perceptron outputs the correctweights,
output = forward_pass(X, values for all inputs in
bias)
the OR gate truth table
print(f"Input: {X}, Output: {output}")

test_perceptron(X_train, weights, bias)

 Code-along
• What would you change for the AND gate?
y_train_and = np.array([0, 0, 0, 1])
• Input data is the same, but the expected output is
different
weights, bias = initialize_perceptron()
weights, bias = train_perceptron(X_train, y_train_and, weights,
bias)

test_perceptron(X_train, weights, bias)

Single neuron
• 10 epochs to learn the OR gate ✅
• 10 epochs to learn the AND gate ✅
• Could it be possible to learn the XOR gate?
• Why yes/no?
• XOR truth table?
Single neuron
• The XOR gate cannot be learned by a single perceptron
• XOR is not linearly separable
• A perceptron can only classify linearly separable data,
meaning it can only draw a single straight line to
separate different classes in a graph
• XOR requires the data points to be separated in a way
that a straight line can’t achieve
Single neuron
• While single neurons have limitations, they serve as a
fundamental building block towards more complex and
powerful neural network architectures
• Will a single neuron suffice for the following?
• Predicting Whether it Will Rain Tomorrow UNLIKELY
• Identifying Spam Emails UNLIKELY
• Determining if a Fruit is Ripe Based on Color LIKELY
• Recognizing Handwritten Digits UNLIKELY
• Classifying News Articles into Different Categories UNLIKELY
• Deciding Whether to Grant a Loan Based on Applicant’s Credit
Score LIKELY
 Multi-Layer Perceptron
• A class of feedforward artificial neural network that
consists of at least three layers of nodes

The final output

Each neuron in this of the MLP:
layer represents a The number of
different feature of neurons in this
the input data layer depends
These neurons on the number
have no bias One or more layers that of output
are used to learn variables
representations of the
input data through
weighted sums and
activation functions
That’s all for today!
Questions?