ISP560

06 – Backpropagation Neural
Network
Mohd Razif Shamsuddin
0193371231
 Machine Learning:
Backpropagation Neural Network
Mohd Razif Shamsuddin :: Faculty of Computer Science & Mathematics
Backpropagation
Backpropagation Neural Network as an Extension of Linear Regression and Perceptron
6.1 Introduction: From Linear Regression to Backpropagation
• Linear Regression:
• In linear regression, the relationship between input features and the target variable is modeled as a
linear equation, where the weights are adjusted to minimize the error.
• Limitations: Linear regression can only model linear relationships and is limited to a single output,
making it unsuitable for more complex tasks.
• Perceptron:
• The perceptron is a simple neural network unit that uses linear functions and applies an activation
function to decide whether to activate the neuron. However, it can only solve linearly separable
problems.
• Backpropagation Neural Network:
• Backpropagation is an extension of both linear regression and perceptrons. It is a multi-layered neural
network where the model can learn non-linear relationships through multiple hidden layers and
activation functions.
• The core idea of backpropagation is to adjust the weights in each layer of the network based on the
error (similar to how gradient descent is used in linear regression).
What is Backpropagation Neural Network?
• Backpropagation: Backward Propagation of Errors
• It is a multilayer neural network and a standard
method of training artificial neural networks.
• This method helps to calculate the gradient of a loss
function with respects to all the weights in the
network.
• It is the method of fine-tuning the weights of a neural
net based on the error rate obtained in the previous
epoch
• Proper tuning of the weights allows you to reduce
error rates and to make the model reliable by
increasing its generalization
..cont
Why BPNN?
• Backpropagation is fast, simple and easy to program
• It has no parameters to tune apart from the numbers of input
• It is a flexible method as it does not require prior knowledge
about the network
• It is a standard method that generally works well
• It does not need any special mention of the features of the
function to be learned.

BPNN has 3 main phases:

• Feedforward
• Backpropagate the error
• Weight changes
Backpropagation Algorithm

2
3
4
How it works?
Important Key Points:
• Simplifies the network structure by elements weighted links that
have the least effect on the trained network
• Although a single-layer net is severely limited in the mappings it
can learn, a multilayer net (with one or more hidden layers) can
learn any continuous map- ping to an arbitrary accuracy. More than
one hidden layer may be beneficial for some applications, but one
hidden layer is sufficient. (Fausette, 1990 pg 290)
• Need to study a group of input and activation values to develop the
relationship between the input and hidden unit layers.
• It helps to assess the impact that a given input variable has on a
network output and the knowledge gained from this analysis should
be represented in rules.
• Backpropagation is especially useful for deep neural networks
working on error-prone projects, such as image or speech
recognition.
Disadvantage of BPNN
• The actual performance of backpropagation on a
specific problem is dependent on the input data.
• Backpropagation can be quite sensitive to noisy data
• Need to use the matrix-based approach for
backpropagation instead of mini-batch.
References:
• Fausett Book
• Artificial Intelligence Book by Rob Callan
• https://www.guru99.com/backpropogation-neural-network.html
• https://wp.wwu.edu/machinelearning/2017/01/22/generalization-and-
overfitting/
• Negnevitsky: Introduction to Artificial Intelligence
• Previous Lecture Notes iTS660
Exercises:
MACHINE LEARNING
REVISION Week 7
What have we learn so far?
• 20/11/2023
By now you should know what is Machine Learning:
-In essence, machine learning is about (1)enabling computers to learn
from experience and (2)make intelligent decisions or predictions based
on that learning, (3)without being explicitly programmed for each task.
-its about mathematics and science and it’s a subfield of A.I.
What have we learn so far?
Why should we learn about Machine Learning & what is its applications in
the real world?
Why? (1) Automation and Efficiency, (2) Solving Complex Problems, (3)Career
Opportunities, (4) Competitive Edge, (5) Pattern, Predictive, Behavioral
Analytics, (6) Research Opportunity

Application:
- Medical Field (Diagnosis)
- Digital Imaging
- Predictive analysis
- Data Analytics
What we have learn so far?
What are the similarities of ANN and BNN?
- Neurons/Nodes : both have input and output.
- Connections/Edges : connects the neurons
- Learning mechanism: how to update weights
- Activation of neuron: how signals are transmitted through the
network
What are the available Machine Learning
Algorithm.
• Algorithm = step by step processes
• Model = trained algorithm
• Architecture = the structure of the ANN

Type of ML – (1) Supervised (2) Unsupervised

Question and Answer Session
Zikry - Because machine learning is an evolvement of technology
that allows the computer to make intellgent decision without the
interference of humans example : Medical field (Diagnosis)
(Supervised)Perceptron
• Major Steps in perceptron?
• Step 1: Initialization – set initial weights, learning rate, threshold, other
parameters and etc..
• Step 2: Activation – the network produces an output, (actual network
output). Y = step(sum((xi*wi)-t))
• Step 3: Weight training – Update the weight by calculating the error.
e = yd – ya, dw = a * x * e, w(p+1) = w(p) + dw
• Step 4: iteration, repeat step number 2.
(Supervised)
Backprop (Multi-layer perceptron)
• Major Steps in BPNN?
• Step 1: Initialization – set initial weights, learning rate, threshold, other parameters
and etc..
• Step 2: Activation – the network produces an output at hidden neuron in the hidden
layers and the output layer (actual network output).
Yj = sigmoid(sum((xi*wi)-tj)) [tanh, relu] (hidden output)
Yk = sigmoid(sum((xj*wj)-tk)) [tanh, relu] (final output)
• Step 3: Weight training – Update the weight by calculating the error.
e = yd – ya,
δek = yk*(1-yk)*ek (at output)
δej = yj*(1-yj)*(sum(δek*wjk)) (at hidden)
dw = a * x * e, (hidden and output)
w(p+1) = w(p) + dw
• Step 4: iteration, repeat step number 2.
 ISP560
07 – Recurrent Neural Network
Mohd Razif Shamsuddin
0193371231
Machine Learning: RNN
Hopfield Network
Mohd Razif Shamsuddin :: Faculty of Computer Science & Mathematics
Recurrent Neural Network
Recurrent Neural Networks (RNNs) are a class of artificial neural networks designed to process
sequential data by maintaining a memory of previous inputs. Here are the must-have features of
an RNN:
1. Sequential Data Processing
• RNNs are designed to handle sequences of data, such as time-series data, text, speech, or any data
that has a temporal or sequential structure. The key feature of RNNs is that they process input one step
at a time and maintain a "memory" of past inputs.
2. Recurrent Connections (Feedback Loops)
• Unlike feedforward neural networks, RNNs have feedback loops where the output of a neuron at one
time step is fed back into the network as part of the input for the next time step. This allows the network
to retain information over time.
3. Hidden State
• The hidden state is a key feature of RNNs. It is a representation of the "memory" of the network that
captures information from previous time steps. The hidden state is updated with each new input and
helps the network to remember past information that is important for making predictions.
Recurrent Neural Network
4. Shared Weights Across Time Steps
• The same set of weights is used for every time step in the sequence. This weight
sharing helps the model to generalize across different sequence lengths and
ensures that the model can handle variable-length input sequences.
5. Unfolding in Time
• RNNs can be "unfolded" or "unrolled" over time, which allows the network to be
viewed as a series of layers where each layer represents a different time step in the
input sequence. This visualization helps understand how data flows through the
network across time steps.
6. Ability to Capture Temporal Dependencies
• RNNs can capture dependencies between data points in a sequence, making them
particularly useful for tasks where previous data influences future data. This
includes tasks like speech recognition, language modeling, and time series
prediction.
Recurrent Neural Network
7. End-to-End Training with Backpropagation Through Time
(BPTT)
• RNNs are trained using a variant of backpropagation called
Backpropagation Through Time (BPTT), which adjusts weights by
calculating the gradients over all time steps. This allows the
network to learn temporal dependencies in the data.
8. Handling Variable-Length Input
• RNNs are inherently flexible and can handle variable-length
sequences. This is crucial for tasks such as machine translation
or text generation, where the input data may not have a fixed size.
Recurrent Neural Network
Optional Features (Enhanced RNN Variants):
• While the above features are must-haves for any RNN, there are enhanced variants
that address certain limitations of standard RNNs, such as:
Long-Term Memory: Standard RNNs struggle to capture long-term dependencies due
to vanishing gradients. Enhanced models like LSTMs (Long Short-Term Memory) and
GRUs (Gated Recurrent Units) overcome this issue by incorporating mechanisms for
remembering and forgetting information over longer time periods.
Gating Mechanisms: LSTMs and GRUs have additional gates (input, forget, and
output gates) to better control the flow of information through the network.
Conclusion:
RNNs must have sequential data processing, recurrent connections, hidden states,
shared weights, and the ability to capture temporal dependencies, making them ideal
for time-dependent tasks.
Recurrent Neural Network
Hopfield Networks Are RNNs:
1.Recurrent Structure:In a Hopfield network, the neurons are fully connected
and feedback to each other, meaning the output of one neuron can influence
others in the same network. This feedback structure is a hallmark of
recurrent neural networks, where connections form cycles.
2.Memory-Like Behavior:Hopfield networks are designed to store and retrieve
memory patterns. When presented with an incomplete or noisy input, the
network iterates over its states until it converges to a stored pattern (this
process is also recurrent).
3.Energy Minimization:The Hopfield network evolves iteratively, adjusting its
states by minimizing an energy function. This recurrence in state evolution
continues until the network reaches a stable state, which corresponds to a
stored memory pattern.
Recurrent Neural Network
Differences from Modern RNNs:
• Hopfield Networks are static (they settle into an equilibrium point),
while modern RNNs (like LSTMs or GRUs) are typically used to process
sequences of data over time, maintaining hidden states that evolve
with each time step.
• Hopfield networks are more often used for associative memory,
whereas modern RNNs handle tasks such as sequence modeling,
speech recognition, and time series prediction.
• In summary, Hopfield networks are recurrent because they have
feedback connections and evolve iteratively, but they serve different
purposes compared to modern recurrent neural networks used in deep
learning.
Hopfield Neural Networks
Hopfield Neural Networks
1

*Once the weights are calculated, they remain fixed.

Hopfield Neural Networks
2
Hopfield Neural Networks
3
Hopfield Neural Networks
Exercise
Exercise – Use case
Memory Problem and BAM's Role

• The memory problem refers to the challenge of efficiently storing and

recalling data without confusion or interference between stored
patterns, especially when inputs are noisy or incomplete.
• Hopfield Network Limitations: Hopfield networks, while powerful for
recalling stored patterns, can suffer from limitations like low capacity
(the number of patterns that can be stored and retrieved), and they
primarily focus on auto-association (storing and recalling the same
pattern).
• BAM as a Solution: BAM expands on this by offering bidirectional
retrieval, allowing better pattern mapping between two datasets. This
reduces the chance of errors in associative memory because it focuses
on relating two sets of patterns rather than just self-association.
Additionally, BAM increases flexibility in tasks requiring complex
mappings (e.g., translation from one set of data to another).
 Machine Learning: RNN
Bidirectional Associative Memory (BAM)
Mohd Razif Shamsuddin :: Faculty of Computer Science & Mathematics
Bidirectional Associative Memory (BAM) as a
Predecessor to Hopfield Networks

Bidirectional Associative Memory (BAM) is a type

of recurrent neural network that acts as a form of
associative memory, which allows for the storage
and retrieval of pattern pairs. It can be seen as a
precursor to Hopfield networks and addresses
some specific aspects of memory retrieval.
Bidirectional Associative Memory (BAM)
Key Features of BAM
Bidirectionality:
• Unlike Hopfield networks, which only retrieve a pattern given a noisy or partial input from
the same domain (i.e., unidirectional association), BAM can retrieve one pattern from a
related pattern in another domain.
• BAM connects two sets of neurons (usually input and output layers) and allows pattern
retrieval to flow in both directions, making it possible to recall a corresponding output
pattern given an input and vice versa.
Associative Learning:
• BAM works on the principle of associative memory. It stores pairs of patterns (one in
each layer) and retrieves the associated pattern when presented with part of the input.
For example, it can map an input pattern to an output pattern and vice versa.
Hetero-Associative Memory:
• While Hopfield networks perform auto-association (retrieving the same pattern when
presented with noisy input), BAM performs hetero-association—meaning it can
associate one pattern with a different, complementary pattern (e.g., image-label pairs).
Bidirectional Associative Memory (BAM)
Bidirectional Associative Memory (BAM)
Bidirectional Associative Memory (BAM)
Bidirectional Associative Memory (BAM)
Bidirectional Associative Memory (BAM)
Exercise (Question & Answer)
• Use other variation of input – output patterns as probe
• What is the similarity between Hopfield network and BAM
• What is the difference between Hopfield network and BAM
• How do we use these neural network to solve real world problems.
What are it’s application? Don’t know? lets find out.
 ISP560
08 – Hebbian Network
Mohd Razif Shamsuddin
0193371231
Machine Learning: Unsupervised
Network
Hebbian Network
Mohd Razif Shamsuddin :: Faculty of Computer Science & Mathematics
Hebbian Neural Network
8.1 Introduction to Hebbian Neural Networks
• Hebbian Neural Networks are among the earliest models of
learning in neural networks. They are based on the principles of
unsupervised learning, where the network learns through
association rather than being provided with explicit target labels.
• Hebbian learning is often summarized by the phrase: "Cells that
fire together, wire together." This means that if two neurons
frequently activate together, their connection strengthens.
Hebbian Neural Network
Hebbian Neural Network
8.2 Hebb's Law
Hebb's Law (also known as the Hebbian learning rule) forms the
core principle of Hebbian neural networks:
Hebbian Neural Network
From the previous delta rule, the weight can only increase, so we
introduce a forgetting factor Φ to control the weight change.
Hebbian Neural Network

2
Hebbian Neural Network

4
Exercise
Hebbian Neural Network

Probe
Hebbian Neural Network
8.3 Strengths of Hebbian Networks
Unsupervised Learning: Hebbian networks learn in an unsupervised manner,
making them highly useful for situations where labeled data is scarce or
unavailable. They are designed to find patterns and relationships within data by
strengthening neuron connections that activate together frequently.

Biological Plausibility: Hebbian learning is inspired by how biological neural

networks in the brain learn. The principle of "neurons that fire together, wire
together" closely mirrors real-world neural processes in synaptic
strengthening.

Simple Learning Rule: The learning rule is straightforward and

computationally inexpensive, making it feasible for large-scale networks. The
weight adjustment depends only on the local information of the neuron
activations, without needing backpropagation or gradient calculations.
Hebbian Neural Network
8.4 Limitations of Hebbian Networks
1.Uncontrolled Weight Growth:Hebbian learning, in its basic form, can lead to uncontrolled
growth of weights because the weights continually increase without bounds if neurons keep
firing together. To combat this, various forms of normalization or weight decay are often
introduced.
2.Lack of Negative Learning: Hebbian learning does not incorporate mechanisms to weaken
connections when neurons do not fire together. In many real-world scenarios, connections
between neurons that should not be associated need to be pruned or weakened.
3.Limited Learning Capacity: Hebbian networks can struggle to learn complex patterns or
relationships because they rely solely on pairwise correlations between neurons. More
sophisticated models like backpropagation allow deeper learning of non-linear and
complex relationships.
4.Local Learning Rule: Hebbian learning updates the weights based only on the local activity
of connected neurons, making it less powerful in handling tasks that require global
information (like supervised tasks with specific target outputs).
Hebbian Neural Network
8.5 Hebbian Networks as Unsupervised Training Models
Despite their limitations, Hebbian neural networks have great potential
as unsupervised learning models in several scenarios:
1.Clustering: Hebbian networks can be used to cluster data by finding
natural groupings based on which neurons activate together. This is
especially useful in exploratory data analysis when the structure of the
data is unknown.
2.Pattern Recognition: Hebbian networks are capable of recognizing
patterns in data. For example, in image recognition, patterns that
frequently co-occur (such as specific shapes or textures) strengthen
synaptic connections, enabling the network to recognize these
patterns in future inputs.
Hebbian Neural Network
3. Cognitive Modeling: Hebbian learning is widely used in
neuroscience and cognitive modeling to simulate learning in
biological brains. It helps in understanding how animals and
humans form memories, learn associations, and respond to
repeated stimuli.
4. Self-Organizing Maps (SOMs): Hebbian learning principles are
used in self-organizing maps (SOMs), a type of unsupervised
neural network that can organize high-dimensional input data into
low-dimensional representations, making it easier to visualize and
interpret.
Machine Learning: Unsupervised Neural
Network
Competitive Learning/Self Organizing
Maps
Mohd Razif Shamsuddin :: Faculty of Computer Science & Mathematics
 Self-Organizing Map (SOM) and Competitive
Learning as an Extension of Hebbian Neural
Networks
9.1 Introduction to Self-Organizing Maps (SOM)
• A Self-Organizing Map (SOM) is a type of unsupervised learning neural
network that uses competitive learning to map high-dimensional input data
to a lower-dimensional (typically 2D) grid.
• SOMs were developed by Teuvo Kohonen and are often referred to as
Kohonen maps. They provide a visualization of complex data structures
while preserving the topological relationships between data points.
• SOM can be considered an extension of Hebbian learning because it
employs a similar principle of neurons strengthening their connections
through frequent activation but introduces competition among neurons to
specialize in representing specific patterns.
Self-Organizing Maps (SOM)
7.2 Competitive Learning in SOM
• Competitive Learning is a neural network process where neurons
compete to become the most activated in response to a given input.
The "winning" neuron gets updated to better represent the input. This
approach is a natural extension of Hebb's Law, with an added
emphasis on competition and specialization.
• In SOM, neurons in the grid compete to respond to input data. The
"winning" neuron (and its neighbors) have their weights adjusted to
more closely match the input pattern.
• This competition encourages specialization, where different neurons
respond to different input regions or patterns.
Self-Organizing Maps (SOM)
Self-Organizing Maps (SOM)
7.3 Steps and Processes of Competitive Learning
• The competitive learning process forms the foundation of SOM,
allowing the network to learn and organize input patterns. The
steps are as follows:
Self-Organizing Maps (SOM)
Exercise
KSOM with 3 vectors
 ISP560
01 - Introduction to Machine
Learning
Mohd Razif Shamsuddin
0193371231
01- Introduction to Machine Learning
Mohd Razif Shamsuddin :: Faculty of Computer Science & Mathematics
Introduction to Machine Learning
What is Machine Learning?
1.1 Core Definition:
• Machine Learning (ML) is a branch of Artificial Intelligence (AI)
focused on building systems that can automatically learn and
improve from experience without being explicitly programmed.
• It involves algorithms that process large amounts of data to find
patterns, make decisions, and adapt to new inputs.
Exercise : Do your own search on ML
definitions
• Use Chatgpt, google, Wikipedia.
• Share your findings.
• What are the similarity of these definitions.
• What are the repetitive keywords.
Introduction to Machine Learning
1.2 Alternative Definitions:
• Definition 1 (Tom M. Mitchell, 1997):
"A computer program is said to learn from experience E with respect to some class of tasks T and performance measure
P, if its performance at tasks in T, as measured by P, improves with experience E."
Explanation: ML allows systems to improve performance on tasks as they gain more experience (data), measured by a
specific metric.

• Definition 2 (IBM Cloud):

"Machine learning is a subset of AI that focuses on the use of data and algorithms to imitate the way humans learn,
gradually improving accuracy."
Explanation: It emphasizes learning in a human-like manner, where accuracy improves with more exposure to data.

• Definition 3 (Wikipedia):
"Machine learning is the study of computer algorithms that improve automatically through experience and by the use of
data."
Explanation: This definition highlights the autonomy of ML algorithms in evolving based on their exposure to data.
Introduction to Machine Learning
1.3 Key Components of Machine Learning:
• Data: The fuel for machine learning models, consisting of historical
records, images, or any form of information the algorithm can process.
• Algorithms: Mathematical models designed to process data, recognize
patterns, and make predictions.
• Experience (Training): The process of feeding data to algorithms to
allow them to learn patterns and adjust their predictions based on
feedback.
• Learning: The algorithm's ability to improve its performance over time
as it is exposed to more data.
Introduction to Machine Learning
1.4 How Machine Learning Generally Works:
• Step 1: Data Collection – Gather historical or real-time data for the task.
• Step 2: Data Preprocessing – Clean and prepare the data to ensure the model
can use it effectively.
• Step 3: Model Selection – Choose the right algorithm (e.g., decision trees,
neural networks).
• Step 4: Training – Feed the data into the model to teach it the patterns and
relationships.
• Step 5: Testing – Evaluate how well the model performs on new, unseen data.
• Step 6: Deployment – Use the trained model to make predictions or
automate tasks in real-world applications.
Introduction to Machine Learning
1.5 Real-World Applications of Machine Learning:
• Healthcare: Predicting patient outcomes, drug discovery,
diagnosing diseases.
• Finance: Fraud detection, algorithmic trading, risk assessment.
• Retail: Personalized recommendations, dynamic pricing, demand
forecasting.
• Technology: Speech recognition (e.g., Siri, Google Assistant), self-
driving cars, facial recognition systems.
Introduction to Machine Learning
1.6 Importance of Machine Learning:
• Efficiency: Automates complex and repetitive tasks.
• Scalability: Can handle large datasets and make accurate
predictions without human intervention.
• Adaptability: Continuously improves as more data becomes
available.
• Decision-making: Helps businesses and systems make data-
driven decisions that improve performance and outcomes.
02 - Types of Machine Learning
Mohd Razif Shamsuddin :: Faculty of Computer Science & Mathematics
Types of Machine Learning
2.1 Supervised Learning
• Definition: Supervised learning involves training a model on a
labeled dataset, where the input data is paired with the correct
output. The model learns to map inputs to the correct outputs by
minimizing the error between its predictions and the actual labels.
• How it Works:
• Training: The model is fed with input-output pairs (features and labels). It
learns a function that maps inputs to outputs by adjusting its parameters
to minimize prediction errors.
• Testing: After training, the model is tested on new data (input features) to
predict outputs.
Types of Machine Learning
2.1 Supervised Learning
Example:
Image Classification: A dataset of images (input) is labeled with the
objects in the images (output). The model learns to classify new, unseen
images based on the patterns it has learned.
Spam Detection: Models are trained on labeled emails (spam or not
spam) and can automatically filter emails.
Predictive Maintenance: In industrial settings, data on equipment
performance (e.g., temperature, vibration) is used to predict equipment
failure.
Medical Diagnosis: Given patient data (input features like symptoms,
test results), the model predicts the presence or absence of a disease.
Types of Machine Learning
2.2 Unsupervised Learning
• Definition: Unsupervised learning works with data that is not
labeled. The model explores the structure of the data, identifying
patterns, groups, or associations within the dataset without
predefined labels.
• How it Works:
• Clustering: The model groups similar data points together based on
certain features.
• Dimensionality Reduction: The model reduces the number of variables
in the data while retaining important information, helping to visualize or
speed up data processing.
Types of Machine Learning
2.2 Unsupervised Learning
Example:
Customer Segmentation: A retail company uses purchase data to group
customers into segments based on behavior, preferences, and
purchasing habits without knowing the labels upfront.
Market Segmentation: Grouping customers into distinct segments for
targeted marketing.
Anomaly Detection: Identifying abnormal patterns in network traffic,
financial transactions, or sensor data to detect fraud or system failures.
Recommender Systems: Clustering users based on their viewing habits
to recommend personalized content on platforms like Netflix or YouTube.
Types of Machine Learning
2.3 Reinforcement Learning
• Definition: Reinforcement learning involves an agent learning to
make decisions by interacting with an environment. The agent
takes actions and receives feedback in the form of rewards or
penalties, learning from this feedback over time.
• How it Works:
• Agent: The decision-making entity.
• Environment: The context or system with which the agent interacts.
• Reward: Feedback given to the agent to reinforce good decisions or
penalize bad decisions.
• Goal: Maximize cumulative rewards over time.
Types of Machine Learning
2.3 Reinforcement Learning
Example:
Game AI: An AI agent learns to play chess by playing games and receiving
feedback (win, lose, or draw) after each game. It improves its strategy by
maximizing its chances of winning.
Autonomous Vehicles: A self-driving car learns to navigate through traffic by
taking actions (steering, accelerating) and receiving rewards based on how well
it avoids obstacles or follows traffic rules.
Robotics: Robots learn to perform tasks like assembling parts or walking by
interacting with their environment and adjusting their actions based on
feedback.
Personalized Marketing: E-commerce platforms use reinforcement learning
to optimize product recommendations by learning which items result in
purchases (reward) or customer disinterest (penalty).
Exercise : Give your own example of ML
applications

Share your thoughts

Types of Machine Learning
2.4 Semi-Supervised Learning (Optional)
Definition: Semi-supervised learning is a combination of supervised and
unsupervised learning, where the model is trained on a small amount of labeled data
and a large amount of unlabeled data. It helps reduce the need for extensive labeling
of datasets.
How it Works:
• The model first identifies patterns in the unlabeled data and then refines those patterns
using the small amount of labeled data.
Example:
• Speech Recognition: A voice assistant (like Google Assistant or Siri) is trained using a large
collection of unlabeled audio data combined with a smaller set of transcribed recordings.
• Web Page Classification: Using a small number of manually labeled pages, the model can
categorize a much larger dataset of unlabeled pages into predefined categories.
• Object Detection: Semi-supervised learning can improve the accuracy of object detection
in images, especially when labeling every image is too time-consuming.
Summary of Applications Across Types of
ML
Supervised Learning:
• Predictive models, medical diagnosis, spam filtering, loan approval
systems.
Unsupervised Learning:
• Customer segmentation, fraud detection, genetic data analysis,
recommender systems.
Reinforcement Learning:
• Robotics, autonomous systems, personalized marketing, game AI,
resource management (e.g., optimizing power grids).
Semi-Supervised Learning:
• Speech recognition, web categorization, text and image classification.
03 -Biological Network vs Artificial
Network
Mohd Razif Shamsuddin :: Faculty of Computer Science & Mathematics
The Connection Between Biological and
Artificial Networks
3.1 Introduction to Neural Networks
Biological Neural Networks:
• Biological neural networks, like those in the human brain, consist of
interconnected neurons that transmit electrical and chemical signals to
process information. These networks enable humans and animals to
think, perceive, learn, and respond to the environment.
Artificial Neural Networks (ANNs):
• Artificial Neural Networks are computational models inspired by the way
biological neurons work. ANNs are composed of artificial neurons (also
called nodes or units) that are arranged in layers and can learn to
recognize patterns in data.
The Connection Between Biological and
Artificial Networks
3.2 Components of Biological Neurons
Neuron: The basic unit of the nervous system. A neuron receives
signals, processes them, and transmits information to other
neurons.
Key Parts:
• Dendrites: Receive input signals from other neurons.
• Cell Body (Soma): Processes incoming signals.
• Axon: Transmits signals to other neurons.
• Synapse: The connection between the axon of one neuron and the dendrite of
another neuron, where information is transmitted.
The Connection Between Biological and
Artificial Networks
3.3 Components of Artificial Neurons (Perceptrons)
Artificial Neuron (Node): The basic unit of an artificial neural
network. It takes in inputs, processes them using a weighted sum,
applies an activation function, and produces an output.
• Key Parts:
• Inputs (x1, x2, ..., xn): Corresponds to the features or data points fed into the
network.
• Weights (w1, w2, ..., wn): Determine the importance of each input.
• Summation Function: Combines the weighted inputs.
• Activation Function: Decides whether the neuron should be activated (fired) based
on the combined input.
• Output: The final value after applying the activation function, passed on to other
neurons in the network.
The Connection Between Biological and
Artificial Networks
The Connection Between Biological and
Artificial Networks
Here is an analogy that compares the components of biological neurons and artificial neurons:
Biological Neuron Artificial Neuron Analogy
Dendrites: Receive signals from Inputs: Data points or features
Both collect input information.
other neurons. fed into the neural network.
Cell Body (Soma): Processes Summation Function: Both perform the initial
incoming signals. Aggregates the weighted inputs. processing of input signals.
Axon: Transmits signals to other Output: The result passed to the Both carry forward the processed
neurons. next layer of neurons. information.
Synapse: The junction between
Weights: Influence the strength Both regulate the signal strength
neurons where signals are
and importance of each input. during transmission.
transmitted.
Action Potential: The threshold Activation Function: Decides
Both determine whether a neuron
that determines if a signal is whether to activate the neuron
will fire based on stimuli.
transmitted. based on the input.
The Connection Between Biological and
Artificial Networks
Biological Neural Network Artificial Neural Network Key Differences
Neurons communicate through
Neurons communicate using Biological signals are slower,
mathematical computations
electrical impulses and chemical while artificial ones are
(weighted sums and activation
signals. computed quickly.
functions).
Learning occurs through
Learning occurs by adjusting the Artificial networks learn through
strengthening or weakening
weights during training. algorithms like backpropagation.
synapses (synaptic plasticity).
Highly complex, with billions of Simpler in comparison, typically
Biological networks are far more
neurons and trillions of with millions of neurons in deep
complex.
connections in the brain. learning models.
Specialized for specific tasks like
Adaptable to a wide range of Artificial networks require large
classification, prediction, or
stimuli in dynamic environments. data and are task-spec
pattern recognition.
The Connection Between Biological and
Artificial Networks
Example in Biological Neurons:When you touch a hot object, the
neurons in your skin (sensory neurons) send signals through
dendrites to the cell body, where the information is processed. If
the stimulus is strong enough (e.g., the object is very hot), the axon
will transmit the signal to other neurons, eventually reaching your
brain, which triggers a response (like pulling your hand away).
The Connection Between Biological and
Artificial Networks
Example in Artificial Neural Networks:In an image recognition
task, the inputs could be pixel values of an image. The artificial
neuron takes these values, applies weights to them (like how strong
each feature is), sums the values, and passes them through an
activation function. If the signal passes a certain threshold, the
neuron activates and passes information to the next layer of the
network. The final output could be a classification label, like “cat”
or “dog” in the case of animal recognition.
The Connection Between Biological and
Artificial Networks
3.8 Conclusion: Inspiration from Biology
• Artificial neural networks are simplified versions of biological
systems but can be very powerful in solving complex
computational tasks.
• Despite the similarities, the human brain remains vastly more
efficient and adaptable, while ANNs excel at specific tasks with
speed and precision in well-defined domains.
• Key Takeaway: Understanding biological networks provides
valuable insights into creating more advanced and efficient
artificial systems.
 ISP560
04 – Machine Learning
(Supervised)
Mohd Razif Shamsuddin
0193371231
Machine Learning (Supervised)
Mohd Razif Shamsuddin :: Faculty of Computer Science & Mathematics
Artificial Neural Network (Quick Revision)
Artificial Neural Networks are being touted as the wave of
the future in computing. They are indeed self learning
mechanisms which don't require the traditional skills of a
programmer. But unfortunately, misconceptions have
arisen. Writers have hyped that these neuron-inspired
processors can do almost anything. These exaggerations
have created disappointments for some potential users
who have tried, and failed, to solve their problems with
neural networks.
Biological Neural Network
•

Biological Neural Network

Biological Neural Network (Cont.)
• An artificial neural network consists of a number of very simple and highly
interconnected processors, also called neurons.
• The neurons are connected by weighted links passing signals from one
neuron to another.
• Each neuron receives a number of input signals through its connections;
however, it never produces more than a single output signal.
• The output signal is transmitted through the neuron’s outgoing connection
(corresponding to the biological axon).
• The outgoing connection, in turn, splits into a number of branches that
transmit the same signal (the signal is not divided among these branches in
any way).
• The outgoing branches terminate at the incoming connections of other
neurons in the network.
Artificial Neural Networks

• Artificial Neural Network

Artificial Neural Networks (Cont.)
• The neurons are connected by links (Input and Output Neuron)
• Each link has a numerical weight associated with it.
• Weights are the basic means of long-term memory in ANNs.
They express the strength, or in other words importance.
• A neural network ‘learns’ through repeated adjustments of these
weights.
Biological Vs Artificial
Supervised Learning Algorithm
• Supervised Learning
• Definition: Supervised learning involves training a model on a labeled
dataset, where the input data is paired with the correct output. The
model learns to map inputs to the correct outputs by minimizing the
error between its predictions and the actual labels.

• How it Works:
• Training: The model is fed with input-output pairs (features and labels).
It learns a function that maps inputs to outputs by adjusting its
parameters to minimize prediction errors.
• Testing: After training, the model is tested on new data (input features)
to predict outputs.
Supervised Learning Algorithm
Supervised Learning (Cont.)

Example:
• Image Classification: A dataset of images (input) is labeled with the objects in
the images (output). The model learns to classify new, unseen images based on
the patterns it has learned.
• Spam Detection: Models are trained on labeled emails (spam or not spam) and
can automatically filter emails.
• Predictive Maintenance: In industrial settings, data on equipment performance
(e.g., temperature, vibration) is used to predict equipment failure.
• Medical Diagnosis: Given patient data (input features like symptoms, test
results), the model predicts the presence or absence of a disease.
 04 – Linear Regression
Mohd Razif Shamsuddin :: Faculty of Computer Science & Mathematics
Linear Regression
• 4.1 Introduction to Regression
• What is Regression?
Regression is a fundamental concept in supervised learning,
where the goal is to predict a continuous outcome (dependent
variable) based on one or more input features (independent
variables).
• Linear Regression: The simplest form of regression, where the
relationship between the input and output is modeled as a straight line.
Linear Regression
Linear Regression (Cont.)
Linear Regression (Cont.)
Linear Regression (Cont.)
Linear Regression (Cont.)
Linear Regression Exercise
Linear Regression Exercise (Cont.)
1

2
Linear Regression Exercise (Cont.)
3

4
Linear Regression
• 4.5 Visualization of Linear Regression
• Scatter Plot: A scatter plot of the data points (size vs. price) with
the regression line superimposed.
• The line represents the best-fit line that minimizes the cost function.
• Gradient Descent Visualization: A plot showing the cost
function's surface and how the weights are adjusted during each
iteration of gradient descent, moving toward the minimum.
Linear Regression
• 4.6 Conclusion
• Why Linear Regression?: Linear regression is simple,
interpretable, and often a good starting point for many machine
learning problems, especially when the relationship between the
features and the target is linear.
• Limitations: Linear regression is limited when the relationship
between the input and output is nonlinear or when there are
complex interactions between features.
05 – Perceptron as a simple computing
element
Mohd Razif Shamsuddin :: Faculty of Computer Science & Mathematics
Neuron as a simple computing element
• A neuron receives several signals from its input links
• Computes a new activation level and sends it as an output
signal through the output links.
• The input signal can be raw data or outputs of other neurons.
• The output signal can be either a final solution to the problem
or an input to other neurons. Figure below shows a typical
neuron.
Input Neurons

Output Neurons
 Activation functions
• What are activation functions?
• Activation function decides, whether a neuron should be
activated or not by calculating weighted sum = x_in and further
adding bias with it. The purpose of the activation function is to
introduce non-linearity into the output of a neuron.

x_in = (x1 * w1) + (x2*w2)

Output Neuron
output
x_in =Σ xi*wi, where i x1 w1
ranges from 1 to n Activation Activation =f(x_in)
Input Neurons Function
w2
x2
Step activation
Binary Step Function

A binary step function is a

Output = f(X)
threshold-based
activation function. If the
X = weighted
input value is above or sum
below a certain
threshold, the neuron is
activated and sends
exactly the same signal to
the next layer.
Sign Activation
• Sign Function
• Bipolar value either will produce
+1 or -1 based on its input.
• Similar to step, sign is a Output = f(X)
threshold-based activation
function. If the input value is X = weighted
above or below a certain sum

threshold, the neuron is

activated and sends exactly the
same signal to the next layer.
Sigmoid / Logistic
• Smooth gradient, preventing
“jumps” in output values.
• Output values
bound between 0 and 1,
normalizing the output of each
neuron. Output = f(X)
• Clear predictions—For X
above 2 or below -2, tends to X = weighted
bring the Y value (the sum
prediction) to the edge of the
curve, very close to 1 or 0. This
enables clear predictions.
Exercise - Find other activation functions.

• https://missinglink.ai/guides/neural-network-
concepts/7-types-neural-network-activation-
functions-right/

• Find the reason these activation functions are used.

• Find the advantages and disadvantaged of the
activation functions.
Perceptron
• How do a Perceptron produce an output
• Output = activation ((ΣXi*Wi)+(bias*θ))
• f(x) = step_activation ((ΣXi*Wi)+(-1*θ))
• f(x) = step_activation ((ΣXi*Wi)-θ)
Bias (+1 or -1)

-1

*Remember bias can be a positive or negative value θ

X1
W1
Activation
Function
W2
X2
Output Neuron
Input Neurons
Perceptron (Step 1: Initialization)
• Step 1:
1
• Initialisation Set initial weights w1, w2, ... wn
• set threshold θ to random numbers in the range between [-0.5 to
0.5].
• Set alpha α value at a low value. Example: 0.1
Perceptron (Step 2: Activation)
• Step 2:
2
• Activate the perceptron by applying inputs X1, X2, ... Xn and
desired output Y1.
• Calculate the actual output at iteration p = 1

• where n is the number of the perceptron inputs, and step is a

step activation function.
 Perceptron (Step 3: Weight training)
• Step 3:
3
• Also known as Weight training / Weight adjustments / Weight updates

1. Calculate error first, error is calculated as

e = Yd(desired) – Ya(actual) *if error = 0, weight values stays the same. No weight updation is needed.
2. Calculate the weight change (delta weight, ΔW1, ΔW2 ... ΔWn)
ΔWn = α * xn * e
3. Update the weight by adding the current weight to the calculated
weight change.
Wn (P+1) = Wn (P) + ΔWn
Perceptron (Step 4: Iteration)
• Step 4: Iteration
4
• Simply repeat step number 2 with the newly updated weight.
• If error = 0, the weight should stay the same value.
• The iteration is ended when the perceptron reaches zero error for
all possible input output pairs.
Exercise
• Do operation OR
with the same
parameter
• Weight
• Alpha
• Threshold

• Can perceptron
learn
• XOR operation?

• Why not?
Linearly and non-linearly separable function
Linearly and non-linearly separable function

• If the input is in three dimension. How would we perceive a

linearly separable function?
• Imagine cutting a square cake, as long as you can cut it into two
pieces, its linearly separable

Linearly and non-linearly separable function

Summary