Neural Networks

Chapter 2- Artificial Neural Networks

Introduction
Artificial Neural Networks (ANNs) are computational models designed to mimic
the functioning of the human brain. They are widely used in pattern
recognition, data classification, and decision-making tasks. This chapter
introduces the fundamental concepts, evolution, models, learning methods,
activation functions, and essential terminologies related to ANNs.

2.1 Fundamental Concepts

An Artificial Neural Network (ANN) is a system inspired by the structure and
function of biological neural networks (human brain). It consists of multiple
interconnected neurons (nodes) that process information in parallel. The key
properties of ANNs include:
• Non-linearity: ANNs can model non-linear relationships, unlike
traditional statistical methods.
• Generalization: Once trained, an ANN can generalize knowledge and
predict outputs for unseen data.
• Fault tolerance: Even if some neurons fail, the network can still function
properly.
 • Adaptability: The network learns from input data and updates itself
accordingly.
Components of an ANN:
• Neurons (Nodes): Basic processing units that take inputs, apply weights,
and pass outputs to the next layer.
• Weights: Numerical values that adjust the influence of inputs on the next
neuron.
• Bias: A constant value added to the weighted sum to adjust the output.
• Activation Function: Determines the output of the neuron based on the
weighted sum.
• Network Layers:
o Input Layer: Receives raw data.
o Hidden Layers: Process and extract features from the data.
o Output Layer: Produces the final decision or classification.

2.2 Biological vs. Artificial Neurons

Biological Neurons:
• A biological neuron consists of a cell body (soma), dendrites, and an
axon.
• Dendrites receive signals from other neurons.
 • The soma processes the signals and, if a certain threshold is reached, the
axon transmits the output.
• Communication occurs through synapses via electrical and chemical
signals.
Artificial Neurons:
• Represented mathematically using weighted inputs and activation
functions.
• Inputs are multiplied by weights and summed up.
• The activation function determines whether the neuron should be
"activated" (i.e., pass the signal forward).
Brain vs. ANN (Key Differences):

Feature Biological Neuron Artificial Neuron

Processing Electrochemical Mathematical

Hebbian Learning (synaptic Weight adjustments via

Learning
adjustments) algorithms

Fault Lower than biological but still

High (due to redundancy)
Tolerance robust

Speed Slow (milliseconds) Faster (microseconds)

Structure Asynchronous, complex Simplified, layer-based

2.3 Evolution of Neural Networks
Historical Developments:
1. 1943: McCulloch-Pitts Model
o Proposed the first mathematical model of a neuron using binary
threshold logic.
2. 1949: Hebbian Learning Rule
o Introduced the concept of strengthening neuron connections
based on repeated activation.
3. 1958: Perceptron (Frank Rosenblatt)
o Introduced a simple neural network with adjustable weights,
capable of learning.
4. 1969: Minsky & Papert's Criticism
o Showed that single-layer perceptrons cannot solve problems like
XOR, leading to reduced interest in ANNs.
5. 1980s: Backpropagation Algorithm
o Allowed multi-layer networks to learn efficiently, leading to
renewed interest in ANNs.
6. 1990s-Present: Deep Learning Revolution
o Multi-layered networks (Deep Learning) achieved breakthroughs
in AI applications.

2.4 Basic Models of Artificial Neural Networks

2.4.1 Types of Neural Networks:
1. Single-layer Perceptron:
o Consists of one layer of neurons with weighted inputs.
o Can solve only linearly separable problems.
2. Multilayer Perceptron (MLP):
 o Includes one or more hidden layers for complex decision-making.
o Uses the backpropagation algorithm for training.
3. Recurrent Neural Networks (RNNs):
o Contains loops allowing information to persist over time (useful for
sequential data like speech or text).
4. Convolutional Neural Networks (CNNs):
o Specialized for image recognition by applying convolutional filters.

1. Single-layer feed-forward network

o This is the simplest type of neural network. It has an input layer
and an output layer, but no hidden layers. Information flows only
in one direction—from input to output—without looping back.
 o

2. Multilayer feed-forward network

o This network has multiple layers, including hidden layers between
the input and output. Each layer processes the data and passes it
forward to the next layer. It helps in learning complex patterns.

3. Single node with its own feedback

o This model consists of a single neuron (node) that loops its output
back to itself. This feedback mechanism allows it to remember
past inputs, which can be useful for time-dependent data.

o
4. Single-layer recurrent network
o In this model, neurons are connected in a way that allows
feedback loops within the layer. This means the output of some
neurons can influence their own inputs or those of other neurons
in the same layer.

5. Multilayer recurrent network

o This is an advanced network with multiple layers where neurons
can send feedback signals to both their own layer and previous
layers. It helps in processing sequential or time-series data, like
speech or handwriting recognition.

o
2.5 Learning Methods in Neural Networks
2.5.1 Supervised Learning:
• The network learns from labeled training data.
• Example: Image classification (labels such as "cat" or "dog").
• Uses techniques like Backpropagation and Gradient Descent.

• During training. the input vector is presented to the network, which

results in an output vector. This output vector is the actual output vector
Then the actual output vector is compared with the desired (target)
output ·vector. If there exists a difference between the two output
vectors then an error signal is generated by the network. This error signal
is used for the adjustment of the weights until the actual output matches
the desired output.
2.5.2 Unsupervised Learning:
• The network finds patterns in unlabeled data.
• Example: Clustering customer data for market segmentation.
• Uses techniques like K-means clustering and Self-Organizing Maps
(SOMs).
• there is no feedback from the environment to inform what the outputs
should be or whether the outputs are correct. In this case, the network
must itself discover pattern regularities, features or categories from the
input data and relations for the input data over the output.
 • In ANNs following unsupervised learning, the input vectors of similar
types are grouped together without the use of training data to specify
how the member of group looks or to which group a number belong. In
the training process, the network receives the input and organizes these
patterns to form clusters. When a new input pattern is applied, the
neural ·· network gives an output response indicating the class in which
the input pattern belongs. If for an input, a pattern class cannot be found
then a new class is generated .

2.5.3 Reinforcement Learning:

• The network learns by interacting with the environment and receiving
rewards or penalties.
• The learning based on this critic information is called reinforcement
learning and the feedback sent is called reinforcement signal
• This learning process is similar to supervised learning. In the case of
supervised learning, the correct target output values are known for each
input pattern. But, in reinforcement learning the exact correct output is
not known less info is available.
• Example: Training AI to play chess or control a robot.

•
2.6 Activation Functions and Their Equations
Activation functions determine how the weighted sum of inputs is
transformed into an output signal. Here are the commonly used activation
functions:

1. Identity Function (Linear Activation Function)

• Definition: The output is directly proportional to the input.

• Equation:

• Characteristics:
o Used in linear regression and some output layers.
o No non-linearity, making it unsuitable for complex problems.

2. Binary Step Function

• Definition: A threshold function that outputs either 0 or 1 based on the
input value.

• Equation:
 •

• Characteristics:
o Used in simple binary classification tasks.
o where thetha represents the threshold value. This function is
most widely used in single-layer nets to convert the net input to
an output that is a binary (1 or 0).

3. Bipolar Step Function

• Definition: A variation of the step function where the output is either -
1 or -1 instead of 0 and 1.

• Equation:
• where thetha represents the threshold value. This function is also used
in single-layer nets to convert the net input to an output that is
bipolar(+ 1 or -1).

•
 • Characteristics:
o Used in threshold-based decision-making tasks.
o Lacks smooth transitions, making it unsuitable for deep learning.

4. Sigmoidal Function (Logistic Function)

• Definition: A smooth, S-shaped function that maps any real-valued
number into a range between 0 and 1.

•
 •

• Characteristics:
• If the nerwork uses a binary data, it is better to convert it to bipolar
form and use the bipolar sigmoidal activation function or hyperbolic
tangent function.
o Introduces non-linearity.
o Used in probabilistic models and logistic regression.
o Suffers from the vanishing gradient problem, making deep
networks difficult to train.

5. Ramp Function (ReLU - Rectified Linear Unit)

• Definition: A function that outputs zero for negative inputs and
remains linear for positive inputs.

• Equation:

• Characteristics:
 o Efficient for deep learning due to its simple derivative.
o Helps mitigate the vanishing gradient problem.
o Can suffer from the dying ReLU problem, where neurons become
inactive for negative inputs.

2.7 Important Terminologies in Neural Networks

1. Weights: Control the influence of inputs on neurons.
2. Bias: Allows shifting the activation function for better flexibility.
3. Threshold: Minimum value needed for neuron activation.
4. Learning Rate: Determines the step size for weight updates.
5. Momentum Factor: Helps prevent oscillations during training.
6. Vigilance Parameter: Used in adaptive networks to control the level of
detail in pattern recognition.

2.8 McCulloch-Pitts Neural Model

Overview
• Introduced in 1943 by Warren McCulloch and Walter Pitts.
• First mathematical model of a neuron, based on binary threshold logic.
• It is a simple feedforward neural network that performs logic operations.
• Used to model basic decision-making processes.
Key Characteristics
• Binary Inputs and Outputs: Only 0 or 1.
• Threshold-Based Decision: Output is 1 if the weighted sum is above the
threshold.
• Performs Logical Operations: AND, OR, NOT operations can be
implemented.
• Limitations: Cannot solve non-linearly separable problems like XOR.
2.9 Hebbian Neural Model
Overview
• Introduced in 1949 by Donald Hebb.
• Based on the principle "Neurons that fire together, wire together."
• It strengthens connections between neurons that activate
simultaneously.

Key Characteristics
 • Hebb's Rule: If two neurons activate together, their connection
strength increases.
• Self-organizing learning: No external supervision is needed.
• Used in associative learning: Helps in memory storage and pattern
recognition.
• Limitations: No weight decay, leading to unbounded growth of weights.

Flowchart of training algorithm

2.10 Linear Separability
Linear separability is a concept in neural networks and machine learning that
refers to the ability to separate different classes of data using a straight line,
plane, or hyperplane.
A dataset is linearly separable if a straight line (for 2D data), a plane (for 3D
data), or a hyperplane (for higher dimensions) can be used to separate one
class from another. If no such separation is possible using a straight boundary,
the dataset is non-linearly separable.
• Example: AND & OR functions are separable, but XOR is not.
 • Multi-layer networks are required for solving non-linearly separable
problems.

2.11 Summary
• ANNs are inspired by biological neural networks.
• They consist of layers of neurons with weighted connections.
• Learning occurs through weight adjustments.
• Activation functions define neuron output behavior.
• Different models and learning techniques are used for various
applications.