Unit 2

ARTIFICIAL NEURAL NETWORKS

Fundamental concepts of Neural Networks: Neural networks are a type of machine learning
model inspired by the structure and function of the human brain. They consist of interconnected
nodes, called neurons, organized into layers. Each neuron receives input, processes it, and
produces an output. The connections between neurons have associated weights that determine
the strength of the signal transmitted between them. Neural networks learn by adjusting these
weights based on input data and desired output.

Biological Neural Network Artificial Neural Network

Created using computer programs or
Made up of billions of neurons and synapses
specialized hardware.
Can learn by experience, but usually does
Learns by experience
this through supervised learning
Slow and limited Fast and powerful
ANNs are widely used in machine learning
BNNs are essential for various cognitive
for various applications, such as image and
functions in living organisms, including
speech recognition, natural language
memory, decision-making etc.
processing, robotics etc

Artificial neuron model: Figure below shows artificial neuron model. Here, x 1, x 2, x 3, ..., xn
are the n inputs to the artificial neuron. w 1, w 2, ... ,wn are the weights attached to the input links.
The weights are real numbers that represent the strength of the connections between neurons.
The larger the weight, the stronger the connection. Hence the total input received by the soma of
the artificial neuron is the sum of the inputs multiplied by their corresponding weights.To
generate the final output y, the weighted input sum is passed onto a non-linear filter called
activation function which releases the output

y=f(net calculated input)

This activation function introduces non-linearity to the model, allowing the neural network to
capture complex patterns and relationships in the data.
Commonly used activation functions include tanh , ReLU, sigmoid, signum, and softmax,
depending on the specific task and network architecture.

Basic models of Neural Networks (or Architectures of ANN): Artificial Neural networks are
computational models inspired by the structure and function of the human brain. Basic models of
Neural Networks are

1. Single Layer Feedforward Network

2. Multilayer Feedforward Network

3. Recurrent Networks (or) Recurrent Neural Networks(RNN)

4. Modular network

Single Layer Feedforward Network: It is also known as the Perceptron. It consists of an input
layer and an output layer, with no hidden layers. The input layer is the first layer of the neural
network and is responsible for receiving the input data. Each node (artificial neuron) in the input
layer represents a feature or attribute of the input data. The number of nodes in the input layer is
determined by the number of input features.The output layer is the final year which is
responsible for producing the network's outputs or predictions. The synaptic links carrying the
weights connect every input neuron to the output neuron but not vice-versa.

Despite the two layers, the network is termed single layer since it is the output layer, alone which
performs computation. The input layer merely transmits the signals to the output layer. Hence,
the name single layer feedforward network. Figure 2.8 illustrates an example network.
Perceptrons are limited to solving linearly separable problems and have relatively limited
learning capabilities.
Multilayer Feedforward Network: It is also known as a Multi-Layer Perceptron (MLP). It
consists of an input layer, one or more hidden layers, and an output layer. Each node in a hidden
layer is connected to every node in the preceding and succeeding layers through weighted
connections as shown in figure below. Information flows in a single direction, from the input to
the output layer, without any feedback loops. The hidden layer aids in performing useful
intermediary computations before directing the input to the output layer.The introduction of
hidden layers and non-linear activation functions allows MLPs to learn complex patterns and
solve more intricate problems. Backpropagation is commonly used to train these networks.

RECURRENT NEURAL NETWORK: A Recurrent Neural Network (RNN) is designed to

work with sequential data such as time series or natural language. RNNs have a feedback loop
that allows them to process information from previous time steps in addition to the current input.
This makes RNNs well-suited for applications such as speech recognition, machine translation,
and image captioning.

The basic structure of an RNN consists of an input layer, a hidden layer, and an output layer. The
hidden layer contains a set of neurons with a feedback connection that allows them to pass
information from one time step to the next.
The diagram above shows the basic structure of an RNN. The input is fed into the RNN at time
step xt, and the hidden state ht-1 from the previous time step is fed back into the RNN. The output
y(t) is a function of the hidden state at time step t and can be fed into another layer or used as the
final output.

The main and most important feature of RNN is its Hidden state, which remembers some
information about a sequence. The state is also referred to as Memory State since it remembers
the previous input to the network. It uses the same parameters for each input as it performs the
same task on all the inputs or hidden layers to produce the output. This reduces the complexity of
parameters, unlike other neural networks.

• One of the challenges of training RNNs is the vanishing gradient problem, which occurs
when the gradient signal becomes too small to be useful during training.

• To address this problem, several variants of RNNs have been developed, such as Long
Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) networks. These
networks use additional memory cells and gating mechanisms to better control the flow
of information through the network.

Modular Neural Network: A Modular Neural Network (MNN) is composed of several,

modules(tiny neural networks), that are connected in a specific way to achieve a specific task.
Each module is responsible for processing a specific type of input or feature. Each module can
be trained separately using different algorithms and optimization techniques.
The diagram above shows the structure of a basic MNN. The input is fed into the network and is
processed by several modules, each of which is responsible for processing a specific type of
feature. The output of each module is combined to form the final output of the network

Fig shows a block diagram of a modular neural network (MNN) for image recognition. The
MNN consists of Module 1, Module 2 ---- Module Z, and gating network.

The input image is first divided into two sub-images. The first sub-image is processed by Module
1, and the second sub-image is processed by Module 2. The outputs of the two modules are then
combined by the gating network to produce a final classification.

The gating network is a neural network that decides which module to use for each input image.
The gating network takes the outputs of the two modules as input and outputs a single value,
which represents the probability that the first module should be used.

Modular Neural Networks Recurrent Neural Networks

Neural networks that have feedback loops,
Made up of several neural network models allowing the output to be fed back into the
that are linked together input
Used for tasks that are difficult to solve with a Used for tasks that involve sequential data,
single neural network, such as natural such as speech recognition and machine
language processing and computer vision translation
Can be used to solve complex tasks and can Can learn long-term dependencies and can
be more robust to noise handle sequential data
Can be more difficult to train and can be Can be difficult to interpret and can be
more computationally expensive sensitive to the order of the data

Learning rules in Neural Networks: Artificial Neural Networks (ANNs) learn from data using
learning rules or algorithms that adjust the weights and biases of the network's connections.
These rules enable the network to gradually improve its performance on a given task.

(or)

Learning rules are a set of instructions that tell an artificial neural network (ANN) how to adjust
its weights and biases to improve its performance.
Learning rules can be categorized into two broad categories based on their underlying
mechanisms and goals:

• 1. Error Correction Learning Rules:

• 2. Memory-Based Learning Rules:

1. Error correction learning rules are primarily used in supervised learning, where the
network aims to minimize the error between predicted and actual output. Key learning
rules in this category include:
• Gradient Descent: Calculates the gradient of the error (loss) function with respect to the
network's weights and updates weights in the opposite direction to reduce the error.
Variants include batch, stochastic, and mini-batch gradient descent.

• Least Mean Squares (LMS): A straightforward error correction rule that updates
weights in the direction of the error, commonly used in linear regression.

• Backpropagation: This is a learning rule that is used to train multilayer feedforward

neural networks. Backpropagation works by propagating the error from the output layer
of the network back to the input layer. The weights of the connections between neurons
are then adjusted in proportion to the error.

• Stochastic: Stochastic learning rules use a random process to update the weights of the
neural network. This type of learning rule is often used for classification problems

2. Memory-Based Learning Rules: Memory-based learning rules are used in unsupervised

learning, where the network aims to discover patterns within data without explicit error-
correction targets. Key learning rules in this category include:
 • Hebbian Learning: Strengthens connections between neurons when they activate
together, promoting associative learning. The Hebbian rule can be stated as follows: Cells
that fire together wire together.

• Competitive Learning: Neurons compete based on input, often used for clustering and
feature extraction.

Training or learning methods for Neural Networks can be broadly classified into three basic
types:

1. Supervised

2. unsupervised

3. Reinforced

Supervised learning: Supervised learning is a type of machine learning in which the

model(artificial neuron) is trained on labeled data. This means that the input data has been
tagged with the desired output. The model learns to map the input data to the output data.The
goal of supervised learning is to minimize the difference between the predicted output and
the actual output.

In neural networks, supervised learning works by adjusting the weights and biases of the
neurons in the network based on the error between the predicted output and the actual output.
This process is repeated many times until the error is minimized and the network can
accurately predict the output for new data.

Supervised learning is commonly used in tasks such as image recognition, speech

recognition, and natural language processing.

Supervised learning is classified into two types: classification and regression.

 1. Classification is the task of predicting a categorical output, such as whether an email is
spam or not spam.

2. Regression is the task of predicting a continuous output, such as the price of a house.

Unsupervised learning: Unsupervised learning is a type of machine learning where the model is
not given any labeled data. This means that the model does not know what the correct output
should be. The model must then find patterns in the data on its own.

Applications: Customer segmentation, anomaly detection, data visualization.

There are two main types of unsupervised learning: Clustering and Dimensionality reduction.

1. Clustering is the process of grouping similar data points together based on their features.
For example, in customer segmentation, clustering can be used to group customers with
similar buying patterns together.

2. Dimensionality reduction is the process of reducing the number of features in a dataset

while preserving its important characteristics. This can be useful in image recognition,
where there may be thousands of features for each image. By reducing the number of
features, the algorithm can learn faster and more accurately.
 Supervised learning Unsupervised learning
In supervised learning, the model is trained on In unsupervised learning, the model is trained on
labeled data unlabeled data.
The goal is to learn a mapping function from Focuses on discovering patterns and structures in
input to output. the data.
It is classified into two types: classification and It is classified into two types: Clustering and
regression. Dimensionality reduction.

Common applications include image recognition, Common applications include customer

speech recognition, and natural language segmentation, anomaly detection, and data
processing. visualization.
Examples include predicting housing prices, Examples include grouping similar documents,
classifying emails as spam or not, and anomaly detection in network traffic, and
recognizing handwritten digits. extracting topics from text.
Error correction is possible through feedback on No explicit feedback for correctness, as there are
incorrect predictions. no predefined outputs.

Examples of algorithms include linear Algorithms include k-means clustering,

regression, support vector machines, and neural hierarchical clustering, and principal component
networks analysis (PCA).

Applications of ANN in power systems:Neural networks have found several applications in

power systems due to their ability to learn patterns and relationships from data. Here are some
common applications:

• Load Forecasting: Neural networks can predict future electricity demand based on
historical usage patterns, weather conditions, and other relevant factors. Accurate load
forecasting helps power companies optimize generation and distribution resources.

• Economic dispatch: Neural networks can be trained to find the optimal set of power
plants to operate in order to meet the demand for electricity at the lowest cost. This can
help to reduce the cost of electricity generation.

• Fault Detection and Diagnosis: Neural networks can analyze power system data to
detect faults, anomalies, or equipment failures. They can learn normal operating patterns
and identify deviations, enabling quicker detection and timely maintenance.
• Voltage and Frequency Control: Neural networks can help in maintaining stable
voltage and frequency levels in power grids by predicting potential issues and suggesting
control strategies.

• Renewable Energy Forecasting: For systems with renewable energy sources like solar
and wind, neural networks can predict power output based on weather conditions,
enabling better integration into the grid.

• Energy Management: Neural networks can optimize energy consumption in industrial

settings by learning patterns in usage data and suggesting efficient scheduling strategies.

• Power Quality Improvement: Neural networks can analyze power quality data and
suggest corrective measures to mitigate issues like harmonics, voltage sags, and flicker.

• Optimal Reactive Power Compensation: Neural networks can determine optimal

settings for reactive power compensation devices to improve voltage stability and power
factor

• Energy Theft Detection: Neural networks can detect abnormal usage patterns that might
indicate energy theft, helping utilities identify and prevent unauthorized usage.

• Transmission Line Monitoring: By analyzing data from sensors on transmission lines,

neural networks can predict line health, detect potential failures, and suggest maintenance
actions.

• Market Price Prediction: Neural networks can predict electricity market prices by
analyzing historical pricing data and considering factors like supply-demand dynamics
and fuel prices.
 Short questions

1.What are the fundamental concepts of neural networks?

Neural networks are computational models inspired by the human brain's neural structure. They
consist of artificial neurons that process data through interconnected layers, with each neuron
applying an activation function to produce an output. Neural networks can learn from data, adjust
their parameters (weights and biases), and make predictions based on the learned patterns.

2.What are the basic models of neural networks, and how do they differ?
The basic models of neural networks include the perceptron model for single-layer networks and
the multi-layer perceptron (MLP) model for multi-layer feedforward networks. The perceptron is
limited to solving linearly separable problems, while the MLP, with hidden layers and non-linear
activation functions, can handle complex, non-linear problems.

3.What are learning rules in neural networks?

Learning rules are algorithms used to adjust the connection weights in neural networks during
the training process. A popular learning rule for supervised learning is the backpropagation
algorithm, which updates weights based on the gradient of the error between the predicted and
actual outputs.

4.How do single layer and multi-layer feedforward and feedback networks differ?
Single-layer feedforward networks consist of one layer of neurons with no feedback connections,
making them suitable for simple tasks. In contrast, multi-layer feedforward networks, with
hidden layers, can handle more complex problems. Feedback networks, such as recurrent neural
networks, have connections that loop back, enabling memory and temporal processing.

5. What are the differences between supervised and unsupervised training methods in neural
networks?
Supervised training involves providing labeled data (input-output pairs) to the network to learn
from, while unsupervised training uses unlabeled data to uncover patterns and relationships
without explicit output targets.

6.What are recurrent networks in neural networks?

Recurrent networks, like recurrent neural networks (RNNs), have feedback connections that
enable them to process sequential data and time-series information. RNNs can model temporal
dependencies and are suitable for tasks such as natural language processing and speech
recognition.
7. What is a modular network in neural networks?
A modular network is composed of multiple specialized sub-networks or modules, each designed
for specific tasks. By integrating these modules, the network can adapt to different requirements,
allowing for flexibility and ease of reconfiguration.

8.How are neural networks applied in power systems?

Neural networks are used in power systems for applications like short-term and long-term load
forecasting, fault detection and diagnosis, economic load dispatch, and optimal power flow. They
leverage historical and real-time data to improve prediction accuracy, system stablity, and energy
optimization.

9. How do artificial neurons learn?

Artificial neurons learn by adjusting the weights of the connections between them. The weights
are adjusted based on the error between the network's output and the desired output. The error is
calculated using a loss function, and the weights are adjusted using an optimization algorithm.

10. What are the different types of activation functions?

Activation functions are used to introduce non-linearity into neural networks. They allow the
network to learn more complex relationships between the inputs and outputs. Some common
activation functions include the sigmoid function, the tanh function, and the ReLU function.

11. What is the Backpropagation algorithm?

The Backpropagation algorithm is a type of supervised learning algorithm that is used to train
artificial neural networks. It works by calculating the error between the network's output and the
desired output. The error is then propagated back through the network, and the weights of the
network are adjusted to reduce the error.

12. What is the Delta rule?

The Delta rule is a type of supervised learning algorithm that is used to train artificial neurons. It
works by calculating the error between the neuron's output and the desired output. The error is
then used to update the weights of the neuron.

13.What is the Gradient Descent algorithm?

The Gradient Descent algorithm is a type of optimization algorithm that is used to find the
minimum of a function. It works by starting at a random point and then moving in the direction
of the negative gradient of the function. The gradient of a function is a vector that points in the
direction of the steepest ascent of the function.
 Multiple Choice questions
1. Which of the following is NOT a fundamental concept of neural networks?

a) Activation function

b) Backpropagation

c) Neuron

d) Tensor

2. What is the key characteristic of a single-layer feed-forward neural network?

a) It contains one hidden layer between the input and output layers.

b) It can have multiple hidden layers.

c) It has feedback connections from the output layer to the input layer.

d) It does not have any hidden layers.

3. The learning rule commonly used in training feed-forward neural networks is:

a) Hebbian learning

b) Delta rule (Widrow-Hoff rule)

c) Reinforcement learning

d) Genetic algorithm

4. A recurrent neural network (RNN) is designed to:

a) Have multiple hidden layers.

b) Process sequential data with feedback connections.

c) Only handle binary inputs.

d) Learn from a limited amount of training data.

5. Which method of training involves providing the neural network with input-output pairs
during the learning process?

a) Unsupervised learning

b) Reinforcement learning

c) Supervised learning

d) Semi-supervised learning

6. A modular neural network consists of:

a) A single large network with numerous hidden layers.

b) Multiple small neural networks, each specialized for a specific task.

c) A feedback loop from the output layer to the input layer.

d) An equal number of neurons in each layer.

7. In a power system application, a neural network can be used for:

a) Real-time control of power generation.

b) Circuit breaker operation.

c) Manual monitoring of energy consumption.

d) Solving linear equations.

8. The primary advantage of using a multi-layer feed-forward network over a single-layer

network is:

a) Faster training time.

b) Better generalization to new data.

c) Simplicity in design.

d) Robustness to noisy input data.

9. Which learning rule enables a neural network to adjust its weights based on the difference
between its output and the desired output?

a) Hebbian learning
 b) Competitive learning

c) Delta rule (Widrow-Hoff rule)

d) Backpropagation

10. An unsupervised learning method is commonly used for:

a) Image classification tasks.

b) Identifying patterns in data without labeled examples.

c) Training deep neural networks.

d) Reinforcement learning tasks.

Answers:

1. d) 2. d) 3. b) 4. b) 5. c) 6. b) 7. a) 8. b) 9. d) 10. b)

Long Questions

1. Explain the artificial neuron model with a relevant diagram

2. Explain the differences between supervised and unsupervised learning in

neural networks

3. Explain the architectures of Single Layer Feedforward and Multilayer

Feedforward in Artificial Neural Networks.
4. Explain the architecture of the Recurrent Neural Network.
5. Explain the architecture of a Modular Network.

6. What are the applications of neural networks in power systems?