Soft Computing

UNIT I:Basics of Artificial

Neural Networks
Course :Soft Computing
Presented By: Dr P Indira priyadarsini

DEPARTMENT OF INFORMATION TECHNOLOGY 1 11/09/2021

DEPARTMENT OF INFORMATION TECHNOLOGY 2 11/09/2021
 Text Books:
1. Satish Kumar (2004), Neural Networks A classroom Approach Tata McGraw Hill
Publication, New Delhi.
2. Lotfi A. Zadeh (1997), Soft computing and Fuzzy Logic, World Scientific Publishing
Co., Inc. River Edge, NJ, USA

Reference Books:
1. B. Yegnanarayana (2006), Artificial Neural Networks, Prentice Hall of India, New
Delhi, India.
2. John Yen, Reza Langari (2006), Fuzzy Logic, Pearson Education, New Delhi, India.
3. S. Rajasekaran, Vijaylakshmi Pari (2003), Neural networks, Fuzzy Logic and
Genetic Algorithms Synthes is and Applications, Prentice Hall of India, New Delhi,
India.

DEPARTMENT OF INFORMATION TECHNOLOGY 3 11/09/2021

DEPARTMENT OF INFORMATION TECHNOLOGY 4 11/09/2021
 -Soft computing is based on natural as well as artificial ideas. 
Soft Computing techniques are Fuzzy Logic, Neural Network, Support Vector
Machines, Evolutionary Computation and Machine Learning and Probabilistic
Reasoning.

some of the applications of soft computing across different industries-

Handwritten Script Recognition.
Image Processing and Data Compression.
Automotive Systems and Manufacturing.
Soft computing based Architecture.
Decision Support System.
Power System Analysis.
Bioinformatics.
Investment and Trading.

The first application of fuzzy logic is to create a decision system that can
predict any sort of risk. The second application is using fuzzy information that
selects the areas which need replacement.
Soft computing uses the method of Artificial Neural Network (ANN) to predict
any instability in the voltage of the power5 system.
DEPARTMENT OF INFORMATION TECHNOLOGY 11/09/2021
 ANN (Artificial Neural Network)
Artificial intelligence and machine learning haven’t just grabbed headlines and made
for blockbuster movies; they’re poised to make a real difference in our everyday
lives, such as with self-driving cars and life-saving medical devices.

In fact, according to Global Big Data Conference, Artificial intelligence is “completely

reshaping life sciences, medicine, and healthcare” and is also transforming voice-
activated assistants, image recognition, and many other popular technologies.

Artificial Intelligence is a term used for machines that can interpret the data, learn
from it, and use it to do such tasks that would otherwise be performed by humans. 

Machine Learning is a branch of Artificial Intelligence that focuses more on training

the machines to learn on their own without much supervision.

DEPARTMENT OF INFORMATION TECHNOLOGY 6 11/09/2021

DEPARTMENT OF INFORMATION TECHNOLOGY 7 11/09/2021
 ANN
• Humans are able to react adaptively to changes in their external and internal
environment, and they use their nervous system to perform these behavior's.

• An appropriate model/simulation of the nervous system should be able to

produce similar responses and behavior's in artificial systems.

• Artificial Neural Networks is a function approximator where it transforms

inputs into outputs to the best of its ability.

• Neural Networks comprises of many neurons that co-operate to perform the

desired function.

DEPARTMENT OF INFORMATION TECHNOLOGY 8 11/09/2021

 • New models of computing to perform pattern recognition tasks are inspired
by the structure and performance of our biological neural network.

• You’ve probably already been using neural networks on a daily basis. When you
ask your mobile assistant to perform a search for you—say, Google or Siri or
Amazon Web—or use a self-driving car, these are all neural network-driven.

• Computer games also use neural networks on the back end, as part of the game
system and how it adjusts to the players, and so do map applications, in processing
map images and helping you find the quickest way to get to your destination.

• A neural network is a system or hardware that is designed to operate like a human

brain.

• Neural networks can perform the following tasks:

 Translate text
 Identify faces
 Recognize speech
 Read handwritten text
DEPARTMENT OF INFORMATION TECHNOLOGY 9 11/09/2021
 Control robots
 ANN
• Neural networks are parallel computing devices, which is basically an attempt to
make a computer model of the brain.
• The main objective is to develop a system to perform various computational
tasks faster than the traditional systems.
• These tasks include pattern recognition and classification, approximation,
optimization, and data clustering.

Definition:
An artificial neural network (ANN) is the piece of a computing system designed to
simulate the way the human brain analyzes and processes information.

• It is the foundation of artificial intelligence (AI) and solves problems that would

prove impossible or difficult by human or statistical standards.

• ANNs have self-learning capabilities that enable them to produce better results
as more data becomes available.

DEPARTMENT OF INFORMATION TECHNOLOGY 10 11/09/2021

 • Similar to a human brain has neurons interconnected to each other,
artificial neural networks also have neurons that are linked to each
other in various layers of the networks. These neurons are known as
nodes.

• There are many types of ANN’s

• For example, Convolutional neural network : Good for image

recognition

• Long Short term memory network : Good for speech recognition

DEPARTMENT OF INFORMATION TECHNOLOGY 11 11/09/2021

 Characteristics of Artificial Neural
Networks
• Any Artificial Neural Network, irrespective of the style and
logic of implementation, has a few basic characteristics.
These are mentioned below.

• An Artificial Neural Network consists of large number of

 “neuron” like processing elements.
• All these processing elements have a large number of
weighted connections between them.
• The connections between the elements provide a
distributed representation of data.
• A Learning Process is implemented to acquire knowledge.

DEPARTMENT OF INFORMATION TECHNOLOGY 12 11/09/2021

DEPARTMENT OF INFORMATION TECHNOLOGY 13 11/09/2021
 Some attractive features of the biological neural network that make
it superior to even the most sophisticated computer system for
pattern recognition tasks are the following:
(a) Robustness and fault tolerance:
The decay of nerve cells does not seem to affect the performance significantly.
Extortion of one or more cells of ANN does not prohibit it from generating
output, and this feature makes the network fault-tolerance.
(b) Flexibility: The network automatically adjusts to a new
environment without using any preprogrammed instructions.
(c) Ability to deal with a variety of data situations: The network
can deal with information that is fuzzy, probabilistic, noisy and
inconsistent.
(d)Collective computation: The network performs routinely many
operations in parallel and also a given task in a distributed manner.

Biological Neural Networks

The features of the biological neural network are attributed to its
structure and function.
The fundamental unit of the network is called a neuron or a nerve
cell.
DEPARTMENT OF INFORMATION TECHNOLOGY 14 11/09/2021
   A Biological Neuron
As shown in the above diagram, a typical neuron consists of the following four parts
with the help of which we can explain its working −
Dendrites − They are tree-like branches, responsible for receiving the information
from other neurons it is connected to. In other sense, we can say that they are like
the ears of neuron.
Soma − It is the cell body of the neuron and is responsible for processing of
information, they have received from dendrites.
Axon − It is just like a cable through which neurons send the information.
Synapses − It is the connection between the axon and other neuron dendrites.
DEPARTMENT OF INFORMATION TECHNOLOGY 15 11/09/2021
 Biological Neuron

• A nerve cell, neuron  is a biological cell that processes information.

• According to an estimation, there are huge number of neurons, approximately

1011 with numerous interconnections, approximately 1015.

• Before taking a look at the differences between Artificial Neural Network ( ANN)
and Biological Neural Network  (BNN), let us take a look at the similarities based
on the terminology between these two.

Biological Neural Network BNN Artificial Neural Network ANN

Soma Node
Dendrites Input
Synapse Weights or Interconnections
Axon Output
DEPARTMENT OF INFORMATION TECHNOLOGY 16 11/09/2021
 ANN versus BNN
The following table shows the comparison between ANN and BNN based on some criteria mentioned.
Criteria BNN ANN
Processing Massively parallel, slow but superior Massively parallel, fast but inferior
than ANN than BNN
Size 1011 neurons and 102 to 104 nodes mainly depends on
1015 interconnections the type of application and network
designer mainly depends on the
type of application and network
designer

Learning They can tolerate ambiguity Very precise, structured and

formatted data is required to
tolerate ambiguity

Fault tolerance Performance degrades with even It is capable of robust performance,

partial damage hence has the potential to be fault
tolerant

Storage capacity Stores the information in the Stores the information in

synapse continuous memory locations

DEPARTMENT OF INFORMATION TECHNOLOGY 17 11/09/2021

DEPARTMENT OF INFORMATION TECHNOLOGY 18 11/09/2021
 Artificial Neural Network : Terminology
The following diagram represents the general model of ANN followed by its
processing.

For the above general model of artificial neural network, the net input can be
calculated as follows −
yin=x1.w1+x2.w2+x3.w3…xm.wm
The output can be calculated by applying the activation function over the net
input.

DEPARTMENT OF INFORMATION TECHNOLOGY 19 11/09/2021

 Structure and Working of a biological
neural network
• Artificial Neural Networks are software implementations of the neural structures
of human brain.
• ANN is a computational system influenced from the structure, processing
capability and learning ability of a human brain.

• let us take a look at the structure of our brain. Human brain contains billions of
neurons that act as organic switches.
• All these neurons are interconnected to form a huge and complex structure
called Neural Network.
• The output of a single neuron is dependent on inputs from thousands of
interconnected neurons.

DEPARTMENT OF INFORMATION TECHNOLOGY 20 11/09/2021

 • Processing unit:
We can consider an artificial neural network (ANN) as a highly simplified model of
the structure of the biological neural network.

• An ANN consists of interconnected processing units.

The general model of a processing unit consists of a summing part followed by an
output part.

• The summing part receives N input values, weights each value, and computes a
weighted sum.

• The weighted sum is called the activation value.

• The output part produces a signal from the activation value. The sign of the
weight for each input determines whether the input is excitatory (positive
weight) or inhibitory (negative weight).
DEPARTMENT OF INFORMATION TECHNOLOGY 21 11/09/2021
 • The inputs could be discrete or continuous data values, and
likewise the outputs also could be discrete or continuous.

• The input and output could also be deterministic or

stochastic or fuzzy.

DEPARTMENT OF INFORMATION TECHNOLOGY 22 11/09/2021

 Interconnections:
• In an artificial neural network several processing units are interconnected
according to some topology to accomplish a pattern recognition task.

• Therefore the inputs to a processing unit may come from the outputs of other
processing units, and/or from external sources.

• The output of each unit may be given to several units including itself.

• The amount of the output of one unit received by another unit depends on the
strength of the connection between the units, and it is reflected in the weight
value associated with the connecting link.

• If there are N units in a given ANN, then at any instant of time each unit will have
a unique activation value and a unique output value.
• The set of the N activation values of the network defines the activation state of
the network at that instant.
• Likewise, the set of the N output values of the network defines the output state
of the network at that instant. Depending on the discrete or continuous nature of
the activation and output values, the state of the network can be described by a
discrete or continuous point in an N-dimensional space.
DEPARTMENT OF INFORMATION TECHNOLOGY 23 11/09/2021
 Operations:
 In operation, each unit of an ANN receives inputs from other connected units
and/or from an external source.
 A weighted sum of the inputs is computed at a given instant of time.

 The activation value determines the actual output from the output function
unit, i.e., the output state of the unit.
 The output values and other external inputs in turn determine the activation
and output states of the other units.
 Activation dynamics determines the activation values of all the units, i.e., the
activation state of the network as a function of time.
 The activation dynamics also determines the dynamics of the output state of
the network.
 The set of all activation states defines the activation state space of the
network.
 The set of all output states defines the output state space
of the network. Activation dynamics determines the trajectory of the path of
the states in the state space of the network. For a given network, defined by
the units and their interconnections with appropriate weights, the activation
states determine the short term memory24function of the network.
DEPARTMENT OF INFORMATION TECHNOLOGY 11/09/2021
 • In order to store a pattern in a network, it is necessary to adjust the weights
of the connections in the network. The set of all weights on all connection in
a network form a weight vector.

• The set of all possible weight vectors define the weight space.
• When the weights are changing, then the synaptic dynamics of the network
determines the weight vector as a function of time.
• Synaptic dynamics is followed to adjust the weights in order to store the
given patterns in the network.
• The process of adjusting the weights is referred to as learning.
• Once the learning process is completed, the final set of weight values
corresponds to the long term memory function of the network.

The procedure to incrementally update each of the weights is called a

learning law or learning algorithm.

Update:
In implementation, there are several options available for both activation and
synaptic dynamics. In particular, the updating of the output states of all the
units could be performed synchronously.
DEPARTMENT OF INFORMATION TECHNOLOGY 25 11/09/2021
In this case, the activation values of all the units are computed at
 • From the activation values, the new output state of the network is derived.

• In an asynchronous update, on the other hand, each unit is updated sequentially,

taking the current output state of the network into account each time.

• For each unit, the output state can be determined from the activation value either
deterministically or stochastically.

DEPARTMENT OF INFORMATION TECHNOLOGY 26 11/09/2021

 Models of Neurons
In this section we will consider three classical models for an artificial
neuron or processing unit.

1.McCulloch-Pitts Model
2. Perceptron model
3. Adaline model

1.McCulloch-Pitts Model
In McCulloch-Pitts (MP) model (Figure 1.2) the activation (x) is given
by a weighted sum of its M input values (αi) and a bias term (θ).

DEPARTMENT OF INFORMATION TECHNOLOGY 27 11/09/2021

DEPARTMENT OF INFORMATION TECHNOLOGY 28 11/09/2021
DEPARTMENT OF INFORMATION TECHNOLOGY 29 11/09/2021
 • A single input and a single output MP neuron with proper weight and threshold
gives an output a unit time later.

• For eg: First input is high and second input is low then the output neuron fires
high.
x1 x2 y
0 0 0
0 1 0
1 0 1
1 1 0

• This unit delay property of the MP neuron can be used to build sequential
digital circuits.

• With feedback, it is also possible to have a memory cell (Figure 1.4c) which can
retain the output indefinitely in the absence of any input.

• In the MP model the weights are fixed.

• Hence a network using this model does not have the capability of learning.
Moreover, the original model allows only binary output states, operating at
DEPARTMENT OF INFORMATION TECHNOLOGY 30 11/09/2021
discrete time steps
 2.Perceptron model
• The Rosenblatt's perceptron model (Figure 1.5) for an artificial neuron
consists of outputs from sensory units to a fixed set of association units, the
outputs of which are fed to an MP neuron.
• The association units perform predetermined manipulations on their inputs.
• The main deviation from the MP model is that learning (i.e., adjustment of
weights) is incorporated in the operation of the unit. The desired or target
output (b) is compared with the actual binary output (s), and the error (6) is
used to adjust the weights.
• The following equations describe the operation of the perceptron model of
a neuron:

DEPARTMENT OF INFORMATION TECHNOLOGY 31 11/09/2021

DEPARTMENT OF INFORMATION TECHNOLOGY 32 11/09/2021
 • There is a perceptron learning law which gives a step-by-step procedure
for adjusting the weights.
• Whether the weight adjustment converges or not depends on the nature
of the desired input-output pairs to be represented by the model.

• Boolean AND is linearly separable but Boolean XOR function is not.

DEPARTMENT OF INFORMATION TECHNOLOGY 33 11/09/2021

3. Adaline model
• ADAptive LINear Element (ADALINE) is a computing model proposed by Widrow
and is shown in Figure 1.6 Widrow, 19621.
• The main distinction between the Rosenblatt's perceptron model and the
Widrow's Adaline model is that, in the Adaline the analog activation value (x) is
compared with the target output (b).
• In other words, the output is a linear function of the activation value (x). The
equations that describe the operation of an Adaline are as follows:

DEPARTMENT OF INFORMATION TECHNOLOGY 34 11/09/2021

DEPARTMENT OF INFORMATION TECHNOLOGY 35 11/09/2021
DEPARTMENT OF INFORMATION TECHNOLOGY 36 11/09/2021
DEPARTMENT OF INFORMATION TECHNOLOGY 37 11/09/2021
 TOPOLOGY
 This section presents a few basic structures which will assist in evolving new
architectures.

 The arrangement of the processing units, connections, and pattern

input/output is referred to as topology Simpson, 1901.
 Artificial neural networks are normally organized into layers of processing
units.
 The units of a layer are similar in the sense that they all have the same
activation dynamics and output function.

 Connections can be made either from the units of one layer to the units of
another layer (interlayer connections) or among the units within the layer
(intralayer connections) or both interlayer and intralayer connections.
 Further, the connections across the layers and among the units within a layer
can be organized either in a feed forward manner or in a feedback manner.
 In a feedback network the same processing unit may be visited more than
once.

DEPARTMENT OF INFORMATION TECHNOLOGY 38 11/09/2021

  Any Artificial Neural Network will become useful only when all the processing
elements are organized in an appropriate manner so that they can accomplish the
task of pattern recognition.
 This organization or arrangement of the processing elements, their
interconnections, inputs and outputs is simply known as Topology.

 Generally, in an ANN, the processing units are arranged into layers and all the
units in a particular layer have the same activation values and output values.

 Connection can be made between layers in multiple ways like processing unit of
one layer connected to a unit of another layer, processing unit of a layer
connected to a unit of same layer, etc.

• Some of the commonly used Topologies in Artificial Neural Networks are:

1)Instar
2)Outstar
3)Group of Instars
4)Group of Outstars
5)Bidirectional Associative Memory
6)Autoassociative Memory
DEPARTMENT OF INFORMATION TECHNOLOGY 39 11/09/2021
 TOPOLOGY

DEPARTMENT OF INFORMATION TECHNOLOGY 40 11/09/2021

 Basic Learning Laws
 The operation of a neural network is governed by neuronal dynamics.
 Neuronal dynamics consists of two parts: one corresponding to the dynamics of
the activation state and the other corresponding to the dynamics of the synaptic
weights.
 The Short Term Memory (STM) in neural networks is modelled by the activation
state of the network.
 The Long Term Memory (LTM) corresponds to the encoded pattern information
in the synaptic weights due to learning.
 We will discuss models of neuronal dynamics in Chapter 2.
 In this section we discuss some basic learning laws [Zurada, 1992, Sec. 2.5;
Hassoun, 1995,
 Learning laws are merely implementation models of synaptic dynamics. Typically,
a model of synaptic dynamics is described in terms of expressions for the first
derivative of the weights. They are called learning equations.

DEPARTMENT OF INFORMATION TECHNOLOGY 41 11/09/2021

 Learning Algorithm
• The final and important elements of any ANN are Learning Algorithms or Laws.
The operation any neural network is governed by Neural Dynamics consisting of
both activation state dynamics and synaptic weight dynamics.

• Learning Algorithms or Laws are implementations of synaptic dynamics and are

described in terms of first derivative of the weights. These learning laws can be
supervised, unsupervised or a hybrid of both.

• Some of the commonly known Learning Algorithms are:

 Hebb’s Law
 Perception Learning Law
 Delta Learning Law
 Widrow & Hoff LMS Learning Law
 Correlation Learning Law
 Winner takes all Rule
 outstar Learning Law

DEPARTMENT OF INFORMATION TECHNOLOGY 42 11/09/2021

  Hebbian learning Law – It identifies, how to modify the weights of nodes of a
network.

 Perceptron learning Law – Network starts its learning by assigning a random value
to each weight.

 Delta learning Law – Modification in sympatric weight of a node is equal to the

multiplication of error and the input.

 Correlation learning Law – The correlation rule is the supervised learning.

 Outstar learning rule – We can use it when it assumes that nodes or neurons in a
network arranged in a layer.

 Winner−takes−all Law
It is concerned with unsupervised training in which the output nodes try to
compete with each other to represent the input pattern. 

DEPARTMENT OF INFORMATION TECHNOLOGY 43 11/09/2021

  Widrow & Hoff LMS Learning Law - Is independent of the activation
function of the neuron.

 It is sometimes called Least Mean Square Learning Rule.

DEPARTMENT OF INFORMATION TECHNOLOGY 44 11/09/2021

DEPARTMENT OF INFORMATION TECHNOLOGY 45 11/09/2021