www.rejinpaul.

com

om
OEC552-SOFT COMPUTING (SC)

.c
ul
UNIT-I

pa
Introduction to Soft Computing

Text book
jin
.re
1. J.S.R.Jang, C.T. Sun and E.Mizutani, “Neuro-Fuzzy and Soft Computing”, PHI / Pearson
Education 2004
w

2. S.N. Sivanandam & S.N. Deepa, “Principles of Soft Computing, 2nd Edition”
by Copyright  2011 Wiley India Pvt. Ltd. All rights reserved
w
w
 www.rejinpaul.com
SOFT COMPUTING (SC)

om
• Solutions for a complex problems –obtained by incorporating certain
processes resembling biological and nature inspired phenomena.

.c
• used intelligently where solutions in polynomial time and remain
intractable to conventional mathematical and analytical methods.

ul
Deals with

pa
Imprecision, uncertainty, partial truth and approximation to achieve
practicability, robustness and low solution cost.

jin
SC techniques includes Expert systems, Artificial Neural Networks,
Fuzzy Logic systems and evolutionary computations.
.re
Evolutionary computation techniques include Evolutionary
algorithms, metaheuristics and Swarm intelligent techniques.
w

Swarm intelligent techniques such as ant colony optimisation,

w

Particle swarm optimisation, bees algorithms and cuckoo search.

w
 www.rejinpaul.com
SOFT COMPUTING (SC)

om
Soft Computing (SC):
The symbiotic use of many emerging problem-solving disciplines.

.c
According to Prof. Zadeh:

ul
"...in contrast to traditional hard computing, soft computing exploits
the tolerance for imprecision, uncertainty, and partial truth to

pa
achieve tractability, robustness, low solution-cost, and better rapport
with reality”
jin
Soft Computing Main Components:
• Approximate Reasoning
.re
• Search & Optimization
 Neural Networks, Fuzzy Logic, Evolutionary Algorithms
w
w
w
 www.rejinpaul.com
PROBLEM SOLVING TECHNIQUES

om
HARD COMPUTING SOFT COMPUTING

.c
ul
Precise Models Approximate Models

pa
jin
Traditional Functional
.re
Symbolic Numerical Approximate Approximation
Logic Modeling and Reasoning and Randomized
Reasoning Search Search
w
w
w
 www.rejinpaul.com
Hard computing vs Soft computing

om
Hard Computing Soft computing
Precisely stated analytical model Imprecision is tolerable

.c
required
More Computation time required As it involves intelligent computational steps,

ul
computational time required is less

pa
It involves binary logic crisp It involves nature inspired systems such as neural
systems and numerical analysis networks, fuzzy logic systems and swarm intelligent
system.
Precision is observed within the
computation jinApproximation is obtained in the computation
.re
Imprecision and uncertainity are Tolerance for imprecision and uncertainty is exploited
undesirable properties to achieve tractability, lower cost, high Machine
intelligence quotient and economy of communication.
w

It produces precise answer It can produce approximate answers

w
w
 www.rejinpaul.com
Hard computing vs Soft computing

om
Hard Computing Soft computing

.c
Programs are written which follow Programs are evolved which require new laws and
standard rules of programming theories to be created and justified while
programming

ul
pa
The outcome is deterministic(i.e., The outcome is stochastic or random in nature and
Every trial run, the output is same) need not be deterministic

It requires exact input data

It strictly follows sequential

jin It can deal with ambiguous and noisy input data

It allows parallel computations

.re
computations

It produces precise answer It can produce approximate answers

w
w
w
 www.rejinpaul.com
OVERVIEW OF TECHNIQUES IN SOFT COMPUTING

om
 Neural Networks

.c
Fuzzy Logic

ul


pa
 Genetic Algorithm

 Hybrid Systems

jin
.re
w
w

“
w
 www.rejinpaul.com
NEURAL NETWORKS

om
DARPA Neural Network Study (1988, AFCEA International
Press, p. 60):

.c
... a neural network is a system composed of many simple processing
elements operating in parallel whose function is determined by network

ul
structure, connection strengths, and the processing performed at
computing elements or nodes.

pa
According to Haykin (1994), p. 2:
jin
A neural network is a massively parallel distributed processor that has a
.re
natural propensity for storing experiential knowledge and making it
available for use. It resembles the brain in two respects:
• Knowledge is acquired by the network through a learning process.
w

• Interneuron connection strengths known as synaptic weights are

w

used to store the knowledge

w
 www.rejinpaul.com

om
According to Nigrin (1993), p. 11:

A neural network is a circuit composed of a very large number of

.c
simple processing elements that are neurally based. Each element
operates only on local information.

ul
Furthermore each element operates asynchronously; thus there is no
overall system clock.

pa
According to Zurada (1992):

jin
Artificial neural systems, or neural networks, are physical cellular
.re
systems which can acquire, store and utilize experiential knowledge.
w
w
w
 www.rejinpaul.com
Advantages of Neural Networks

om
 Adaptive learning- -based on data given for training / initial
experience

.c
 Self organization – create own representation of information it

ul
receives during training

pa
 Real time operation - computation carried in parallel. Special
H/W designed and manufactured

 jin
Fault tolerance via redundant information coding - partial
destruction leads to corresponding degradation of performance.
.re
Some network capabilities may be retained even after major
network damage.
w
w
w
 www.rejinpaul.com
MULTIDISCIPLINARY VIEW OF NEURAL NETWORKS

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
Conventional Computing vs Neuro Computing

om
Conventional Computing Neuro Computing
Computational process is Computational process is not sequential and
sequential and deterministic necessarily deterministic

.c
A single processing unit is present- Many simple processing unit-present. They only takes
complex central processor the weighted sum of their inputs from other

ul
processors.
Respond to any programmed Do not respond to programmed instruction. But

pa
instruction respond in parallel such as simulated or actual
responses for the pattern of inputs present to it.
Separate addresses for storing No separate memory addresses for storing data;
data
jin however information is contained in the overall
activation state of the network
.re
Knowledge is centrally located : if Knowledge is distributed, data retrieval may be possible
certain parts of data is lost, if a certain part of data is lost.(similar to the memory
retrieval in not possible retrieval in human brain)
w

Not suited in situations where Well suited to situations where algorithmic solutions
there are no clear cut algorithmic are not possible.
w

solutions
w

Cannot handle noisy imprecise Can manage noisy imprecise data.

data
 www.rejinpaul.com
FUZZY LOGIC

om
 Origins: Multivalued Logic for treatment of imprecision and
vagueness

.c
• 1930s: Post, Kleene, and Lukasiewicz attempted to

ul
represent undetermined, unknown, and other possible

pa
intermediate truth-values.

• 1937: Max Black suggested the use of a consistency profile to

jin
represent vague (ambiguous) concepts.

• 1965: Zadeh proposed a complete theory of fuzzy sets (and its

.re
isomorphic fuzzy logic), to represent and manipulate ill-defined
concepts.
w
w
w
 www.rejinpaul.com
FUZZY LOGIC – LINGUISTIC VARIABLES

om
 Fuzzy logic gives us a language (with syntax and local semantics)
in which we can translate our qualitative domain knowledge.

.c
Linguistic variables to model dynamic systems

ul


pa
 These variables take linguistic values that are characterized by:

• a label - a sentence generated from the syntax

jin
• a meaning - a membership function determined by a local
semantic procedure
.re
w
w
w
 www.rejinpaul.com
FUZZY LOGIC – REASONING METHODS

om
 The meaning of a linguistic variable may be interpreted as an
elastic constraint on its value.

.c
 These constraints are propagated by fuzzy inference operations,

ul
based on the generalized modus-ponens.

pa
 An FL Controller (FLC) applies this reasoning system to a
Knowledge Base (KB) containing the problem domain heuristics.


jin
The inference is the result of interpolating among the outputs of all
.re
relevant rules.

The outcome is a membership distribution on the output space,

w


which is defuzzified to produce a crisp output.
w
w
 www.rejinpaul.com
GENETIC ALGORITHM EVOLUTIONARY PROCESS

om
.c
ul
pa
jin
Definition of GA : The genetic algorithm is a probabalistic search
algorithm that iteratively transforms a set (called a population) of
.re
mathematical objects (typically fixed-length binary character strings),
each with an associated fitness value, into a new population of
w

offspring objects using the Darwinian principle of natural

selection and using operations that are patterned after naturally
w

occurring genetic operations, such as crossover (sexual recombination)

and mutation.
w
 www.rejinpaul.com
STEPS INVOLVED IN GENETIC ALGORITHM

om
The genetic algorithms follow the evolution process in the nature to
find the better solutions of some complicated problems. Foundations

.c
of genetic algorithms are given in Holland (1975) and Goldberg (1989)
books.

ul
Genetic algorithms consist the following steps:

pa
 Initialization
 Selection
 jin
Reproduction with crossover and mutation
.re
Selection and reproduction are repeated for each generation until a
solution is reached.
w

During this procedure a certain strings of symbols, known as

chromosomes, evaluate toward better solution.
w
w
 www.rejinpaul.com
HYBRID SYSTEMS

om
Hybrid systems enables one to combine various soft computing
paradigms and result in a best solution.

.c
The major three hybrid systems are as follows:

ul
 Hybrid Fuzzy Logic (FL) Systems

pa
 Hybrid Neural Network (NN) Systems


jin
Hybrid Evolutionary Algorithm (EA) Systems
.re
w
w
w
 www.rejinpaul.com
SOFT COMPUTING: HYBRID FL SYSTEMS

om
Approximate Reasoning Functional Approximation/ Randomized
Search

.c
Probabilistic Multivalued & Neural Evolutionary

ul
Models Fuzzy Logics Networks Algorithms

pa
Fuzzy Multivalued
Systems Algebras

Fuzzy Logic
Controllers jin
.re
HYBRID FL SYSTEMS

NN modified by FS FLC Tuned by NN FLC Generated

w

(Fuzzy Neural (Neural Fuzzy and Tuned by EA

Systems) Systems)
w
w
 www.rejinpaul.com
SOFT COMPUTING: HYBRID NN SYSTEMS

om
Approximate Reasoning Functional Approximation/ Randomized
Search

.c
Probabilistic Multivalued & Neural Evolutionary

ul
Models Fuzzy Logics Networks Algorithms

pa
Feedforward Recurrent
NN NN

RBF jin
Single/Multiple
Layer Perceptron
Hopfield SOM ART
.re
HYBRID NN SYSTEMS
NN parameters NN topology and/or
(learning rate h
w

weights
momentum a )
controlled by FLC generated by EAs
w
w
 www.rejinpaul.com
SOFT COMPUTING: HYBRID EA SYSTEMS

om
Approximate Reasoning Functional Approximation/ Randomized
Search

.c
ul
Probabilistic Multivalued & Neural Evolutionary
Models Fuzzy Logics Networks Algorithms

pa
Evolution Genetic
Strategies Algorithms

jin Evolutionary
Programs
Genetic
Programs
.re
HYBRID EA SYSTEMS
EA parameters EA-based search EA parameters
w

(N, P cr, Pmu) inter-twined with (Pop size, selection)

controlled by FLC hill-climbing controlled by EA
w
w
 www.rejinpaul.com
PROPERTIES OF SOFT COMPUTING

om
 Learning from experimental data
 Soft computing techniques derive their power of generalization

.c
from approximating or interpolating to produce outputs from
previously unseen inputs by using outputs from previous learned

ul
inputs
 Generalization is usually done in a high dimensional space.

pa
ADVANTAGES OF SOFT COMPUTING

jin
Models based on human reasoning.
.re
 Models can be
• linguistic,
w

• simple (no number crunching),

• comprehensible (no black boxes),
w

• fast when computing,

• good in practice.
w
 www.rejinpaul.com
APPLICATIONS OF SOFT COMPUTING

om
 Handwriting Recognition
 Image Processing and Data Compression

.c
 Automotive Systems and Manufacturing
Soft Computing to Architecture

ul

 Decision-support Systems

pa
 Soft Computing to Power Systems
 Neuro Fuzzy systems
 Fuzzy Logic Control


jin
Machine Learning Applications
Speech and Vision Recognition Systems
.re
 Process Control and So on
w
w
w
 www.rejinpaul.com
SOFT COMPUTING APPLICATIONS: CONTROL

om
.c
• Heavy industry (Matsushita,
Siemens, Stora-Enso).

ul
• Home appliances (Canon,

pa
Sony, Goldstar, Siemens).

jin • Automobiles (Nissan,

Mitsubishi, Daimler-Chrysler,
BMW, Volkswagen).
.re
• Spacecrafts (NASA).
w
w
w
 www.rejinpaul.com
SOFT COMPUTING APPLICATIONS: BUSINESS

om
 sales prognosis for mail order
 supplier evaluation for sample
house,

.c
testing,
 multi-criteria optimization, etc.
 customer targeting,

ul
 sequencing,
 scheduling,

pa
 optimizing R&D,
 projects,
 knowledge-based prognosis,


fuzzy data analysis
jin
hospital stay prediction,
.re
 TV commercial slot evaluation,
 address matching,
 fuzzy cluster analysis,
w
w
w
 www.rejinpaul.com
SOFT COMPUTING APPLICATIONS: FINANCE

om
 fuzzy scoring for mortgage applicants,
 creditworthiness assessment,

.c
 fuzzy-enhanced score card for lease risk assessment,
 risk profile analysis,

ul
 insurance fraud detection,
 cash supply optimization,

pa
 foreign exchange trading,
 insider trading,


trading surveillance,
jin
investor classification, etc.
.re
SOFT COMPUTING APPLICATIONS: OTHERS
w

Statistics, Social sciences, Behavioral sciences, Robotics, Biology, Medicine

w
w
 www.rejinpaul.com
ARTIFICIAL NEURAL NETWORKS: AN INTRODUCTION

om
DEFINITION OF NEURAL NETWORKS

According to the DARPA Neural Network Study (1988, AFCEA

.c
International Press, p. 60):
• ... a neural network is a system composed of many simple processing

ul
elements operating in parallel whose function is determined by network
structure, connection strengths, and the processing performed at

pa
computing elements or nodes.

jin
According to Haykin (1994), p. 2:
A neural network is a massively parallel distributed processor that has a
natural propensity for storing experiential knowledge and making it
.re
available for use. It resembles the brain in two respects:
• Knowledge is acquired by the network through a learning process.
w

• Interneuron connection strengths known as synaptic weights are

used to store the knowledge.
w

“
w
 www.rejinpaul.com
BRAIN COMPUTATION

om
The human brain contains about 10 billion nerve cells, or
neurons. On average, each neuron is connected to other

.c
neurons through approximately 10,000 synapses.

ul
pa
jin
.re
w
w
w
 www.rejinpaul.com

om
INTERCONNECTIONS

.c
IN BRAIN

ul
pa
BIOLOGICAL
jin
.re
(MOTOR) NEURON
w
w
w
 www.rejinpaul.com
ARTIFICIAL NEURAL NET

om
 Information-processing system.
 Neurons process the information.

.c
 The signals are transmitted by means of connection links.
 The links possess an associated weight.

ul
 The output signal is obtained by applying activations to the net
input.

pa
MOTIVATION FOR NEURAL NET
jin
.re
 Scientists are challenged to use machines more effectively for
tasks currently solved by humans.
Symbolic rules don't reflect processes actually used by humans.
w


 Traditional computing excels in many areas, but not in others.
w
w
 www.rejinpaul.com

om
The major areas being:

 Massive parallelism

.c
 Distributed representation and computation

ul
 Learning ability

pa
 Generalization ability

 Adaptivity jin
.re
 Inherent contextual information processing

Fault tolerance
w


w

 Low energy consumption.

w
 www.rejinpaul.com
ARTIFICIAL NEURAL NET

om
W1
Y Simple artificial neural net with two
X1
input neurons (X1, X2) and one

.c
output neuron (Y). The inter

ul
connected weights are given by W1
W2 and W2.

pa
X2

ASSOCIATION OF BIOLOGICAL NET

jin
WITH ARTIFICIAL NET
.re
w
w
w
 www.rejinpaul.com
PROCESSING OF AN ARTIFICIAL NET

om
The neuron is the basic information processing unit of a NN. It consists
of:
1. A set of links, describing the neuron inputs, with weights W1, W2,

.c
…, Wm.
2. An adder function (linear combiner) for computing the weighted

ul
sum of the inputs (real numbers):
3. Activation function for limiting the amplitude of the neuron output.

pa
m
u = ∑ W jX j y = ϕ (u + b)
j =1

jin
BIAS OF AN ARTIFICIAL NEURON x1-x2= -1
.re
The bias value is added to the
x2 x1-x2=0

weighted sum ∑wixi so that we x1-x2= 1

can transform it from the origin.

w

Yin = ∑wixi + b, x1
w

where b is the bias

w
 www.rejinpaul.com
MULTI LAYER ARTIFICIAL NEURAL NET

om
INPUT: records without class attribute with normalized attributes

.c
values.

ul
INPUT VECTOR: X = { x1, x2, …, xn} where n is the number of
(non-class) attributes.

pa
INPUT LAYER: there are as many nodes as non-class attributes, i.e.

jin
as the length of the input vector.
.re
HIDDEN LAYER: the number of nodes in the hidden layer and the
number of hidden layers depends on implementation.
w
w
w
 www.rejinpaul.com

om
OPERATION OF A NEURAL NET

.c
ul
Bias
-

pa
x0 w0j

∑
x1 w1j
jin f Output y
xn wnj
.re
Input Weight Weighted Activation
w

vector x vector sum function

w

w
w
 www.rejinpaul.com
WEIGHT AND BIAS UPDATION

om
Per Sample Updating
• updating weights and biases after the presentation of each sample.

.c
Per Training Set Updating (Epoch or Iteration)
• weight and bias increments could be accumulated in variables and

ul
the weights and biases updated after all the samples of the

pa
training set have been presented.

STOPPING CONDITION

jin
All change in weights (∆wij) in the previous epoch are below some
.re
threshold, or
 The percentage of samples misclassified in the previous epoch is
below some threshold, or
w

 A pre-specified number of epochs has expired.

w

 In practice, several hundreds of thousands of epochs may be

required before the weights will converge.
w
 www.rejinpaul.com
BUILDING BLOCKS OF ARTIFICIAL NEURAL NET
Network Architecture (Connection between Neurons)

om

 Setting the Weights (Training)
 Activation Function

.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
LAYER PROPERTIES

om
 Input Layer: Each input unit may be designated by an attribute
value possessed by the instance.
Hidden Layer: Not directly observable, provides nonlinearities for

.c

the network.

ul
 Output Layer: Encodes possible values.

pa
TRAINING PROCESS
 Supervised Training - Providing the network with a series of

jin
sample inputs and comparing the output with the expected
responses.
.re
 Unsupervised Training - Most similar input vector is assigned to
the same output unit.
 Reinforcement Training - Right answer is not provided but
w

indication of whether ‘right’ or ‘wrong’ is provided.

w
w
 www.rejinpaul.com
ACTIVATION FUNCTION

om
 ACTIVATION LEVEL – DISCRETE OR CONTINUOUS

.c
 HARD LIMIT FUCNTION (DISCRETE)
• Binary Activation function

ul
• Bipolar activation function

pa
• Identity function

 SIGMOIDAL ACTIVATION FUNCTION (CONTINUOUS)

jin
• Binary Sigmoidal activation function
• Bipolar Sigmoidal activation function
.re
w
w
w
 www.rejinpaul.com
ACTIVATION FUNCTION

om
Activation functions:

.c
(A) Identity

ul
(B) Binary step

pa
(C) Bipolar step

jin (D) Binary sigmoidal

.re
(E) Bipolar sigmoidal

(F) Ramp
w
w
w
 www.rejinpaul.com
CONSTRUCTING ANN

om
 Determine the network properties:
• Network topology

.c
• Types of connectivity
• Order of connections

ul
• Weight range

pa
 Determine the node properties:
• Activation range


jin
Determine the system dynamics
.re
• Weight initialization scheme
• Activation – calculating formula
w

• Learning rule
w
w
 www.rejinpaul.com
PROBLEM SOLVING

om
 Select a suitable NN model based on the nature of the problem.
 Construct a NN according to the characteristics of the application

.c
domain.
Train the neural network with the learning procedure of the

ul


selected model.

pa
 Use the trained network for making inference or solving problems.

NEURAL NETWORKS
 jin
Neural Network learns by adjusting the weights so as to be able
to correctly classify the training data and hence, after testing phase,
.re
to classify unknown data.
 Neural Network needs long time for training.
w

 Neural Network has a high tolerance to noisy and incomplete

data.
w
w
 www.rejinpaul.com
SALIENT FEATURES OF ANN

om
 Adaptive learning
 Self-organization
 Real-time operation

.c
 Fault tolerance via redundant information coding

ul
 Massive parallelism
 Learning and generalizing ability

pa
 Distributed representation
FEW APPLICATIONS OF NEURAL NETWORKS
jin
.re
w
w

.
w
 www.rejinpaul.com
INTRODUCTION TO FUZZY LOGIC,
CLASSICAL SETS AND FUZZY SETS

om
FUZZY LOGIC

.c
 Fuzzy logic is the logic underlying approximate, rather than

ul
exact, modes of reasoning.

pa
 It is an extension of multivalued logic: Everything, including
truth, is a matter of degree.
 It contains as special cases not only the classical two-value logic


jin
and multivalue logic systems, but also probabilistic logic.
A proposition p has a truth value
.re
• 0 or 1 in two-value system,
• element of a set T in multivalue system,
w

• Range over the fuzzy subsets of T in fuzzy logic.

w

“Principles of Soft Computing, 2nd Edition”

w

by S.N. Sivanandam & SN Deepa

Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
 www.rejinpaul.com

om
 Boolean logic uses sharp distinctions.
 Fuzzy logic reflects how people think.
 Fuzzy logic is a set of mathematical principles for

.c
ul
pa
jin
.re
 Fuzzy logic is a set of mathematical principles for knowledge
representation and reasoning based on degrees of membership.
w
w

“Principles of Soft Computing, 2nd Edition”

w

by S.N. Sivanandam & SN Deepa

Copyright  2011 Wiley India Pvt. Ltd. All rights reserved.
 www.rejinpaul.com
TYPES AND MODELING OF UNCERTAINTY

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
FUZZY vs PROBABILITY

om
 Fuzzy ≠ Probability
 Probability deals with uncertainty an likelihood
 Fuzzy logic deals with ambiguity an vagueness

.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
NEED OF FUZZY LOGIC

om
 Based on intuition and judgment.
 No need for a mathematical model.
 Provides a smooth transition between members and nonmembers.
Relatively simple, fast and adaptive.

.c

 Less sensitive to system fluctuations.

ul
 Can implement design objectives, difficult to express mathematically, in
linguistic or descriptive rules.

pa
CLASSICAL SETS (CRISP SETS)
Conventional or crisp sets are

jin
Binary. An element either belongs
to the set or does not.
.re
{True, False} {1, 0}
w
w
w
 www.rejinpaul.com
OPERATIONS ON CRISP SETS

om
 UNION:

.c
ul
 INTERSECTION:

pa
 COMPLEMENT:

 DIFFERENCE: jin
.re
w
w
w
 www.rejinpaul.com
PROPERTIES OF CRISP SETS

om
The various properties of crisp sets are as follows:

.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
PROPERTIES OF CRISP SETS

om
.c
ul
pa
jin
.re
w
w
w
FUZZY SETS www.rejinpaul.com

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
OPERATIONS ON FUZZY SETS

om
.c
ul
pa
jin
.re
PROPERTIES OF FUZZY SETS
w
w
w
 www.rejinpaul.com

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
CLASSICAL RELATIONS AND FUZZY

om
RELATIONS
RELATIONS

.c
 Relations represent mappings between sets and connectives in logic.
 A classical binary relation represents the presence or absence of a

ul
connection or interaction or association between the elements of two sets.
 Fuzzy binary relations are a generalization of crisp binary relations, and

pa
they allow various degrees of relationship (association) between elements.

CRISP CARTESIAN PRODUCT

jin
.re
w
w
w
 www.rejinpaul.com
CRISP RELATIONS

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
CRISP BINARY RELATIONS

om
Examples of binary relations

.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
OPERATIONS ON CRISP RELATIONS

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
PROPERTIES OF CRISP RELATIONS
The properties of crisp sets (given below) hold good for crisp relations as well.

om
 Commutativity,
 Associativity,
 Distributivity,

.c
 Involution,
 Idempotency,

ul
 DeMorgan’s Law,
 Excluded Middle Laws.

pa
COMPOSITION ON CRISP RELATIONS

jin
.re
w
w
w
 www.rejinpaul.com
FUZZY CARTESIAN PRODUCT

om
Let R be a fuzzy subset of M and S be a fuzzy subset of N. Then the Cartesian
product R × S is a fuzzy subset of N × M such that

.c
ul
Example:

pa
Let R be a fuzzy subset of {a, b, c} such that R = a/1 + b/0.8 + c/0.2 and S be
a fuzzy subset of {1, 2, 3} such that S = 1/1 + 3/0.8 + 2/0.5. Then R x S is
given by

jin
.re
w
w
w
FUZZY RELATION www.rejinpaul.com

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
OPERATIONS ON FUZZY RELATION

om
The basic operation on fuzzy sets also apply on fuzzy relations.

.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
PROPERTIES OF FUZZY RELATIONS

om
The properties of fuzzy sets (given below) hold good for fuzzy relations as well.
 Commutativity,
 Associativity,

.c
 Distributivity,
 Involution,

ul
 Idempotency,
 DeMorgan’s Law,

pa
 Excluded Middle Laws.

COMPOSITION OF FUZZY RELATIONS

jin
.re
w
w
w
 www.rejinpaul.com

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
CLASSICAL EQUIVALENCE RELATION

om
.c
ul
pa
jin
CLASSICAL TOLERANCE RELATION
.re
w
w
w
 www.rejinpaul.com
FUZZY EQUIVALENCE RELATION

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
FUZZY TOLERANCE RELATION

om
A binary fuzzy relation that possesses the properties of reflexivity and
symmetry is called fuzzy tolerance relation or resemblance relation.

.c
The equivalence relations are a special case of the tolerance relation. The fuzzy

ul
tolerance relation can be reformed into fuzzy equivalence relation in the same
way as a crisp tolerance relation is reformed into crisp equivalence relation,

pa
i.e.,

jin
.re
where ‘n’ is the cardinality of the set that defines R1.
w
w
w
 www.rejinpaul.com
GENETIC ALGORITHM

om
General Introduction to GAs
 Genetic algorithms (GAs) are a technique to solve problems which

.c
need optimization.

ul
 GAs are a subclass of Evolutionary Computing and are random

pa
search algorithms.

GAs are based on Darwin’s theory of evolution.


jin
.re
History of GAs:
w

• Evolutionary computing evolved in the 1960s.

• GAs were created by John Holland in the mid-1970s.
w
w
 www.rejinpaul.com
Biological Background (1) – The Cell

om
 Every cell is a complex of many small “factories”
working together.

.c

 The center of this all is the cell nucleus.

ul
 The nucleus contains the genetic information.

pa
jin
.re
w
w
w
 www.rejinpaul.com
Biological Background (2) – Chromosomes

om
 Genetic information is stored in the chromosomes.

.c
 Each chromosome is build of DNA.

ul
 Chromosomes in humans form pairs.

pa
 There are 23 pairs.

 jin
The chromosome is divided in parts: genes.
.re
 Genes code for properties.
w

 The posibilities of the genes for one property is called: allele.

w

 Every gene has an unique position on the chromosome: locus.

w
 www.rejinpaul.com
Biological Background (3) – Genetics

om
 The entire combination of genes is called genotype.
 A genotype develops into a phenotype.

.c
 Alleles can be either dominant or recessive.
Dominant alleles will always express from the genotype to the

ul

fenotype.

pa
 Recessive alleles can survive in the population for many
generations without being expressed.

jin
.re
w
w
w
 www.rejinpaul.com
Biological Background (4) – Reproduction

om
 Reproduction of genetical
information:

.c
• Mitosis,
Meiosis.

ul
•

pa
 Mitosis is copying the same
genetic information to new
offspring: there is no
jin
exchange of information.
.re
 Mitosis is the normal way of
growing of multicell structures,
w

like organs.
w
w
 www.rejinpaul.com
Biological Background (5) – Reproduction

om
 Meiosis is the basis of sexual
reproduction.

.c
After meiotic division 2 gametes

ul

appear in the process.

pa
 In reproduction two gametes
conjugate to a zygote wich will
jin
become the new individual.
.re
 Hence genetic information is
shared between the parents in
w

order to create new offspring.

w
w
 www.rejinpaul.com
Biological Background (6) – Natural Selection

om
 The origin of species: “Preservation of favorable variations and
rejection of unfavorable variations.”

.c
There are more individuals born than can survive, so there is a

ul

continuous struggle for life.

pa
 Individuals with an advantage have a greater chance for survive:
survival of the fittest. For example, Giraffes with long necks.


jin
Genetic variations due to crossover and mutation.
.re
w
w
w
 www.rejinpaul.com
Genetic Algorithm (1) – Search Space

om
 Most often one is looking for the
best solution in a specific subset

.c
of solutions. 2.5

This subset is called the search

ul

space (or state space). 2

pa
 Every point in the search space is 1.5

a possible solution.
 Therefore every point has a
jin
1

fitness value, depending on the 0.5

problem definition.
.re
 GAs are used to search the search
0
0 100 200 300 400 500 600 700 800 900 1000

space for the best solution, e.g. a

w

minimum.
 Difficulties are the local minima
w

and the starting point of the

w

search.
 www.rejinpaul.com
Genetic Algorithm (2) – Basic Algorithm

om
 Starting with a subset of n randomly chosen solutions from the
search space (i.e. chromosomes).
This is the population.

.c
 This population is used to produce a next generation of
individuals by reproduction.

ul
 Individuals with a higher fitness have more chance to reproduce
(i.e. natural selection).

pa
Comparison of Natural and GA Terminology
Natural
Chromosome
jin Genetic Algorithm
String
.re
Gene Feature or character
Allele Feature value
Locus String position
w

Genotype Structure
w

Phenotype Parameter set, a decoded structure

w
 www.rejinpaul.com
UNIT-III

om
ARTIFICIAL NEURAL NETWORKS

.c
McCULLOCH–PITTS NEURON

ul
 Neurons are sparsely and randomly connected

pa
 Firing state is binary (1 = firing, 0 = not firing)


cells)
jin
All but one neuron are excitatory (tend to increase voltage of other
.re
• One inhibitory neuron connects to all other neurons
w

• It functions to regulate network activity (prevent too many

firings)
w
w
 www.rejinpaul.com
LINEAR SEPARABILITY

om
 Linear separability is the concept wherein the separation of the
input space into regions is based on whether the network response

.c
is positive or negative.

ul
 Consider a network having

pa
positive response in the first
quadrant and negative response
in all other quadrants (AND
jin
function) with either binary or
bipolar data, then the decision
.re
line is drawn separating the
positive response region from
w

the negative response region.

w
w
 www.rejinpaul.com
HEBB NETWORK

om
Donald Hebb stated in 1949 that in the brain, the learning is performed
by the change in the synaptic gap. Hebb explained it:

.c
“When an axon of cell A is near enough to excite cell B, and repeatedly
or permanently takes place in firing it, some growth process or

ul
metabolic change takes place in one or both the cells such that A’s
efficiency, as one of the cells firing B, is increased.”

pa
HEBB LEARNING

jin
The weights between neurons whose activities are positively
correlated are increased:
dw ij
.re
~ correlation ( x i , x j )
dt

 Associative memory is produced automatically

w

 The Hebb rule can be used for pattern association, pattern

w

categorization, pattern classification and over a range of other

areas.
w
 www.rejinpaul.com
SUPERVISED LEARNING NETWORKS

om
 Training and test data sets

.c
 Training set; input & target are specified

ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
PERCEPTRON NETWORKS

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com

om
 Linear threshold unit (LTU)

.c
ul
x1 w1
w0

pa
w2
x2 Σ o
. n

.
. wn
jin
Σ
i=0
wi xi
.re
n
xn 1 if Σ wi xi >0
f(xi)= { i=0
-1 otherwise
w
w
w
 www.rejinpaul.com
PERCEPTRON LEARNING

om
wi = wi + ∆wi
∆wi = η (t - o) xi

.c
where
t = c(x) is the target value,

ul
o is the perceptron output,

pa
η Is a small constant (e.g., 0.1) called learning rate.

 If the output is correct (t = o) the weights wi are not changed


jin
If the output is incorrect (t ≠ o) the weights wi are changed such
.re
that the output of the perceptron for the new weights is closer to t.

The algorithm converges to the correct classification

w


• if the training data is linearly separable
w

• η is sufficiently small
w
 www.rejinpaul.com
LEARNING ALGORITHM

om
 Epoch : Presentation of the entire training set to the neural
network.

.c
In the case of the AND function, an epoch consists of four sets of

ul

inputs being presented to the network (i.e. [0,0], [0,1], [1,0],

pa
[1,1]).

 Error: The error value is the amount by which the value output by
jin
the network differs from the target value. For example, if we
required the network to output 0 and it outputs 1, then Error = -1.
.re
 Target Value, T : When we are training a network we not only
w

present it with the input but also with a value that we require the
network to produce. For example, if we present the network with
w

[1,1] for the AND function, the training value will be 1.

w

www.rejinpaul.com
Output , O : The output value from the neuron.

om
 Ij : Inputs being presented to the neuron.

 Wj : Weight from input neuron (Ij) to the output neuron.

.c
ul
 LR : The learning rate. This dictates how quickly the network
converges. It is set by a matter of experimentation. It is typically

pa
0.1.

TRAINING ALGORITHM

jin
Adjust neural network weights to map inputs to outputs.
.re
 Use a set of sample patterns where the desired output (given the
inputs presented) is known.
 The purpose is to learn to
w

• Recognize features which are common to good and bad

w

exemplars
w
 www.rejinpaul.com
MULTILAYER PERCEPTRON

om
Output Values

.c
ul
Output Layer

pa
Adjustable
jin Weights
.re
Input Layer
w

Input Signals (External Stimuli)

w
w
 www.rejinpaul.com
LAYERS IN NEURAL NETWORK

om
 The input layer:
• Introduces input values into the network.

.c
• No activation function or other processing.

ul
 The hidden layer(s):

pa
• Performs classification of features.
• Two hidden layers are sufficient to solve any problem.
• Features imply more layers may be better.

 The output layer:

jin
.re
• Functionally is just like the hidden layers.
• Outputs are passed on to the world outside the neural
w

network.
w
w
 www.rejinpaul.com
ADAPTIVE LINEAR NEURON (ADALINE)

om
In 1959, Bernard Widrow and Marcian Hoff of Stanford developed
models they called ADALINE (Adaptive Linear Neuron) and MADALINE
(Multilayer ADALINE). These models were named for their use of

.c
Multiple ADAptive LINear Elements. MADALINE was the first neural
network to be applied to a real world problem. It is an adaptive filter

ul
which eliminates echoes on phone lines. ADALINE MODEL

pa
jin
.re
w
w
w
 www.rejinpaul.com
ADALINE LEARNING RULE

om
Adaline network uses Delta Learning Rule. This rule is also called as
Widrow Learning Rule or Least Mean Square Rule. The delta rule for

.c
adjusting the weights is given as (i = 1 to n):

ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
USING ADALINE NETWORKS

om
 Initialize
Initialize • Assign random weights to all links

.c
 Training

ul
• Feed-in known inputs in random sequence
• Simulate the network

pa
• Compute error between the input and the
Training output (Error Function)
• Adjust weights (Learning Function)

jin
• Repeat until total error < ε

 Thinking
.re
Thinking • Simulate the network
• Network will respond to any input
• Does not guarantee a correct solution even
w

for trained inputs

w
w
 www.rejinpaul.com
MADALINE NETWORK

om
MADALINE is a Multilayer Adaptive Linear Element. MADALINE was the
first neural network to be applied to a real world problem. It is used in

.c
several adaptive filtering process.

ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
BACK PROPAGATION NETWORK

om
.c
ul
pa

jin
A training procedure which allows multilayer feed forward Neural
.re
Networks to be trained.
 Can theoretically perform “any” input-output mapping.
 Can learn to solve linearly inseparable problems.
w
w
w
 www.rejinpaul.com
MULTILAYER FEEDFORWARD NETWORK

om
Inputs

.c
Hiddens

ul
I0
Outputs

pa
h0
I1 o0

jin h1
.re
I2 o1
h2 Outputs
w

I3 Hiddens
w

Inputs
w
 www.rejinpaul.com
MULTILAYER FEEDFORWARD NETWORK:

om
ACTIVATION AND TRAINING
 For feed forward networks:

.c
• A continuous function can be

ul
• differentiated allowing
• gradient-descent.

pa
• Back propagation is an example of a gradient-descent technique.
• Uses sigmoid (binary or bipolar) activation function.

In multilayer
jin
networks, the activation function is
.re
usually more complex than just a threshold function,
like 1/[1+exp(-x)] or even 2/[1+exp(-x)] – 1 to allow for
w

inhibition, etc.
w
w
 www.rejinpaul.com
GRADIENT DESCENT

om
 Gradient-Descent(training_examples, η)
 Each training example is a pair of the form <(x1,…xn),t> where

.c
(x1,…,xn) is the vector of input values, and t is the target output

ul
value, η is the learning rate (e.g. 0.1)
 Initialize each wi to some small random value

pa
 Until the termination condition is met, Do
• Initialize each ∆wi to zero

jin
• For each <(x1,…xn),t> in training_examples Do
 Input the instance (x1,…,xn) to the linear unit and compute the
.re
output o
 For each linear unit weight wi Do
w

• ∆wi= ∆wi + η (t-o) xi

• For each linear unit weight wi Do
w

• wi=wi+∆wi
w
 www.rejinpaul.com
MODES OF GRADIENT DESCENT

om
 Batch mode : gradient descent
w=w - η ∇ED[w] over the entire data D

.c
ED[w]=1/2Σd(td-od)2

ul
 Incremental mode: gradient descent

pa
w=w - η ∇Ed[w] over individual training examples d
Ed[w]=1/2 (td-od)2

 jin
Incremental Gradient Descent can approximate Batch Gradient
Descent arbitrarily closely if η is small enough.
.re
w
w
w
 www.rejinpaul.com
SIGMOID ACTIVATION FUNCTION

om
x0=1
x1 w1
w0 net=Σi=0n wi xi o=σ(net)=1/(1+e-net)

.c
w2
x2 Σ o

ul
.
.

pa
. wn σ(x) is the sigmoid function: 1/(1+e-x)
xn
jin
dσ(x)/dx= σ(x) (1- σ(x))
.re
Derive gradient decent rules to train:
• one sigmoid function
w

∂E/∂wi = -Σd(td-od) od (1-od) xi

• Multilayer networks of sigmoid units backpropagation
w
w
 www.rejinpaul.com
BACKPROPAGATION TRAINING ALGORITHM

om
 Initialize each wi to some small random value.

.c
 Until the termination condition is met, Do

ul
• For each training example <(x1,…xn),t> Do

pa
• Input the instance (x1,…,xn) to the network and compute the
network outputs ok
• For each output unit k
jin
– δk=ok(1-ok)(tk-ok)
• For each hidden unit h
.re
– δh=oh(1-oh) Σk wh,k δk
• For each network weight w,j Do
w

• wi,j=wi,j+∆wi,j where
– ∆wi,j= η δj xi,j
w
w
BACKPROPAGATION www.rejinpaul.com

om
 Gradient descent over entire network weight vector
 Easily generalized to arbitrary directed graphs
 Will find a local, not necessarily global error minimum -in practice
often works well (can be invoked multiple times with different

.c
initial weights)
Often include weight momentum term

ul

∆wi,j(t)= η δj xi,j + α ∆wi,j (t-1)

pa
 Minimizes error training examples
 Will it generalize well to unseen instances (over-fitting)?
 Training can be slow typical 1000-10000 iterations (use Levenberg-

APPLICATIONS 
jin
Marquardt instead of gradient descent)
Load forecasting problems in power systems.
.re
 Image processing.
 Fault diagnosis and fault detection.
w

 Gesture recognition, speech recognition.

Signature verification.
w


 Bioinformatics.
w

 Structural engineering design (civil).

 www.rejinpaul.com
RADIAL BASIS FUNCTION NETWORK

om
 The radial basis function (RBF) is a classification and functional
approximation neural network developed by M.J.D. Powell.

.c
 The network uses the most common nonlinearities such as

ul
sigmoidal and Gaussian kernel functions.
 The Gaussian functions are also used in regularization networks.

pa
 The Gaussian function is generally defined as

jin
.re
w
w
w
 www.rejinpaul.com
RADIAL BASIS FUNCTION NETWORK

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com

om
Discussed on the several supervised learning networks like

.c
 Perceptron,

ul
 Adaline,
 Madaline,

pa
 Backpropagation Network,
 Radial Basis Function Network.

jin
Apart from these mentioned above, there are several other supervised
.re
neural networks like tree neural networks, wavelet neural network,
functional link neural network and so on.
w
w
w
 www.rejinpaul.com
ASSOCIATIVE MEMORY NETWORKS

om
PATTERN ASSOCIATION

.c
 Associating patterns which are
• similar,

ul
• contrary,

pa
• in close proximity (spatial),
• in close succession (temporal).

 Associative recalljin
• evoke associated patterns,
.re
• recall a pattern by part of it,
• evoke/recall with incomplete/noisy patterns.
w
w
w
 www.rejinpaul.com
ASSOCIATIVE MEMORY (AM) NETWORK

om
 Two types of associations exist. For two patterns s and t

.c
• hetero-association (s != t): relating two different patterns (s –

ul
input, t – target).
• auto-association (s = t): relating parts of a pattern with other

pa
parts.


jin
Architectures of NN associative memory:
• single layer (with/out input layer),
.re
• two layers (for bidirectional association)

 Learning algorithms for AM:

w

• Hebbian learning rule and its variations,

w

• gradient descent.
w
 www.rejinpaul.com
ASSOCIATIVE MEMORY NETWORK

om
 WORKING PROCESS

.c
• Recall a stored pattern by a noisy input pattern.

ul
• Using the weights that capture the association.

pa
• Stored patterns are viewed as “attractors”, each has its
“attraction basin”.

•
jin
Often call this type of NN “associative memory” (recall by
.re
association, not explicit indexing/addressing).
w
w
w
 www.rejinpaul.com
TRAINING ALGORITHM FOR ASSOCIATIVE

om
MEMORY NETWORK
 Network structure: single layer

.c
• one output layer of non-linear units and one input layer.

ul
s_1 x_1 w_1 y_1 t_1

pa
1
w_1m
w_n

 Goal of learning: jin

s_n x_n 1
w_n
m
y_m t_m
.re
• to obtain a set of weights w_ij from a set of training pattern
pairs {s:t} such that when s is applied to the input layer, t is
w

computed at the output layer,

w

• for all training pairs s:t, tj = f(sTwj) for all j.

w
 www.rejinpaul.com
HEBB RULE FOR PATTERN ASSOCIATION

om
 Algorithm (bipolar or binary patterns):

.c
• For each training samples s:t: ∆wij = si ⋅ t j

ul
• ∆wij increases if both si and t j are ON (binary) or have the same
sign (bipolar).

pa
If ∆wij = 0, then, after updates for all P training patterns,
P
wij = ∑ si ( p )t j ( p ),
P =1
jin W = { wij }
.re
• Instead of obtaining W by iterative updates, it can be
computed from the training set by calculating the outer
w

product of s and t.
w
w
 www.rejinpaul.com
OUTER PRODUCT FOR PATTERN ASSOCIATION

om
Let s and t be row vectors.

.c
Then for a particular training pair s:t

ul
 s1   s1t1...s1t m  ∆w11...∆w1m 
   s t ...s t   

pa
∆W ( p ) = s ( p ) ⋅ t ( p ) =   [t1 ,..., t m ] = 
T 2 1 2 m = 
     
     
 n  n 1 n m   n1 ∆ ∆ nm 

and
jin s s t ...s t w ... w
.re
P
W ( p) = ∑ sT ( p) ⋅ t ( p)
p =1
w
w
w
 www.rejinpaul.com
HETERO-ASSOCIATIVE MEMORY NETWORK

om
• Binary pattern pairs s:t with |s| = 4 and |t| = 2.

.c
• Total weighted input to output units: y _ in j = ∑ x i w ij
y _ in j > 0
i
Activation function: threshold 1 if

ul
• yj = 
0 if y _ in j ≤ 0

pa
• Weights are computed by Hebbian rule (sum of outer products
of all training pairs) P
W = ∑ s i ( p) t j ( p)
• Training samples:
jin
s(p)
p =1
T

t(p)
.re
p=1 (1 0 0 0) (1, 0)
p=2 (1 1 0 0) (1, 0)
w

p=3 (0 0 0 1) (0, 1)
w

p=4 (0 0 1 1) (0, 1)
w
 www.rejinpaul.com
COMPUTING THE WEIGHTS

om
1  1 0 1  1 0
       

.c
0 0 0 1  1 0
s T (1) ⋅ t (1) =  (1 0 ) =  s T (2) ⋅ t (2) =  (1 0 ) = 
0 0 0 0 0 0
       

ul
0
 
0
 0  0
 
0
 0 

pa
0 0 0  0 0 0
       
0 0 0  0 0 0
s T (3) ⋅ t (3) =  (0 1) =  s T (4) ⋅ t (4) =  (0 1) = 
0 1
0
 
 
1  jin

0
0

1 
1
 
1 
 

0

0

1 
.re
2 0
 
1 0
W =
0 1
w

 
0
 2

w
w
 www.rejinpaul.com
TEST/ RECALL THE NETWORK

om
x = [1 0 0 0] x = [0 1 1 0]
2 0 2 0
 

.c
  0
(0 1 1 0)
1
(1 0 0 0)
1 0
= (2 0 ) = (1 1)
 0 1

ul

0 1
  
0 2 
0
 2 


pa
y1 = 1, y 2 = 0 y1 = 1, y 2 = 1

x = [0 1 0 0] similar to s(1) and s(2) (1 0 0 0), (1 1 0 0) class (1, 0)

2

(0 1 0 0)
1
0
0
 jin
= (1 0 )
(0 0 0 1), (0 0 1 1) class (0, 1)
(0 1 1 0) is not sufficiently similar
.re
0 1 
  to any class
0
 2 
w

y1 = 1, y 2 = 0
w
w
 www.rejinpaul.com
AUTO-ASSOCIATIVE MEMORY NETWORK

om
• Same as hetero-associative nets, except t(p) =s (p).
• Used to recall a pattern by a its noisy or incomplete version.

.c
(pattern completion/pattern recovery)

ul
• A single pattern s = (1, 1, 1, -1) is stored (weights computed
by Hebbian rule or outer product rule.

pa
1 1 1 − 1
1 1 − 1
jin
W =
1
− 1
1
1
−1
1 − 1

− 1 1 
.re
training pattern (111 − 1) ⋅W = (4 4 4 − 4) → (111 − 1)
noisy pattern (− 111 − 1) ⋅W = (2 2 2 − 2) → (111 − 1)
w

missing info (0 0 1 − 1) ⋅W = (2 2 2 − 2) → (111 − 1)

(− 1 − 11 − 1) ⋅W = (0 0 0 0) not recognized
w

more noisy
w
 www.rejinpaul.com
AUTO-ASSOCIATIVE MEMORY NETWORK –

om
DIAGONAL ELEMENTS
• Diagonal elements will dominate the computation when

.c
multiple patterns are stored (= P).
• When P is large, W is close to an identity matrix. This causes

ul
output = input, which may not be any stoned pattern. The

pa
pattern correction power is lost.
• Replace diagonal elements by zero.

jin 0 1 1 − 1
1 0 1 − 1
W0 = 
−

.re
 1 1 0 1 
− 1 − 1 − 1 0 
(1 1 1 − 1)W ' = (3 3 3 − 3) → (1 1 1 − 1)
w

(−1 1 1 − 1)W ' = (3 1 1 − 1) → (1 1 1 − 1)

(0 0 1 − 1)W ' = (2 2 1 − 1) → (1 1 1 − 1)
w

(−1 − 1 1 − 1)W ' = (1 1 − 1 1) → wrong

w
 www.rejinpaul.com
STORAGE CAPACITY

om
• Number of patterns that can be correctly stored & recalled by a
network.

.c
• More patterns can be stored if they are not similar to each
other (e.g., orthogonal).

ul
• Non-orthogonal
0 0 −2 2

pa
0 0 0 0 
(1 − 1 − 1 1) → W0 =   (1 − 1 − 11) ⋅ W0 = (1 0 − 1 1)
(1 1 − 1 1) − 2 0 0 − 2
It is not stored correctly
2 0 −2 0 

• Orthogonal
jin
.re
0 −1 −1 − 1
(1 1 − 1 − 1) − 1 0 −1 − 1
(−1 1 1 − 1) → W0 =  
w

(−1 1 − 1 1) − 1 −1 0 − 1 All three patterns can be

− 1 −1 −1 0 correctly recalled
w
w
 www.rejinpaul.com
BIDIRECTIONAL ASSOCIATIVE MEMORY (BAM)

om
NETWORK
Architecture:

.c
• Two layers of non-linear units: X-layer, Y-layer.
• Units: discrete threshold, continuing sigmoid (can be either

ul
binary or bipolar).

pa
Weights:

P
Wn×m = ∑ s T ( p ) ⋅ t ( p )
p =1
jin
(Hebbian/outerproduct)
.re
Symmetric: w ij = w ji
Convert binary patterns to bipolar when constructing
W.
w
w
w
 www.rejinpaul.com
RECALL OF BAM NETWORK

om
Bidirectional, either by X (to recall Y) or by Y (to recall X).
Recurrent:

.c
y (t ) = [ f ( y _ in1 (t ),..., f ( y _ inm (t )]

ul
n
where y _ in j (t ) = ∑ wi j ⋅ xi (t − 1)

pa
i =1

x(t + 1) = [ f ( x _ in1 (t + 1),..., f ( x _ inn (t + 1)]

m
where x _ in i (t + 1) = ∑ wij ⋅ y j (t )

jin j =1

Update can be either asynchronous (as in hetero-associative memory)

.re
or synchronous (change all Y units at one time, then all X units the
next time).
w
w
w
 www.rejinpaul.com
ACTIVATION FUNCTIONS IN BAM

om
The activation function is based on whether the input target vector
pairs used are binary or bipolar.

.c
ul
Activation function for the Y- Activation function for the X-
layer layer

pa
With binary input vectors With binary input vectors

jin
.re
With bipolar input vectors With bipolar input vectors
w
w
w
 www.rejinpaul.com
DISCRETE HOPFIELD NETWORK (DHN)

om
 A single-layer network
• each node as both input and output units.

.c
More than an Associative Memory, Discrete Hopfield Network can

ul

be used in several applications.

pa
• Other applications such as combinatorial optimization.

 Different forms: discrete & continuous.


jin
Major contribution of John Hopfield to NN:
.re
• Treating a network as a dynamic system.
• Introduction of energy function into Neural Network Research.
w
w
w
 www.rejinpaul.com
ARCHITECTURE OF DHN

om
 Architecture

.c
• Single-layer (units serve as both input and output):
 nodes are threshold units (binary or bipolar).

ul
 weights: fully connected, symmetric, and zero diagonal.

pa
w ij = w ji
w ii = 0

jin
xi are external inputs, which
.re
may be transient or
permanent.
w
w
w
 www.rejinpaul.com

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
STORAGE CAPACITY OF DHN

om
P : maximum number of random patterns of dimension n can be stored
in a DHM of n nodes

.c
ul
P
Hopfield’s observation: P ≈ 0.15n, ≈ 0.15
n

pa
n P 1
Theoretical analysis: P≈ , ≈
2 log 2 n n 2 log 2 n

jin
P / n decreases because larger n leads to more interference between
stored patterns.
.re
Some work to modify HM to increase its capacity to close to n, W is
w

trained (not computed by Hebbian rule).

w
w
 www.rejinpaul.com
CONTINUOUS HOPFIELD NET

om
 Architecture

.c
• Continuous node output, and continuous time

ul
• Fully connected with symmetric weights wij = w ji , wii = 0

pa
dui (t ) n
• Internal activation ui : with = ∑ wij x j (t ) + θ i = neti (t )

•
jin
Output (state) xi (t ) = f (ui (t ))
dt j =1
.re
where f is a sigmoid function to ensure binary/bipolar output. E.g. for
bipolar, use hyperbolic tangent function:
w

e x − e− x
w

f ( x) = tanh( x) = x
e + e−x
w
 www.rejinpaul.com
CONTINUOUS HOPFIELD NET

om
Computation: all units change their output (states) at the same time,

.c
based on states of all others.

ul
n
• Compute net: neti (t ) = ∑ wij x j (t ) + θi

pa
j =1

• Compute internal activationui (t ) by first-order Taylor expansion

jin
ui (t + δ ) = ∫ ini (t )dt ≈ ui (t ) +
dui (t )
dt
⋅ δ + ... = ui (t ) + neti ⋅ δ
.re
• Compute output
xi (t ) = f (ui (t ))
w
w
w
 www.rejinpaul.com
ITERATIVE ASSOCIATIVE MEMORY NETWORK

om
Example

.c
 0 1 1 −1 
 1 0 1 − 1 Output units are
x = (1, 1, 1, − 1) W = 
1 1 0 −1 

ul
  threshold units
 − 1 − 1 − 1 0 

pa
An incomplete recall input : x' = (1, 0, 0, 0)
Wx' = (0, 1, 1, − 1) = x"
Wx" = (3, 2, 2, − 2) → (1, 1, 1, − 1) = x

jin
In general: using current output as input of the next iteration
x(0) = initial recall input
.re
x(I) = S(Wx(I-1)), I = 1, 2, ……
until x(N) = x(K) for some K < N
w
w
w
 www.rejinpaul.com

om
Dynamic System: State vector x(I)
If K = N-1, x(N) is a stable state (fixed point)

.c
f(Wx(N)) = f(Wx(N-1)) = x(N)

ul
If x(K) is one of the stored pattern, then x(K) is called a genuine

pa
memory

jin
Otherwise, x(K) is a spurious memory
talk/interference between genuine memories)
(caused by cross-
.re
Each fixed point (genuine or spurious memory) is an attractor (with
different attraction basin)
w
w
w
 www.rejinpaul.com

om
If K != N-1, limit-circle,

The network will repeat

.c
ul
x(K), x(K+1), …, x(N) = x(K) when iteration continues.

pa
Iteration will eventually stop because the total number of distinct state
is finite (3^n) if threshold units are used. If patterns are continuous,

jin
the system may continue evolve forever (chaos ) if no such K exists.
.re
w
w
w
 www.rejinpaul.com
UNSUPERVISED LEARNING NETWORKS

om
 No help from the outside.
 No training data, no information available on the desired output.

.c
 Learning by doing.
 Used to pick out structure in the input:

ul
• Clustering,

pa
• Reduction of dimensionality  compression.
 Example: Kohonen’s Learning Law.

jin
There exists several networks under this category, such as
 Max Net,
.re
 Mexican Hat,
 Kohonen Self-organizing Feature Maps,
w

 Learning Vector Quantization,

 Counterpropagation Networks,
w

 Hamming Network,
w

 Adaptive Resonance Theory.

 www.rejinpaul.com
COMPETITIVE LEARNING

om
 Output units compete, so that eventually only one neuron (the one
with the most input) is active in response to each output pattern.

.c
The total weight from the input layer to each output neuron is

ul

limited. If some connections are strengthened, others must be

pa
weakened.

 A consequence is that the winner is the output neuron whose

jin
weights best match the activation pattern.
.re
w
w
w
 www.rejinpaul.com
MAX NET

om
 Max Net is a fixed weight
competitive net.

.c
Max Net serves as a subnet for

ul

picking the node whose input is

pa
larger. All the nodes present in this
subnet are fully interconnected and
there exist symmetrical weights in
jin
all these weighted interconnections.
.re
 The weights between the neurons
are inhibitory and fixed.
w

 The architecture of this net is as

w

shown:
w
 www.rejinpaul.com
MEXICAN HAT NETWORK

om
 Kohonen developed the Mexican hat network which is a more
generalized contrast enhancement network compared to the earlier

.c
Max Net.
Here, in addition to the connections within a particular layer of

ul

neural net, the neurons also receive some other external signals.

pa
This interconnection pattern is repeated for several other neurons in
the layer.The architecture for the network is as shown below:

jin
.re
w
w
w
 www.rejinpaul.com
MEXICAN HAT NETWORK

om
 The lateral connections are used to create a competition between
neurons. The neuron with the largest activation level among all
neurons in the output layer becomes the winner. This neuron is the

.c
only neuron that produces an output signal. The activity of all other
neurons is suppressed in the competition.

ul
pa
 The lateral feedback connections produce excitatory or inhibitory
effects, depending on the distance from the winning neuron. This is
achieved by the use of a M ex ican Hat function which describes
jin
synaptic weights between neurons in the output layer.
.re
MEXICAN HAT FUNCTION OF
LATERAL CONNECTION
w
w
w
 www.rejinpaul.com
HAMMING NETWORK

om
 The Hamming network selects stored classes, which are at a
maximum Hamming distance (H) from the noisy vector presented

.c
at the input.

ul
 The Hamming distance between the two vectors is the number of

pa
components in which the vectors differ.

 The Hamming network consists of two layers.

jin
• The first layer computes the difference between the total
number of components and Hamming distance between the
.re
input vector x and the stored pattern of vectors in the feed
forward path.
w

• The second layer of the Hamming network is composed of

Max Net (used as a subnet) or a winner-take-all network
w

which is a recurrent network.

w
 www.rejinpaul.com
ARCHITECTURE OF HAMMING NET

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
SELF-ORGANIZATION

om
 Network Organization is fundamental to the brain
• Functional structure.

.c
• Layered structure.
• Both parallel processing and serial processing require

ul
organization of the brain.

pa
SELF-ORGANIZING FEATURE MAP
jin
Our brain is dominated by the cerebral cortex, a very complex structure
of billions of neurons and hundreds of billions of synapses. The cortex
.re
includes areas that are responsible for different human activities
(motor, visual, auditory, etc.) and associated with different sensory
w

inputs. One can say that each sensory input is mapped into a
corresponding area of the cerebral cortex. The cortex is a self-
w

organizing com putational m ap in the hum an brain.

w
 www.rejinpaul.com
SELF-ORGANIZING NETWORKS

om
 Discover significant patterns or features in the input data.
 Discovery is done without a teacher.
 Synaptic weights are changed according to local rules.

.c
 The changes affect a neuron’s immediate environment until a final
configuration develops.

ul
pa
KOHONEN SELF-ORGANIZING FEATURE MAP (KSOFM)
 The Kohonen model provides a topological mapping.
It places a fixed number of input patterns from the input layer



jin
into a higher dimensional output or Kohonen layer.
Training in the Kohonen network begins with the winner’s
.re
neighborhood of a fairly large size. Then, as training proceeds,
the neighborhood size gradually decreases.
w

 Kohonen SOMs result from the synergy of three basic processes

• Competition,
w

• Cooperation,
w

• Adaptation.
 www.rejinpaul.com
ARCHITECTURE OF KSOFM

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
COMPETITION OF KSOFM

om
 Each neuron in an SOM is
assigned a weight vector with the

.c
same dimensionality N as the
input space.

ul
pa
 Any given input pattern is
compared to the weight vector of
each neuron and the closest
jin
neuron is declared the winner.
.re
 The Euclidean norm is commonly
used to measure distance.
w
w
w
 www.rejinpaul.com
CO-OPERATION OF KSOFM

om
 The activation of the winning neuron is spread to neurons in its
immediate neighborhood.

.c
• This allows topologically close neurons to become sensitive to
similar patterns.

ul
pa
 The winner’s neighborhood is determined on the lattice topology.
• Distance in the lattice is a function of the number of lateral
connections to the winner.


jin
The size of the neighborhood is initially large, but shrinks over
.re
time.
• An initially large neighborhood promotes a topology-preserving
w

mapping.
• Smaller neighborhoods allow neurons to specialize in the latter
w

stages of training.
w
 www.rejinpaul.com
ADAPTATION OF KSOFM

om
During training, the winner neuron and
its topological neighbors are adapted to

.c
make their weight vectors more similar
to the input pattern that caused the

ul
activation.

pa
Neurons that are closer to the winner
will adapt more heavily than neurons
that are further away.
jin
The magnitude of the adaptation is
.re
controlled with a learning rate, which
decays over time to ensure convergence
w

of the SOM.
w
w
 www.rejinpaul.com
KSOFM ALGORITHM

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
EXAMPLE OF KSOFM

om
.c
ul
pa
jin
Find the winning neuron using the Euclidean distance:
.re
w
w
w
 www.rejinpaul.com

om
Neuron 3 is the winner and its weight vector W3 is updated
according to the competitive learning rule:

.c
ul
pa
The updated weight vector W3 at iteration (p+1) is
determined as:

jin
.re
The weight vector W3 of the winning neuron 3 becomes closer
w

to the input vector X with each iteration.

w
w
 www.rejinpaul.com
LEARNING VECTOR QUANTIZATION (LVQ)

om
 This is a supervised version of vector quantization. Classes are
predefined and we have a set of labeled data.

.c
The goal is to determine a set of prototypes that best represent

ul

each class.

pa
jin
.re
w
w
w
 www.rejinpaul.com
BASIC SCHEME OF LVQ

om
Step 1: Initialize prototype vectors for different classes.
Step 2: Present a single input.

.c
Step 3: Identify the closest prototype, i.e., the so-called winner.

ul
Step 4: Move the winner
• closer toward the data (same class),

pa
• away from the data (different class).

VARIANTS OF LVQ
jin
.re
• LVQ 1
• LVQ 2
w

• LVQ 2.1
• LVQ 3
w
w
 www.rejinpaul.com
COUNTERPROPAGATION NETWORK

om
 Another variant of the BPN is the counterpropagation network
(CPN).
 Although this network uses linear neurons, it can learn nonlinear

.c
functions by means of a hidden layer of competitive units.
 Moreover, the network is able to learn a function and its inverse at

ul
the same time.

pa
TYPES OF COUNTERPROPAGATION NETWORK
 FULL COUNTERPROPAGATION NET

jin
• Full counterpropagation net (full CPN) efficiently represents a
large number of vector pairs x:y by adaptively constructing a
.re
look-up-table.
FORWARD-ONLY COUNTERPROPAGATION NET
• A simplified version of full CPN is the forward-only CPN. The
w

approximation of the function y = f(x) but not of x = f(y) can

be performed using forward-only CPN.
w

• In forward-only CPN only the x-vectors are used to form the

w

clusters on the Kohonen units.

 www.rejinpaul.com
BASIC STRUCTURE OF FULL CPN

om
.c
ul
pa
FIRST PHASE OF CPN

jin
.re
w
w
w
 www.rejinpaul.com
SECOND PHASE OF CPN

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
CPN LEARNING PROCESS

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
COUNTERPROPAGATION NETWORK

om
 After the first phase of the training, each hidden-layer neuron is
associated with a subset of input vectors.

.c
The training process minimized the average angle difference

ul

between the weight vectors and their associated input vectors.

pa
 In the second phase of the training, we adjust the weights in the
network’s output layer in such a way that, for any winning hidden-
jin
layer unit, the network’s output is as close as possible to the
desired output for the winning unit’s associated input vectors.
.re
 The idea is that when we later use the network to compute
w

functions, the output of the winning hidden-layer unit is 1 and the

output of all other hidden-layer units is 0.
w
w
 www.rejinpaul.com

om
 In the first training phase, if a hidden-layer unit does not win for
a long period of time, its weights should be set to random values
to give that unit a chance to win subsequently.

.c
 There is no need for normalizing the training output vectors.

ul
 After the training has finished, the network maps the training

pa
vectors onto output vectors that are close to the desired ones. The
more hidden units, the better the mapping.


jin
CPN can be widely used in data compression and image
.re
compression applications.
w
w
w
 www.rejinpaul.com
ADAPTIVE RESONANCE THEORY (ART)

om
NETWORK
 Adaptive Resonance Theory (ART) is a family of algorithms for

.c
unsupervised learning developed by Carpenter and Grossberg.

ul
 ART is similar to many iterative clustering algorithms where each
pattern is processed by

pa
• finding the "nearest" cluster (a.k.a. prototype or template) to
that exemplar (desired).

jin
• updating that cluster to be "closer" to the exemplar.
.re
ARCHITECTURES OF ART NETWORK

ART1, designed for binary features.

w


 ART2, designed for continuous (analog) features.
w

 ARTMAP, a supervised version of ART.

w
 www.rejinpaul.com
FUNDAMENTAL ALGORITHM OF ART NETWORK

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
BASIC ARCHITECTURE OF ART1

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
ART1 UNITS

om
ART1 Network is made up of two units
 Computational units
• Input unit (F1 unit – input and interface).

.c
• Cluster unit (F2 unit – output).
• Reset control unit (controls degree of similarity).

ul
 Supplemental units
• One reset control unit.

pa
• Two gain control units.
ART2 NETWORK
jin
.re
w
w
w
 www.rejinpaul.com
BASIC ARCHITECTURE OF ART2

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
ART2 ALGORITHM

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
APPLICATIONS OF NEURAL NETWORKS

om
.c
ul
pa
jin
.re
w
w
w
 NEURAL NETWORK MATLAB TOOLBOX
www.rejinpaul.com
NEURAL NETWORK TOOLBOX

om
 The Matlab neural network toolbox provides a complete set of
functions and a graphical user interface for the design,

.c
implementation, visualization, and simulation of neural networks.
 It supports the most commonly used supervised and unsupervised

ul
network architectures and a comprehensive set of training and
learning functions.

pa
KEY FEATURES
 Graphical user interface (GUI) for creating, training, and simulating


jin
your neural networks.
Support for the most commonly used supervised and unsupervised
.re
network architectures.
 A comprehensive set of training and learning functions.
A suite of Simulink blocks, as well as documentation and
w


demonstrations of control system applications.
w

 Automatic generation of Simulink models from neural network objects.

 Routines for improving generalization.
w
 www.rejinpaul.com
GENERAL CREATION OF NETWORK

om
net = network
net=
network(numInputs,numLayers,biasConnect,inputConnect,layerConnec,

.c
outputConnect,targetConnect)
Description

ul
NETWORK creates new custom networks. It is used to create
networks that are then customized by functions such as NEWP,

pa
NEWLIN, NEWFF, etc.

numLayers
jin
NETWORK takes these optional arguments (shown with default values):
numInputs - Number of inputs, 0.
- Number of layers, 0.
.re
biasConnect - numLayers-by-1 Boolean vector, zeros.
inputConnect - numLayers-by-numInputs Boolean matrix, zeros.
w

layerConnect - numLayers-by-numLayers Boolean matrix, zeros.

outputConnect - 1-by-numLayers Boolean vector, zeros.
w

targetConnect - 1-by-numLayers Boolean vector, zeros, and returns,

w

NET - New network with the given property values.

 www.rejinpaul.com
TRAIN AND ADAPT

om
1. Incremental training : updating the weights after the
presentation of each single training sample.

.c
2. Batch training : updating the weights after each presenting the

ul
complete data set.

pa
When using adapt, both incremental and batch training can be
used . When using train on the other hand, only batch training will
jin
be used, regardless of the format of the data. The big plus of train
is that it gives you a lot more choice in training functions (gradient
.re
descent, gradient descent w/ momentum, Levenberg-Marquardt,
etc.) which are implemented very efficiently .
w
w
w
 www.rejinpaul.com
The difference between train and adapt: the difference between passes
and epochs. When using adapt, the property that determines how many

om
times the complete training data set is used for training the network is
called net.adaptParam.passes. Fair enough. But, when using train, the
exact same property is now called net.trainParam.epochs.

.c
>> net.trainFcn = 'traingdm';

ul
>> net.trainParam.epochs = 1000;
>> net.adaptFcn = 'adaptwb';

pa
>> net.adaptParam.passes = 10;

TRAINING FUNCTIONS
jin
There are several types of training functions:
.re
1. Supported training functions,
2. Supported learning functions,
3. Transfer functions
w

4. Transfer derivative functions,

w

5. Weight and bias initialize functions,

6. Weight derivative functions.
w
 www.rejinpaul.com
SUPPORTED TRAINING FUNCTIONS

om
trainb – Batch training with weight and bias learning rules
trainbfg – BFGS quasi-Newton backpropagation

.c
trainbr – Bayesian regularization
trainc – Cyclical order incremental update

ul
traincgb – Powell-Beale conjugate gradient backpropagation
traincgf – Fletcher-Powell conjugate gradient backpropagation

pa
traincgp – Polak-Ribiere conjugate gradient backpropagation
traingd – Gradient descent backpropagation
traingda – Gradient descent with adaptive learning rate backpropagation
traingdm
traingdx
trainlm
jin
– Gradient descent with momentum backpropagation
– Gradient descent with momentum & adaptive linear ackpropagation
– Levenberg-Marquardt backpropagation
.re
trainoss – One step secant backpropagations
trainr – Random order incremental update
trainrp – Resilient backpropagation (Rprop)
w

trains – Sequential order incremental update

trainscg – Scaled conjugate gradient backpropagation
w
w
 www.rejinpaul.com
SUPPORTED LEARNING FUNCTIONS

om
learncon – Conscience bias learning function
learngd – Gradient descent weight/bias learning function

.c
learngdm – Gradient descent with momentum weight/bias learning
function

ul
learnh – Hebb weight learning function
learnhd – Hebb with decay weight learning rule

pa
learnis – Instar weight learning function
learnk – Kohonen weight learning function
learnlv1
learnlv2 jin
– LVQ1 weight learning function
– LVQ2 weight learning function
.re
learnos – Outstar weight learning function
learnp – Perceptron weight and bias learning function
learnpn – Normalized perceptron weight and bias learning function
w

learnsom – Self-organizing map weight learning function

learnwh – Widrow-Hoff weight and bias learning rule
w
w
 www.rejinpaul.com
TRANSFER FUNCTIONS

om
compet - Competitive transfer function.
hardlim - Hard limit transfer function.

.c
hardlims - Symmetric hard limit transfer function.
logsig - Log sigmoid transfer function.

ul
poslin - Positive linear transfer function.

pa
purelin - Linear transfer function.
radbas - Radial basis transfer function.
satlin - Saturating linear transfer function.
satlins
softmax
jin
- Symmetric saturating linear transfer function.
- Soft max transfer function.
.re
tansig - Hyperbolic tangent sigmoid transfer function.
tribas - Triangular basis transfer function.
w
w
w
 www.rejinpaul.com
TRANSFER DERIVATIVE FUNCTIONS

om
Dhardlim - Hard limit transfer derivative function.
dhardlms - Symmetric hard limit transfer derivative function

.c
dlogsig - Log sigmoid transfer derivative function.
dposlin - Positive linear transfer derivative function.

ul
dpurelin - Hard limit transfer derivative function.

pa
dradbas - Radial basis transfer derivative function.
dsatlin - Saturating linear transfer derivative function.
dsatlins - Symmetric saturating linear transfer derivative

dtansig
jin
function.
- Hyperbolic tangent sigmoid transfer derivative
.re
function.
dtribas - Triangular basis transfer derivative function.
w
w
w
 www.rejinpaul.com
WEIGHT AND BIAS INITIALIZATION FUNCTIONS

om
initcon - Conscience bias initialization function.
initzero - Zero weight/bias initialization function.

.c
midpoint - Midpoint weight initialization function.
randnc - Normalized column weight initialization function.

ul
randnr - Normalized row weight initialization function.

pa
rands - Symmetric random weight/bias initialization function.

jin
WEIGHT DERIVATIVE FUNCTIONS
.re
ddotprod - Dot product weight derivative function.
w
w
w
 www.rejinpaul.com
NEURAL NETWORK TOOLBOX GUI

om
1. The graphical user interface (GUI) is designed to be simple and
user friendly. This tool lets you import potentially large and

.c
complex data sets.
2. The GUI also enables you to create, initialize, train, simulate, and

ul
manage the networks. It has the GUI Network/Data Manager

pa
window.
3. The window has its own work area, separate from the more
familiar command line workspace. Thus, when using the GUI, one
jin
might "export" the GUI results to the (command line) workspace.
Similarly to "import" results from the command line workspace to
.re
the GUI.
4. Once the Network/Data Manager is up and running, create a
w

network, view it, train it, simulate it and export the final results to
the workspace. Similarly, import data from the workspace for use in
w

the GUI.
w
 www.rejinpaul.com

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com

om
A graphical user interface can thus be used to

1. Create network,

.c
2. Create data,

ul
3. Train the networks,

pa
4. Export the networks,

jin
5. Export the data to the command line workspace.
.re
w
w
w
 UNIT-III FUZZY LOGIC
www.rejinpaul.com
MEMBERSHIP FUNCTIONS

om
CRISP MEMBERSHIP FUNCTIONS
 Crisp membership functions (µ) are either one or zero.

.c
 Consider the example: Numbers greater than 10. The membership
curve for the set A is given by

ul
pa
jin
REPRESENTING A DOMAIN IN FUZZY LOGIC
.re
w
w
w
 www.rejinpaul.com
FUZZY MEMBERSHIP FUNCTIONS

om
.c
ul
pa
jin
.re
The set B of numbers
approaching 2 can be
w

represented by the
membership function
w
w
 www.rejinpaul.com
FUZZINESS vs PROBABILITY

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
LINGUISTIC VARIABLE

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
LINGUISTIC VARIABLE

om
 Let x be a linguistic variable with the label “speed”.
 Terms of x, which are fuzzy sets, could be “positive low”, “negative

.c
high” from the term set T:

ul
T = {PostiveHigh, PositiveLow, NegativeLow,

pa
NegativeHigh, Zero}

jin
 Each term is a fuzzy variable defined on the base variable which
might be the scale of all relevant velocities.
.re
w
w
w
 www.rejinpaul.com
MEMBERSHIP

om
FUNCTIONS

.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
FEATURES OF MEMBERSHIP FUNCTIONS

om
 CORE:
 SUPPORT:

.c
BOUNDARY:

ul
pa
FUZZIFICATION

jin
.re
w
w
w
 www.rejinpaul.com

om
 Use crisp inputs from the user.

 Determine membership values for all the relevant classes (i.e., in

.c
right Universe of Discourse).

ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
EXAMPLE - FUZZIFICATION

om
.c
ul
pa
FUZZIFICATION OF HEIGHT

jin
.re
w
w
w
 FUZZIFICATION OF WEIGHT www.rejinpaul.com

om
.c
ul
pa
METHODS OF MEMBERSHIP VALUE ASSIGNMENT

jin
The various methods of assigning membership values are:
 Intuition,
.re
 Inference,
 Rank ordering,
 Angular fuzzy sets,
w

 Neural networks,
w

 Genetic algorithm,
 Inductive reasoning.
w
 www.rejinpaul.com
DEFUZZIFICATION

om
 Defuzzification is a mapping process from a space of fuzzy control
actions defined over an output universe of discourse into a space

.c
of crisp (nonfuzzy) control actions.

ul
 Defuzzification is a process of converting output fuzzy variable into

pa
a unique number.

 Defuzzification process has the capability to reduce a fuzzy set into

jin
a crisp single-valued quantity or into a crisp set; to convert a fuzzy
matrix into a crisp matrix; or to convert a fuzzy number into a crisp
.re
number.
w
w
w
 www.rejinpaul.com
LAMBDA CUT FOR FUZZY SETS

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
LAMBDA CUT FOR FUZZY RELATIONS

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
METHODS OF DEFUZZIFICATION

om
Defuzzification is the process of conversion of a fuzzy quantity into a
precise quantity.

.c
Defuzzification methods include:

ul
pa
 Max-membership principle,
 Centroid method,
 Weighted average method,

 Center of sums,
jin
Mean-max membership,
.re
 Center of largest area,
 First of maxima, last of maxima.
w
w
w
 www.rejinpaul.com
FUZZY DECISION

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
MAX MEMBERSHIP METHOD

om
 Fuzzy set with the largest membership value is selected.
 Fuzzy decision: Fn = {P, F. G, VG, E}

.c
 Fn = {0.6, 0.4, 0.2, 0.2, 0}
Final decision (FD) = Poor Student

ul

 If two decisions have same membership max, use the average of

pa
the two.

jin
.re
w
w
w
 www.rejinpaul.com
CENTROID METHOD

om
This method is also known as center-of-mass, center-of-area, or
center-of-gravity method. It is the most commonly used defuzzification

.c
method. The defuzzified output x* is defined as

ul
pa
jin
.re
where the symbol ∫ denotes an algebraic integration.
w
w
w
 www.rejinpaul.com
WEIGHTED AVERAGE METHOD

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
MEAN MAX MEMBERSHIP METHOD

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
CENTER OF SUMS

om
This method employs the algebraic sum of the individual fuzzy subsets
instead of their unions. The calculations here are very fast

.c
but the main drawback is that the intersecting areas are added twice.

ul
The defuzzified value x* is given by

pa
jin
.re
w
w
w
 www.rejinpaul.com
CENTER OF LARGEST AREA

om
This method can be adopted when the output consists of at least two
convex fuzzy subsets which are not overlapping. The output in this

.c
case is biased towards a side of one membership function. When
output fuzzy set has at least two convex regions then the center-of-

ul
gravity of the convex fuzzy subregion having the largest area is used to
obtain the defuzzified value x*.

pa
This value is given by

jin
.re
where is the convex subregion that has the largest area making up
w
w
w
 www.rejinpaul.com
FIRST OF MAXIMA (LAST OF MAXIMA)

om
The steps used for obtaining crisp values are as follows:

.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com

om
 Defuzzification process is essential because some engineering
applications need exact values for performing the operation.

.c
Example: If speed of a motor has to be varied, we cannot instruct

ul

to raise it “slightly”, “high”, etc., using linguistic variables; rather, it

pa
should be specified as raise it by 200 rpm or so, i.e., a specific
amount of raise should be mentioned.

 jin
The method of defuzzification should be assessed on the basis of
the output in the context of data available.
.re
w
w
w
 www.rejinpaul.com
FUZZY ARITHMETIC AND FUZZY MEASURES

om
FUZZY ARITHMETIC

.c
 Distributions do not have to be precise.
Requires no assumption about correlations.

ul

 Fuzzy measures are upper bounds on probability.

pa
 Fuzzy arithmetic might be a conservative way to do risk
assessments.

 Fuzzy numbers
jin
FUZZY NUMBERS AND THEIR ARITHMETIC
.re
• Fuzzy sets of the real line,
• Unimodal,
w

• Reach possibility level one.

 Fuzzy arithmetic
w

• Interval arithmetic at each possibility level.

w
 www.rejinpaul.com
FEATURES OF FUZZY ARITHMETIC

om
 Fully developed arithmetic and logic
• Addition, subtraction, multiplication, division, min, max;
• Log, exp, sqrt, abs, powers, and, or, not;

.c
• Backcalculation, updating, mixtures, etc.
 Very fast calculation and convenient software.

ul
 Very easy to explain.
 Distributional answers (not just worst case).

pa
 Results robust to choice about shape.

TYPES OF NUMBERS jin

Scalars are well-known or mathematically defined integers and real
.re

numbers.
 Intervals are numbers whose values are not know with certainty but
w

about which bounds can be established.

 Fuzzy numbers are uncertain numbers for which, in addition to
w

knowing a range of possible values, one can say that some values are
more plausible than others.
w
 www.rejinpaul.com
FUZZY NUMBERS

om
1
no no YES
 Fuzzy set that is unimodal and reaches 1. 0.5

 Nested stack of intervals. 0

.c
ul
LEVEL-WISE INTERVAL ARITHMETIC

pa
jin
.re
w
w
w
 www.rejinpaul.com
FUZZY INTERVALS BY TRAPEZOIDAL SHAPES

om
.c
ul
pa
FUZZY ADDITION
jin 1
A B A+B
.re
0.5
0
0 2 4 6 8
w

 Subtraction, multiplication, division, minimum, maximum,

w

exponentiation, logarithms, etc. are also defined.

 If distributions are multimodal, possibility theory (rather than just
w

simple fuzzy arithmetic) is required.

 www.rejinpaul.com
FUZZY ARITHMETIC

om
Interval Arithmetic

.c
[a, b] + [d, e] = [a+d, b+e]
[a, b] - [d, e] = [a-e, b-d]

ul
[a, b] x [d, e] = [min(a.d, a.e, b.d, b.e), max(a.d, a.e, b.d, b.e)]
[a, b]/[d, e] = [min(a/d, a/e, b/d, b/e), max(a/d, a/e, b/d, b/e)]

pa
jin
.re
w
w
w
 www.rejinpaul.com

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
EXTENSION PRINCIPLE

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
VARIOUS FUZZY MEASURES

om
 Belief and Plausibility Measure.

.c
 Probability Measure.
 Possibility and Necessity Measure.

ul
pa
POSSIBILITY MEASURE

 jin
No single definition.
Many definitions could be used:
.re
• Subjective assessments.
• Social consensus.
• Measurement error.
w

• Upper betting rates (Giles)

• Extra-observational ranges (Gaines).
w
w
 www.rejinpaul.com
FUZZY SET AS POSSIBILITY MEASURE

om
.c
ul
pa
jin
.re
w
w
w
GETTING FUZZY INPUTS www.rejinpaul.com

om
 Subjective assignments:
• Make them up from highest, lowest and best-guess estimates.
 Objective consensus:

.c
• Stack up consistent interval estimates or bridge inconsistent
ones.

ul
 Measurement error:

pa
• Infer from measurement protocols.
SUBJECTIVE ASSIGNMENTS


jin
Triangular fuzzy numbers, e.g. [1,2,3].
Trapezoidal fuzzy numbers, e.g. [1,2,3,4].
.re
1 1
Possibility

0.5
0.5
w

0
0 0 1 2 3 4 5
w

0 1 2 3 4
 www.rejinpaul.com
OBJECTIVE CONSENSUS

om
 [1000, 3000]
 [2000, 2400]

.c
 [500, 2500]
 [800, 4000]

ul
1
 [1900, 2300]

pa
Possibility
jin 0.5
.re
w

0
w

0 2000 4000
w
 www.rejinpaul.com
MEASUREMENT ERROR

om
.c
1

Possibility
 46.8 ± 0.3

ul
 [46.5, 46.8, 47.1]

pa
0
46.5 46.7 46.9 47.1

jin 1

Possibility
.re
 [12.32]
 [12.315, 12.32, 12.325]
w

0
12.31 12.32 12.33
w
w
 www.rejinpaul.com
SHAPE OF X AFFECTING aX+b

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
ROBUSTNESS OF THE ANSWER

om
Different choices for the fuzzy number X all yield very similar
distributions for aX + b

.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
PROBABILITY MEASURE

om
 A fuzzy number F is said to “enclose” a probability distribution P if,
• the left side of F is larger than P(x) for each x,
• the right side of F is larger than 1-P(x) for each x.

.c
 For every event X < x and x < X, possibility is larger than the
probability, so it is an upper bound.

ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
ADVANTAGES OF FUZZY ARITHMETIC

om
 Requires little data.
 Applicable to all kinds of uncertainty.
 Fully comprehensive.

.c
 Fast and easy to compute.
 Doesn’t require information about correlations.

ul
 Conservative, but not hyper-conservative
 In between worst case and probability.

pa
 Backcalculations easy to solve.

LIMITATIONS OF FUZZY ARITHMETIC

 Controversial. jin
.re
 Are alpha levels comparable for different variables?
 Not optimal when there're a lot of data.
 Can’t use knowledge of correlations to tighten answers.
w

 Not conservative against all possible dependencies.

w

 Repeated variables make calculations cumbersome.

w
 www.rejinpaul.com
SOFTWARES FOR FUZZY ARITHMETIC

om
 FuziCalc
(Windows 3.1) FuziWare, 800-472-6183

.c
•

 Fuzzy Arithmetic C++ Library

ul
• (C code) anonymous ftp to mathct.dipmat.unict.it and get
\fuzzy\fznum*.*

pa
 Cosmet (Phaser)
• (DOS, soon for Windows) [email protected]

 Risk Calc jin

(Windows) 800-735-4350; www.ramas.com
.re
•
w
w
w
 www.rejinpaul.com
SCREEN SHOT OF FUZZY CALCULATOR - FzCalc

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
FUZZY RULE BASE AND APPROXIMATE

om
REASONING
FUZZY RULES AND REASONING

.c
The degree of an element in a fuzzy set corresponds to the truth value
of a proposition in fuzzy logic systems.

ul
LINGUISTIC VARIABLES

pa
 A linguistic variable is a fuzzy variable.
• The linguistic variable speed ranges between 0 and 300 km/h
jin
and includes the fuzzy sets slow, very slow, fast, …
• Fuzzy sets define the linguistic values.
.re
 Hedges are qualifiers of a linguistic variable.
• All purpose: very, quite, extremely
w

• Probability: likely, unlikely

• Quantifiers: most, several, few
w

• Possibilities: almost impossible, quite possible

w
 www.rejinpaul.com
LINGUISTIC HEDGES (LINGUISTIC QUANTIFIERS)

om
 Hedges modify the shape of a fuzzy set.

.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
TRUTH TABLES

om
Truth tables define logic functions of two propositions. Let X and Y be
two propositions, either of which can be true or false.

.c
The operations over the propositions are:
1. Conjunction (∧): X AND Y.

ul
2. Disjunction (∨): X OR Y.
3. Implication or conditional (⇒): IF X THEN Y.

pa
4. Bidirectional or equivalence (⇔): X IF AND ONLY IF Y.

FUZZY RULES
jin
A fuzzy rule is defined as the conditional statement of the form
.re
If x is A THEN y is B
w

where x and y are linguistic variables and A and B are linguistic values
w

determined by fuzzy sets on the universes of discourse X and Y.

w

www.rejinpaul.com
The decision-making process is based on rules with sentence
conjunctives AND, OR and ALSO.

om
 Each rule corresponds to a fuzzy relation.
 Rules belong to a rule base.
 Example: If (Distance x to second car is SMALL) OR (Distance y

.c
to obstacle is CLOSE) AND (speed v is HIGH) THEN (perform

ul
LARGE correction to steering angle θ) ALSO (make MEDIUM
reduction in speed v).

pa
 Three antecedents (or premises) in this example give rise to two
outputs (consequences).

FUZZY RULE FORMATION jin

.re
IF height is tall THEN weight is heavy.
Here the fuzzy classes height and weight have a given range (i.e., the
w

universe of discourse).
w

range (height) = [140, 220] range (weight) = [50, 250]

w
 www.rejinpaul.com
FORMATION OF FUZZY RULES

om
Three general forms are adopted for forming fuzzy rules. They are:
 Assignment statements,

.c
 Conditional statements,
 Unconditional statements.

ul
pa
Assignment Statements Unconditional Statements

jin
.re
w

Conditional Statements
w
w
 DECOMPOSITION OF FUZZY RULES
www.rejinpaul.com
A compound rule is a collection of several simple rules combined together.

om
Multiple conjunctive antecedent, Multiple disjunctive antecedent,
Conditional statements (with ELSE and UNLESS).

.c
Multiple Conjunctive antecedants Multiple disjunctive antecedent

ul
pa
jin
Conditional Statements ( With Else and Unless)
.re
w
w
w
 www.rejinpaul.com
AGGREGATION OF FUZZY RULES

om
Aggregation of rules is the process of obtaining the overall consequents
from the individual consequents provided by each rule.
 Conjunctive system of rules.

.c
 Disjunctive system of rules.

ul
pa
Conjunctive system of rules

jin
.re
w
w
w
 www.rejinpaul.com
Disjunctive system of rules

om
.c
ul
pa
jin
.re
FUZZY RULE - EXAMPLE
Rule 1: If height is short then weight is light.
w

Rule 2: If height is medium then weight is medium.

w

Rule 3: If height is tall then weight is heavy.

w
 www.rejinpaul.com

om
Problem: Given
(a) membership functions for short, medium-height, tall, light,
medium-weight and heavy;

.c
(b) The three fuzzy rules;
(c) the fact that John’s height is 6’1”

ul
estimate John’s weight.

pa
Solution:
jin
(1) From John’s height we know that
John is short (degree 0.3)
.re
John is of medium height (degree 0.6).
John is tall (degree 0.2).
w

(2) Each rule produces a fuzzy set as output by truncating the

consequent membership function at the value of the antecedent
w

membership.
w
 www.rejinpaul.com

om
.c
ul
pa
jin
.re
w
w

 The cumulative fuzzy output is obtained by OR-ing the output from

w

each rule. Cumulative fuzzy output (weight at 6’1”).

 www.rejinpaul.com

om
1. De-fuzzify to obtain a numerical estimate of the output.
2. Choose the middle of the range where the truth value is
maximum.

.c
3. John’s weight = 80 Kg.

ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
FUZZY REASONING

om
There exist four modes of fuzzy approximate reasoning, which include:
1. Categorical reasoning,
2. Qualitative reasoning,

.c
3. Syllogistic reasoning,
4. Dispositional reasoning.

ul
REASONING WITH FUZZY RULES

pa
 In classical systems, rules

 In fuzzy systems, truthjin

with true antecedents fire.
.re
(i.e., membership in some
class) is relative, so all
rules fire (to some extent).
w
w
w
 www.rejinpaul.com
SINGLE RULE WITH SINGLE ANTECEDANT

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
MULTIPLE ANTECEDANTS

om
IF x is A AND y is B THEN z is C
IF x is A OR y is B THEN z is C

.c
Use unification (OR) or intersection (AND) operations to calculate a

ul
membership value for the whole antecedent.

pa
jin
.re
w
w
w
 www.rejinpaul.com
MULTIPLE RULE WITH MULTIPLE ANTECEDANTS

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
MULTIPLE CONSEQUENTS

om
IF x is A THEN y is B AND z is C

.c
ul
Each consequent is affected equally by the membership in the
antecedent class(es).

pa
E.g., IF x is tall THEN x is heavy AND x has large feet.

jin
.re
w
w
w
 www.rejinpaul.com
FUZZY INFERENCE SYSTEMS (FIS)

om
 Fuzzy rule based systems, fuzzy models, and fuzzy expert systems
are also known as fuzzy inference systems.

.c
 The key unit of a fuzzy logic system is FIS.
The primary work of this system is decision-making.

ul

 FIS uses “IF...THEN” rules along with connectors “OR” or “AND” for

pa
making necessary decision rules.
 The input to FIS may be fuzzy or crisp, but the output from FIS is
always a fuzzy set.

output.
jin
When FIS is used as a controller, it is necessary to have crisp
.re
 Hence, there should be a defuzzification unit for converting fuzzy
variables into crisp variables along FIS.
w
w
w
 www.rejinpaul.com
BLOCK DIAGRAM OF FIS

om
.c
ul
pa
jin
.re
TYPES OF FIS
w

There are two types of Fuzzy Inference Systems:

w

 Mamdani FIS(1975)
 Sugeno FIS(1985)
w
 www.rejinpaul.com
MAMDANI FUZZY INFERENCE SYSTEMS (FIS)

om
 Fuzzify input variables:
• Determine membership values.
 Evaluate rules:

.c
• Based on membership values of (composite) antecedents.
 Aggregate rule outputs:

ul
• Unify all membership values for the output from all rules.

pa
 Defuzzify the output:
• COG: Center of gravity (approx. by summation).

SUGENO FUZZY INFERENCE SYSTEMS (FIS)

jin
The main steps of the fuzzy inference process namely,
.re
1. fuzzifying the inputs and
2. applying the fuzzy operator are exactly the same as in MAMDANI
FIS.
w

The main difference between Mamdani’s and Sugeno’s methods is that

w

Sugeno output membership functions are either linear or constant.

w
 www.rejinpaul.com
SUGENO FIS

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
FUZZY EXPERT SYSTEMS

om
An expert system contains three major blocks:
 Knowledge base that contains the knowledge specific to the
domain of application.

.c
 Inference engine that uses the knowledge in the knowledge base
for performing suitable reasoning for user’s queries.

ul
 User interface that provides a smooth communication between the
user and the system.

pa
Examples of Fuzzy Expert
System include Z-II, MILORD.
jin
.re
w
w
w

BLOCK DIAGRAM OF FUZZY EXPERT SYSTEMS

 www.rejinpaul.com

om
 Advantages of fuzzy logic

.c
• Allows the use of vague linguistic terms in the rules.

ul
pa
 Disadvantages of fuzzy logic
• Difficult to estimate membership function
• There are many ways of interpreting fuzzy rules, combining the
jin
outputs of several fuzzy rules and de-fuzzifying the output.
.re
w
w
w
 www.rejinpaul.com
FUZZY DECISION MAKING

om
Decision making is a very important social, economical and scientific
endeavor.

.c
The decision-making process involves three steps:

ul
1. Determining the set of alternatives,
2. Evaluating alternatives,

pa
3. Comparison between alternatives.

jin
Certain fuzzy decision-making process are as follows:
.re
 Individual decision making,
 Multiperson decision making,
 Multiobjective decision making,
w

 Multiattribute decision making,

w

 Fuzzy Bayesian decision making.

w
 www.rejinpaul.com
INDIVIDUAL DECISION MAKING

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
MULTIPERSON DECISION MAKING

om
In multiperson decision making, the decision makers have access to

.c
different information upon which to base their decision.

ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
MULTIOBJECTIVE DECISION MAKING

om
In making a decision when there are several objectives to be realized,
then the decision making is called multiobjective decision making.

.c
Many decision processes may be based on single objectives such as
cost minimization, time consumption, profit maximization and

ul
so on.

pa
The main issues in multiobjective decision making are:
 To acquire proper information related to the satisfaction of the
objectives by various alternatives.
jin
 To weigh the relative importance of each objective.
.re
MULTIATTRIBUTE DECISION MAKING
 The evaluation of alternatives can be carried out based on several
w

attributes of the object called multiattribute decision making.

w

 The attributes may be classified into numerical data, linguistic data

and qualitative data.
w
 www.rejinpaul.com
MULTIATTRIBUTE DECISION MAKING

om
The problem of decision-making structure in multiattributes deals with
determination of an evaluation structure for the multiattribute decision

.c
making from the available multiattribute data xi (i = 1 to n) and
alternative evaluations y shown in the table.

ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
FUZZY BAYESIAN DECISION MAKING

om
 Probability theory.

.c
 Bayesian inference:
• Use probability theory and information about independence.

ul
• Reason diagnostically [from evidence (effects) to conclusions

pa
(causes)] or casually (from causes to effects).

 Bayesian networks:
jin
• Compact representation of probability distribution over a set of
propositional random variables.
.re
• Take advantage of independence relationships.
w
w
w
 www.rejinpaul.com
FUZZY BAYESIAN DECISION MAKING

om
.c
ul
pa
Expected Utility jin
.re
w
w
w
 www.rejinpaul.com

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
APPLICATIONS OF FUZZY LOGIC

om
.c
ul
pa
Examples

jin
.re
w
w
w
 www.rejinpaul.com

om
In the various decision-making process,

In many decision-making situations, the goals, constraints and

.c
consequences of the defined alternatives are known imprecisely, which

ul
is due to ambiguity and vagueness.

pa
Methods for addressing this form of imprecision are important for
dealing with many of the uncertainties we deal within human systems.

jin
.re
w
w
w
 www.rejinpaul.com

om
INTRODUCTION TO FUZZY LOGIC TOOLBOX IN MATLAB

FUZZY LOGIC TOOLBOX

.c
 Fuzzy logic in Matlab can be dealt very easily due to the existing

ul
new Fuzzy Logic Toolbox.

pa
 This provides a complete set of functions to design an implement
various fuzzy logic processes.


• fuzzification,
jin
The major fuzzy logic operations include:
.re
• defuzzification,
• fuzzy inference.
w

 These all are performed by means of various functions and even

w

can be implemented using the Graphical User Interface.

w
 www.rejinpaul.com

om
The features are:

 It provides tools to create and edit Fuzzy Inference Systems (FIS).

.c
It allows integrating fuzzy systems into simulation with SIMULINK.

ul
 It is possible to create stand-alone C programs that call on fuzzy
systems built with MATLAB.

pa
The Toolbox provides three categories of tools:

jin
• Command line functions,
• Graphical or interactive tools,
.re
• Simulink blocks.
w
w
w
 www.rejinpaul.com
COMMAND LINE FIS FUNCTIONS

om
Addmf - Add membership function to FIS
addrule - Add rule to FIS.

.c
addvar - Add variable to FIS.
defuzz - Defuzzify membership function.
evalfis - Perform fuzzy inference calculation

ul
evalmf - Generic membership function evaluation.
gensurf - Generate FIS output surface.

pa
getfis - Get fuzzy system properties.
mfstrtch - Stretch membership function.
newfis - Create new FIS.
plotfis
plotmf
readfis
-
-
-
jin
Display FIS input-output diagram.
Display all membership functions for one variable.
Load FIS from disk.
.re
rmmf - Remove membership function from FIS
rmvar - Remove variable from FIS.
Setfis - Set fuzzy system properties.
showfis - Display annotated FIS.
w

Showrule - Display FIS rules

writefis - Save FIS to disk.
w
w
 www.rejinpaul.com
MEMBERSHIP FUNCTIONS

om
Dsigmf - Difference of two sigmoid membership functions.
gauss2mf - Two-sided Gaussian curve membership function.

.c
gaussmf - Gaussian curve membership function.
gbellmf - Generalized bell curve membership function.

ul
pimf - Pi-shaped curve membership function.

pa
psigmf - Product of two sigmoid membership functions.
smf - S-shaped curve membership function.
sigmf - Sigmoid curve membership function.
trapmf
trimf
jin
- Trapezoidal membership function.
- Triangular membership function.
.re
zmf - Z-shaped curve membership function.
w
w
w
 www.rejinpaul.com
GRAPHICAL USER INTERFACE EDITORS

om
(GUI TOOLS)

.c
anfisedit - ANFIS training and testing UI tool.
findcluster - Clustering UI tool.

ul
fuzzy - Basic FIS editor.

pa
mfedit - Membership function editor.
ruleedit - Rule editor and parser.
ruleview - Rule viewer and fuzzy inference diagram.
surfview - jin
Output surface viewer.
.re
w
w
w
 www.rejinpaul.com
FIS EDITOR (MAMDANI)

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
FIS EDITOR (SUGENO)

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
FIS MEMBERSHIP FUNCTION EDITOR

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
FIS RULE EDITOR

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
FIS RULE VIEWER

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
FIS SURFACE VIEWER

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
SIMULINK BLOCKS

om
Once fuzzy system is created using GUI tools or some other
method, can be directly embedded into SIMULINK using the Fuzzy

.c
Logic Controller block.

ul
pa
ADVANCED TECHNIQUES
anfis -
jin
Training routine for Sugeno-type FIS (MEX only).
.re
fcm - Find clusters with fuzzy c-means clustering.
genfis1 - Generate FIS matrix using generic method.
w

genfis2 - Generate FIS matrix using subtractive clustering.

subclust - Estimate cluster centers with subtractive clustering.
w
w
 www.rejinpaul.com
UNIT –IV

om
GENETIC ALGORITHM

.c
Genetic Algorithm (3) – Basic Algorithm

ul
Outline of the basic algorithm

pa
0 START : Create random population of n chromosomes
1 FITNESS : Evaluate fitness f(x) of each chromosome in the
population
2 NEW POPULATION jin
1 REPRODUCTION/SELECTION : Based on f(x)
.re
2 CROSS OVER : Cross-over chromosomes
3 MUTATION : Mutate chromosomes
w

3 REPLACE : Replace old with new population: the new generation

w

4 TEST : Test problem criterium

w

5 LOOP : Continue step 1 – 4 untill criterium is satisfied

 Flowchart of GA www.rejinpaul.com

om
.c
ul
pa
jin
.re
w
w

• All individuals in population evaluated by fitness function.

• Individuals allowed to reproduce (Selection), Crossover, mutate.
w
 www.rejinpaul.com
Genetic Algorithm – Reproduction Cycle

om
1. Select parents for the mating pool (size of mating pool=population size).
2. Shuffle the mating pool.
3. For each consecutive pair apply crossover.

.c
4. For each offspring apply mutation (bit-flip independently for each bit).
5. Replace the whole population with the resulting offspring.

ul
pa
Genetic Algorithm – Coding
 Chromosomes are encoded by bitstrings.

solution. jin
Every bitstring therefore is a solution but not necessarily the best
.re
 The way bitstrings can code differs from problem to problem.
1
0
w

Either sequence of on/off or the number 9 0

w

1
w
 www.rejinpaul.com
Genetic Algorithm – Crossover (Single Point)

om
 Choose a random point on the two parents.
 Split parents at this crossover point.

.c
 Create children by exchanging tails.

ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
Genetic Algorithm – Crossover (n Points)

om
 Choose n random crossover points.
 Split along those points.

.c
 Glue parts, alternating between parents.
Generalization of 1 point.

ul


pa
jin
.re
w
w
w
 www.rejinpaul.com
Genetic Algorithm – Uniform Crossover

om
Generate uniformly random number.

.c
X1 = 0110001010

ul
X2 = 1100000111

pa
Uniformly generated = 1 0 0 0 0 0 1 0 0 0

jin
As a result, the new population becomes,
.re
X1 = 1110000010
w

X2 = 0100001111
w
w
 www.rejinpaul.com
Genetic Algorithm – Mutation

om
 Alter each gene independently with a probability pm
 pm is called the mutation rate

.c
• Typically between 1/pop_size and 1/ chromosome_length

ul
pa
00010101 0011001 01111000 00000000 00001000 00000010
00100100 10111010 11110000 10000010 00000010 00000000
jin
11000101 01011000 01101010
11000101 01011000 01101010
00000000 00010000 00000000
00010000 00000000 01000000
.re
00010101 00110001 01111010
w

10100110 10111000 11110000

11000101 01111000 01101010
w

11010101 01011000 00101010

w
 www.rejinpaul.com
Genetic Algorithm – Selection

om
 Main idea: Better individuals get higher chance:
• Chances are proportional to fitness.

.c
• Implementation: Roulette wheel technique
» Assign to each individual a part of the roulette wheel.

ul
» Spin the wheel n times to select n individuals.

pa
jin
1/6 = 17%
fitness(A) = 3
.re
A B fitness(B) = 1
C
w

3/6 = 50% 2/6 = 33% fitness(C) = 2

w
w
 www.rejinpaul.com
Genetic Algorithm – An Example

om
 Simple problem: max x2 over {0, 1, …, 31}

.c
 GA approach:
Representation: binary code, e.g. 01101 ↔ 13

ul
•
• Population size: 4

pa
• 1-point xover, bitwise mutation
• Roulette wheel selection
Random initialisation


•
jin
One generational cycle performed manually is shown here.
.re
w
w
w
 www.rejinpaul.com
Example : Selection

om
.c
ul
pa
jin
.re
4 1
31% 14%
w

3
w

5%
49%
2
w
 www.rejinpaul.com
Example : Crossover

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
Example : Mutation

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
Simple Genetic Algorithm

om
 Has been subject of many (early) studies
• still often used as benchmark for novel Gas.
 Shows many shortcomings, e.g.

.c
• Representation is too restrictive.
• Mutation & crossovers only applicable for bit-string & integer

ul
representations.
• Selection mechanism sensitive for converging populations with

pa
close fitness values.

Comparison of GA with Traditional Optimization

Techniques jin
.re
 GA works with the coding of solution set and not with the solution
itself.
w

 GA uses population of solutions rather than a single solution for

searching.
w

 GA uses fitness function for evaluation rather the derivatives.

GA uses probabilistic transition and not deterministic rules.
w


 www.rejinpaul.com
APPLICATIONS OF GENETIC ALGORITHM

om
.c
ul
pa
 Several Scheduling Problems,


jin
Traveling Salesman Problem,

Energy Optimization and so on.

.re

w
w
w
 www.rejinpaul.com
References

om
• Holland, J. (1992), Adaptation in natural and artificial systems , 2nd
Ed. Cambridge: MIT Press.
• Davis, L. (Ed.) (1991), Handbook of genetic algorithms. New York:

.c
Van Nostrand Reinhold.
• Goldberg, D. (1989), Genetic algorithms in search, optimization and

ul
machine learning. Addison-Wesley.
• Fogel, D. (1995), Evolutionary computation: Towards a new

pa
philosophy of machine intelligence. Piscataway: IEEE Press.
• Bäck, T., Hammel, U., and Schwefel, H. (1997), ‘Evolutionary

jin
computation: Comments on the history and the current state’, IEEE
Trans. On Evol. Comp. 1, (1)
.re
Online Resources
• http://www.spectroscopynow.com
w

• http://www.cs.bris.ac.uk/~colin/evollect1/evollect0/index.htm\
w

• IlliGAL (http://www-illigal.ge.uiuc.edu/index.php3)
• GAlib (http://lancet.mit.edu/ga/)
w
 www.rejinpaul.com
GENETIC ALGORITHM MATLAB TOOLBOX

om
 The genetic algorithm is a method for solving both constrained and
unconstrained optimization problems that is based on natural

.c
selection, the process that drives biological evolution.

ul
 The genetic algorithm repeatedly modifies a population of

pa
individual solutions. At each step, the genetic algorithm selects
individuals at random from the current population to be parents
and uses them produce the children for the next generation.


jin
Over successive generations, the population "evolves" toward an
.re
optimal solution.
w
w
w
 www.rejinpaul.com

om
The Genetic Algorithm and Direct Search Toolbox is a collection of
functions that extend the capabilities of the Optimization Toolbox and
the MATLAB numeric computing environment. The Genetic Algorithm

.c
and Direct Search Toolbox includes routines for solving optimization
problems using

ul
 GENETIC ALGORITHM

pa
 DIRECT SEARCH

jin
These algorithms enable you to solve a variety of optimization
problems that lie outside the scope of the Optimization Toolbox.
.re
w
w
w
 www.rejinpaul.com

om
The genetic algorithm uses three main types of rules at each step to
create the next generation from the current population:

.c
 Selection rules select the individuals, called parents, that
contribute to the population at the next generation.

ul
 Crossover rules combine two parents to form children for the next

pa
generation.


form children.
jin
Mutation rules apply random changes to individual parents to
.re
w
w
w
 www.rejinpaul.com

om
The genetic algorithm at the command line, call the genetic algorithm
function ga with the syntax

.c
[x fval] = ga(@fitnessfun, nvars, options) where
@fitnessfun is a handle to the fitness function.

ul
 nvars is the number of independent variables for the fitness

pa
function.


jin
options is a structure containing options for the genetic algorithm.
If you do not pass in this argument, ga uses its default options.
.re
The results are given by
 x — Point at which the final value is attained.
w
w

 fval — Final value of the fitness function.

w
 www.rejinpaul.com
COMMAND LINE FUNCTIONS

om
Bin2int - BINary string to INTeger string conversion
bin2real - BINary string to REAL vector conversion
bindecod - BINary DECODing to binary, integer or real numbers

.c
compdiv - COMPute DIVerse things of GEA Toolbox
compdiv2 - COMPute DIVerse things of GEA Toolbox

ul
compete - COMPETition between subpopulations

pa
comploc - COMPute LOCal model things of toolbox
compplot - COMPute PLOT things of GEA Toolbox
geamain2 - MAIN function for Genetic and Evolutionary

initbp
jin
Algorithm toolbox for matlab
- CReaTe an initial Binary Population
.re
initip - CReaTe an initial (Integer value) Population
mutate - high level MUTATion function
mutbin - MUTation for BINary representation
w

mutbmd - real value Mutation like Discrete Breeder genetic

w

algorithm
mutcomb - MUTation for combinatorial problems
w
 www.rejinpaul.com

om
mutes1 - MUTation by Evolutionary Strategies 1, derandomized
Self Adaption
mutexch - MUTation by eXCHange

.c
mutint - MUTation for INTeger representation
mutinvert - MUTation by INVERTing variables

ul
mutmove - MUTation by MOVEing variables
mutrand - MUTation RANDom

pa
mutrandbin - MUTation RANDom of binary variables
Mutrandbin - MUTation RANDom of binary variables
mutrandint
jin
- MUTation RANDom of integer variables
mutrandperm - MUTation RANDom of binary variables
.re
Mutrandreal - MUTation RANDom of real variables
Rankgoal - perform goal preference calculation between
multiple objective values
w

ranking - RANK-based fitness assignment, single and

w

multi objective, linear and nonlinear

w
 www.rejinpaul.com

om
rankplt - RANK two multi objective values Partially Less
Than
rankshare - SHARing between individuals

.c
recdis - RECombination DIScrete
recdp - RECombination Double Point

ul
recdprs - RECombination Double Point with Reduced
Surrogate

pa
recgp - RECombination Generalized Position
recint - RECombination extended INTermediate
reclin
reclinex
jin
- RECombination extended LINe
- EXtended LINe RECombination
.re
recmp - RECombination Multi-Point, low level
function recombin - high level RECOMBINation function
recpm - RECombination Partial Matching
w

recsh - RECombination SHuffle

w

recsp - RECombination Single Point

w
SELECTION FUNCTIONS www.rejinpaul.com
selection - high level SELECTion function

om
sellocal - SELection in a LOCAL neighbourhood
selrws - SELection by Roulette Wheel Selection
selsus - SELection by Stochastic Universal Sampling

.c
seltour - SELection by TOURnament

ul
seltrunc - SELection by TRUNCation
OBJECTIVE FUNCTIONS

pa
initdopi - INITialzation function for DOuble Integrator objdopi
initfun1 - INITialzation function for de jong's FUNction 1
mopfonseca1
mopfonseca2
jin
- MultiObjective Problem: FONSECA's function 1
- MultiObjective Problem: FONSECA's function 1
.re
moptest - MultiObjective function TESTing
obj4wings - OBJective function FOUR-WINGS.
objbran - OBJective function for BRANin rcos function
w

objdopi - OBJective function for DOuble Integrator

w

objeaso - OBJective function for EASom function

objfletwell - OBJective function after FLETcher and PoWELL
w
 www.rejinpaul.com
GA GRAPHICAL USER INTERFACE

om
The Genetic Algorithm Tool is a graphical user interface that enables
you to use the genetic algorithm without working at the command line.

.c
To open the Genetic Algorithm Tool, enter

ul
Gatool

pa
at the MATLAB command prompt.

jin
.re
w
w
w
 www.rejinpaul.com

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com

om
To use the Genetic Algorithm Tool, you must first enter the following
information:
Fitness function — The objective function you want to minimize.

.c
Enter the fitness function in the form @fitnessfun, where fitnessfun.m
is an M-file that computes the fitness function.

ul
Number of Variables – The number of variables in the given fitness
function should be given.

pa
1. Best fitness
PLOT OPTIONS 2. Best individual
jin 3.
4.
Distance
Expectation
.re
5. Genealogy
6. Range
w

7. Score Diversity
8. Scores
w

9. Selection
w

10. Stopping
 www.rejinpaul.com

om
POPULATION

In this case population type, population size and creation function

.c
may be selected.
The initial population and initial score may be specified, if not, the

ul
‘Ga tool’ creates them. The initial range should be given.

pa
FITNESS SCALING

jin
The fitness scaling should be any of the following
a. Rank
.re
b. Proportional
c. Top
d. Shift Linear
w

e. Custom
w
w
 www.rejinpaul.com
SELECTION

om
.c
ul
pa
jin REPRODUCTION
.re
w
w
w
 www.rejinpaul.com
MUTATION

om
.c
ul
pa
jin CROSSOVER
.re
w
w
w
 www.rejinpaul.com
MIGRATION

om
.c
ul
pa
jin HYBRID FUCNTION
.re
w
w
w
 www.rejinpaul.com
STOPPING CONDITION

om
.c
ul
pa
RUNNING AND
jin SIMULATION
.re
w
w
w
 www.rejinpaul.com
EXAMPLE 1 – FUCNTION WITH SINGLE VARIABLE

om
Define the given function f(x) = x2+3x+2 in a separate m-file

.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
OUTPUT

om
The output showing the best fitness for 50 generations.

.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com

om
The status and results for this functions for 50 generations

.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
EXAMPLE 2 – FUNCTION WITH TWO VARIABLES

om
Define the given function f(x1,x2)=4x1+5x2 in a separate m-file.

.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
OUTPUT

om
The output for 50 generations.

.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
The status and result for this function.

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
EXAMPLE 3 – SINE FUNCTION

om
f (x1, x2, x3) = -5sin(x1)sin(x2)sin(x3)+-sin(5x1)sin(5x2)sin(x3)
where 0 <= xi <= pi, for 1<= i <= 3.

.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com

om
.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
OUTPUT

om
The output for 100 generations.

.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com

om
The status and result for this function

.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
UNIT –V

om
HYBRID SOFT COMPUTING TECHNIQUES

.c
HYBRID SYSTEMS
Neural Network Systems

ul
Neural networks are the simplified models of the human nervous
systems mimicking our ability to adapt to certain situations and to

pa
learn from the past experiences.

Fuzzy Logic
jin
Fuzzy logic or fuzzy systems deal with uncertainty or vagueness
existing in a system and formulating fuzzy rules to find a solution
.re
to problems.

Genetic Algorithm
w

Genetic algorithms inspired by the natural evolution process are

w

adaptive search and optimization algorithms.

w
 www.rejinpaul.com

om
The main aim of the concept of hybridization is to overcome the
weakness in one technique while applying it and bringing out the
strength of the other technique to find solution by combining them.

.c
Neural networks are good at recognizing patterns but they are not

ul
good at explaining how they reach their decisions.

pa
On the contrary, fuzzy logic is good at explaining the decisions but
cannot automatically acquire the rules used for making the decisions.
Also, the tuning of membership functions becomes an important issue
jin
in fuzzy modeling. Genetic algorithms offer a possibility to solve this
problem.
.re
These limitations act as a central driving force for the creation of
hybrid soft computing systems where two or more techniques are
w

combined in a suitable manner that overcomes the limitations of

individual techniques.
w
w
 www.rejinpaul.com
The use of hybrid systems is growing rapidly with successful

om
applications in areas such as
 engineering design
 stock market analysis and prediction

.c
 medical diagnosis
 process control

ul
 credit card analysis and
 few other cognitive simulations.

pa
VARIOUS HYBRID SYSTEMS
jin
The following three different hybrid systems are :
.re
 Neuro fuzzy hybrid system;
 Neuron genetic hybrid system;
w

 Fuzzy genetic hybrid systems.

w
w
 www.rejinpaul.com
NEURO FUZZY HYBRID SYSTEMS

om
Definition:
A neuro-fuzzy hybrid system (also called fuzzy neural hybrid) is a

.c
learning mechanism that utilizes the training and learning algorithms
from neural networks to find parameters of a fuzzy system.

ul
pa
Advantages of neuro fuzzy hybrid systems:

 It can handle any kind of information (numeric, linguistic, logical,


etc.). jin
It can manage imprecise, partial, vague or imperfect information.
.re
 It can resolve conflicts by collaboration and aggregation.
 It has self-learning, self-organizing and self-tuning capabilities.
w

 It doesn’t need prior knowledge of relationships of data.

w
w
 www.rejinpaul.com
ARCHITECTURE OF NEURO FUZZY HYBRID

om
SYSTEMS
The general architecture of neuro-
fuzzy hybrid system

.c
The architecture is a three-layer feed

ul
forward neural network model. It can also
be observed that the first layer

pa
corresponds to the input variables, and the
second and third layers correspond to the
fuzzy rules and output variables,
jin
respectively. The fuzzy sets are converted
to (fuzzy) connection weights.
.re
TYPES OF NEURO FUZZY HYBRID SYSTEMS
w

NFSs can be classified into the following two systems:

w

1. Cooperative NFSs.
2. General neuro-fuzzy hybrid systems.
w
 www.rejinpaul.com
CO-OPERATIVE NEURO FUZZY SYSTEMS

om
In this type of system, both artificial neural network (ANN) and fuzzy
system work independently from each other. The ANN attempts to
learn the parameters from the fuzzy system.

.c
GENERAL NEURO FUZZY HYBRID SYSTEMS

ul
General neuro-fuzzy hybrid systems (NFHS) resemble neural networks

pa
where a fuzzy system is interpreted as a neural network of special kind.

jin
The rule base of a fuzzy
system is assumed to be
.re
a neural network; the
fuzzy sets are regarded
as weights and the rules
w

and the input and

w

output variables as
neurons.
w
 www.rejinpaul.com
GENERAL NEURO HYBRID SYSTEMS

om
Definition:
neuro-genetic hybrid or a genetic_neuro-hybrid system is one in which

.c
a neural network employs a genetic algorithm to optimize its structural
parameters that define its architecture.

ul
pa
Properties of genetic neuro-hybrid systems:
1. The parameters of neural networks are encoded by genetic
algorithms as a string of properties of the network, that is,
jin
chromosomes. A large population of chromosomes is generated,
which represent the many possible parameter sets for the given
.re
neural network.
w

2. Genetic Algorithm _Neural Network, or GANN, has the ability to

locate the neighborhood of the optimal solution quickly, compared
w

to other conventional search strategies.

w
 www.rejinpaul.com
BLOCK DIAGRAM OF GENETIC NEURO HYBRID

om
SYSTEMS

.c
ul
pa
jin
.re
w
w
w
 www.rejinpaul.com
ADVANTAGES OF GENETIC NEURO HYBRID

om
SYSTEMS

The various advantages of neuro-genetic hybrid are as follows:

.c
ul
 GA performs optimization of neural network parameters with
simplicity, ease of operation, minimal requirements and global

pa
perspective.


jin
GA helps to find out complex structure of ANN for given input and
the output data set by using its learning rule as a fitness function.
.re
 Hybrid approach ensembles a powerful model that could
significantly improve the predictability of the system under
w

construction.
w
w
 www.rejinpaul.com
GENETIC FUZZY HYBRID SYSTEMS
The hybridization of genetic algorithm and fuzzy logic can be performed

om
in the following two ways:
1. By the use of fuzzy logic based techniques for improving genetic
algorithm behavior and modeling GA components. This is called fuzzy

.c
genetic algorithms (FGAs).

ul
2. By the application of genetic algorithms in various optimization and
search problems involving fuzzy systems.

pa
ADVANTAGES OF GENETIC FUZZY HYBRID SYSTEMS

jin
 GAs allow us to represent different kinds of structures, such as
weights, features together with rule parameters, etc., allowing us to
.re
code multiple models of knowledge representation. This provides a
wide variety of approaches where it is necessary to design specific
genetic components for evolving a specific representation.
w

 Genetic algorithm efficiently optimizes the rules, membership

w

functions, DB and KB of fuzzy systems. The methodology adopted is

simple and the fittest individual is identified during the process.
w
 www.rejinpaul.com
SOFT COMPUTING APPLICATIONS

om
 A Fusion Approach of Multispectral Images with SAR (Synthetic
Aperture Radar) Image for Flood Area Analysis.

.c
ul
 Optimization of Traveling Salesman Problem using Genetic Algorithm
Approach.

pa
 Genetic Algorithm based Internet Search Technique.

jin
 Soft Computing Based Hybrid Fuzzy Controllers.
.re
 Soft Computing Based Rocket Engine Control.
w
w
w
 www.rejinpaul.com
Soft Computing …

om
.c
ul
pa
jin
.re
w
w
w