Hybrid intelligent systems:

Neuro-fuzzy systems
 Introduction
 Neuro-fuzzy systems
 ANFIS: Adaptive Neuro-Fuzzy Inference
System
 Summary

 Negnevitsky, Pearson Education, 2011 1

 Introduction
 A hybrid intelligent system is one that combines
at least two intelligent technologies. For example,
combining a neural network with a fuzzy system
results in a hybrid neuro-fuzzy system.
 The combination of probabilistic reasoning, fuzzy
logic, neural networks and evolutionary
computation forms the core of soft computing, an
emerging approach to building hybrid intelligent
systems capable of reasoning and learning in an
uncertain and imprecise environment.

 Negnevitsky, Pearson Education, 2011 2

  Although words are less precise than numbers,
precision carries a high cost. We use words
when there is a tolerance for imprecision. Soft
computing exploits the tolerance for uncertainty
and imprecision to achieve greater tractability
and robustness, and lower the cost of solutions.
 We also use words when the available data is
not precise enough to use numbers. This is
often the case with complex problems, and
while “hard” computing fails to produce any
solution, soft computing is still capable of
finding good solutions.

 Negnevitsky, Pearson Education, 2011 3

  Lotfi Zadeh is reputed to have said that a good
hybrid would be “British Police, German
Mechanics, French Cuisine, Swiss Banking and
Italian Love”. But “British Cuisine, German
Police, French Mechanics, Italian Banking and
Swiss Love” would be a bad one. Likewise, a
hybrid intelligent system can be good or bad – it
depends on which components constitute the
hybrid. So our goal is to select the right
components for building a good hybrid system.

 Negnevitsky, Pearson Education, 2011 4

 Comparison of Fuzzy Systems,
Neural Networks and Genetic Algorithms
FS NN GA
Knowledge representation
Uncertainty tolerance
Imprecision tolerance
Adaptability
Learning ability
Explanation ability
Knowledge discovery and data mining
Maintainability
* The terms used for grading are:
- bad, - rather bad, - rather good and - good

 Negnevitsky, Pearson Education, 2011 5

 Neuro-fuzzy systems
 Fuzzy logic and neural networks are natural
complementary tools in building intelligent
systems. While neural networks are low-level
computational structures that perform well when
dealing with raw data, fuzzy logic deals with
reasoning on a higher level, using linguistic
information acquired from domain experts.
However, fuzzy systems lack the ability to learn
and cannot adjust themselves to a new
environment. On the other hand, although neural
networks can learn, they are opaque to the user.

 Negnevitsky, Pearson Education, 2011 6

  Integrated neuro-fuzzy systems can combine the
parallel computation and learning abilities of
neural networks with the human-like knowledge
representation and explanation abilities of fuzzy
systems. As a result, neural networks become
more transparent, while fuzzy systems become
capable of learning.

 Negnevitsky, Pearson Education, 2011 7

  A neuro-fuzzy system is a neural network which
is functionally equivalent to a fuzzy inference
model. It can be trained to develop IF-THEN
fuzzy rules and determine membership functions
for input and output variables of the system.
Expert knowledge can be incorporated into the
structure of the neuro-fuzzy system. At the same
time, the connectionist structure avoids fuzzy
inference, which entails a substantial
computational burden.

 Negnevitsky, Pearson Education, 2011 8

  The structure of a neuro-fuzzy system is similar
to a multi-layer neural network. In general, a
neuro-fuzzy system has input and output layers,
and three hidden layers that represent
membership functions and fuzzy rules.

 Negnevitsky, Pearson Education, 2011 9

 Neuro-fuzzy system
Layer 1 Layer 2 Layer 3 Layer 4 Layer 5

A1 A1 R1
x1 
R1

x1 x1
A2  R2
x1 A2 
R2
wR3 C1 
C1
A3 A3 R3  wR6
R3
y

wR1
  wR2
B1 B1 R4 R4

x2 wR4 C2
 C2
x2 x2  R5 wR5
B2
B2 R5
x2 
R6

B3
B3 R6

 Negnevitsky, Pearson Education, 2011 10

 Each layer in the neuro-fuzzy system is associated
with a particular step in the fuzzy inference process.
Layer 1 is the input layer. Each neuron in this layer
transmits external crisp signals directly to the next
layer. That is,
yi(1)  xi(1)
Layer 2 is the fuzzification layer. Neurons in this
layer represent fuzzy sets used in the antecedents
of fuzzy rules. A fuzzification neuron receives a
crisp input and determines the degree to which
this input belongs to the neuron’s fuzzy set.
 Negnevitsky, Pearson Education, 2011 11
 The activation function of a membership neuron is
set to the function that specifies the neuron’s fuzzy
set. We use triangular sets, and therefore, the
activation functions for the neurons in Layer 2 are
set to the triangular membership functions. A
triangular membership function can be specified by
two parameters {a, b} as follows:
 ( 2) b
 0, if xi a 
2

 2 xi( 2)  a
 b b
yi( 2) 1  ( 2)
, if a   xi  a 
 b 2 2
 ( 2) b
 0, if xi a 
2

 Negnevitsky, Pearson Education, 2011 12
 Triangular activation functions
 
1 1
a = 4, b =6 a = 4.5, b =6 a = 4, b =6
0.8 0.8
a = 4, b =4
0.6 0.6

0.4 0.4

0.2 0.2

0
0 1 2 3 4 5 6 7 8
X 0
0 1 2 3 4 5 6 7 8
X

(a) Effect of parameter a. (b) Effect of parameter b.

 Negnevitsky, Pearson Education, 2011 13

 Layer 3 is the fuzzy rule layer. Each neuron in this
layer corresponds to a single fuzzy rule. A fuzzy
rule neuron receives inputs from the fuzzification
neurons that represent fuzzy sets in the rule
antecedents. For instance, neuron R1, which
corresponds to Rule 1, receives inputs from
neurons A1 and B1.
In a neuro-fuzzy system, intersection can be
implemented by the product operator. Thus, the
output of neuron i in Layer 3 is obtained as:

yi(3)  x1(i3) x2(3i )  xki

(3) y R(31)  A1  B1  R1

 Negnevitsky, Pearson Education, 2011 14

 Layer 4 is the output membership layer. Neurons
in this layer represent fuzzy sets used in the
consequent of fuzzy rules.
An output membership neuron combines all its
inputs by using the fuzzy operation union.
This operation can be implemented by the
probabilistic OR. That is,
( 4)
yi( 4)  x1(i4)  x2( 4i )    xli( 4) yC1  R3   R 6 C1

The value of C1 represents the integrated firing

strength of fuzzy rule neurons R3 and R6.

 Negnevitsky, Pearson Education, 2011 15

 Layer 5 is the defuzzification layer. Each neuron
in this layer represents a single output of the
neuro-fuzzy system. It takes the output fuzzy sets
clipped by the respective integrated firing
strengths and combines them into a single fuzzy
set.
Neuro-fuzzy systems can apply standard
defuzzification methods, including the centroid
technique.
We will use the sum-product composition
method.
 Negnevitsky, Pearson Education, 2011 16
 The sum-product composition calculates the crisp
output as the weighted average of the centroids of
all output membership functions. For example, the
weighted average of the centroids of the clipped
fuzzy sets C1 and C2 is calculated as,

C1 aC1 bC1  C 2 aC 2 bC 2

y
C1 bC1  C 2 bC 2

 Negnevitsky, Pearson Education, 2011 17

 How does a neuro-fuzzy system learn?
A neuro-fuzzy system is essentially a multi-layer
neural network, and thus it can apply standard
learning algorithms developed for neural networks,
including the back-propagation algorithm.

 Negnevitsky, Pearson Education, 2011 18

  When a training input-output example is presented
to the system, the back-propagation algorithm
computes the system output and compares it with
the desired output of the training example. The
error is propagated backwards through the network
from the output layer to the input layer. The
neuron activation functions are modified as the
error is propagated. To determine the necessary
modifications, the back-propagation algorithm
differentiates the activation functions of the
neurons.
 Let us demonstrate how a neuro-fuzzy system
works on a simple example.
 Negnevitsky, Pearson Education, 2011 19
 Training patterns

0
1

0
0

 Negnevitsky, Pearson Education, 2011 20

 The data set is used for training the five-rule neuro-
fuzzy system shown below.
Five-rule neuro-fuzzy system
1
S 1 wR1
x1 0.99 wR5
0.8
L 2 0
S y 0.6
wR3 wR4

Weight
3 0.72 
L 0.4
0.61
S 4
x2 0.2 wR2
0.79
L 5 0
0 10 20 30 40 50
Epoch

(a) Five-rule system. (b) Training for 50 epochs.

 Negnevitsky, Pearson Education, 2011 21

  Suppose that fuzzy IF-THEN rules incorporated
into the system structure are supplied by a
domain expert. Prior or existing knowledge can
dramatically expedite the system training.
 Besides, if the quality of training data is poor,
the expert knowledge may be the only way to
come to a solution at all. However, experts do
occasionally make mistakes, and thus some rules
used in a neuro-fuzzy system may be false or
redundant. Therefore, a neuro-fuzzy system
should also be capable of identifying bad rules.

 Negnevitsky, Pearson Education, 2011 22

  Given input and output linguistic values, a neuro-
fuzzy system can automatically generate a complete
set of fuzzy IF-THEN rules.
 Let us create the system for the XOR example.
This system consists of 22  2 = 8 rules. Because
expert knowledge is not embodied in the system
this time, we set all initial weights between Layer 3
and Layer 4 to 0.5.
 After training we can eliminate all rules whose
certainty factors are less than some sufficiently
small number, say 0.1. As a result, we obtain the
set of four fuzzy IF-THEN rules that represents the
inverse of the XOR operation.
 Negnevitsky, Pearson Education, 2011 23
 Eight-rule neuro-fuzzy system
1 0.8
S 0 wR2 wR8
x1 2
0.7
0.78
0.6 wR3 wR5
L 3 0.69 S
0.5
4 y
0  0.4
5 0.62
0.3 wR6 & wR7
S 6 0 L
0.2
x2 0 wR1
7 0.1
0.80 wR4
L 8 0
0 10 20 30 40 50
Epoch

(a) Eight-rule system. (b) Training for 50 epochs.

 Negnevitsky, Pearson Education, 2011 24

 Neuro-fuzzy systems: summary
 The combination of fuzzy logic and neural
networks constitutes a powerful means for
designing intelligent systems.
 Domain knowledge can be put into a neuro-fuzzy
system by human experts in the form of linguistic
variables and fuzzy rules.
 When a representative set of examples is available,
a neuro-fuzzy system can automatically transform
it into a robust set of fuzzy IF-THEN rules, and
thereby reduce our dependence on expert
knowledge when building intelligent systems.
 Negnevitsky, Pearson Education, 2011 25
 ANFIS:
Adaptive Neuro-Fuzzy Inference System
The Sugeno fuzzy model was proposed for generating
fuzzy rules from a given input-output data set. A typical
Sugeno fuzzy rule is expressed in the following form:
IF x1 is A1
AND x2 is A2
. . . . .

AND xm is Am
THEN y = f (x1, x2, . . . , xm)
where x1, x2, . . . , xm are input variables; A1, A2, . . . , Am
are fuzzy
 Negnevitsky, sets.
Pearson Education, 2011 26
  When y is a constant, we obtain a zero-order
Sugeno fuzzy model in which the consequent of
a rule is specified by a singleton.
 When y is a first-order polynomial, i.e.
y = k0 + k1 x1 + k 2 x2 + . . . + km xm
we obtain a first-order Sugeno fuzzy model.

 Negnevitsky, Pearson Education, 2011 27

 Adaptive Neuro-Fuzzy Inference System
Layer 1 Layer 2 Layer 3 Layer 4 Layer 5 Layer 6
x1 x2
A1 1 N1 1
x1

A2 2 N2 2
y

B1 3 N3 3
x2

B2 4 N4 4

 Negnevitsky, Pearson Education, 2011 28

 Layer 1 is the input layer. Neurons in this layer
simply pass external crisp signals to Layer 2.

Layer 2 is the fuzzification layer. Neurons in this

layer perform fuzzification. In Jang’s model,
fuzzification neurons have a bell activation
function.

 Negnevitsky, Pearson Education, 2011 29

 Layer 3 is the rule layer. Each neuron in this layer
corresponds to a single Sugeno-type fuzzy rule.
A rule neuron receives inputs from the respective
fuzzification neurons and calculates the firing
strength of the rule it represents. In an ANFIS,
the conjunction of the rule antecedents is
evaluated by the operator product. Thus, the
output of neuron i in Layer 3 is obtained as,
k
yi(3)  x (ji3) y(3) =   =  ,
1 A1 B1 1
j 1

where the value of 1 represents the firing

strength, or the truth value, of Rule 1.
 Negnevitsky, Pearson Education, 2011 30
 Layer 4 is the normalisation layer. Each neuron in
this layer receives inputs from all neurons in the
rule layer, and calculates the normalised firing
strength of a given rule.
The normalised firing strength is the ratio of the
firing strength of a given rule to the sum of firing
strengths of all rules. It represents the contribution
of a given rule to the final result. Thus, the
output of neuron i in Layer 4 is determined as,
xii( 4) i ( 4) 1
yi( 4)   i y N1  1
n n 1   2  3   4
( 4)
 ji
x  j
j 1 j 1
 Negnevitsky, Pearson Education, 2011 31
 Layer 5 is the defuzzification layer. Each neuron
in this layer is connected to the respective
normalisation neuron, and also receives initial
inputs, x1 and x2. A defuzzification neuron
calculates the weighted consequent value of a
given rule as,
yi(5)  xi(5) ki 0  ki1 x1  ki 2 x 2 i ki 0  ki1 x1  ki 2 x 2

where is the input and is the output of

defuzzification neuron i in Layer 5, and ki0, ki1 and
ki2 is a set of consequent parameters of rule i.

 Negnevitsky, Pearson Education, 2011 32

 Layer 6 is represented by a single summation
neuron. This neuron calculates the sum of
outputs of all defuzzification neurons and
produces the overall ANFIS output, y,
n n
y  xi(6)  i ki 0  ki1 x1  ki 2 x 2
i 1 i 1

 Negnevitsky, Pearson Education, 2011 33

 Can an ANFIS deal with problems where we
do not have any prior knowledge of the rule
consequent parameters?

It is not necessary to have any prior knowledge of

rule consequent parameters. An ANFIS learns
these parameters and tunes membership functions.

 Negnevitsky, Pearson Education, 2011 34

 Learning in the ANFIS model
 An ANFIS uses a hybrid learning algorithm that
combines the least-squares estimator and the
gradient descent method.
 In the ANFIS training algorithm, each epoch is
composed from a forward pass and a backward
pass. In the forward pass, a training set of input
patterns (an input vector) is presented to the
ANFIS, neuron outputs are calculated on the layer-
by-layer basis, and rule consequent parameters are
identified.

 Negnevitsky, Pearson Education, 2011 35

  The rule consequent parameters are identified by
the least-squares estimator. In the Sugeno-style
fuzzy inference, an output, y, is a linear function.
Thus, given the values of the membership
parameters and a training set of P input-output
patterns, we can form P linear equations in terms
of the consequent parameters as:
 y (1)  (1) f(1)  (1) f(1)  
 d   n(1) fn(1)
 y (2)  (2) f(2)  (2) f(2)  
 d   n(2) fn(2)
 
 yd (p)  (p) f(p)  (p) f(p)    (p) f (p)
  n n
 
   (P) f (P)
 yd (P)  (P) f(P)  (P) f(P)   n n
 Negnevitsky, Pearson Education, 2011 36
 In the matrix notation, we have
yd = A k,
where yd is a P  1 desired output vector,
 yd (1)   (1) (1) x(1)  (1) x (1)   (1)  (1) x(1)   (1) x (1) 
   m n n n m 
 yd (2)   (2) (1) x(2)  (2) x (2)   (2)  (2) x(2)   (2) x (2) 
 m n n n m 
    
yd    A            
 (p)  (p) x(p)   (p) x (p) 
y (p)   (p) (p) x(p) (p) xm(p) 
 d    
n n

n m

          

 (P) (P) x(P) (P) xm(P)  (P)  (P) x(P)  (P) x (P)
 yd (P) n n n m

and k is an n (1 + m)  1 vector of unknown consequent

parameters,
k = [k10 k11 k12 … k1m k20 k21 k22 … k2m … kn0 kn1 kn2 … kn m]T
 Negnevitsky, Pearson Education, 2011 37
  As soon as the rule consequent parameters are
established, we compute an actual network output
vector, y, and determine the error vector, e
e = yd  y
 In the backward pass, the back-propagation
algorithm is applied. The error signals are
propagated back, and the antecedent parameters
are updated according to the chain rule.

 Negnevitsky, Pearson Education, 2011 38

 In the ANFIS training algorithm suggested by
Jang, both antecedent parameters and
consequent parameters are optimised. In the
forward pass, the consequent parameters are
adjusted while the antecedent parameters
remain fixed. In the backward pass, the
antecedent parameters are tuned while the
consequent parameters are kept fixed.

 Negnevitsky, Pearson Education, 2011 39

 Function approximation using the ANFIS model
 In this example, an ANFIS is used to follow a
trajectory of the non-linear function defined by
the equation
cos(2 x1)
y
x2
e
 First, we choose an appropriate architecture for
the ANFIS. An ANFIS must have two inputs – x1
and x2 – and one output – y.
 Thus, in our example, the ANFIS is defined by
four rules, and has the structure shown below.

 Negnevitsky, Pearson Education, 2011 40

 An ANFIS model with four rules
Layer 1 Layer 2 Layer 3 Layer 4 Layer 5 Layer 6
x1 x2
A1 1 N1 1
x1

A2 2 N2 2
y

B1 3 N3 3
x2

B2 4 N4 4

 Negnevitsky, Pearson Education, 2011 41

  The ANFIS training data includes 101 training
samples. They are represented by a 101  3
matrix [x1 x2 yd], where x1 and x2 are input
vectors, and yd is a desired output vector.
 The first input vector, x1, starts at 0, increments
by 0.1 and ends at 10.
 The second input vector, x2, is created by taking
sin from each element of vector x1, with elements
of the desired output vector, yd, determined by the
function equation.

 Negnevitsky, Pearson Education, 2011 42

 Learning in an ANFIS with two membership
functions assigned to each input (one epoch)
y
Training Data
2 ANFIS Output

-1

-2

-3
1
0.5 10
8
0 6
-0.5 4
2
-1 0
x2 x1

 Negnevitsky, Pearson Education, 2011 43

 Learning in an ANFIS with two membership
functions assigned to each input (100 epochs)
y
Training Data
2 ANFIS Output

-1

-2

-3
1
0.5 10
8
0 6
-0.5 4
2
-1 0
x2 x1

 Negnevitsky, Pearson Education, 2011 44

 We can achieve some improvement, but much
better results are obtained when we assign three
membership functions to each input variable. In
this case, the ANFIS model will have nine rules,
as shown in figure below.

 Negnevitsky, Pearson Education, 2011 45

 An ANFIS model with nine rules
x1 x2
A1 1 N1 1
2 N2 2
x1
A2
3 N3 3

A3 4 N4 4
y
5 N5 5 

B1 6 N6 6
7 N7 7
x2
B2
8 N8 8

B3 9 N9 9

 Negnevitsky, Pearson Education, 2011 46

 Learning in an ANFIS with three membership
functions assigned to each input (one epoch)
y
Training Data
2 ANFIS Output

-1

-2

-3
1
0.5 10
8
0 6
-0.5 4
2
-1 0
x2 x1

 Negnevitsky, Pearson Education, 2011 47

 Learning in an ANFIS with three membership
functions assigned to each input (100 epochs)
y
Training Data
2 ANFIS Output

-1

-2

-3
1
0.5 10
8
0 6
-0.5 4
2
-1 0
x2 x1

 Negnevitsky, Pearson Education, 2011 48

Initial and final membership functions of the ANFIS

1 1

0 .8 0 .8

0 .6 0 .6

0 .4 0 .4

0 .2 0 .2

0 0
0 1 2 3 4 5 6 7 8 9 10 -1 -0 .8 -0 .6 -0 .4 -0 .2 0 0 .2 0 .4 0 .6 0 .8 1
x1 x2
(a ) In itia l m e m b e rs h ip fu n c tio n s .
1 1

0 .8 0 .8

0 .6 0 .6

0 .4 0 .4

0 .2 0 .2

0 0
0 1 2 3 4 5 6 7 8 9 10 -1 -0 .8 -0 .6 -0 .4 -0 .2 0 0 .2 0 .4 0 .6 0 .8 1
x1 x2
(b ) M e m b e rs h ip fu n c tio n s a fte r 1 0 0 e p o c h s o f tra in in g .

 Negnevitsky, Pearson Education, 2011 49