Introduction

• ANFIS: Artificial Neuro-Fuzzy Inference Systems

• ANFIS are a class of adaptive networks that are
funcionally equivalent to fuzzy inference systems.
• ANFIS represent Sugeno e Tsukamoto fuzzy
models.
• ANFIS uses a hybrid learning algorithm

Fuzzy Logic-ANFIS – p. 3/53

Sugeno Model

• Assume that the fuzzy inference system has two

inputs x and y and one output z .
• A first-order Sugeno fuzzy model has rules as the
following:
• Rule1:
If x is A1 and y is B1 , then f1 = p1 x + q1 y + r1
• Rule2:
If x is A2 and y is B2 , then f2 = p2 x + q2 y + r2

Fuzzy Logic-ANFIS – p. 4/53

Sugeno Model - I

A1 B1

W1

X Y

A2 B2

W2
X Y
x y
w1.f1+w2.f2
f1=p1x+q1y+r1 f2=p2x+q2y+r2 f=
w1+w2

Fuzzy Logic-ANFIS – p. 5/53

 ANFIS Architecture

Layer1 Layer2 Layer3 Layer4 Layer5

x y
A1
x W1 W1f1
Prod Norm
f
A2 Sum
W2
Prod Norm W1f2

B1 x y

y
B2

Fuzzy Logic-ANFIS – p. 6/53

Layer 1 - I

• Ol,i is the output of the ith node of the layer l.

• Every node i in this layer is an adaptive node with
a node function
O1,i = µAi (x) for i = 1, 2, or
O1,i = µBi−2 (x) for i = 3, 4
• x (or y ) is the input node i and Ai (or Bi−2 ) is a
linguistic label associated with this node
• Therefore O1,i is the membership grade of a fuzzy
set (A1 , A2 , B1 , B2 ).

Fuzzy Logic-ANFIS – p. 7/53

Layer 1 - II

• Typical membership function:

1
µA (x) =
1 + | x−c i 2bi
ai |

• ai , bi , ci is the parameter set.

• Parameters are referred to as premise parameters.

Fuzzy Logic-ANFIS – p. 8/53

Layer 2

• Every node in this layer is a fixed node labeled

Prod.
• The output is the product of all the incoming
signals.
• O2,i = wi = µAi (x) · µBi (y), i = 1, 2

• Each node represents the fire strength of the rule

• Any other T-norm operator that perform the AN D
operator can be used

Fuzzy Logic-ANFIS – p. 9/53

Layer 3

• Every node in this layer is a fixed node labeled

Norm.
• The ith node calculates the ratio of the ith rulet’s
firing strenght to the sum of all rulet’s firing
strengths.
• O3,i = w i = wi , i = 1, 2
w1 +w2
• Outputs are called normalized firing strengths.

Fuzzy Logic-ANFIS – p. 10/53

Layer 4

• Every node i in this layer is an adaptive node with

a node function:
O4,1 = wi fi = w i (px + qi y + ri )

• wi is the normalized firing strenght from layer 3.

• {pi , qi , ri } is the parameter set of this node.
• These are referred to as consequent parameters.

Fuzzy Logic-ANFIS – p. 11/53

Layer 5

• The single node in this layer is a fixed node

labeled sum, which computes the overall output as
the summation of all incoming signals:
P P
• overall output = O5,1 = i wi fi
P
i w i fi = wi
i

Fuzzy Logic-ANFIS – p. 12/53

 Alternative Structures

• There are other structures

Layer1 Layer2 Layer3 Layer4 Layer5

x y
A1
x W1 W1f1
Prod
W1f1+W2f2
f
A2 Sum /
W2
Prod W1f2

B1 x y

y
B2 Sum

Fuzzy Logic-ANFIS – p. 13/53

Learning Algorithm

Fuzzy Logic-ANFIS – p. 14/53

Hybrid Learning Algorithm - I

• The ANFIS can be trained by a hybrid learning

algorithm presented by Jang in the chapter 8 of
the book.
• In the forward pass the algorithm uses
least-squares method to identify the consequent
parameters on the layer 4.
• In the backward pass the errors are propagated
backward and the premise parameters are
updated by gradient descent.

Fuzzy Logic-ANFIS – p. 15/53

Hybrid Learning Algorithm - II

Forward Pass Backward Pass

Premise Parameters Fixed Gradient Descent
Consequent Parameters Least-squares estimator Fixed
Signals Node outputs Error signals

Two passes in the hybrid learning algorithm for ANFIS.

Fuzzy Logic-ANFIS – p. 16/53

Basic Learning Rule Definitions
• Suppose that an adptive network has L layers and
the k th layer has #(k ) nodes.
• We can denote the node in the ith position of the
k th layer by (k, i).
• The node function is denoted by Oik .
• Since the node output depends on its incoming
signals and its parameter set (a, b, c), we have

Oik = Oik (Oik−1 , . . . , O#(k−1)

k−1
, a, b, c)

• Notice that Oik is used as both node output and

node function.

Fuzzy Logic-ANFIS – p. 17/53

Error Measure
• Assume that a training data set has P entries.
• The error measure for the pth entry can be defined as the sum of
the squared error

X
#(L)
Ep = L
(Tm,p − Om,p )2
m=1

• Tm,p is the mth component of the pth target.

L
• Om,p is the mth component the actual output vector.
• The overall error is
X
P
E= Ep
p=1

Fuzzy Logic-ANFIS – p. 18/53

Error Rate for each output
• In order to implement the gradient descent in E we calculate the
∂E
error rate ∂O for the pth training data for each node output O.
• The error rate for the output note at (L, i) is

∂Ep L
L
= −2(T i,p − Oi,p ) (1)
∂Oi,p

• For the internal node at (k, i), the error rate can be derived by the
chain rule:
∂Ep X ∂Ep ∂Om,p
#(k+1) k+1

k
= k+1 ∂O k
, (2)
∂Oi,p m=1
∂Om,p i,p

where 1 ≤ k ≤ L − 1
• The error rate of an internal node is a linear combination of the
error rates of the nodes in the next layer.
Fuzzy Logic-ANFIS – p. 19/53
Error Rate for each parameter
• Consider α one of the parameters.
• Therefore
∂Ep X ∂Ep ∂O∗
= , (3)
∂α ∂O ∗ ∂α
∗ O ∈S

where S is the set of nodes whose outputs depend on α

• The derivative of the overall error with respect to α is

∂E X ∂Ep
P
= , (4)
∂α p=1
∂α

• The update formula for α is

∂E
∆α = η
∂α

Fuzzy Logic-ANFIS – p. 20/53

Learning Paradigms

• If the parameters are to be updated after each

input-output pair (on-line training) then the update
formula is:
∂Ep X ∂Ep ∂O∗
= (5)
∂α ∂O ∗ ∂α
O∗ ∈S
• With the batch learning (off-line learning) the
update formula is based on the derivative of the
overall error with respect to α:

∂E X
P
∂Ep
= , (6)
∂α ∂α
p=1

Fuzzy Logic-ANFIS – p. 21/53

Gradient Problems

• The method is slow.

• It is likely to be trapped in local minima.

Fuzzy Logic-ANFIS – p. 22/53

Hybrid Learning Rule

Fuzzy Logic-ANFIS – p. 23/53

Hybrid Learning Rule

• Combines:
• the gradient rule;
• the least squares estimate.

Fuzzy Logic-ANFIS – p. 24/53

Definitions

• Considere that the adptive network has only one output.

• output = F (I, S)
• I is the vector of input variables.
• S is the set of parameters.
• F is the function implemented by the ANFIS.
• If there exists a function H such that the composite function H ◦ F
is linear in some elements of S then these elements can be
identified by LSM.

Fuzzy Logic-ANFIS – p. 25/53

Continuing Definitions
• More formally, if the parameter set S can be decomposed into two
sets S = S1 ⊕ S2 (⊕ direct sum), such that H ◦ F is linear in the
elements of S2
• then applying H to output = F (I, S) we have

H(output) = H ◦ F (I, S) (7)

which is linear in the elements of S2 .

• Given values of elements of S1 , it is possible to plug P training
data in equation 7.
• As a result we obtain a matrix equation Aθ = y where θ is the
unknown vector whose elements are parameters in S2 .
• This is the standard linear least-square problem.

Fuzzy Logic-ANFIS – p. 26/53

Combining LSE and gradient descent
- forward pass

• In batch mode, each epoch is composed of a

forward pass and a backward pass.
• In the forward pass an input vector is presented
and the output is calculated creating a row in the
matrices A and y.
• The process is repeated for all training data and
the parameters S2 are identified by BLS or RLS.
• After S2 is identified the error for each pair is
computed.

Fuzzy Logic-ANFIS – p. 27/53

Combining LSE and gradient descent
- backward pass

• The derivative of the error measure with respect to each node

output propagate from the output toward the input.
• The derivatives are:
∂Ep L
L
= −2(Ti,p − Oi,p )
∂Oi,p

∂Ep X
#(k+1) k+1
∂Ep ∂Om,p
k
= k+1 ∂O k
∂Oi,p m=1
∂Om,p i,p

• The parameters in S2 are updated by the gradient method

∂E
∆α = −η
∂α

Fuzzy Logic-ANFIS – p. 28/53

Universal Aproximator

Fuzzy Logic-ANFIS – p. 29/53

ANFIS is a Universal Aproximator

• When the number of rules is not restricted, a

zero-order Sugeno model has unlimited
approximation power for matching well any
nonlinear function arbitrarily on a compact set.
• This can be proved using the Stone-Weierstrass
theorem.
• Let domain D be a compact space of N
dimensions, and let F be a set of continuous
real-valued functions on D satisfying the following
criteria:

Fuzzy Logic-ANFIS – p. 30/53

 Stone-Weierstrauss theorem - I

Indentity function: The constant f (x) = 1 is in F .

Separability:For any two points x1 6= x2 in D, there is an
f in F such that f (x1 ) 6= f (x2 ).
Algebraic closure:If f and g are any two functions in F ,
then f g and af + bg are in F for any two real
numbers a and b.

Fuzzy Logic-ANFIS – p. 31/53

Stone-Weierstrauss theorem - II

• Then F is dense on C(D), the set of continuous

real-valued functions on D.
• For any ǫ > 0 and any function g in C(D), there is a
function f in F such that |g(x) − f (x)| < ǫ for all
x ∈ D.
• The ANFIS satisfies all these requirements.

Fuzzy Logic-ANFIS – p. 32/53

Stone-Weierstrauss theorem - III

• In applications of fuzzy inference systems, the

domain is almost always compact.
• It is possible, applying this theorem to prove the
universal approximation power of the zero-order
Sugeno model.

Fuzzy Logic-ANFIS – p. 33/53

Indentity Function

• Indentity function: The constant f (x) = 1 is in F .

• The first hypothesis requires that our fuzzy
inference system be able to compute the identity
function f (x) = 1.
• An obvious solution is to set the consequence part
of each rule equal to one.
• A fuzzy inference system with only one rule is able
to compute the identity function.

Fuzzy Logic-ANFIS – p. 34/53

Separability

• Separability: For any two points x1 6= x2 in D,

there is an f in F such that f (x1 ) 6= f (x2 ).
• The second hypothesis requires that our fuzzy
inference system be able to compute functions that
have different values for different points.
• This is achievable by any fuzzy inference system
with appropriate parameters.

Fuzzy Logic-ANFIS – p. 35/53

Algebraic Closure - Addition I

• Algebraic closure addition: If f and g are any

two functions in F , then af + bg are in F for any two
real numbers a and b.
• Suppose that we have two fuzzy inference
systems S and Ŝ ; each of them has two rules.
• The final output of each system is specified as
w 1 f1 + w 2 f2
S:z=
w1 + w2
ŵ1 f1 + ŵ2 f2
Ŝ : ẑ =
ŵ1 + ŵ2

Fuzzy Logic-ANFIS – p. 36/53

 Algebraic Closure - Addition II

• sum of z and ẑ is:

w1 f1 + w2 f2 ŵ1 f1 + ŵ2 f2
az + bẑ = a +b
w1 + w2 ŵ1 + ŵ2
w1 ŵ1 (af1 + bfˆ1 ) + w1 ŵ2 (af1 + bfˆ2 ) + w2 ŵ1 (af2 + bfˆ1 ) + w2 ŵ2 (af2 + bfˆ2 )
=
w1 ŵ1 + w1 ŵ2 + w2 ŵ1 + w2 ŵ2

Fuzzy Logic-ANFIS – p. 37/53

Algebraic Closure - Addition III

• Therefore, it is possible to construct a four-rule

inference system that computes az + bẑ .
• The firing strength of each rule is given by
wi ŵj (i, j = 1 or 2)
• The output of each rule is given by
afi + bfˆj (i, j = 1 or 2)

Fuzzy Logic-ANFIS – p. 38/53

Algebraic Closure - Multiplication I

• Algebraic closure multiplication: If f and g are

any two functions in F , then f g are in F .
• product of z and ẑ is:

w1 ŵ1 f1 fˆ1 + w1 ŵ2 f1 fˆ2 + w2 ŵ1 f2 fˆ1 + w2 ŵ2 f2 fˆ2

z ẑ =
w1 ŵ1 + w1 ŵ2 + w2 ŵ1 + w2 ŵ2
• Therefore, it is possible to construct a four-rule
inference system that computes z ẑ .
• The firing strength and output of each rule is
defined by wi ŵj and fi fˆj (i, j = 1 or 2) respectively.

Fuzzy Logic-ANFIS – p. 39/53

Conclusion

• ANFIS architectures that compute z ẑ and az + bẑ

are of the same class as those of S and Ŝ if and
only if the membership functions used are
invariant under multiplication.
• The Gaussian membership functions satisfy this
property.
x−ci 2
• µA = ki e [−( ai
) ]
i

Fuzzy Logic-ANFIS – p. 40/53