Benchmark Definition File, ver.
17March2002
Page 1 of 20
Benchmark Definition
1.
VERSION INFO
2.
INTRODUCTION
3.
STEP I
4.
STEP II
5.
STEP III
APPENDIX I GLOSSARY
APPENDIX II PERFORMANCE INDEXES
APPENDIX III FDI METHOD FORM S1-MF
11
APPENDIX IV FAULT TEST FORM S1-FF-FXX
12
APPENDIX V EXAMPLE OF ADDITIONAL APPENDIX FOR S1-FF-FXX FORM
14
APPENDIX VI SCENARIO FORM OF STEP II ... S2-SF-X
15
APPENDIX VII SCENARIO FORM OF STEP III ... S3-SF-X
16
APPENDIX VIII ACTUATOR DESCRIPTION
17
APPENDIX IX EXAMPLES OF ACTUATORS IN REAL PROCESSES
20
WUT Team
with cooperation with UPC
Preliminary, ver. 1.0
March 17, 2002
Benchmark Definition File, ver. 17March2002
Page 2 of 20
1. Version Info
version #
draft
1.0
preliminary
Remarks
Date
02-March-2002
Changes in performance indexes
definitions
Introduction of new performance indexes
Updates in glossary
Changes in the forms due to the
changes in performance indexes
17-March-2002
Benchmark Definition File, ver. 17March2002
Page 3 of 20
2. Introduction
Benchmark scope
The benchmark concerns on actuator consisting of: control valve, pneumatic linear servo motor
and positioner.
The general description of actuator is given in Appendix VIII. More detailed description can be
found in document Specification of Actuators Intended to Use for Benchmark Definition.
This document can be downloaded from DAMADICS Information Website, section Benchmark /
Introduction.
Objectives
Comparing the properties of fault detection and isolation methods intended to be applied for
actuator diagnosis in industrial environment.
Assumptions
1.
Benchmark is based on:
actuator Simulink-Matlab model,
real process data files with artificial generated faults.
2.
Analytical model of actuator is not available. This is typical situation when considering
industrial implementation. Available are real process data files.
3.
Due to unlimited number of possible fault scenarios the important limitations were introduced:
number of considered actuator faults is equal to 19,
two fault simulation scenarios are assumed: abrupt and incipient,
only single fault scenarios are considered and simulated.
4.
Benchmark structure:
Step I. Benchmark basing on Simulink model. Step I is suited for evaluation a
features of FDI methods. Fault scenarios are well defined to allow evaluation.
Step II. Benchmark basing on Simulink model. Step II is suited for testing a features
of FDI methods. Data files will be generated with hidden faults. Fault scenarios will be
defined by project coordinator.
Step III. Benchmark basing on real process data. Step III is suited for approving
applicability of FDI methods.
5.
Benchmark is designed to ensure comparability of the results achieved by applying different
FDI approaches.
Benchmark Definition File, ver. 17March2002
Page 4 of 20
3. Step I
The aim of this benchmark part is investigating of partners FDI methods. Step I is based on the
data generated by data generator.
Every project partner has to show FDI results for all fault scenarios considered in Step I. In every
scenario they are defined: kind of fault (fault number) and failure mode.
In Step I:
Only two failure modes are available: abrupt and incipient. Modes are defined in document
Using Damadics Actuator Benchmark Library (DABLib) ; in section DGen functional
block / Failure modes.
Total of 44 fault scenarios must be investigated. Scenarios are given in document: Using
Damadics Actuator Benchmark Library (DABLib) in section DGen functional block /
Fault scenarios.
Step I in details
For all scenarios three time zones were fixed:
First zone, Set-up zone, was set to avoid taking into account false FDI decisions which can
occur at the beginning, e.g., some FDI methods need to be tuned properly before starting.
This zone is limited by the time ton (see Figure 1)
Second zone is a Benchmark zone. All results (figures, performance indexes, etc.) are
referring to this zone. This zone is limited by the time horizon thor:
t hor = t from + t fd + t ov
where tov is a preset period of time form the moment where the fault strength reaches its
maximum value.
Third zone, Out of interest zone, is outside of the scope of the benchmark.
Fault strength
|fs max |
Set-up
zone
Benchmark zone
ton
0
0
Out of
interest
thor
tfro m
tfd
tov
Figure 1. Definition of time parameters of a benchmark step I scenarios.
Table 1 presents values time parameters of benchmark step I scenarios.
Time
Benchmark Definition File, ver. 17March2002
Page 5 of 20
Table 1. Parameters of failure modes
Parameter
|fsmax|
ton
tfrom
tfd
short
medium
long
Fault type
Abrupt
Incipient
according to Scenario
300 s
900 s
tov
thor
short
medium
long
900 s
3600 s
84000 s
0s
600 s
1500 s
2400 s
5100 s
85500 s
For evaluating of benchmark Step I results a set of performance indexes was given (see
Appendix II). The calculated indexes should be presented on proper form.
Reporting results
To make easier the comparison of achieved results, the special booklet of forms was designed.
There are two kinds of forms:
Form presenting features of applied FDI method. This form will be further called as S1-MF
form (see Appendix III)
Forms presenting results of FDI method applied for fault scenarios foreseen in Step I. This
forms will be further called as S1-FF-fxx forms (where xx denotes fault number, see
Appendix IV)
Results of benchmark - Step I will be collected together. The results achieved will be a good base
for DAMADICS partners common paper.
4. Step II
Step II of benchmark is suited for testing FDI methods.
Four fault scenarios are foreseen, one per each fault group (see fault definitions in Using
Damadics Actuator Benchmark Library (DABLib) ; in section Act block / Simulation of
faults). These fault scenarios will be set by project coordinator. Fault scenarios will be unknown for
all partners taking part in the benchmark. Basing on the chosen scenarios data files will be
generated using Simulink data generator block. Data files will be spread among project partners
using internet.
Reporting results
To make easier the comparison of achieved results, the special form was designed. This form will
be further called as S2-SF-x forms (where x denotes scenario number, see Appendix VI).
These scenarios will be published after sending results from all partners to project coordinator.
Common evaluation of results will be made.
Benchmark Definition File, ver. 17March2002
Page 6 of 20
5. Step III
Step III of benchmark is suited for approving applicability of FDI methods.
Step III is based on data from three real actuators installed in the Lublin Sugar Factory. A general
view of actuator placement in technological installation is given in Appendix IX. More detailed can
be found in document Specification of Sugar Production Process Connected to the
Actuators Intended to Use for Benchmark Definition. This document can be downloaded from
DAMADICS Information Website, section Benchmark / Introduction.
Five fault scenarios will be chosen from those that were simulated in Lublin Sugar Factory. These
fault scenarios will be chosen by project coordinator. Fault scenarios will be unknown for all
partners taking part in the benchmark. Proper data files will be prepared and spread among project
partners using internet.
Reporting results
To make easier the comparison of achieved results, the special form was designed. This form will
be further called as S3-SF-x forms (where x denotes scenario number, see ).
These scenarios will be published after sending results from all partners to project coordinator.
Common evaluation of results will be made.
Benchmark Definition File, ver. 17March2002
Page 7 of 20
Appendix I
Glossary
Abrupt fault.
A fault where the effect develops rapidly (IFAC definition).
Incipient fault.
A fault where the effect develops slowly (IFAC definition).
Detection threshold.
Limit value of a residuals deviation from zero, so if exceeded, a fault
is declared as detected (based of IFAC Threshold definition).
Isolation threshold.
Limit value of a FI decision deviation from zero, so if exceeded, the
fault is declared as isolated (based of IFAC Threshold definition).
Elementary diagnosis.
An elementary diagnosis is a set of faults that are defined in FDI
algorithm as unisolable. The FDI diagnosis decision pointing out fault
free state and undetectable faults is treated also as elementary
diagnosis. Such a elementary diagnosis is denoted as DGN0.
OK
Fault free
f1
f2
f3
f4
f5
f6
f7
f8
Undetectable faults
Elementary diagnosis:
DGN0 = {OK., f1, f2, f3, f4, f5}
f9
f10
f11
Detectable faults
f12
f19
Unisolable faults
Unisolable faults
Unisolable faults
DGN1 ={f6, f7, f8}
DGN2 = {f9, f10, f11}
DGN3 = {f12, ..., f19}
Figure 2. Explanation of elementary diagnosis definition.
Primary Diagnosis
The elementary diagnosis which includes fault that where simulated or
was marked as simulated (for hidden faults).
Complementary Diagnosis
All elementary diagnosis except of primary one and DGN0.
Fault Free Diagnosis
The elementary diagnosis which points out fault free state and
undetectable faults.
Detection Decision DD
The binary signal that indicates the existence of a fault in the system.
Isolation Decision ID.
The isolation decision is a set of elementary diagnosis, e.g., FDI
decision: {DGN2, DGN3}.
Primary Isolation Decision PID
The binary signal that indicates primary diagnosis.
Complementary Isolation Decision CID The signal that indicates complementary diagnosis. Its
value equals the number of elementary diagnosis pointed in the
complementary diagnosis, i.e., 0...N-2 where N is the number of
elementary diagnosis.
Fault Free Isolation Decision FFID The binary signal that indicates fault free diagnosis.
Benchmark Definition File, ver. 17March2002
Page 8 of 20
Appendix II
Performance indexes
Fault strength
|fs max |
fs d
ton
thor
tfro m
Time
Out of interest
Detection decision DD
True
False
0
Time
False detection
tdt
True detection
Figure 3. Explanation of parameters used in performance indexes for FD.
Detection time tdt . Period of time from the begin of fault start-up tfrom up to the moment of the last
leading edge of DD signal.
Detection recovery time tdrt . Period of time from the end of fault simulation tto up to the moment
of the last falling edge of DD signal (used in case of data files with hidden
faults in Step II and III).
Detection moment tdm . Time of the last leading edge of DD signal, starting from the beginning of
benchmark file (used in case of data files with hidden faults in Step II and III).
Detection recovery moment tdrm . Time of the last falling edge of DD signal, starting from the
beginning of benchmark file (used in case of data files with hidden faults in
Step II and III).
False detection rate rfd .
r fd =
where
i , DD
fd
t from t on
t ifd, DD is a ith period of high DD signal value between ton to tfrom (see
red circles on Figure 3)
True detection rate rtd .
rtd =
where
i , DD
td
t hor t from
t tdi , DD is a ith period of high DD signal value between tfrom to thor (see
green circles on Figure 3)
Benchmark Definition File, ver. 17March2002
Page 9 of 20
Fault detection sensitivity factor fsd . The value of fault strength in the moment of the last
leading edge of DD signal (see Figure 3).
***************
Fault strength
|fs max |
fs i
t on
t hor
t fro m
Time
Out of interest
True isolation
Primary Isolation Decision PID
True
False
Time
0
t it
False isolation
Co mplementary Isolation Decision CID
2
1
0
Time
0
Fault Free Isolation Dec ision FFID
True
False
0
Time
Figure 4. Explanation of parameters used in performance indexes for FD.
Isolation time tit . Period of time from the begin of fault start-up tfrom up to the moment of the last
leading edge of PID signal.
Isolation recovery time tirt . Period of time from the end of fault simulation tto up to the moment of
the last falling edge of PID signal (used in case of data files with hidden faults
in Step II and III).
Isolation moment tim . Time of the last leading edge of PID signal, starting from the beginning of
benchmark file (used in case of data files with hidden faults in Step II and III).
Isolation recovery moment tirm . Time of the last falling edge of PID signal, starting from the
beginning of benchmark file (used in case of data files with hidden faults in
Step II and III).
False isolation rate rfi .
r fi =
where
i , PID
fi
t from t on
t ifi, PID is a ith period of high PID signal value between ton to tfrom (see
red circles on Figure 4).
Benchmark Definition File, ver. 17March2002
Page 10 of 20
True isolation rate rti .
rti =
where
i , PID
ti
t hor t from
t tii , PID is a ith period of high PID signal value between tfrom to thor (see
green circles on Figure 4)
Mismatch isolation rate rmi .
thor
rmi =
CID dt
ton
t hor t on
Fault isolation sensitivity factor fsi . The value of fault strength in the moment of the last leading
edge of PID signal (see Figure 4).
***************
i
Theoretical diagnosis accuracy dacct .
daccti =
1
L
where: i the number of elementary diagnosis, method dependent,
st
L - the number of faults indicated in i elementary diagnosis, for DGN0
the fault free state (OK) is also counted.
Theoretical mean diagnosis accuracy dacctm .
1
dacctm =
N
N 1
dgn
i =0
i
t
where N is a number of elementary diagnosis.
Diagnosis accuracy dacc.
dacc =
1
L
where L is a number of faults indicated in FDI decision when the permanent
true detection / isolation is achieved.
Benchmark Definition File, ver. 17March2002
Page 11 of 20
Appendix III
Form:
FDI method form S1-MF
Partner:
S1-MF
Method name:
WUT
FDI based of partial FNN for FI and F-DTS for FD
General method description:
The method is based on application of fuzzy neural networks model for residual generation purpose. The isolation of faults is
based on F-DTS method of reasoning. For more details please refer to:
Smith J. (2001), My revolutionary FDI method, UNSAFEPROCESS 2001, pp. 234-235
Appendix S1-MF_A1 If any appendix considered please construct the name according to example.
Questionnaire:
Set of considered faults:
Does the method base on models?
If YES then what kind of model and used for what?
Does the method need training data with faults?
f1, f2, f3, f4, f10, f13 and f18
Yes
FNN are used to build partial models
No
Elementary diagnosis and indexes:
Elementary
diagnosis
DGN0
DGN1
DGNN
Index
dacctm
Component
faults
OK
f1, f2, f3, f10
f13, f15, f19
Value
0.38
dgnti
1/1
1/4
1/3
Comments
denotes the fault free state
Comments
Detection
Methodology description:
Six residuals are designed. Five of then are based on five partial models (five different FNN).
please include necessary tables, figures, etc., e.g. set of residuals in a table
Fuzzy residual evaluation is used. Two achieve crisp FI decision, a detection threshold equal to 0.5 was used.
A fault is detected in any of the residuals exceeds defined threshold.
Analysis of detectability:
Seven faults are detectable: f1, f2, f3, f4, f10, f13 and f18.
Isolation
Methodology description:
Isolation is based on analysis of symptoms. The symptoms are expressed is a fuzzy terms.
please include necessary tables, figures, etc., e.g. diagnostic matrix
To elaborate FI decision a F-DTS method is used. See reference xxx.
Analysis of isolability:
Faults (f2, f3, f4) and (f13, f18) are not isolable.
Benchmark Definition File, ver. 17March2002
Page 12 of 20
Appendix IV
Form:
Fault test form S1-FF-fxx
Partner:
S1-FF-f13
Failure mode
WUT
Method name:
Incipient
FDI based of partial FNN for FI and F-DTS for FD
Fault type:
f13 Rod displacement sensor fault
Group of not isolable faults:
f13, f18
Performance indexes
Index
tdt
rfd
rtd
fsd
tit
Value
250 s
33 %
80 %
0.45
500 s
rfi
rti
25 %
45 %
5%
0.8
0.5
rmi
fsi
dacc
Comments
the moment where the permanent detection was achieved
the moment where the permanent isolation was achieved, even if other groups of faults were
distinguished
for 1800 s
Questionnaire:
to be defined in future.
...
Notes:
It is the best result that we had ever achieved in the history of WUT.
Detection
Fault detection signal plot:
From:
0s
To:
ff13
13
fsd
0
300
1
0
Comments:
No comments
600
900
1200
1500
1800
2100
2400
DD
DD
tt [s]
[s]
2400 s
Benchmark Definition File, ver. 17March2002
Page 13 of 20
Isolation
Fault isolation signal plot:
From:
fsin
0s
To:
ff13
13
0
300
600
900
1200
1500
1800
2100
2400
PID
PID (f13,
(f13, f18)
f18)
1
tt [s]
[s]
t1
t2
tin
CID
CID
1
tt [s]
[s]
t3
t4
Comments:
see Appendix S1-FF-f13_A1 If any appendix considered please construct the name according to example.
2400 s
Benchmark Definition File, ver. 17March2002
Appendix V
Page 14 of 20
Example of additional appendix for
S1-FF-fxx form
Appendix S1-FF-f13_A1
1
ff1
1
0
1
300
600
900
1200
1500
1800
2100
2400
300
600
900
1200
1500
1800
2100
2400
(f
, f ))
(f13
13, f18
18
Used FI algorithm generates fault certainty factors. To achieve crisp FI decision a isolation
threshold equal to 0.5 was chosen.
True value of Isolation Decision for all the other groups in period (t3, t4) was caused by false
isolation of f1.
etc ...
etc ...
Benchmark Definition File, ver. 17March2002
Appendix VI
Form:
Scenario form of Step II ... S2-SF-x
Partner:
S2-SF-1
Page 15 of 20
Method name:
WUT
FDI based of partial FNN for FI and F-DTS for FD
Overall FDI results:
We found that f13 or f18 was simulated.
Only one fault (or a group of not isolable faults) must be declared as a result of FDI. All other results of FDI are treated as
false and must be reported.
Performance indexes
Index
tdm
tdrm
tim
tirm
...
Value
2300 s
2400 s
-
Comments
No fault decay was detected
No fault decay was detected
Questionnaire:
to be defined in future.
...
Notes:
None.
Detection
Fault detection signal plot:
From:
0s
To:
3400 s
To:
3400 s
DD
DD
tt [s]
[s]
0
1300
1600
1900
2200 t
2500
2800
3100
3400
dt
The plot should cover the whole time horizon of data file.
Comments:
No comments
Isolation
FI results:
Please give all the faults(or group of faults) that were isolated for the whole time horizon.
Fault isolation signal plot:
From:
0s
PID
PID (f13,
(f13, f18)
f18)
1
tit
CID
CID
1
tt [s]
[s]
0
1300
1600
1900
The plot should cover the whole time horizon of data file.
Comments:
No comments
2200
2500
2800
3100
3400
Benchmark Definition File, ver. 17March2002
Page 16 of 20
Appendix VII
Form:
Scenario form of Step III ... S3-SF-x
Partner:
S3-SF-1
Method name:
WUT
FDI based of partial FNN for FI and F-DTS for FD
Overall FDI results:
We found that f13 or f18 was simulated.
Only one fault (or a group of not isolable faults) must be declared as a result of FDI. All other results of FDI are treated as
false and must be reported.
Performance indexes
Index
tdm
tdrm
tim
tirm
...
Value
2001 s
3020 s
2201 s
2612 s
Comments
-
Questionnaire:
to be defined in future.
...
Notes:
None.
Detection
Fault detection signal plot:
From:
0s
To:
2400 s
To:
400 s
DD
DD
tt [s]
[s]
0
1300
1600
1900 t
dt
2200
2500
2800
3100
3400
tedt
3700
The plot should cover the time:
600 s before the fault was permanent detected and / or isolated
600 s after the fault was removed (if such a moment was detected)
the time units should be calculated form the beginning of data file.
Comments:
No comments
Isolation
FI results:
Please give all the faults(or group of faults) that were isolated for the whole time horizon.
Fault isolation signal plot:
From:
0s
PID
PID (f13,
(f13, f18)
f18)
1
tit
teit
CID
CID
1
tt [s]
[s]
0
1300
1600
1900
2200
2500
2800
The plot should cover the time:
600 s before the fault was permanent detected and / or isolated
600 s after the fault was removed (if such a moment was detected)
the time units should be calculated form the beginning of data file.
Comments:
No comments
3100
3400 3700
Benchmark Definition File, ver. 17March2002
Appendix VIII
Page 17 of 20
Actuator description
General description
The benchmark actuator selected is a final control element or simply named actuator, which
interacts with the controlled process. The input of the actuator is the output of the process
controller (flow or level controller) and the actuator modifies the position of the valve allowing a
direct effect on the primary variable in order to follow the flow or level set-point (Figure 5).
Figure 5. View of the typical industrial control valve actuator.
The actuator consists in three main components (Figure 6):
control valve,
spring-and-diaphragm pneumatic servo-motor,
positioner.
Control valve is the mean used to prevent and/or limit the flow of fluids. Changing the state of the
control valve is accomplished by a servomotor.
A spring-and-diaphragm pneumatic servomotor can be defined as a compressible (air) fluid
powered device in which the fluid acts upon the flexible diaphragm, to provide linear motion of the
servomotor stem.
Positioner is a device applied to eliminate the control-valve-stem miss-positions produced by the
external or internal sources such as friction, pressure unbalance, hydrodynamic forces etc. It
consists in a inner loop with a P controller of a cascade control structure, including the output signal
of the outer loop of the flow or level controller and the inner loop of the position controller.
The components available in this actuator are:
Pneumatic servo-motor S
Control valve V
Positioner P
ZC
- position P controller (internal loop)
C
- flow or pressure PI controller (external loop)
E/P
- electro-pneumatic transmitter
ZT
- rod position transmitter
D/A
- digital to analogue transducer
Additional external components:
V1, V2
V3
PSP
PT
FT
TT
- cut-off valves
- by-pass valve
- positioner supply pressure
- pressure transmitter
- volume flow rate transmitter
- temperature transmitter
Benchmark Definition File, ver. 17March2002
Page 18 of 20
PSP
Pz
Pneumatic servo S
motor
CVI
Ps
E/P
CV
ZC
P
Positioner
X
ZT
X
FT
Valve V
T1
Fv
P1
P2
V1
V2
Fv3
V3
TT
PT
PT
Figure 6. The actuator scheme.
Set of basic measured physical values:
external ( flow or level) controller output - CV
flow sensor measurement - F
valve input pressure P1
valve output pressure P2
liquid temperature T1
rod displacement X
Set of additional physical values that are realistic to measure:
positioner supply pressure - Pz
pneumatic servo-motor chamber pressure - Ps
position P controller output - CVI
Additional variables are not available for benchmark. These variables are available in EXTACT
block in DABLib Simulink actuator library. EXTACT block is not used in a benchmark steps,
however can be applied for additional FDI research.
Additional set of unmeasurable physical values that are used in structural analysis:
flow through the valve V - Fv
flow through the valve V3 - Fv3
Vena-contracta force Fvc
By-pass valve opening ratio X3
Additional unmeasurable physical values can not be used as an inputs of FDI algorithms, unless
they are reconstructed basing on measured variables.
All above mentioned variables are called main variables. The selection of main variables
corresponds to physical variables appearing when entering first level of Simulink actuator model
structure.
Benchmark Definition File, ver. 17March2002
Page 19 of 20
Structural Analysis
The structural analysis allows to define the relation between variables.
Table 2 summarizes the relations of the actuator main variables.
Table 2. Components and variables relations
Component
Relations
Pneumatic servo-motor, S
X = r1(Ps, Fvc)
Control valve, V
Fv = r2(X, P1, P2, T1)
Fvc = r3(P1, P2, X, T1)
Bypass valve, V3
Fv3=r4(P1, P2, T1, X3)
P position controller, ZC
Electro-pneumatic transducer, E/P
Other external
CVI=r5(CV, X)
Ps=r6(CVI, Pz, X)
F=r7(Fv,Fv3)
The Table 3 shows the structural matrix and Figure 7 is the resulting causal graph.
Table 3. Components and variables relations
CV
P1
P2
T1
r1
r2
x
x
x
r3
x
x
x
r4
x
x
x
r5
x
r6
r7
Pz
X3
CVI
Ps
x
Fv
Fv3
Fvc
x
x
x
x
x
x
x
X
x
x
x
x
x
x
x
Legend: _ - measured variables
x variables in relations
Pz
CV
r5
CVI
r6
Ps
r1
Fvc
r2
r3
Fv
r4
r7
Fv3
Measured
Measuredphysical
physicalvalues
values
Physical
Physicalvalues
valuesrealistic
realisticto
tomeasure
measure
Unmeasurable
Unmeasurablephysical
physicalvalues
values
P1
P2
Figure 7. Causal graph of the benchmark main variables.
T1
X3
Benchmark Definition File, ver. 17March2002
Appendix IX
Page 20 of 20
Examples of actuators in real processes
The actuators chosen for Benchmark Step III are installed in Lublin Sugar Factory (Poland).
Three valves have been selected. Two of them are installed at Evaporation Station (see Figure 8):
st
actuator controlling thin juice level in the 1 stage of evaporation station,
th
actuator controlling thick juice outflow from the 5 stage of evaporation station.
Figure 8. Schematic Diagram of Evaporation Station.
The third one is installed at steam boiler (see Figure 9):
th
actuator controlling water level in the 4 boiler station.
Figure 9. Schematic Diagram of Boiler Station.
