IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 16, NO.

9, SEPTEMBER 2020 6023

Complex Fuzzy System Based Predictive

Maintenance Approach in Railways
Mehmet Karakose , Senior Member, IEEE, and Orhan Yaman , Member, IEEE

Abstract—In this article, the Industry 4.0 compliant Rail transport historically has been an indispensable part of
method for condition monitoring and diagnostics on rail- our lives. Railway vehicles are used for both passenger and cargo
ways has been developed. A complex fuzzy system-based transportation across the world. This type of transportation is
thermography approach is proposed for predictive mainte-
nance on electric railways. The first contribution requires very advantageous because of its ability to carry heavy loads [3].
the use of a complex fuzzy approach, as estimator main- Heavy construction on railways causes defects in the railway
tenance methods in rail systems depend on seasonal con- line, and arcs formed in the pantograph catenary system used
ditions, environmental conditions, daylight, and especially in electric railway vehicles can cause the pantograph surface
periodic effects such as train speed. The second contri-
to malfunction. These faults can be repaired at a low cost if
bution is that both the rail surface and the pantograph
catenary system are very susceptible to thermal changes, detected early. If the faults are not detected early, then they
as diagnosing faults in image processing and monitored could lead to bigger accidents and significant interruptions to
systems requires a significant amount of work. In the liter- railway traffic during the maintenance work. There are many
ature, in some studies, there is no real-time and reasonably studies in the literature on how to best detect failures in railway
effective method of using thermal imaging in rail systems.
components [3]–[7]. Johansson et al. [5] developed a method
In order to validate these novel approaches, the proposed
method is applied to experimental data. The experimental for detecting wear failures due to friction and corrosion during
results obtained demonstrate the performance of the pro- switches and crossings by monitoring the switches and crossing
posed complex fuzzy system and the performance achieved components. When switches were passed, the contact with the
by processing thermal images. rail was simulated. Bocciolone et al. [6] developed a signal
Index Terms—Complex fuzzy system, image processing, processing-based method for detecting faults occurring on the
Industry 4.0, pantograph system, predictive maintenance, rail surface. In the proposed method, as the train travels on the
railways, thermography. railway, the generated vibration signals are used for the detection
of anomalies. As mentioned above, there are many different
methods of current, voltage, or model-based methods for de-
I. INTRODUCTION
tecting faults in pantograph catenary systems. These methods
HE development of the Internet and information technol-
T ogy has changed almost everything in our lives. Internet
technology, which is effective in every field, has revolutionized
have been run in real time, and failures that can occur in the
pantograph catenary system have been detected [8]–[11]. Li
et al. [12] presented an image-processing algorithm to detect
the Industry. The concept of Industry 4.0, which first emerged in the erosion that occurs in the pantograph contact area. In that
2011, has brought about a major change, especially in the field method, to obtain the edges in the image, wavelet transform,
of the production. Industry 4.0 is a broad term. With Industry and Hough transform were used and the bottom and top edges
4.0 technology, many concepts have developed such as Internet of the pantograph contact strip were detected. Boguslavskii’s
of Things (IoT), big data and analytics, smart machines, cloud [13] computer image processing techniques and pantographs
computing (CBM), and cyber–physical systems (CPS) [1]. As a proposed a real-time method for geometric model detection.
result of the application of Industry 4.0 technology in the field of In the proposed method in that study, the top and side regions
industry, production costs decrease, and productivity increases. of the pantograph were detected. Landi et al. [14] proposed a
In addition, Industry 4.0 makes it easy to monitor systems thermal imaging and Hough transform-based method for state
and diagnose failures. In this article, vision-based methods for monitoring in pantograph catenary systems. Using the thermal
condition monitoring and fault diagnosis are developed with images from the pantograph catenary system, defects in the
Industry 4.0 [2]. pantograph contact area and catenary contact were detected. A
summary of the studies in the literature is given in Table I.
Manuscript received July 30, 2019; revised October 23, 2019 and When the studies in the literature are examined, condition
December 28, 2019; accepted January 24, 2020. Date of publication monitoring and fault detection are generally performed on nor-
February 11, 2020; date of current version May 26, 2020. Paper no. mal images. In this article, an adaptive method was proposed
TII-19-3455. (Corresponding author: Orhan Yaman.)
The authors are with the Department Computer Engineering, on thermal images. Both thermal images and normal images
Firat University, Elazig 23119, Turkey (e-mail: [email protected]; are affected by environmental conditions. Also, in this article,
[email protected]). image preprocessing was performed to reduce the environmental
Color versions of one or more of the figures in this article are available
online at http://ieeexplore.ieee.org. effects of thermal image. The rail surface and pantograph contact
Digital Object Identifier 10.1109/TII.2020.2973231 area were determined by image preprocessing. Then, rail surface

1551-3203 © 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.

Authorized licensed use limited to: Auckland University of Technology. Downloaded on June 07,2020 at 04:22:31 UTC from IEEE Xplore. Restrictions apply.
6024 IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 16, NO. 9, SEPTEMBER 2020

TABLE I the dataset, the performance results of the method, and compar-
SUMMARY OF STUDIES IN THE LITERATURE
isons along with error analysis are given in Section V. Section VI
concludes this article.

II. INDUSTRY 4.0

Industry 4.0 can be defined as the ability to implement cyber–
physical structures, the concept of the IoT and the field of cloud
computing. The concept of Industry 4.0, which first emerged in
2011, is becoming more widespread with the development of
technology. The goal of Industry 4.0 is to increase production
and reduce costs by developing new generation software and
hardware. Interoperability, virtualization, autonomous manage-
ment, real-time capability, service orientation, and modularity
are the most important principles of Industry 4.0. Interoperabil-
ity provides the ability of CPS to communicate with people and
intelligent systems via the Internet of objects. Virtualization is
and pantograph contact area were cut over the main image performed by connecting the sensor data used in the developed
and condition monitoring was performed on these sections. No system with a virtual environment and simulation models. In
measures have been taken for the environmental effects on the systems using Industry 4.0 technology, condition monitoring
cut thermal images. and fault diagnosis are made easy. Systems and components
The purpose of using thermal images in this article is to be are aware of conventional systems. Furthermore, it has many
able to visualize seasonal changes on thermal images. Thus, advantages such as high efficiency, cost reduction, and develop-
even with seasonal changes, the recommended method success is ment of new service and business models. The main objective of
higher than normal images. In this context, it is meant to perform Industry 4.0 is to establish a system that can communicate with
condition monitoring and abnormal condition detection with each other, detect the environment using sensors and perform
the predictive maintenance approach. A maintenance approach data analysis [1], [2].
has been proposed that can change the membership functions
according to the seasonal conditions and monitor the situation III. RAILWAY COMPONENTS AND FAULTS
on the rail surface and pantograph contact area. The main contri- The railways have two component groups. The first group is
butions of the proposed method are given follows. In the studies the rail line component. The second group is the pantograph
in the literature, condition monitoring is generally performed by catenary system. The failures that can occur in these structures
using classical (RGB) images. In this research, it was aimed to affect the railway transportation in the negative direction. For
obtain better results from the literature by using thermal images. this reason, rails and railway pantograph systems are critical
In the railway condition monitoring, faults usually occur on the components in railway transportation.
rail surface are monitored and detected. However, joint points
of the rails have never been monitored. If the distance between A. Rail Line Components and Faults
the two rails increases at the rail joint, the train may exit the
rail line and major accidents may occur. Monitoring of the rail There are many failures in railway components due to many
joint shows that the study is original. In this article, a complex reasons. The biggest reason for the occurrence of these failures
fuzzy system is developed and an adaptive system is proposed. is the tension on the rail. Since the expansion of the rails affects
Thus, a fuzzy system is designed, which can vary depending on the relationship between the wheel and the rail, a rail failure can
seasonal conditions. There is very little information available in occur [15]–[18]. Rail flaws in railways have been published by
the literature about the application of complex fuzzy systems the International Railway Association (UIC) with UIC 712 R
in industry. This article demonstrate the applicability of the code [15]. With this fault code, a lot of information such as the
complex fuzzy system in the field of industry. In the literature, position and type of the rail fault is obtained. Codes consisting of
there is no common method developed for both the railway three or four digits indicating the type of rail failures are given.
line and the pantograph contact point, whereas in this article, a By looking at these codes, it can be understood in which region
complex fuzzy system has been developed for both component the rail fault has occurred and what the fault is.
groups.
B. Pantograph Catenary System and Faults
The pantograph catenary system consists of the pantograph
A. Study Outline system located on the locomotive and the catenary system in-
The rest of this article is organized as follows. Section II of this stalled along the railway line [19]. The electrical energy of the
article describes Industry 4.0. In Section III, the general faults catenary system is transmitted to the train via the pantograph
that occur due to rail line components and the pantograph cate- system. If the contact point does not change constantly, then it
nary system are explained. Section IV introduces the proposed causes friction and overheating. Some defects such as overheat-
method and describes the complex fuzzy system. Explanation of ing and arc formation occur due to the friction generated at the

Authorized licensed use limited to: Auckland University of Technology. Downloaded on June 07,2020 at 04:22:31 UTC from IEEE Xplore. Restrictions apply.
KARAKOSE AND YAMAN: COMPLEX FUZZY SYSTEM BASED PREDICTIVE MAINTENANCE APPROACH IN RAILWAYS 6025

is located and where the data are stored. In the subtraction

section, fuzzy results are obtained using entries and rules. In
the defuzzification section, blurred results are converted into
numerical values.
A complex fuzzy system is a logic system expressed by
complex fuzzy sets [21], [22]. This is the generalization of tra-
ditional fuzzy logic based on complex fuzzy sets. The complex
fuzzy structure resembles the type-1 fuzzy sets. A complex
fuzzy system can be defined by specific choices of connection,
division, and complementary operators [21]. Inference rules are
constructed like traditional fuzzy logic. Complex fuzzy sets are
characterized by complex, valuable membership functions. In
traditional fuzzy logic, membership functions range from [0, 1]
to a complex fuzzy system up as a unit circle in virtual plane.
This structure provides a way to define membership in the set
as a complex number. The characteristics of complex numbers
Fig. 1. Architecture of the proposed method.
and complex fuzzy sets are advantageous over conventional
fuzzy logic.
contact point in the pantograph catenary systems. Failures occur- Complex fuzzy sets are a new development in the theory of
ring in pantograph catenary systems lead to substantial material fuzzy systems. In this article, membership values are complex
damage. In addition, the disruption of railway transportation numbers drawn in the unit circle of the complex plane. The
can cause problems such as passenger and cargo security. Early definition of a complex fuzzy set is defined by a complex number
detection in pantograph catenary systems is very important to that effectively transforms a set member into a two-dimensional
prevent the resulting failures from causing larger outcomes [20]. concept [21], [22].
In fuzzy sets, R[0,1] and C[0,1] are the set of real and com-
IV. PROPOSED APPROACH plex numbers, respectively [22]. The real range is R[0,1] =
{a ∈ R : 0 ≤ a ≤ 1} and the complex interval is C[0,1] =
In this article, a new thermography-based method for mon- {c ∈ C : |c| ≤ 1}. A z complex number is expressed in two
itoring and diagnosing the condition of both the railway and different forms: Descartes and polar coordinates. The represen-
pantograph system is proposed. In the proposed method, feature tation of a complex number of Descartes is given follows:
extraction is done by image processing on thermal images. In
the pantograph system, the heat generated at the contact point z =x+j·y (1)
of the catenary wire is monitored while the joint points of the
and the polar representation given as follows:
rails are monitored on the rail line. The general architecture of
the proposed method is given in Fig. 1. r = ej·ω . (2)
The results obtained from the thermal images and the esti-
mated results of the complex fuzzy system were used to monitor A complex fuzzy system can be defined by using both equa-
the situation. Friction may occur between the wheel and the rail tions. There is a relationship between (1) and (2), given by
due to expansion in the rail. At the same time, warming may x = r cos(ω), y = r sin(ω). (3)
occur at the contact point as a result of the expansion of the
catenary wire. In this article, seasonal conditions were taken U is a discourse universe, and the membership function of Ã
into account in order to monitor the temperature in both the rail complex expression defined in U is μÃ. The polar representation
line and the pantograph contact area more successfully. Seasonal of the membership function μà is defined as follows:
variations in temperature affect the expansion of the rail line and μà : U → C[0,1] , u → r(u) · ej·ω(u) (4)
the heating of the pantograph surface. A complex fuzzy system
was, therefore, developed to allow for seasonal conditions to be where r(u) is an actual value defined from the value U to R[0,1] .
used in the proposed method. ω(u) is the periodic function. The value of r(u) is the modulo
part of A while the value of ω(u) is the part of the phase.
A. Complex Fuzzy System The Deccartes representation of the membership function of the
complex expression à defined in U is given as follows:
A complex fuzzy is a method derived from the fuzzy logic. 
The fuzzy logic concept is a successful method that represents à = u, x(u) + j · y(u) : |x(u)| , |y(u)| ∈ R[0,1]
a mathematical model and is a natural way of creating systems 
× x2 (u) + y 2 (u) ≤ 1 . (5)
that mimic human decision making processes. There are four
basic parts of the fuzzy logic structure. In the blur zone, the It is quite difficult to identify the phase fraction introduced at
input values are converted to fuzzy values using the membership the complex membership level in fuzzy sets. For this reason, the
function. The knowledge base is the place where the rules table phase section is limited and the operations are made concrete.

Authorized licensed use limited to: Auckland University of Technology. Downloaded on June 07,2020 at 04:22:31 UTC from IEEE Xplore. Restrictions apply.
6026 IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 16, NO. 9, SEPTEMBER 2020

Fig. 2. Temperature and daylight graph for input membership func- Fig. 3. Complex input membership functions. (a) Complex tempera-
tions. (a) Estimated three year temperature graph. (b) Annual daylight ture membership. (b) Complex daylight membership function.
graph.

Usually, in the literature studies, the phase fraction is limited to

[22]. Under normal conditions, the phase section has no limit.
In operation, the phase section is limited in terms of ease of
operation.
In this study, the features obtained from thermal images for
condition monitoring and fault detection are used together with
the complex fuzzy system. The purpose of using the complex
fuzzy system is to correlate the properties obtained from the Fig. 4. Complex fuzzy predictive maintenance method for rail joint.
thermal images with the seasonal conditions. Thus, it is planned
to monitor the situation more successfully and to diagnose the
fault. Using (6), the complex temperature membership function
In this article, a complex fuzzy system with two inputs for rail given in Fig. 3(a) is used in (7) to obtain the daylight complex
line and pantograph situation monitoring was developed. In this membership function given in Fig. 3(b).
complex fuzzy system, the first membership function is annual In the complex input membership functions given in Fig. 3,
temperature change, and the second membership function is the the module is limited to the range 0–1 and the phase 0 to 2π.
annual daylight period. The same entry membership functions Complex fuzzy functions do not normally have a specific limit
are used for both rail tracing and pantograph status monitoring. on the phase value. The reason for limiting the phase value to
The rule tables and output membership functions for the rail line 2π in this article is to simplify the calculation process, as has
and pantograph system are different. For the first entry member- been done in previous studies.
ship function, first an annual temperature graph of a country is
created. This graph considering air temperature measurements
was presumably created in Turkey. The three-year estimated B. Proposed Rail Line Condition Monitoring and Fault
temperature data are given in Fig. 2(a), where each will form Detection Method
a period. The annual daylight graph, the second membership In the proposed method, railway images are obtained using a
function of the complex fuzzy system, is given in Fig. 2(b). thermal camera. The rail surface is detected primarily on the rail
So that the graphs given in Fig. 2 can be used as input mem- line image. Using the obtained rail surface, rail joint points are
bership functions of the complex fuzzy system, the temperature determined. The intersection between two rails at the rail joint
graph is transformed using the following equations: point is investigated by the complex fuzzy method. In this article,
⎧ω  a flowchart of the proposed method of rail surface detection and
⎪ , ω ∈ 0, π the method of fixing the rail joint are given in Fig. 4.
⎨π 
r(ω) = 1.0, ω ∈ π, 3π (6) In this article, a correlation-based method is used for rail
⎪
⎩ ω−2π  3π 2 surface detection and rail joint detection on the thermal image.
3π/ 2−2π , ω ∈ 2 , 2π When generating an attribute array from an image, the corre-
ω
 lation value for that column is calculated using all the rows of
π, ω ∈ 0, π
r(ω) = ω−2π  . (7) each column in the thermal image. Correlation is a method that
−π , ω ∈
π, 2π repeats multiple times on the image [23]. Calculation of the

Authorized licensed use limited to: Auckland University of Technology. Downloaded on June 07,2020 at 04:22:31 UTC from IEEE Xplore. Restrictions apply.
KARAKOSE AND YAMAN: COMPLEX FUZZY SYSTEM BASED PREDICTIVE MAINTENANCE APPROACH IN RAILWAYS 6027

Fig. 6. Proposed output membership functions for monitoring the rail

joint.

TABLE II
PROPOSED FUZZY RULE TABLE FOR RAIL LINE

Fig. 5. Detection of rail surface on a sample image and measurement

of rail joint. (a) Detection of rail surface. (b) Measurement of rail joint.

correlation value is given as follows:

N −τ −1
1
ri (τ ) = x(t + τ )x(t) (8)
N t=0

where N is the total number of samples, t is the time, τ is the

scrolling value at the time, x is the normalized signal value, and
r is the autocorrelation value. The correlation signal is applied
to the image to obtain the feature signal. The smoothing process Fig. 7. Proposed method for pantograph condition monitoring and fault
is applied in order to eliminate the noises and further regulate detection.
the attribute signal. The smoothing process is given as follows:
x(t) = (x(t − 2) + x(t − 1) + x(t) + x(t + 1) + x(t + 2))/5.
conditions. The rule table developed for this process is given in
(9)
Table II.
After softening, the differential signal is obtained for both rail
The rule table in Table II is based on the annual temperature
surface and rail joint detection. Equation (10) is used once and
and daylight charts, which are complex fuzzy entries. The mem-
the difference signal is obtained as follows:
bership functions for temperature and daylight used in this article
x(t) = |x(t + 6) − x(t − 6)| . (10) may differ for other countries. Complex membership functions
should be updated, taking into account the seasonal conditions
After the difference signal is obtained, the peak points formed
of the country where the work will be implemented.
on the positive side of the signal are called the rail surface end,
and the peak point formed on the negative side of the rail surface
is called the rail surface finish. The feature signal, the smoothed C. Proposed Pantograph Condition Monitoring and Fault
signal, and the difference signal obtained from a sample image Detection Method
are shown in Fig. 5(a), and the calculation of the rail joint is In this method, a pantograph contact area is detected on the
given in Fig. 5(b). pantograph image. When a pantograph region is detected, a
In the proposed method, the distance between the rails is correlation-based method is used. The flowchart of the proposed
monitored by determining the joint between the two rails. To method for detecting the pantograph contact area is given in
determine whether the distance between the two rails is normal, Fig. 7. The correlation graph shown in Fig. 8 is obtained.
the complex input membership functions and the output mem- As shown in Fig. 8, the heat generated in the contact area is
bership function are used. The proposed output membership measured after the pantograph contact area has been detected. As
function for this method is given in Fig. 6. a result of the complex fuzzy system, considering the seasonal
The output membership function given in Fig. 6 is used to- conditions, the temperature of the pantograph contact area is
gether with the complex input membership functions to estimate estimated according to environmental temperature. The output
the distance between two rails according to the environmental membership function used for the proposed method is given in

Authorized licensed use limited to: Auckland University of Technology. Downloaded on June 07,2020 at 04:22:31 UTC from IEEE Xplore. Restrictions apply.
6028 IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 16, NO. 9, SEPTEMBER 2020

Measurement range −20–350 °C, resolution 0.2 °C (30 °C),

0.1 °C (S/N 30 °C), spectral range 8–13 μm, image refresh
rate 8.5 Hz, angle of vision 28° × 21°. For the accuracy of the
thermography-based method proposed in this article, thermal
images of both the rail line and the pantograph system were used.
Here, a ready set of data was not used. An experimental device
was established and thermal imaging camera was used to obtain
images from both the railway track and pantograph system. A
total of 200 images were obtained from 100 thermal images of
Fig. 8. Detection of the pantograph contact area in a sample image. the rail line and 100 thermal images from the pantograph system.
The images obtained were individually measured and labeled.
The proposed complex fuzzy system method is applied for 100
frame rail line image and 100 frame pantograph contact region
image.

A. Experimental Results for Rail Line

The thermal image was transformed into a signal by the
Fig. 9. Proposed output membership function for the pantograph con- correlation method developed on the thermal images. The rail
tact area. surface was detected by processing the signal. The distance
between the two rails was measured by detecting the rail joining
point on the rail surface image. Then, the complex fuzzy results
TABLE III
PROPOSED FUZZY RULE TABLE FOR PANTOGRAPH CONTACT AREA were compared with the estimated distance between the two rails
according to the seasonal conditions and the distance between
the two rails per the thermal image.
As shown in Fig. 10, the rail surfaces were detected by
obtaining correlation signals on the thermal images taken in
different seasonal conditions. An image taken at the switches
and the crossing point was used in Fig. 10(c). The proposed
method on this image was successfully detected for both rail
lines. The rail connection point was detected by applying a
correlation signal on the detected rail surface. Fig. 10 shows the
rail line image taken in four different seasons. First, in Fig. 10, the
correlation values were calculated using the thermal image on
the left. A difference signal was obtained as shown in the middle
image. By making the necessary calculations on this signal, the
Fig. 9. The output membership function given in Fig. 9 is used
rail surface is determined as shown in the image on the right.
together with the complex input membership functions to esti-
In Fig. 10(c), because the image is taken in the transition area,
mate the temperature of the pantograph contact area according
there are two rail surfaces in the image. Although there were
to environmental conditions. The rule table developed for this
two rail surfaces in the image, it still detected the rail surfaces.
process is given in Table III.
Fig. 11 shows the detection of rail joint points from the thermal
Using the rule table given in Table III, the temperature of the
rail surface images obtained under different seasonal conditions.
pantograph contact area is estimated according to seasonal con-
In Fig. 11, the distance between two rails was measured after
ditions. The estimated temperature value is expressed in pixel
the rail joint points were detected. The measured distances were
value. The values with the highest pixel value are selected in the
compared with the complex fuzzy system end result. These com-
pantograph contact area and compared with the complex fuzzy
parisons and results are given in Table IV. Complex fuzzy input
result. If the estimated temperature value in the complex fuzzy
values and estimation results are given for the example images in
result is lower than the actual pantograph temperature value,
Table IV. When estimated results were obtained, the phase parts
then the result is that the fault is in the pantograph contact area.
of the complex fuzzy membership values were neglected. The
annual temperature change was calculated according to the last
V. EXPERIMENTAL RESULTS
three periods. When the annual temperature value was selected
In this article, images were taken of a railway line and a as the input value of the complex fuzzy system, the complex
pantograph system using a thermal camera in different seasonal value having the greatest value of the three periods was used
conditions at different times. The results were obtained by as the input. The ideal distance between two rails is estimated
combining the images with a method developed in a MATLAB in terms of pixels according to seasonal conditions in the board
environment. Thermal images were taken using a NEC F30W table. According to these conditions, the distance between two
thermal camera. The features of the camera are as follows. rails is normally short. From this result, it is seen that the

Authorized licensed use limited to: Auckland University of Technology. Downloaded on June 07,2020 at 04:22:31 UTC from IEEE Xplore. Restrictions apply.
KARAKOSE AND YAMAN: COMPLEX FUZZY SYSTEM BASED PREDICTIVE MAINTENANCE APPROACH IN RAILWAYS 6029

Fig. 10. Rail surface detection in thermal images taken under different seasonal conditions. (a) Spring. (b) Summer. (c) Autumn. (d) Winter.

Fig. 11. Detection and measurement of rail joint in different seasonal conditions. (a) Spring. (b) Summer. (c) Autumn. (d) Winter.

Authorized licensed use limited to: Auckland University of Technology. Downloaded on June 07,2020 at 04:22:31 UTC from IEEE Xplore. Restrictions apply.
6030 IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 16, NO. 9, SEPTEMBER 2020

TABLE IV
COMPLEX FUZZY RESULTS ON SAMPLE RAIL IMAGES

TABLE V
COMPLEX FUZZY RESULTS ON SAMPLE PANTOGRAPH IMAGES

Fig. 12. Pantograph contact area detection on sample thermal images. (a) Robust pantograph image. (b) Warmer pantograph image.

expansion ratio of the rail is small. In this case, it is predicted that to 60 °C. The exemplary thermal images and pantograph contact
the structure of the railway line may deteriorate when it expands area detection used in this article are given in Fig. 12.
in hot weather. The difference between the estimated value and As shown in Fig. 12, the thermal images were rotated by 90°,
the actual value gives the size of the faulty rail. and the pantograph contact area was detected by the correlation
In Table V, the estimated value in our experiment was 28.7 method. The pixel values of the hottest region in the detected
while the actual value was 37 for the sample value taken in the pantograph contact area were assessed. For this process, the
spring. According to these conditions, the distance between the largest pixel value in the pantograph contact area was taken.
two rails is normally long. In this case, it is anticipated that The complex fuzzy results with sample pantograph images are
the distance between the two rails will increase when the rail given in Table V.
shrinks in cold weather. If the distance between the two rails is The temperature and daylight values of the two images given
more than normal, then crushing of the rail mushroom can cause in Table V are the same because the two images were obtained
many failures such as cracks on the rails. Also, in the case of on the same day. The complex fuzzy result was the predicted
an example taken in autumn, the actual value is smaller than the temperature pixel value for the pantograph contact area, which
estimated value. was 168. The actual temperature value in the contact area in
the image shown in Fig. 12(a) is 176. The predicted value and
the actual value for this image are close to each other. Thus,
B. Experimental Results for Pantograph
it is understood that there is no problem in the pantograph
Two different images taken during the autumn month were contact area in this image. In Fig. 12(b), the image shows a
used to monitor the temperature change in the pantograph con- contact in the actual temperature value that is measured to be
tact area. The thermal images used range in temperature from 0 205 pixels. The actual measured pixel value in the contact area is

Authorized licensed use limited to: Auckland University of Technology. Downloaded on June 07,2020 at 04:22:31 UTC from IEEE Xplore. Restrictions apply.
KARAKOSE AND YAMAN: COMPLEX FUZZY SYSTEM BASED PREDICTIVE MAINTENANCE APPROACH IN RAILWAYS 6031

TABLE VI
CLASSICAL FUZZY LOGIC AND THE RESULTS OF THE PROPOSED COMPLEX FUZZY

TABLE VII
PERFORMANCE COMPARISON OF THE PROPOSED METHOD AND LITERATURE

considerably greater than the estimated pixel value. In this im- processor. The proposed method can work in real time. But it is
age, it is understood that the pantograph contact area is warmer not fast enough to work in real time. In future studies, methods
than the seasonal conditions. This warming in the contact area is will be developed to make the method available for high-speed
caused by the friction at the end of the pantograph and the contact trains by using high-processing computers.
of the catenary wire. Fig. 12(b) shows that there is warming in the 1) Error Analysis: Both thermal images and normal images
contact area for the image and that there is an abnormal situation are affected by environmental conditions. In this article, the
here. Accuracy, precision, specificity, recall, and F-measure surface of the rail and the contact area of the pantograph are
values were calculated for classical fuzzy logic and complex determined by image preprocessing on thermal images. Then,
fuzzy logic. The performance results of the proposed method these sections are removed from the main image and the process
are given in Table VI and the literature comparison is given in is done. Thus, the size of the image was reduced to reduce envi-
Table VII. ronmental impacts. However, environmental effects on thermal
When the studies in Table VII are examined, it is seen that no images could not be neglected. In this article, vibration and blur
studies performed rail joint detection and condition monitoring. parameters were not taken into consideration while obtaining
When compared with the studies in the literature in general, thermal image. Rail transport is interrupted during the fixation
it is seen that the proposed method has a good running time of the cameras on the train and moving them on the rail line.
and success rate. It is also seen that the proposed method can Therefore, a limited number of images were taken and this
work in real time. The images obtained from the thermal camera dataset was used. In the future studies, it is aimed to increase the
are generated according to the temperature data. In this article, dataset and collect images for different fault conditions.
adaptive rules according to seasonal conditions and daylight are
formed by using complex fuzzy logic. Thus, different rules are
VI. CONCLUSION
applied for images taken in summer and images taken in winter.
In addition, the complex fuzzy system takes into account the As in all types of transportation, safety of life is very im-
expansion situation in the rails. If classical fuzzy logic was used portant in railway transport vehicles. In this article, a new
in this article, the same rules would apply to all cases. The complex thermography method with a complex fuzzy base for
success rate would, therefore, be lower than complex fuzzy logic. the railway and pantograph catenary system was developed. In
The operating time of the proposed method is approximately this method, image and signal processing methods were used
122.5 ms. The recommended method runs about 8 frames per on thermal images. The features obtained from the thermal
second when run on an 8 GB RAM capable computer with an i5 images were compared with the complex fuzzy result and the

Authorized licensed use limited to: Auckland University of Technology. Downloaded on June 07,2020 at 04:22:31 UTC from IEEE Xplore. Restrictions apply.
6032 IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 16, NO. 9, SEPTEMBER 2020

rail line and pantograph contact areas were observed. A complex [12] M. Li, W. Ze-yong, G. Xiao-rong, W. Li, and Y. Kai, “Edge detection on
fuzzy system was developed using the annual temperature and pantograph slide image,” in Proc. Int. Congr. Image Signal Process., 2009,
pp. 1–3.
annual daylight information of the medium. Here, the predictive [13] A. A. Boguslavskii and S. M. Sokolov, “Detecting objects in images in
maintenance method recommended for rail systems depends on real-time computer vision systems using structured geometric models,”
periodic effects such as seasonal conditions and daylight. For Pleiades Publishing, vol. 32, pp. 177–187, 2006.
[14] A. Landi, L. Menconi, and L. Sani, “Hough transform and thermo-vision
this reason, the complex fuzzy system gives more successful for monitoring pantograph-catenary system,” Proc. Inst. Mech. Eng. Part
results than the traditional fuzzy system. In this article, the com- F J. Rail Rapid Transit, vol. 220, no. 4, pp. 435–447, Jan. 2006.
plex fuzzy membership functions were formed on the complex [15] Australian Rail Track Corporation, Manual for non-destructive testing of
rail, ARTC, Adelaide, SA, Australia, 2009.
plane and the phase interval was limited to 2π. Also, when the [16] M. Kerr, Rail Defects Handbook. Adelaide, SA, Australia: Australian Rail
results were obtained, the phase part of the complex results was Track Corp., 2012.
neglected. Thus, the proposed estimator maintenance method [17] B. Lichtberger, Track Compendium: Formation, Permanent Way, Mainte-
nance. Hamburg, Almanya: Economics, Eurail Press, 2005.
was transformed into a simple and feasible structure. Thermal [18] UIC-712 R, Rail defects, Paris, France: International Union of Railways
images of the railway line were taken at different seasons and the (UIC), 2002.
results were obtained for all seasons. In the proposed method, [19] A. K. Hallgrímsson, “Dynamic behavior of contact lines for railways with
laboratorial model setup according to Norwegian conditions,” Norwegian
both the rail line and the pantograph condition monitoring have Uni. Sci. Technol., p. 129, 2013.
made a contribution according to the studies in the literature in [20] E. Karakose, M. T. Gencoglu, M. Karakose, O. Yaman, I. Aydin, and
the proposed method. E. Akin, “A new arc detection method based on fuzzy logic using
S-transform for pantograph–catenary systems,” J. Intell. Manuf., vol. 29,
In future studies, the complex fuzzy system based methods pp. 839–856, 2018.
will be developed in order to detect failures on rail surface. Thus, [21] A. Y. Deshmukh, A. B. Bavaskar, P. R. Bajaj, and A. G. Keskar, “Imple-
the availability of complex fuzzy systems for fault detection in mentation of complex fuzzy logic modules with VLSI approach,” Int. J.
Comput. Sci. Netw. Secur., vol. 8, pp. 172–178, 2008.
the field of Industry 4.0 will be increased. In this article, results [22] J. Ma, G. Zhang, and J. Lu, “A method for multiple periodic factor
were obtained by using only thermal images. In future studies, prediction problems using complex fuzzy sets,” IEEE Trans. Fuzzy Syst.,
results will be obtained on both normal and thermal images and vol. 20, no. 1, pp. 32–45, Feb. 2012.
[23] O. Yaman, M. Karakose, and E. Akin, “A vision based diagnosis approach
comparison will be made. Thus, it will be clearly seen whether for multi rail surface faults using fuzzy classificiation in railways,” in Proc.
there is a difference in success between thermal and normal Int. Conf. Comput. Sci. Eng., 2017, pp. 713–718.
images. In addition, the problem of vibrations and blur in the [24] Y. Xia, F. Xie, and Z. Jiang, “Broken railway fastener detection based
on adaboost algorithm,” in Proc. IEEE Int. Conf. Optoelectron. Image
image will be solved in future studies. Process., 2010, vol. 1, pp. 313–316.
[25] J. Liu et al., “Learning visual similarity for inspecting defective railway
fasteners,” IEEE Sensors J., vol. 19, no. 16, pp. 6844–6857, Aug. 2019.
REFERENCES [26] Y. Zhan, K. Linb, H. Zhan, Y. Guo, and G. Sun, “A unified framework
for fault detection of freight train images under complex environment,” in
[1] M. Aazam, S. Zeadally, and K. A. Harras,” Deploying fog computing in
Proc. 25th IEEE Int. Conf. Image Process., 2018, pp. 1348–1352.
industrial Internet of Things and Industry 4.0,” IEEE Trans. Ind. Informat.,
[27] I. Aydin, M. Karakose, and E. Akin, “Anomaly detection using a modified
vol. 14, no. 10, pp. 4674–4682, Oct. 2018.
kernel-based tracking in the pantograph–catenary system,” Expert Syst.
[2] M. Schluse, M. Priggemeyer, L. Atorf, and J. Rossmann, “Experimentable
With Appl., vol. 42, no. 2, pp. 938–948, 2015.
digital twins—Streamlining simulation-based systems engineering for
Industry 4.0,” IEEE Trans. Ind. Informat., vol. 14, no. 4, pp. 1722–1731,
Apr. 2018.
[3] M. Karakose, O. Yaman, K. Murat, and E. Akin, “A new approach for
condition monitoring and detection of rail components and rail track
in railway,” Int. J. Comput. Intell. Syst., vol. 11, no. 1, pp. 830–845, Mehmet Karakose (Senior Member, IEEE) re-
2018. ceived the B.S. degree in electrical engineering,
[4] L. Zhuang, L. Wang, Z. Zhang, and K. L. Tsui, “Automated vision the M.S. and Ph.D. degrees in computer engi-
inspection of rail surface cracks: A double-layer data-driven framework,” neering from Firat University, Elazig, Turkey, in
Transp. Res. Part C: Emerg. Technol., vol. 92, pp. 258–277, 2018. 1998, 2001, and 2005, respectively,
[5] A. Johansson et al., “Simulation of wheel–rail contact and damage in He is currently a Professor Doctor with the
switches & crossings,” Wear, vol. 271, no. 1, pp. 472–481, 2011. Department of Computer Engineering, Firat Uni-
[6] M. Bocciolone, A. Caprioli, A. Cigada, and A. Collina, “A measurement versity. His research interests include fuzzy sys-
system for quick rail inspection and effective track maintenance strategy,” tems, intelligent systems, quantum computing,
Mech. Syst. Signal Process., vol. 21, no. 3, pp. 1242–1254, 2007. simulation and modeling, fault diagnosis, com-
[7] G. Karaduman, M. Karakose, and E. Akin, “Experimental fuzzy diagnosis puter vision, railway inspection systems, and
algorithm based on image processing for rail profile measurement,” in photovoltaic systems.
Proc. IEEE 15th Int. Conf. MECHATRONIKA, 2012.
[8] D. Zhang, S. Gao, L. Yu, G. Kang, D. Zhan, and X. Wei, “A robust
pantograph–catenary interaction condition monitoring method based on
deep convolutional network,” IEEE Trans. Instrum. Meas., to be published,
doi: 10.1109/TIM.2019.2920721.
[9] Z. Huang, L. Chen, Y. Zhang, Z. Yu, H. Fang, and T. Zhang, “Ro- Orhan Yaman (Member, IEEE) received the
bust contact-point detection from pantograph–catenary infrared images B.S., M.S., and Ph.D. degrees in computer en-
by employing horizontal-vertical enhancement operator,” Infrared Phys. gineering from Firat University, Elazig, Turkey, in
Technol., vol. 101, pp. 146–155, 2019. 2012, 2014, and 2018, respectively.
[10] Y. Gao and G.Y. Tian, “Emissivity correction using spectrum correlation He is currently an Assistant Professor with
of infrared and visible images,” Sensors Actuators A, Phys., vol. 270, the Department of Computer Technology, Firat
pp. 8–17, 2018. University. His research interests include image
[11] Y. Liu, G. W. Chang, and H. M. Huang, “Mayr’s equation-based model processing, fault diagnosis, railway inspection
for pantograph arc of high-speed railway traction system,” IEEE Trans. systems, and fuzzy systems.
Power Del., vol. 25, no. 3, pp. 2025–2027, Jul. 2010.

Authorized licensed use limited to: Auckland University of Technology. Downloaded on June 07,2020 at 04:22:31 UTC from IEEE Xplore. Restrictions apply.