Available online at www.sciencedirect.

com

ScienceDirect
Procedia CIRP 100 (2021) 816–821
www.elsevier.com/locate/procedia

31st CIRP Design Conference 2021 (CIRP Design 2021)

Augmented reality for IT/OT failures in maintenance operations of

digitized trains: current status, research challenges and future directions
Sara Scheffer*, Alberto Martinetti, Roy Damgrave, Leo van Dongen

Department of Design, Production & Management, University of Twente, 7500 AE, Enschede, The Netherlands

* Corresponding author. Tel.: +31-53-489-2875. E-mail address: [email protected]

Abstract

The railway industry is moving towards a complex socio-technological system that relies on computer control and human-machine interfaces.
Digitization and convergence of information technology (IT) with operational technology (OT) becomes increasingly important. Therefore,
enabling innovations, lowering costs, increasing safety systems, enhancing performance, and increasing flexibility gains. In this scenario,
opportunities arise for Augmented Reality (AR) capabilities to enhance maintenance operations of digitized trains. Initial research and tests have
enabled training technicians and assistance of train drivers during specific activities by facilitating interactions with machines that go beyond
traditional training manuals. Despite the positive results AR has shown, there are no specific indications available regarding the ‘deployability’
of the technology related to maintenance tasks. This position paper presents arguable boundaries and challenges for a structural appraisal of AR
on maintenance operations of IT/OT converged trains. Potential AR solutions and specifications were based on current maintenance operations.
Semi-structured interviews and surveys were conducted with maintenance providers to determine existing challenges. Moreover, the paper
comprises an analysis to reveal the potential application AR has in maintenance procedures of digitized trains. Finally, this paper provides future
research directions of AR technologies related to maintenance procedures of trains.
© 2021 The Authors. Published by Elsevier Ltd.
This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0)
Peer-review under responsibility of the scientific committee of the 31st CIRP Design Conference 2021.
Keywords: Augmented Reality; Maintenance; Digitized trains; IT/OT failures

1. Introduction blockchains that innovate the railway system. The railway

industry aims to have safe, high-quality, and sustainable
Over the last few years, the railway industry is experiencing transport and maintenance operations. Maintenance costs
a tidal wave of digital transformation whose impact is complex. depict a significant proportion of the life-cycle costs of trains,
This industry is a socio-technical system that is becoming where fleet maintenance costs significantly exceed the initial
increasingly concerned with digitization. Key performance investment costs. A fundamental part of supporting trains
indicators (KPIs) for railway operations remain the same, but throughout the entire life-cycle of a train is therefore
each of them will show a positive impact from the application maintenance.
of digital technologies. Technological innovations of trains and Digitization and convergence of information technology
traffic management systems create new services and (IT) with operational technology (OT) becomes increasingly
commercial business cases [1]. Overall competitiveness and important for railway operations to achieve less maintenance
performance improve by exploiting data-based services with cost, efficient, and reliable operations. Here, IT is defined as an
for example: artificial intelligence techniques, merging real and engine that accepts data flow as an input to deliver new data
virtual worlds, autonomous driving, Internet of Things, and flow but does not interfere with the physical world [2]. OT is a

2212-8271 © 2021 The Authors. Published by Elsevier Ltd.

This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0)
Peer-review under responsibility of the scientific committee of the 31st CIRP Design Conference 2021.
10.1016/j.procir.2021.05.038

This is a resupply of March 2023 as the template used in the publication of the original article contained errors. The content of the article has remained unaffected.
 Sara Scheffer et al. / Procedia CIRP 100 (2021) 816–821 817

set of devices and processes that act in real-time on physical 2. AR status in maintenance operations
operational systems. Because of digitization in the railway
industry, maintenance of new trains is more often focused on When assessing the current AR status in maintenance
(remote) inspections based on data rather than on physical operations, a brief specification of AR application fields is
assembling and disassembling components. An additional required. A clear AR tool description helps to understand the
result of digitization is more IT and OT systems are technology which will be used in this research. System
implemented into trains. IT/OT convergence lowers costs, boundaries are presented next to evaluate the current
reduces risks, enhances train performance, and reaches technology implementation status. All information results in
flexibility gains [2]. Convergence generates more data, data positioning AR tools in different maintenance fields.
relationships, and dependencies, thereby requiring early
identification of critical interfaces. However, the convergence 2.1 Specification of AR technology
of IT/OT is highly complex due to the involvement of different
independent systems, technologies, stakeholders, equipment, AR techniques can be used to visualize contextualized
and methods [3]. Finally, failure within the boundaries of virtual information at the appropriate time and location within
IT/OT converged systems is thereby largely unpredictable and the working environment. Potential applications of AR in
increasingly involute. industrial ecosystems are primarily focused on maintenance,
Based on a case study, it was found that failures are not yet service, inspection, device diagnostics, repair of complex
fully understood by today’s asset and fleet managers. In machinery, and training [5]. Easier understandable instructions
addition, knowledge of IT is limited and only present in a few are provided as computer-generated information superimposed
departments. Furthermore, the collaboration between IT and upon the actual equipment, and step-by-step explanation of
maintenance departments lacks coherence. As a result, only a tasks. A report on mechanics’ capability of viewing virtual
few employees have competence in both IT and OT of the train, information panels that overly the real image of engines and
creating an IT/OT failure knowledge gap. Therefore, see through the machinery without disassembling components
understanding and structuring IT/OT systems is of great was already proposed in 1993 [6].
importance for the railway industry. In order to meet this Typical AR systems include components such as
interest, this research aims to examine the use of Augmented visualization/capturing device, interaction device, and tracking
Reality (AR) as a tool for identifying, prioritizing, and system [6]. Capturing technologies are collecting
providing solutions to failures related to IT/OT converged environmental data and can be viewed by the user with
systems. AR is capable of showing the right information in the superimposed information. Visualization devices are used to
right context to support users in finding this malfunction [4]. display the results of image processing. Interaction devices are
AR is a technology, mature enough to be widely deployed in used for commands that affect information processing and
maintenance operations, testing, and training of converged IT displaying. Tracking technologies are essential for identifying
and OT trains. However, challenges of deployability and user position and provides correct information when
integration of the technology must be dealt with. An overview augmenting the environmental scene itself. Technical and
of this position paper is presented in figure 1. technological issues related to development and
implementations of AR are debatable elements. Most studies
that review technological solutions for visualization either use
mobile devices or Head-Mounted Displays (HMDs) [7].
Mobile devices need to be held in the hand when used and
therefore potentially hinder maintenance tasks. HMD solutions
leave the hand free, allowing for a more natural and intuitive
hand-based interaction with virtual objects. Two main
categories of tracking systems can be defined: marker-based
tracking and marker-less tracking [9]. Marker-based solutions
use tags placed on elements that let the AR system recognize it
and allows it to convey additional information. Marker-less
systems are for example hybrid tracking methods, feature
Figure 1. Position paper outline. tracking or natural markers. Nowadays, marker-less AR is the
preferred image recognition method depending on the
This research investigates the use of AR as a tool mainly for environment’s real features rather than identifying markers.
IT/OT failure analysis. Failure analysis is a method used in This method eliminates the need for object tracking systems.
product development to determine weaknesses and technical Marker-less systems require computer-aided design (CAD)
risks. As a result, it optimizes the use of resources. models, 3D point clouds, or plane segments. Therefore, a
Identification, prioritizing, and structuring of IT/OT failures complete and accurate data structure is essential.
enables designers to perform failure analysis during the
conceptual design. This analysis through AR is recognized as a 2.2 Prior research AR in maintenance
powerful tool to improve quality and reliability during product
development. A case study in aircraft maintenance training and operations
support suggests that AR technology can improve maintenance

This is a resupply of March 2023 as the template used in the publication of the original article contained errors. The content of the article has remained unaffected.
 818 Sara Scheffer et al. / Procedia CIRP 100 (2021) 816–821

task’s efficiency [10]. The production environment also Interviews were held online in times of Covid-19. Online
provides multiple cases in which AR performs like a work platforms are suitable for qualitative research to conduct
support system in a challenging and complex environment [11]. individual interviews as well as small focus groups.
All cases demonstrate positive results regarding AR and Participants were informed about the topic of interest prior to
complex maintenance procedures in terms of efficient, reliable, the interview. Each interview was scheduled to take
and safe operations. approximately one hour. Semi-structured interviews were used
to allow specifying initial questions, if necessary existing
2.3 Current status of application AR in the railway industry questions were specified or innovative questions were
proposed to be more precise. During the interview, analytic
Previous research has been conducted within the railway notes were made. A summary of the interview was shared and
industry for adopting AR technologies to train technicians and discussed with the participants to verify the outcome.
to assist train drivers during specific operations [12]. A proof
of concept was proposed by targeting the main advantages: 3.3 Field study analysis
increasing operational efficiency and training. Despite the
positive results AR can achieve, several questions remain Qualitative data was analyzed by performing theoretical
unanswered regarding the deployability of these emergent latent analysis [14]. Six steps are required to process data from
solutions. the interviews: (1) familiarize with data, (2) generate initial
codes, (3) search for latent themes, (4) review themes (5) define
3. Methodology and name themes, and (6) product report. To systematically
analyze complex phenomena hidden in the unstructured data,
To examine whether AR can be a suitable tool for ATLAS.ti is used [15]. The analysis was based on open coding
identifying, prioritizing, and providing solutions to failures procedures in which codes are not pre-set but developed and
related to IT/OT converged systems, a case study is performed modified during the coding process. Quotes of interviewees
for the Dutch Railway company (NS). A qualitative field study were assessed and many potential themes are generated for
is used to support the initial root cause examination by coding. After scoping the content of each theme, thematic
providing more and detailed data analysis of the current analysis can be performed. Throughout the entire study, a
situation. This field study supports the motivation of the constant comparative analysis is performed to find
research by verifying whether the failure of complex IT/OT consistencies and differences enabling refinement of the
systems can be detected and further investigated by using AR concepts.
tools.
4. Root cause analysis through Ishikawa diagram
3.1 Characteristics interviewees
Preliminary analysis of utilizing AR technologies within
Participants of this field study were employees of NS. The maintenance procedures of NS revealed multiple complex and
majority of the group was all within the department of NS- interdependent problems. Understanding, structuring, and
technology. Participants were selected based on their prioritizing problems allows obtaining clear and transparent
knowledge on IT/OT convergence, data collection and problem identification. A widely-used method in the rail
management, innovative technology and maintenance transport sector is Root Cause Analysis (RCA) by recognizing
operations knowledge. In total, 28 individuals were selected for the problem in a structured way. Different problem sources are
interviews representing 35% of the total sample size. This distinguished and all directly causing an overall effect on
sample size is sufficient within this specific field of knowledge identifying IT/OT convergence failures. Quick identification of
to generate correct information [13]. Table 1 shows the field of root cause problems in NS can be achieved by using the causal
expertise of the interviewees, some experts had knowledge diagram of Ishikawa, one of the multiple options for
within multiple fields. performing an RCA. This tool is a suitable instrument in
problem-solving methodologies and not only for the analysis
Table 1. Expertise knowledge. of quality characteristics. Recognizing root cause problems
Focus area Number of experts enables companies to implement focused and specific
IT/OT convergence 14 improvement actions [16]. The Ishikawa diagram for finding
Data collection and management 8
proper tools to identify, structure, and solve IT/OT problems in
the current maintenance procedure of NS is presented in figure
Innovative technology 3
2. The diagram is based on interviews and documents collected
Maintenance operations 24 from NS. Firstly, the factors that affect the utilization of AR
technologies are specified. The cause for the failure leads to
3.2 Field study procedure different failure sources. Secondly, failure causes are grouped
into major categories to identify and classify these failure
The dataset was gathered by using a combination of sources. Thirdly, the root causes of the problem can be
theoretical sampling and purposive sampling. This results in a specified. Lastly, the company can address root causes by
process of data collection whereby codes are collected and data allowing specific improvement actions.
analyzed. Based on the analysis, theories can be generated.

This is a resupply of March 2023 as the template used in the publication of the original article contained errors. The content of the article has remained unaffected.
 Sara Scheffer et al. / Procedia CIRP 100 (2021) 816–821 819

Figure 2. Root cause analysis using the Ishikawa diagram.

5.2 IT/OT convergence
5. Field study results
Unsurprisingly, 17 individuals are experiencing difficulties
with IT/OT convergence. The interviewees mentioned that the
Participants were questioned about four main topics: IT/OT
convergence of IT and OT systems is becoming more important
convergence, data gathering and management, transition
in the next decade. IT/OT convergence failure is mostly caused
maintenance procedure, and AR potential application fields.
by new or incompatible IT systems, software bugs, connection
failure, and complex configuration management. Table 3
5.1 Qualitative analysis results
presents the most occurring failure causes of IT/OT systems.
IT/OT complexity represents 41% of all failure causes.
From the interview results, 44 codes were generated. The
codes were divided into 8 themes. Table 2 represents examples Table 3. Most occurring failure causes of digitized trains in NS.
of how quotes were categorized and form a theme.
Failure type Number of times mentioned
by interviewee [%]
Table 2. Quote, category and theme division.
Connection failure 15.9
Quote Code Theme
IT/OT complexity 41.3
AR can be performed on the AR for support Potential
revision of components, brake maintenance work applications for Mechanical failure 11.1
cylinders, bearings and low voltage AR Software bug 20.6
installations.
Configuration management 11.1
Sensitive data needs to be protected Data related to Data complexity
at all times complex problems
Preventive maintenance of IT/OT Comparison of Maintenance Participants did recognize the importance of digitization,
systems is in the development phase maintenance procedures citing benefits such as: increased reliability and operability
and currently only used for high- procedures of trains, less downtime, less maintenance cost, and on-time
risk cyber-security operations traveling. Other often mentioned topics were about the
IT/OT convergence is difficult due IT/OT complexity IT/OT failure importance of cybersecurity in which confidential data needs
to the complexity of the systems, to be protected from cyber-attacks or data leakage.
involving different technologies,
employees, equipment and methods
5.3 Data gathering and management
A continuous connection between Connection failure Failure causes
shore and operation is required but
not always established The results of the field study revealed that data gathering
It becomes possible to predict when Future potential of Future steps
and usage were both considered to be highly complex tasks.
a failure occurs using data data Digitized trains generate a lot of data; this large amount of

This is a resupply of March 2023 as the template used in the publication of the original article contained errors. The content of the article has remained unaffected.
 820 Sara Scheffer et al. / Procedia CIRP 100 (2021) 816–821

data is stored and protected. Different data types can be accept the new technology if the tool is user-friendly and
distinguished: (1) event data based on diagnostic system and supports the work. The AR technologies should be developed
failures, and (2) sensor data which relies on current component in such a way that the use of them does not require intense
condition. However, not all data is analyzed or used. In-house additional learning trajectories.
knowledge is not sufficient to be able to transfer all data.
Currently, too much data is generated such that it becomes 6. Discussion
difficult to distinguish and prioritize data. The quality of data
highly depends on the data source, this should be accurately Due to different systems and technologies, machine
representing real and actual data. Finally, data is partially interfaces are highly complex. From the qualitative analysis,
owned by NS, depending on train type and contracts drafted the following benefits of digitization can be drawn: (1) to have
with stakeholders. safe and reliable operations, (2) less downtime, (3) less
maintenance cost, (4) and on-time traveling. The Ishikawa
5.4 Potential AR applications diagram showed that the following root causes can be
formulated: (1) increased complexity of IT/OT converged
Figure 3 presents potential AR application solutions systems, (2) entanglement of failure types, (3) involvement of
specifically mentioned by the interviewee. The most frequently different stakeholders, and (4) transition towards digitized
mentioned examples were: (1) fault detecting, (2) providing maintenance operations.
work descriptions, (3) inspecting components, (4) remote
control, and (5) assistance for specialized work. Different AR 6.1 Root cause analysis
application fields can be distinguished. Maintenance
application fields offer the best potential for AR technologies. IT/OT becomes increasingly important for the railway
In total, 72% of the interviewees consider maintenance as a industry in the next decade. Operating systems such as
good application field. cybersecurity and operating physical components are relevant
for IT/OT converged trains. Not all converged IT/OT systems
25,00
have the same, safety requirements, reporting structure, and
Number of times mentioned by

20,00 environment. This means that configuration management must

15,00 always be up to date.
interviewee [%]

10,00 Failure types highly depend on component lifecycle,

5,00 knowledge of systems, and complexity of procedures. Based on
- the Ishikawa diagram and the qualitative field study, the most
common failures are: (1) connection failure, (2) IT/OT
AR for passenger
AR for inspections

AR for train driver

AR for unsolicited
AR for support

AR for fault detection

AR for educating and

AR for specific cases

AR for remote control

maintenance

complexity, (3) mechanical failure, (4) software bugs, and (5)

comfort

advice

insufficient configuration management. Most failures are

training

caused by complicated IT/OT systems which are often badly

understood by the asset manager. This complicatedness is
caused by having different and more IT systems on the train
Application field
and the interdependency of those systems. Due to the
digitization of trains, new trains possess more and different
Maintenance Passenger functionalities. Current maintenance operations need to be
Training Train operation support
transferred into digital-oriented and data-based approaches.
Figure 3. Potential AR applications based on qualitative field study. Structured, complete, and clear data is required for AR tools.
Proper data sources are not always available, resulting in
5.5 Transition maintenance procedure inaccurate data. Also, data exchange systems are not up-to-date
in all occasions. Train data, such as sensor and event data, is
Maintenance operations are currently more based on gathered and stored in a centralized data repository. The
performing corrective and preventive tasks rather than transition to a digitalized maintenance concept creates
predictive tasks. New IT/OT systems require a different opportunities to use AR for failures related to converged IT/OT
maintenance setup and thus a new and improved maintenance trains.
concept. The life cycle of IT systems is significantly shorter Different stakeholders are involved such as system
compared to hardware systems. Digitized trains can self- integrators, (sub)contractors, and technicians or engineers.
diagnose failures by sensing and monitoring current train Clear collaboration is required to transfer knowledge between
component conditions. In the future, NS aims to perform stakeholders. Currently, technicians and engineers have limited
primarily predictive maintenance by gathering and analyzing IT knowledge which is required for configuration management
data on the way to Prognostics and Health Management. and maintenance of those new technologies.
Failure complexity and transition of maintenance procedures Specific AR maintenance application fields are (1) fault
are important to consider as well. When new technologies are detection, (2) inspections, (3) unsolicited advice, (4) remote
introduced, local ambassadors are often used. Local control, (5) special cases, and (6) support of maintenance tasks.
ambassadors test and promote new technologies. Based on the Applying AR in maintenance operations ensures early
field study, it is estimated that 75% of the technicians will identification and a better understanding of failures, analysing

This is a resupply of March 2023 as the template used in the publication of the original article contained errors. The content of the article has remained unaffected.
 Sara Scheffer et al. / Procedia CIRP 100 (2021) 816–821 821

and visualizing component degradation, remote control and Acknowledgements

assistance, work instructions, assistance for specialized work,
and to alert technicians when performing high-risk tasks. This research is co-financed with a PPP surcharge for
Acceptance of new technology is key when it comes to Research and Innovation from the Dutch Ministry of Economic
introducing AR. Concepts need to be introduced properly, Affairs and Climate, the Netherlands Railways, and the
preferably by using local ambassadors. The added value of the University of Twente.
new technology must be clear and user-friendly.
References
6.2 Future directions
[1] Villalba AB. How to Speed up Digitization in the
In the next few years, the transition into digitized Railway. IEEE Electrif Mag. 2020;8(1):75–6.
maintenance operations will be extremely important for the [2] Kraeling M, Fletcher D, Kraeling M. Railroad assets:
railway industry as data and real-time monitoring are paving information and operational convergence. 2017.
the way towards predictive maintenance procedures. To [3] Lim FP. A Research Analysis on the Convergence of
achieve this, data needs to be exploited for both analysis and Information and Operational Technology. 2016.
making causal connections. Data regarding IT/OT failures [4] Titu AM, Stanciu A. Merging Operations Technology
needs to be more structured, organized, and prioritized before with Information Technology. Proc 12th Int Conf
being able to getting a better insight into systems. Information Electron Comput Artif Intell ECAI 2020. 2020.
management must be clear and ensure secure operations. Both [5] Ong SK, Yuan ML, Nee AYC. Augmented reality
IT and OT departments are required to collaborate for broader applications in manufacturing: a survey. 2008.
knowledge sharing. AR opens the way towards structuring and [6] Fernández del Amo I, Erkoyuncu JA, Roy R, Palmarini R,
recognizing failures of digitized trains. It is therefore strongly Onoufriou D. A systematic review of Augmented
believed that AR is a proper tool to be deployed for Reality content-related techniques for knowledge
maintenance operations in the railway industry. transfer in maintenance applications. Comput Ind.
2018;103:47–71.
7. Conclusion [7] Bottani E, Vignali G. Augmented reality technology in the
manufacturing industry: A review of the last decade.
The research presented in this paper suggests that IISE Trans. 2019;51(3):284–310.
identification, prioritizing, and solving failures of IT/OT [8] Kaufmann H, Csisinko M. Multiple Head Mounted
converged trains can be performed by using a tool such as AR. Displays in Virtual and Augmented Reality
Relevant research challenges and future directions in applying Applications. 2006;5(3):1–8.
AR to complex environments are proposed. Regarding the [9] Platonov J, Meier P. A mobile markerless AR system for
deployability of technology, it has been shown that the tool can maintenance and repair. 2006;105–8.
be used in different applications of the maintenance operation. [10] Crescenzio F De, Fantini M, Persiani F, Stefano L Di,
In this article, we proposed a novel direction for providing Azzari P, Salti S. Augmented Reality for Aircraft
future directions of AR in maintenance operations for IT/OT Maintenance Training and Operations Support. 2011.
converged systems which allow non-experts to take top-level [11] Lorenz M. Industrial Augmented Reality: Requirements
decisions. AR can be used as a tool to support maintenance for an Augmented Reality Maintenance Worker
operations by identifying, structuring, and provide solutions to Support System. :1–3.
IT/OT failures of digitized trains. [12] Martinetti A, Hart K, Damgrave R, Dongen LAM Van,
Although AR has been proven to have good potential in Turkenburg R, Nouwens A. There is no spoon:
maintenance, repair, service, and inspection, many of these applying virtual reality for maintenance training of
applications are still at the prototype stage and have not found rolling stock technicians. 2018;8(4):398–415.
significant adoption in the industry. Looking at the industrial [13] Malterud K, Siersma VD, Guassora AD. Sample Size in
sectors where AR has been used, it is difficult to identify a Qualitative Interview Studies. 2016.
sector that is especially benefitting from AR technologies [17]. [14] Maguire M, Delahunt B. Doing a thematic analysis: a
Many technical studies were carried out only in laboratory practical, step-by-step guide for learning and teaching
settings, without implementing the AR system in a real context scholars. All Irel J Teach Learn High Educ.2017;8(3).
[7]. Existing real case scenarios lack AR solutions that target [15] ATLAS.ti. The world of data in your hand. cited 2020
specific areas. Prototyping of AR technology in combination Nov 26. Available from: https://atlasti.com/
with maintenance operations should be evaluated. Different [16] Suárez-barraza MF, Rodríguez-gonzález FG.
case studies are required to analyze the effect AR has as a tool Cornerstone root causes through the analysis of the
for identifying, structuring, and solving IT/OT failures. Experts Ishikawa diagram, is it possible to find them? A first
from the industry can verify proposed solutions before direct research approach. 2019;(1985):302–16.
adoption of results. Other future works will include the [17] Martinetti A, Marques HC, Singh S, Dongen L Van.
implementation in the process of a tool for assessing the Applied Sciences Reflections on the Limited
ergonomics and economic aspects of the AR application. Pervasiveness of Augmented Reality in Industrial
Sectors. 2019.

This is a resupply of March 2023 as the template used in the publication of the original article contained errors. The content of the article has remained unaffected.