______________________________________________________PROCEEDING OF THE 26TH CONFERENCE OF FRUCT ASSOCIATION

Case Study of Using Virtual and Augmented

Reality in Industrial System Monitoring
Dmitry Pavlov, Igor Sosnovsky, Vyacheslav Dimitrov, Vasilii Melentyev, Dmitry Korzun
Petrozavodsk State University (PetrSU)
Petrozavodsk, Russia
{dpavlov, sosnovsk, dimitrov}@cs.petrsu.ru, [email protected], [email protected]

Abstract—An equipment monitoring system is responsible In this study, we select several reference analytical ser-
for analytical services that assist the personnel in equipment vices that come from our pilot development with metalcutting
diagnostics, fault prediction, and post-analysis. In this work, we industry in Russia. The two service models are proposed:
consider a case study of using virtual and augmented reality
(VR/AR) for effective analytics delivery to the production equip- (a) Situational Center and (b) Virtual Assistant. We identify
ment personnel. The reference analytical services for our study the VR/AR advantages compared with the traditional style of
come from our pilot development with metalcutting industry in visualizing analytical services on computer screens. Our study
Russia. The two service models are proposed: (a) Situation Center is supported with prototype implementation of the VR/AR
and (b) Virtual Assistant. We identify the VR/AR advantages system for analytics delivery to the equipment personnel.
compared with the traditional style of visualizing analytical
services on computer screens. Our study is supported with The rest of the paper is organized as follows. Sec-
prototype implementation of the VR/AR system for analytics tion II provides an overview of analytical services useful in
delivery to the personnel. production equipment monitoring. Section III considers our
VR model for constructing analytical services for a situational
I. I NTRODUCTION center. Section IV considers our VR model for constructing
Modern enterprises use advanced information systems analytical services for a situational center. Section V describes
for equipment monitoring to optimize the production operation our prototype implementation of VR and AR based variants
and quickly respond to faults [1]. Multiple sensors are installed of analytical services.
on machines to collect data and to transfer the data to server
II. A NALYTICAL S ERVICES IN P RODUCTION E QUIPMENT
computers, where they are converted into a convenient form for
M ONITORING
the specialist to understand. The traditional method of check-
ing equipment is long, and searching for faulty equipment can A. Problem
lead to long downtime of the entire production line, while One of the tasks in maintenance is to analyze the
the equipment monitoring system in the enterprise has the condition of the equipment. Technical personnel need to detect
ability to check machine components in real-time and report problems related to equipment components in a timely manner
the failure of specific equipment. in order to eliminate them with minimal harm to production.
An equipment monitoring system collects big data to If the problem was not noticed, the situation may become
analyze the state of the entire production. Then a person critical, which may later stop the production line.
(equipment personnel) makes decisions depending on the An equipment monitoring system is installed in enter-
situation. In order to speed up understanding of the problem prises to help detect problems. A system capable of collecting
and finding a solution, the information from sensors and data from sensors on equipment for subsequent interpretation
services needs representing in a human-friendly form [2], [3]. in a form that is understandable to a person. It is also possible
which can help virtual and augmented reality—VR and AR, to perform a primary analysis, in which the monitoring system
respectively. is able to give advice to technical personnel on how to fix
The VR technology provides a new level of image the found vulnerability or notify them of the found fault in a
display through the use of a helmet [4]. The application of timely manner. Work [6] describes some representation of the
the stereoscopic effect, which gives a full immersion in the monitoring system shown in Fig. 1, which is divided into two
virtual world and allows you to feel the depth of the image, global worlds: physical and cyber. The physical world includes
with which you can determine the distance to the object and real objects involved in production (machines, equipment,
better perceive three-dimensional objects. people). Cyber world contains a software part consisting of
The AR technology is an auxiliary tool for assessing services, modules, and algorithms and is able to control the
(diagnosing) the equipment status situation [5]. By superim- physical world.
posing additional information on real objects, it is possible to The monitoring system uses large data collected from a
evaluate the machine condition based on sensor data. Technical variety of sensors installed on the equipment. For their analy-
personnel can easier check the equipment and determine the sis, it is necessary to process the received data for subsequent
cause of problems. analysis by a human [7]. The more clearly the information is

ISSN 2305-7254

Authorized licensed use limited to: University of Exeter. Downloaded on June 17,2020 at 15:36:43 UTC from IEEE Xplore. Restrictions apply.
 ______________________________________________________PROCEEDING OF THE 26TH CONFERENCE OF FRUCT ASSOCIATION

Fig. 2. Control panel example

Fig. 1. Monitoring system scheme

presented, the more confident and faster the person will be

able to make a conclusion and make a decision [2].
Virtual reality technologies are able to display three-
dimensional objects and make it possible to interact with
them. As an example, you can imagine a three-dimensional
graph with data displayed on three axes at once and see the
entire volume of the graph due to the stereoscopic effect.
It is also possible to visualize a three-dimensional model
of a production facility (machine, workplace) and show the
location of sensors directly on the model for more detailed
identification of the problem.
Augmented reality differs from virtual reality in that
it overlays three-dimensional models and images on real
objects. Thus, this technology can be used to quickly assess
Fig. 3. Report example
the condition of equipment when it is checked for faults by
using data that is usually obtained from computers or paper
documentation, while being near the equipment being checked. describes list of operations which the user should perform.
Due to the development of technologies related to The system fix start time of step, finish time of step, state of
data visualization, it is possible to improve the perception system and allows the user to add some descriptions, photos,
of information and thus simplify and speed up the work of videos, etc.
technical personnel and help them solve problems related to All these three types of user interface are static. The in-
the maintenance of production equipment. terface binds the user to computer screen (desktop or mobile),
B. Existing solutions where all the analytic information from services are visualized.
Many existing industrial equipment monitoring systems
C. Advantages of VR/AR technologies
suggest three types of user interfaces. The first type is dash-
board for real time state in whole system (e.g., values of The types of interfaces for human interaction discussed
sensors, different charts, new and current tasks, user activities in section II-B can be implemented in virtual reality, which
and so on). The dashboard is designed for quick decisions. A will improve data perception and take advantage of the tech-
dashboard example is shown in Fig. 2 from Softweb Solutions nology used to improve data perception. The advantages of
Inc. (www.softwebsolutions.com/). VR, as well as examples of using VR, are shown in table I.
The second type of user interfaces is reports. Typical Overlay-based AR provides an alternative view of the
reports are generated for selected period and are shown system object in question, either replacing the entire object with
changes in this period. The report is designed for medium and an augmented view, or replacing part of the object with an
long decisions. A reporting example is shown in Fig. 3. augmented view. In this case, object recognition plays a vital
The third type of user interface is assistance. The logical role. if the app does not know what it is looking at, it
assistance provides sequence steps for operator. Each step can’t replace the original view with an augmented one [8].

---------------------------------------------------------------------------- 368 ----------------------------------------------------------------------------

Authorized licensed use limited to: University of Exeter. Downloaded on June 17,2020 at 15:36:43 UTC from IEEE Xplore. Restrictions apply.
 ______________________________________________________PROCEEDING OF THE 26TH CONFERENCE OF FRUCT ASSOCIATION

Depending on what type of presentation is required, the According to the description of services in Table III, one
technology can be used for several purposes [8]. can say that for displaying information from most services,
1) Doctors can use this technology to examine a patient visualization can be used in the form of graphs or color
from various points in real time. A live broadcast from indicators, as well as in the form of text data. For other
an x-ray machine can be used to superimpose an x-ray services, one can use data visualization in the form of 3D
image of a patient’s body part onto a real body part to models, checklists, or text and visual instructions.
better understand bone damage. The app can be made to
III. S ITUATION C ENTER : U SE OF V IRTUAL R EALITY
work with a head-up display or special glasses. In other
cases, the image may be shown on a screen where the The situation center uses an autonomous Oculus Quest
video is taken from a real camera and x-ray vision may virtual reality helmet to view data. Management in the virtual
be applied to it [8]. space is carried out using Oculus Touch controllers, which
2) In military applications, an AR-based overlay can pro- are presented inside the situation center in the form of virtual
vide multiple views of the target object without showing hands. The application requires a free space of more than 4
additional information in the text and blocking the square meters and an Internet connection via Wi-Fi.
soldier’s vision from other important objects around. If The situation center model is shown in Fig. 4. The
you were shooting at enemies with a computer mouse, concept model includes four levels:
you would already know what it would look like. 1) the target machine is located on the first level
Superimposing an infrared view or a radioactive view 2) at the second level, sensors capture data from the ma-
of an object or area can help save lives or win wars [8]. chine.
3) Overlaying ancient paintings on real objects can provide 3) at the third level, services process data from sensors.
interesting views of historical places. Broken monu- 4) at the fourth level, the situation center accepts data from
ments can come back to life in all their original glory. services and converts it into graphs and status indicators.
Perhaps different eras complete with landscapes can be it is also possible to change some parameters to change
re-lived with AR [8]. the request for data from services, for example, to output
4) Letting a tiger or snake get close to you can be an data for a certain period of time.
unpleasant experience with dangerous consequences, The situation center is a virtual room with a menu in
except when AR overlay is used. Placing a person in the form of a screen fixed in the air. Fig. 5 shows how the
a place or situation that is otherwise dangerous can be menu will look on the screen and its variations depending on
done safely using AR overlay [8]. the user’s choice. The screen shows the interface, which has
5) Overlaying a real object with its internal appearance can buttons with the corresponding text. The initial screen is used
also be useful in education, for example, for studying the
structure of bones [8].
According to the information provided in Table II, the
advantage of AR over desktop PCs and text documentation
is that in AR all data are presented in real time right in
front of the person’s eyes and, therefore, they do not have
to spend time accessing the PC or text documentation. Also,
in AR, the system itself can offer the necessary instructions
for setting up, performing diagnostics or Troubleshooting,
without being distracted by searching for information in the
text documentation. AR will help reduce the time spent on site
maintenance and reduce the likelihood of human error.

D. Reference analytical services

The monitoring system uses services for processing raw
data from sensors that take readings from the machine, thereby
tracking its status. Various services can read data from specific
sensors to obtain specific conclusions, based on which the
question of the serviceability of the machine component is
decided.
Table III provides a list of services that are used in the
monitoring system and their purpose. Each service is used in
certain cases, these cases are described in the 3rd column of
the table. The second column contains the name of the service,
which indicates the scope of this service. Fig. 4. Situation center model

---------------------------------------------------------------------------- 369 ----------------------------------------------------------------------------

Authorized licensed use limited to: University of Exeter. Downloaded on June 17,2020 at 15:36:43 UTC from IEEE Xplore. Restrictions apply.
 ______________________________________________________PROCEEDING OF THE 26TH CONFERENCE OF FRUCT ASSOCIATION

5"#-&
*
VR
ADVANTAGES

Advantages Example of use Justification

Real-time data visualization with easier Virtualitics recently launched its VR product that puts the In contrast to conventional data visualization models,
pattern recognition in large data sets and researcher directly into the data. It also uses machine learning virtual reality allows users to walk and view data from
more intuitive data understanding to help analysts determine which factors will ensure the best different angles. in addition, it is easier for people to
display and creation of a multidimensional view of data. [9]. perceive information in the form of three-dimensional
shapes, objects, compared to flat images.
Reduce production time and costs One of the world’s leading car manufacturers, Ford, has created In VR, the user reacts faster to all events because the
a virtual reality laboratory. This allows engineers to quickly entire interface is intuitive, which allows faster data
and easily improve the quality of early concepts by actually analysis.
designing new car designs. With VR, Ford manufacturing crew
can now design cars before creating a physical prototype. This
speeds up time to market and reduces production costs [9].
Building and implementing various In VR, you can display any 3D objects and manipulate
three-dimensional shapes and objects 1) A similar concept is already used in Balfour Beatty Rail. them, which allows you to understand what the object
This railway infrastructure contractor integrates the ben- from which the readings come looks like, and display
efits of VR into the design process. It provides planning, the color statuses of individual nodes of this object.
prototyping and construction of objects, helping in the
implementation of projects [10].
2) Another example is the design of a car. The car itself
is a complex structure. There are many important details
in the design of even the simplest cars. A car design
created in VR can increase the efficiency and productiv-
ity of creating a real car design. The JLR virtual reality
center in the UK takes advantage of virtual reality and
turns VR into a work of art. This service offers both
semi-immersive and CAVE systems with projection and
advanced tracking. Probably the biggest part is that this
VR center is being used to design the next-generation
Land Rover [10].

TABLE
II
"3
"%7"/5"(&4

Advantages Example of use Justification

Free access to technical and expert in- Elevator manufacturer thyssenkrupp has announced that its AR allows a person to reduce the time needed to
formation on site specialists will start using Microsoft HoloLens technology as solve a problem by setting up equipment, performing
a tool in service operations. Using HoloLens, service techni- maintenance, or identifying and fixing a problem that
cians can visualize and identify problems with elevators before has occurred.
working, as well as have remote, free access to technical and
expert information on site. As the first fully Autonomous holo-
graphic computer running Windows 10, Microsoft HoloLens
helps companies and industries innovate in completely new
ways. Using the Skype app without any additional development,
thyssenkrupp service engineers can now perform their work
more safely and efficiently. Sorting out service requests in
advance of a visit and getting a hands-free holographic guide
when you are on site has reduced the average duration of
thyssenkrupp service calls by up to four times [11].
Superimposing digital information on a AR allows you to overlay a digital image on a real
real object 1) AccuVein, a new York-based company, uses AR to help object, which allows you to analyze the situation, plan
nurses find veins easier when installing IVS. This makes an action plan without leaving the object itself.
life easier for nurses and patients, increasing the success
rate of the IV by 350% [12].
2) AR also helps some patients in their recovery process.
One company, called NuEyes, uses special AR glasses
to help people with severe visual impairments. With this
technology, NuEyes can help blind children see well
enough to read and recognize their classmates [12].
3) There is evidence that AR can help reduce the excru-
ciating phantom limb pain experienced by amputees.
By projecting a digital limb onto a patient, researchers
were able to trick their brain into thinking that the
amputated limb was still there. This projection, paired
with electrodes, allowed patients to practice relaxing the
digital limb to ease pain [12].

---------------------------------------------------------------------------- 370 ----------------------------------------------------------------------------

Authorized licensed use limited to: University of Exeter. Downloaded on June 17,2020 at 15:36:43 UTC from IEEE Xplore. Restrictions apply.
 ______________________________________________________PROCEEDING OF THE 26TH CONFERENCE OF FRUCT ASSOCIATION

TABLE
III
SERVICES
AND
THEIR
APPLICATION

# Service Application
1 ”Control of the swivel head installation” Service is used to determine the correct positioning of the swivel head
2 ”Fixing various errors that occur on the machine screen” the Service is designed to recognize errors that appear on the machine monitor in
order to determine the most complete description of new errors that appear (their
interpretation) and notify the working staff about their appearance
3 ”Creating tasks for technical personnel” the Function allows you to create maintenance or inspection points for technical
personnel, as well as to record the implementation of these points (fixing the passed
stage, adding photos, adding descriptions and comments)
4 ”Mobile element impact Detection” the Service records the occurrence of a mobile element impact event, records the
readings of all sensors, video sequences from surveillance cameras, and a set of
events preceding and following the impact for a certain period before and after the
event
5 ”Monitoring the current of the motor axes and spindle” when detecting abnormal consumption or current changes informs the user of sudden
increases in load or deviations from normal values, and also allows you to analyze
the readings from all current collectors
6 ”Help in hardware diagnostics” the System provides the ability to display data from various sensors and services on
a single screen, which allows the employee to track the status of the system as a
whole
7 ”Control of spindle bearings and motor” Service allows you to track the occurrence of vibrations and warn the user about it
8 ”Temperature control of auxiliary devices” Service allows you to monitor the temperature of pumps and inform users about
deviations from the norm, which will allow you to service or replace the unit in
time
9 ”Control of coolant supply during processing” the Service is used to display the current status of the coolant supply (present, absent)
and in case of atypical behavior of the feed notifies the responsible persons
10 ”Control of counterweight installation of the processing Service controls the installation of counterweights for the processing head
head”
11 ”Tracking the location of the operator” the Service allows you to track the presence of the machine operator and warn
interested parties in case of his absence from the workplace or when he performs
actions for which he does not have sufficient rights (for example: opening the
electrician’s Cabinet, entering the machine during operation)
12 ”Monitoring of machine load” Using data obtained from installed sensors, the service will allow you to de-
termine the level of machine load and generate reports indicating the time of
operation/downtime of the machine and the nodes involved
13 ”Diagnostics of linear guide axes X, Y, Z” the Service provides the user with an interface (”assistant”) for performing a
sequence of steps for diagnosing linear guide axes

for a quick assessment of the situation. it shows the status of

all machines, the workplace (whether there is a person in the
workplace), and a column with notifications. The status of the
machines is shown as a set of lines, each of which contains
a unique ID of the machine and its status as a color indicator
(green - everything is normal, yellow-needs repair, red-out of
order). The workplace is shown as a list with a description
of the location, who should go to work and an indicator
of the person’s presence in the workplace. The notification
column displays a list of recent notifications and sorts them by
importance. An example of a situation center implementation
can be seen in Fig. 6.
When you select a specific machine, a new screen
appears that includes:
1) list of sensors used in the machine
2) recent notifications related to this machine,
3) the location of the machine.
Each sensor in the list of sensors has a unique identifier, and Fig. 5. On-screen menu
when you select one of the identifiers, a new screen opens,
which displays a visual representation of data from this sensor
in the form of a graph, chart, etc. 2) for the purpose of the workplace,
When selecting a specific job from the list, a screen 3) information about the employee who should currently
appears that contains information consisting of: be at the selected workplace,
1) location of the workplace, 4) images from a security camera.

---------------------------------------------------------------------------- 371 ----------------------------------------------------------------------------

Authorized licensed use limited to: University of Exeter. Downloaded on June 17,2020 at 15:36:43 UTC from IEEE Xplore. Restrictions apply.
 ______________________________________________________PROCEEDING OF THE 26TH CONFERENCE OF FRUCT ASSOCIATION

Fig. 6. Example of the interface

Events of varying degrees of importance appear in

the notification column. The most important notifications are
displayed first in the order they appear (events of equal
importance are sorted by the time they appear in the column).
When you select a specific notification, a screen opens with a
description that indicates the reason for the notification and the
associated equipment (machine, sensor), and the notification
can also be associated with the workplace (the employee is
absent from the workplace for a long time, is late, etc.). Fig. 7. Virtual assistant model
Graph visualization is a three-dimensional object in the
form of a screen that can be grasped, stretched, compressed, or
fixed in space. Multiple graphs can be combined into a single The assistant can also be installed on a mobile phone/pad that
graph based on data from two other graphs. Services that use supports ARCore.
graphs to display data are shown in Table III under the items This app can highlight the position of the sensors on the
4, 5, 6, 7, 8. machine in a color that indicates the state of the sensor (for
Another way to provide information are statuses. Using example, red-the sensor readings are out of the normal range,
color indicators, you can quickly understand the status of the green-does not go out, yellow-close to the limit), when the
equipment or the importance of information, which gives a AR device camera is pointed at the machine. After that, the
quick assessment of the situation. Status indicators are used user can view information about the sensor of interest in more
in services under items 9, 10, 11, 12 in Table III. detail by selecting it with a hand gesture, focusing the camera
Some services use a method for delivering important on the object or tapping on the phone screen on the object of
information, such as notifications. Using this method, you can interest to us. More detailed information can contain graphs,
find out important information in a timely manner and study tables, and color and text status displays, with a description of
the situation in more detail. Notification support is provided the status and offer instructions for correcting situations, which
for services under items 1, 2, and 3 in Table III. can also be selected by focusing the AR device’s camera, using
a hand gesture or tap.
IV. V IRTUAL A SSISTANT: U SE OF V IRTUAL R EALITY Instructions can be presented in the form of checklists,
The virtual assistant entity is shown in Fig. 7. The text or visual algorithms for actions to correct the situation
picture shows that the entity consists of 5 layers. Sensors that caused the sensor indicators to increase above the norm.
collect information from the machine, then transmit it to the The app can also display a check list, for example, for
server. The server, in turn, processes this information and passing something that the user marks on their own.
passes the information to the virtual assistant upon request. It is possible to display the current date and time with
The virtual assistant interacts with the user and generates an accuracy of seconds.
requests to the server based on their actions. In AR, graph visualization is a small screen that appears
Virtual assistant is an application for augmented reality when you hover over the module. You can stretch the graph
glasses or a mobile phone / pad that allows you to add using gestures, head movements, or taps on the phone screen.
useful information to the world around you in real time. The Services that use graph data visualization are listed in Table III
assistant can be installed on hololens 2 augmented reality under numbers 4, 5, 6, and 8.
glasses. Control in this case is performed by simple hand Also, in AR, visualization can be represented as statuses
gestures, focusing the eye on the object or voice commands. (color or text indicators) of States. They can be used to quickly

---------------------------------------------------------------------------- 372 ----------------------------------------------------------------------------

Authorized licensed use limited to: University of Exeter. Downloaded on June 17,2020 at 15:36:43 UTC from IEEE Xplore. Restrictions apply.
 ______________________________________________________PROCEEDING OF THE 26TH CONFERENCE OF FRUCT ASSOCIATION

Fig. 8. Example of virtual assistant interface

assess the status of the sensor, module, machine as a whole, or

the importance of information. Visualization using indicators
can be used to display services presented in Table III with
numbers 4, 5, 9, 11, and 12. Fig. 9. Example of displaying graph data
In addition to schedules and statuses, a virtual assistant
can be organized in AR, which will provide graphical or text
instructions for resolving a problem, configuring equipment,
or performing maintenance. The virtual assistant can be used
in the services listed under items 6 and 13 in Table III.
Two other possible ways to display data are reports on
events that occurred prior to the failure, and a checklist for
personnel to perform diagnostics or configure equipment. This
data display can be used in services with numbers 3 and 13
in Table III.
In Service 1 of Table III, data can be displayed as a 3D
model corresponding to the current position of the rotating
head, indicating the current position of the rotation angle.
Fig. 8 shows an example of displaying the virtual as-
sistant interface. This interface is displayed when the machine
image is recognized. Successfully recognizing machine, the
assistant will highlights place, or will form a small 3D model
of the sensor with the color corresponding to the current state Fig. 10. Example of a checklist in the expanded state
of the sensor with a small status, for example, the green color
sensor values in the normal range, yellow - readings a little
above normal, but are not critical and red - readings are off
the charts, it is necessary to conduct urgent maintenance.
When you hover over a specific sensor model, a graph
will open with its readings, which can be adjusted to receive
real-time data or to display the history. An example of this is
shown in Fig. 9.
In addition, there will be a check list on the side of
the entire interface, which can be either hidden or opened. A
checklist example is shown in Fig. 10.

V. P ROTOTYPE I MPLEMENTATION Fig. 11. Hardware and software components

A. High-Level Architecture
The module system gathers data from the machine
Hardware and software components of implementation
sensors and save to the data base or notify about this server
are represented in Fig. 11. Each user uses AR/VR device
(API). The RabbitMQ message system is used for this purpose.
with application for interaction with UI. Application has
the following function: requesting the server for data and
B. Data Scaling
visualizing the data for the user.
The application uses specified API for getting data from Sensors of the module system generates huge values
server. API extracts data from database or getting those direct of data (for example, 200000 values in one second). In this
from the module system for visualization data in real time. case visualization of this data using some components (for

---------------------------------------------------------------------------- 373 ----------------------------------------------------------------------------

Authorized licensed use limited to: University of Exeter. Downloaded on June 17,2020 at 15:36:43 UTC from IEEE Xplore. Restrictions apply.
 ______________________________________________________PROCEEDING OF THE 26TH CONFERENCE OF FRUCT ASSOCIATION

Fig. 12. Example of calculating the detail levels with the max aggregation
function

example, chart) is not a trivial task. All data in one chart is

non informative, and user cannot analyze this chart. Fig. 13. Interaction scheme
We develop a concept for visualization this data in more
informative way. The idea is using levels with details. Each
new level has new details. This principle is used in interactive he/she may select this period and chart loaded new levels of
maps (for example, Google Maps, Apple Mas, Yandex Maps details. Each selected point transforms to next level of details.
and OpenStreetMaps): when the user increases the detail level First version of implementation was done for WEB
of new raster images are loaded with new details. When the client. The server splits data into levels of details, the client
user decreases level of details new raster images are loaded selected necessary data from levels, constructed and redraw
with less details. In this case, for representation of whole world charts. But loading all levels of details from server to client in
more than 10 images are used. one request may spend a lot of time (for example in case data
For solving our problem with visualization of large in period several months). For levels cache was developed. In
amount of data we used the same principle. The more data first request server sends to client only max level of data and
for visualization, the less details on the chart. For decreasing caching other levels. In case user want to get next detailed data
data on the chart aggregation function is used (for different next request will be done. Common scheme of interaction is
sensors different aggregation function may be used (max, min, represented in Fig. 13.
avg, and so on). After testing algorithm and interaction with WEB sys-
tem, the component was be moved in API for AR/VR appli-
One of the main parameters of details is “window”
cation. And WEB client uses API now.
of aggregation. This parameter defines amount of data for
applying aggregation function. In case amount of data less C. API implementation
than this parameter, then levels of details are not required.
For getting data HTTP-requests are used. There are two
For example, we have 24 values of data and window of
types of requests:
aggregation is 4 and max as function of aggregation. Zero
level of aggregation (without any data processing): 1) Getting list of services (parameters not needed)
4195|6720|8135|8357|8184|6746 Result of this request is JSON document with list of
Amount of data more than defined window aggregation, service, where each element is JSON document with
and we decided construct next (first) level of details. This identification, name and description of service.
level has values which are calculated by applying aggregation 2) Request for data of service (parameters: identification of
function (in our case is max) for which data sets with window service, start and finish time of data for period in UNIX
aggregation amount: timestamps)
9 7 8 8 |8 7 Result of this request is list of data where each element
is JSON document with value and timestamp of data.
First level of details has amount of data more than
defined window aggregation again. We applied aggregation The Unreal Engine 4 can do requests due to TCP
function another time and now we have next (second) level of connection plugin:
aggregation. 1) Connect to IP-address and TCP-port.
88 2) Send request to server.
Now we have three levels of details. These levels may 3) Wait response from server.
have loaded for user as necessary. The example of calculating 4) Get request in one string.
levels of details is shown in Fig. 12. 5) Close TCP connection.
The algorithm splits the data in levels after user request After the data is received from the server, the data is
for data and user has view chart with max level (minimum ”pulled” from the string and further processed. This part is
data). In case the user understands that some data is interesting, implemented by describing the function in C++ using Visual

---------------------------------------------------------------------------- 374 ----------------------------------------------------------------------------

Authorized licensed use limited to: University of Exeter. Downloaded on June 17,2020 at 15:36:43 UTC from IEEE Xplore. Restrictions apply.
 ______________________________________________________PROCEEDING OF THE 26TH CONFERENCE OF FRUCT ASSOCIATION

Studio C++ and specifying metadata so that the function can [2] M. D. Lee, M. A. Butavicius, and R. E. Reilly, “Visualizations of
be used as a blueprint. After data is extracted from the server binary data: A comparative evaluation,” International Journal of Human-
Computer Studies, vol. 59, no. 5, pp. 569–602, 2003.
response, it is written to a structure that stores arrays of
[3] F. Ansari, P. Hold, and M. Khobreh, “A knowledge-based approach
data: an array of values, an array of timestamps in UNIX for representing jobholder profile toward optimal human–machine col-
Timestamp, and an array of timestamps in the classic date laboration in cyber physical production systems,” CIRP Journal of
and time representation. Manufacturing Science and Technology, vol. 28, pp. 87–106, 2020.
Next, the data is displayed using a loop on the graph. [4] H. Qin and J. Lei, “The application of virtual reality technology
in advertising communication,” in 2019 International Conference on
Each point of the graph has two values that correspond to the Virtual Reality and Intelligent Systems (ICVRIS), 2019, pp. 73–76.
index elements of the value and timestamp arrays.
[5] P. Fraga-Lamas, T. M. Fernández-Caramás, Ó. Blanco-Novoa, and M. A.
Vilar-Montesinos, “A review on industrial augmented reality systems for
VI. C ONCLUSION the industry 4.0 shipyard,” IEEE Access, vol. 6, pp. 13 358–13 375, 2018.
This paper considered a case study of using virtual [6] F. Tao and Q. Qi, “New IT driven service-oriented smart manufacturing:
and augmented reality for effective analytic delivery to the Framework and characteristics,” IEEE Transactions on Systems, Man,
equipment personnel. We selected several reference analyti- and Cybernetics: Systems, vol. 49, no. 1, pp. 81–91, 2019.
cal services from our pilot development with metal cutting [7] F. Tao, Q. Qi, A. Liu, and A. Kusiak, “Data-driven smart manufacturing,”
Journal of Manufacturing Systems, vol. 48, pp. 157–169, 2018, special
industry in Russia. The two service models were proposed: Issue on Smart Manufacturing.
(a) Situation Center (VR-based) and (b) Virtual Assistant [8] Different types of augmented reality. Digit
(AR-based). Our prototype implementation indicated that the community, chapter 3 in FastTrack To Augmented
proposed service delivery models have certain potential for Reality [accessed 01.04.2020]. [Online]. Available:
https://www.digit.in/technology-guides/fasttrack-to-augmented-reality/
production equipment monitoring in industrial enterprise. different-types-of-augmented-reality.html
ACKNOWLEDGMENT [9] Y. Velykholova, “6 ways to empower the enterprise with virtual
reality solutions,” Dec. 2017. [Online]. Available: https://www.n-ix.
This research is financially supported by the Ministry com/ways-empower-enterprise-virtual-reality-solutions/
of Science and Higher Education of Russia within project no. [10] Rooney, “6 examples of virtual reality applications and
075-11-2019-088 (Creating the high-tech production of mobile how it works,” blog in Category “Virtual Reality” [ac-
cessed 01.04.2020]. [Online]. Available: https://howtocreateapps.com/
microprocessor computing modules based on SiP and PoP examples-virtual-reality-applications/
technology for smart data collection, mining, and interaction
[11] I. Wright, “What can augmented reality do
with surrounding sources). for manufacturing?” May 2017. [Online]. Avail-
able: https://www.engineering.com/AdvancedManufacturing/ArticleID/
R EFERENCES 14904/What-Can-Augmented-Reality-Do-for-Manufacturing.aspx
[1] D. Hasselquist, A. Rawat, and A. Gurtov, “Trends and detection avoid- [12] A. Collins, “The ultimate guide to augmented reality,”
ance of internet-connected industrial control systems,” IEEE Access, Oct. 2018. [Online]. Available: https://blog.hubspot.com/marketing/
vol. 7, pp. 155 504–155 512, 2019. augmented-reality-ar

---------------------------------------------------------------------------- 375 ----------------------------------------------------------------------------

Authorized licensed use limited to: University of Exeter. Downloaded on June 17,2020 at 15:36:43 UTC from IEEE Xplore. Restrictions apply.