See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/334726493

Manufacturing Assembly Simulations in Virtual and Augmented Reality

Chapter · July 2019

CITATIONS READS

2 1,058

3 authors:

Wenjin Tao Ze-Hao Lai

Missouri University of Science and Technology Missouri University of Science and Technology
16 PUBLICATIONS   106 CITATIONS    6 PUBLICATIONS   29 CITATIONS   

SEE PROFILE SEE PROFILE

Ming C. Leu
National Taiwan University
335 PUBLICATIONS   8,118 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Cyber-Physical Sensing, Modeling, and Control with Augmented Reality for Smart Manufacturing Workforce Training and Operations Management View project

Design of Lattice Structure for Additive Manufacturing View project

All content following this page was uploaded by Wenjin Tao on 27 July 2019.

The user has requested enhancement of the downloaded file.

 Manufacturing Assembly Simulations in
Virtual and Augmented Reality

Wenjin Tao, Ze-Hao Lai, and Ming C. Leu

Missouri University of Science and Technology, Rolla, MO 65401

Abstract:
Virtual Reality (VR) and Augmented Reality (AR) technologies have been well
researched for decades, and recently they have been introduced to the consumer
market and are being applied to many fields. This chapter focuses on utilizing
VR/AR technologies for assembly simulations in advanced manufacturing. First,
some basic terminologies and concepts are clarified, and the VR/AR technologies
are outlined to provide a brief introduction to this topic. State-of-the-art
methodologies including modeling, sensing, and interaction that enable the
VR/AR assembly simulations are then reviewed and discussed. This is followed
by assembly examples applying the technologies, and a hands-on case study is
used to provide a demonstration of and practical guide to implementing a VR/AR
assembly simulation application. Finally, the limitations and challenges of current
VR/AR technologies and future research needs are discussed.

1.1 Introduction

Virtual Reality (VR) and Augmented Reality (AR) technologies have been studied
and prototyped in labs for decades without much public attention. However, in
recent years, they have been introduced to the consumer market and are being
applied to a wide range of fields due to the cost reduction of VR/AR hardware and
the algorithm improvement of software. In the manufacturing assembly area,
VR/AR technologies have been used to simulate the costly processes beforehand
and help train the workforce using a more interactive way. This section introduces
some basic terminologies and concepts to provide a brief introduction to this topic.

1.1.1 Virtual Reality (VR)

Although Virtual Reality (VR) is one of the current technology buzzwords, it is

not a brand-new concept and actually has a history over 60 years. The Sensorama
simulator [35] invented in the 1950s is believed to be the first functional VR

3
4 Augmented, Virtual, and Mixed Reality Applications in Advanced Manufacturing

machine that could provide users a realistic experience. From then on, many
researchers have worked on this area, and various prototypes have been developed.
The term VR refers to a series of techniques for humans to visualize,
manipulate and interact with computers in a realistic virtual environment, which
provides an interactive graphics interface enhanced by non-visual modalities such
as auditory, haptic, and smell feedback to enable the user feeling the presence of
a real physical environment [9, 15, 39]. As shown in Fig. 1, an ideal VR system
can generate a virtual environment for the user by providing the five senses,
including sight, hearing, touch, smell, and taste. There are different VR systems
such as CAVE VR systems [21, 81], desktop VR systems, and head-mounted VR
systems.

Real/Virtual Environment
Sight

Taste Hearing
Human
Senses

Smell Touch

Figure 1: Five human senses to perceive the real/virtual environment.

The cost reduction of hardware and the algorithm improvement of software

over the years have been gradually pushing the VR technology from labs to the
consumer market. In 2012, the company Oculus VR was founded and announced
an affordable VR headset, which makes VR experiences easily accessible to
consumers. Two years later, Facebook acquired Oculus VR for more than 2 billion
dollars [19], which further boosted the VR market and attracted lots of companies,
researchers, and entrepreneurs to enter this field. Meanwhile, the VR technology
has been employed to provide various solutions to all kinds of industries.

1.1.2 Augmented Reality (AR)

For VR, a virtual environment is generated to provide users with fully immersive
experiences. For Augmented Reality (AR), instead of generating a virtual
 Manufacturing Assembly Simulations in Virtual and Augmented Reality 5

environment, the real-world environment is “augmented” by computer-generated

perceptual information, ideally across multiple sensory modalities, including
visual, auditory, haptic, etc., which is how the term AR is originally coined. The
augmented information is appropriately overlaid onto the physical environment
and presented in the user’s view through an AR device, such as a hand-held display
or a see-through Head Mounted Display (HMD). To have the optimal correlation
between the virtual information and the physical world, it is required to keep
sensing the physical world and tracking the AR device in the spatial context.
In addition to AR, the term Mixed Reality (MR) has also been used in the
literature to represent the VR related technologies that involve the merging of real
and virtual worlds. Milgram et al. [54] defined a “virtuality continuum” (see Fig.
2) and divided MR into AR and Augmented Virtuality (AV). For simplicity and
following the common terminology, this chapter emphasizes AR and it has the
same scope as MR shown in Fig. 2.

Mixed
Reality (MR)

Real Augmented Augmented Virtual

Environment Reality (AR) Virtuality (AV) Environment
Figure 2: Milgram’s “virtuality continuum” [54].

There are some commercial AR devices currently available in the market.

Google Glasses was first released in 2013, which is an AR product that needs to
be paired with a smartphone. Microsoft released its AR glasses HoloLens in 2016.
It is a standalone AR device that does not need to connect to a computer or
smartphone. Recently, more smartphone-based AR technologies have been
developed and are increasingly popular in a wide range of fields.

1.1.3 Manufacturing Assembly Simulation (MAS)

To keep pace with the vibrant technology revolution planned in the Industry 4.0
era, more and more manufacturers have been reconsidering their assembly
systems. More flexible and efficient assembly methods and strategies have to be
developed to meet the dynamic needs of customers and the shortened product
lifecycle. In such a context, the current assembly systems must be upgraded in
order for products to gain success and maintain competitiveness in the market.
These goals could be achieved through assembly simulation in a virtual
environment before launching a real factory to identify potential problems without
6 Augmented, Virtual, and Mixed Reality Applications in Advanced Manufacturing

the use of physical mockups, thus shortening the design cycle and improving
product quality.
With the help of VR/AR technologies, which puts humans in the loop and
takes the human’s experience at the first place, the assembly process can be
simulated in a more immersive, interactive manner, especially for worker-involved
assembly tasks. Also, assembly training can be conducted in VR/AR environments
to train workers and improve their skills. The main building blocks needed for
assembly simulation in VR/AR are briefly described below.
Modeling. One of the primary tasks for developing a MAS application is to
create the digital models. The 3D model of a physical part can be generated using
CAD software or a 3D reconstruction process with the acquisition of part
geometric data in digital form from an existing physical part. Unlike gaming
VR/AR, the digital 3D models in the aspect of MAS should be accurate in
dimensions, realistic in physics, and functional as real parts in the virtual
environment.
Environment Sensing & Pose Estimation. In VR, we need to track the
user’s pose to change the digital contents in the user’s view accordingly to make
the user feel that he/she is in the real world performing a real task. In AR, sensing
the environment and tracking the user’s pose is even more crucial, because it
requires real-time registration and tracking to realize the optimal mapping between
the virtual world and the real world.
Interface and Interaction. In VR/AR, the visual interface is the most
important one. In addition, auditory and haptic interfaces, and even smell and taste
interfaces can be used to further augment the visual interface to make the user feel
fully immersed in a VR/AR environment. In terms of interaction, how the user
manipulates and interacts with the virtual objects as he/she does with physical
objects should be developed.

1.2 Methodologies for Assembly Simulations in VR/AR

This section reviews and discusses the state-of-the-art methodologies that enable
VR/AR manufacturing assembly simulations.

1.2.1 MAS modeling

Creating digital models for virtual objects is one of the primary tasks in MAS.
The conventional approach for building 3D models is to use commercial CAD
software, such as NX [47], SolidWorks [25], and CATIA [24], to create CAD
models for virtual objects. Another approach is reconstruction of 3D models from
 Manufacturing Assembly Simulations in Virtual and Augmented Reality 7

real objects, which starts with data acquisition from physical objects in a real
environment and ends with digital models representing these objects on the
computer [64]. With this approach, the often time-consuming modeling process
done by human designers can be automated, which significantly reduces the
development time and cost. As shown in Fig. 3, the process of 3D model
reconstruction includes data acquisition, data processing, positional registration,
modeling, and rendering [27, 28].

Data Positional
Preprocessing Modeling Rendering
Acquisition Registration

Figure 3: Process of 3D model reconstruction.

To create a 3D model from a real object, the first step is to acquire the digital
data of the 3D object’s surface. Many data acquisition techniques have been
developed, and they can be categorized as mechanical and optical approaches.
There are some other methods such as ultrasonography, radiography and Magnetic
Resonance Imaging (MRI) for data acquisition purpose. They are not discussed
here as they are costly and not commonly used for MAS applications. A
classification of these techniques is illustrated in Fig. 4. Mechanical data
acquisition is a mature and well-established method that has been used for many
years in reverse engineering and industrial inspection. It utilizes a Coordinate
Measuring Machine (CMM) or a robotic arm to measure the coordinates of points
on the object’s surface to acquire the shape. Currently, optical methods are more
widely used as they can provide non-contact, relatively more efficient data
acquisition. In addition to acquiring the shape, they can capture the appearance as
well, which further reduces the time consumed for modeling and rendering. The
optical methods can be divided into passive and active ones. Passive methods do
not emit light onto the target object, and the shape of the object is estimated from
the perceived images, using techniques such as shape from shading [36], shape
from silhouette [18, 51], and stereo vision [30, 31, 69]. Active methods often
project light onto the target object before sensing. The structural light based
method [67] projects a certain pattern onto the object and then retrieves the depth
information by analyzing the distorted pattern from the captured image. The Time-
of-Flight based method [22] measures the time taken for the light to travel from
the transmitter to the object surface and back to the receiver, based on which the
depth information can be calculated [46]. The light source can be a laser or a near-
infrared LED.
8 Augmented, Virtual, and Mixed Reality Applications in Advanced Manufacturing

CMM
Mechanical
Shape from
Robotic Arm
Shading

Data Acquisition Shape from

Passive
Silhouette

Stereo Vision
Optical

Structural Light
Active
Time-of-Flight

Figure 4: 3D model data acquisition techniques for MAS.

After the procedure of data acquisition, the 3D model of the object can be
generated by surface reconstruction, the objective of which is to determine the
object’s surface geometry from a given finite set of measurements. Usually, the
acquired data are disorganized and noisy. Furthermore, the surface may not have
a specific topological type and could be arbitrary in shape. The steps to generate
3D models from the acquired digital data include the following [64]: (i): Pre-
processing: Erroneous data are removed, and the noise in the data is smoothed out;
(ii) Determination of the surface’s global topology; (iii) Generation of the
polygonal surface; and (iv) Post-processing: Edge correction, triangle insertion,
hole filling, and polygon editing are used to optimize and smooth the shape.
The data acquisition devices are usually called 3D scanners. Besides
expensive industrial scanners, there are some low-cost 3D scanners in the market
that are affordable and suitable for MAS development, such as Microsoft Kinect
[53], MakerBot Digitizer [50], and 3D Systems Sense [1], which are shown in
Figure 5. The 3D model reconstruction software is also provided to generate the
digital models from 3D scanning.

Figure 5: Example low-cost 3D scanners for data acquisition [53, 50, 1].
 Manufacturing Assembly Simulations in Virtual and Augmented Reality 9

1.2.2 Sensors and sensing technologies

This section reviews different kinds of sensors and sensing technologies for
environment perception and human pose estimation, including marker-based and
marker-less tracking technologies. Human pose estimation methods are also
reviewed.

1.2.2.1 Marker-based tracking for AR

Real-time tracking is one of the crucial tasks in AR applications because the

generated virtual information needs to be superimposed upon the real world with
intuitive mapping in real time to avoid causing discomfort for the user. Marker-
based tracking is the most widely used method for real-time tracking due to its ease
of implementation and high accuracy. As illustrated in Fig. 6, a predefined marker
is captured by a camera and it is detected and tracked on each frame. Then the
position and orientation of the marker relative to the camera is estimated. Finally,
a virtual model is rendered using the estimated pose. There are some commercially
available tools for marker-based tracking, such as the classic ARToolKit [42] and
Vuforia [82], which will be discussed in detail in Section 1.4.1.

Figure 6: Illustration of marker-based tracking for AR.

1.2.2.2 Marker-less tracking for AR

Since markers can be designed with optimal tracking patterns, marker-based

methods are very accurate for camera pose estimation. However, These methods
are not convenient in some cases, especially for objects with small size or uneven
surface, which is not feasible to mount markers on. Furthermore, the tracking can
be lost when the marker is occluded. Therefore, researchers have developed
marker-less tracking methods for AR technology [48, 83]. Instead of detecting a
specific marker, a marker-less method extracts features from the real environment,
such as intersections of walls and floors and edges of an object, then these features
are used to estimate the object’s position and orientation.
10 Augmented, Virtual, and Mixed Reality Applications in Advanced Manufacturing

Simultaneous Localization and Mapping (SLAM) technology is a technology

initially developed in the robotics field for self-navigation, which gets the sensing
data (e.g., Lidar, odometer, GPS, visual) to construct or update a map of an
unknown environment while simultaneously keeps track of an agent’s location
within this map [16, 26, 72]. A SLAM technique using cameras is also called visual
SLAM (vSLAM) because it only uses visual information [72]. SLAM is getting
increasingly attractive in recent years due to the prevalence of domestic robots,
self-driving cars, and smart drones. Since it is useful for perceiving and
understanding the real physical world, researchers and companies have been
utilizing SLAM for AR applications. As examples, Apple released their ARKit [5]
for AR developers in 2017, and Google launched ARCore [32] in 2018 to replace
their previous experimental AR project Tango [41]. ARKit and ARCore both
incorporate SLAM algorithms to detect and track planes such as floors and walls
in the real-world scene, then the digital content can be overlaid onto these planes.
At present, SLAM is an active research area with lots of new algorithms being
developed.

1.2.2.3 Human pose estimation

Human pose estimation is very useful for human-computer interaction in the

VR/AR environment. According to the sensing techniques used, human pose
estimation can be classified into two categories: optical and non-optical [46].
Optical methods utilize optical cameras to capture digital images, from where the
poses are estimated.
(1) Marker-based. Marker-based methods need specific trackable markers to
be mounted on the human body to represent the positions of the body part. These
markers can be passive markers, which are usually small balls coated with a retro-
reflective material. The tracking area is illuminated by an Infrared (IR) light and
an IR camera is used to capture the markers. Companies such as OptiTrack [62],
Vicon [80], and ART [7] have such passive optical tracking systems that are
commercially available. The markers also can be active markers, which are IR
LEDs and emit IR light by themselves. An IR camera is used to capture the
markers. The Wiimote controller from Nintendo uses such techniques for tracking
the player’s pose. These systems can use the same marker tracking algorithm as
the passive marker systems but are much less expensive than the passive maker
systems like OptiTrack. Zhu et al. [91, 92, 93, 94] developed a low-cost assembly
simulation system using Wiimote for tracking.
(2) Marker-less. A marker-based system can achieve high accuracy because
the markers are designed and optimized for tracking purpose. However, it has
 Manufacturing Assembly Simulations in Virtual and Augmented Reality 11

shortcomings such as inconvenience in setting up the tracking system and

movement interference with the attached markers. Marker-less tracking provides
non-invasive solutions to address these issues that marker-based methods have.
Microsoft Kinect [53] falls into this category and is popularly used. It has a depth
sensor to perceive the image depth, from which the skeleton-based human pose
estimation is calculated. Some other depth cameras are commercially available,
such as Intel RealSense [37] and Structure Sensor [58]. The recent unprecedented
development in the artificial intelligence (AI) area, especially deep learning, makes
it possible to accurately estimate the human pose directly from a digital image
without the help of depth information. OpenPose [17, 70, 85], initially released in
2017, is the first real-time multi-person system to estimate the human pose from a
single image. It can detect human body, hands, and facial key points. OpenPose is
a promising solution for human pose estimation because it does not need special
cameras, but it is computationally intensive and relies on high-end GPUs to realize
real-time human tracking, which is one of its limitations.
(3) Non-optical. Non-optical sensors also have been used to estimate the
human pose. A magnetic tracking system, such as MotionStar developed by
Ascension Technology Corp. in 1997 [8], calculates an object’s position and
orientation based on the earth’s magnetic field. It has some drawbacks, e.g., it may
be interfered by metallic objects in the tracking area. An inertia-based tracking
system, such as Moven from Xsens Corp. [88], uses accelerometers and
gyroscopes to capture the human body’s movement and estimate the human’s pose.
It does not measure positions directly, thus accumulated errors may occur. An
acoustic tracking system uses the time-of-flight technique to estimate the position
of the emitter relative to the receivers. The emitter is attached to the body and the
receivers are mounted at known location in the environment.
(4) Sensor fusion. To compensate for the shortcomings of each method,
sensor fusion methods integrating different modalities have been developed.
Researchers at the Missouri University of Science and Technology used a Kinect
to provide skeleton information and a Myo armband to provide inertial and surface
electromyography (sEMG) measurements. Then a multimodal deep learning
method was used for worker behavior understanding, achieving promising results
for future endeavors [4, 74].

1.2.3 Human-computer interface for rendering and interaction

Human-computer interfaces provide the user with different visual, haptic and
auditory sensations, which play a vital role in a virtual assembly system for
increasing the degree of immersion in the VR/AR environment. Figure 7 shows a
12 Augmented, Virtual, and Mixed Reality Applications in Advanced Manufacturing

typical VR/AR system configuration with physical input and output devices that
transmit information between the user and the environment. These sensing
technologies are essential to the realism of the VR/AR system. The critical
technologies addressed here include visual interface, auditory modeling and
rendering, and haptic modeling and rendering.

Figure 7: Typical VR/AR configuration with human-computer interfaces [46]

1.2.3.1 Visual interface

Visual interface is the most import interface in VR/AR. There exists different
hardware providing visual interfaces, such as desktop displays, hand-held displays,
projection displays, and Head Mounted Displays (HMD) that have been the
mainstream due to the fully immersive experiences they provided. VR HMDs can
be divided into three classes, tethered HMDs that need to be connected to a PC or
console, mobile-dependent HMDs that need to attach onto a smartphone, and
standalone headsets that do not need to be connected to other devices. Figure 8
shows three VR glasses available in the market that fall into these three classes and
their costs range from $100 to $500. Some of the mobile-dependent products cost
less than $100, which are affordable to most consumers. AR HMDs are relatively
more expensive than VR HMDs because they require more complicated optical
designs to realize the see-through ability. Figure 9 shows three AR HMDs in the
market. The selection of visual hardware for a given application should take into
consideration of the resolution, update rate, field-of-view, head tracking accuracy,
latency, etc.
 Manufacturing Assembly Simulations in Virtual and Augmented Reality 13

Figure 8: VR headsets of Sony PlayStation VR, Oculus Rift, and Google

Daydream (from left to right) [34].

Figure 9: AR headsets of Microsoft HoloLens, Magic Leap One, and Google

Glasses (from left to right) [52, 49, 33].

1.2.3.2 Auditory modeling and rendering

Audio clues can be used to augment visual interfaces for assembly simulation in
VR/AR. Auditory rendering is especially helpful when haptic feedback is not
available. Physics-based sound modeling is too computationally expensive for
real-time rendering required in virtual assembly. Synthetic sound can be used to
approximate the real sound generated from the physical assembly, making the
simulation more realistic. Spectral modeling can be used as the basis for sound
synthesis in virtual assembly [57, 87]. Sound generation using deep learning
methods also can be used in this case [79].
In the virtual environment, a sound generator can be attached to a virtual
object to generate 3D audio for auditory rendering. 3D audio can provide listeners
a spatial hearing experience that enables them to sense where they are relative to
the sound sources, further immersing them in the virtual world.

1.2.3.3 Haptic modeling and rendering

Haptic devices provide force or tactile feedback to the users can further improve
the realism besides visual and auditory sensations. Such devices usually are
embedded with sensors and actuators to measure the user’s contact position with a
virtual object and apply force or other haptic feedback to the user at the contact
position. The current haptic devices can be classified into three categories: (1)
handheld devices such as the Touch from 3D Systems [2] that has been used widely
as a force feedback device; (2) hand wearable devices such as the Sense Glove [68]
14 Augmented, Virtual, and Mixed Reality Applications in Advanced Manufacturing

which provides feedback to individual fingers to simulate how the users interact
with the real objects; and (3) haptic suits such as the Teslasuit [75] which can
provide haptic feedback to the body covered by the suit. These haptic devices have
some drawbacks such as having strict geometry, placement and workspace
requirements, and being cumbersome for wearing, which limit them from being
widely used.

Figure 10: Haptic devices of 3D System Touch, Sense Glove, and Teslasuit (from
left to right) [2, 68, 75].

1.3 Application and Examples of MAS using VR/AR

Manufacturing assembly simulation using VR/AR can provide intuitive,

immersive and interactive solutions for manufacturers to improve the capability
and efficiency of the various assembly processes, covering assembly design,
process verification, and workforce training.
Many companies have utilized MAS with VR/AR in their assembly processes
and greatly benefitted from it [65]. For example, in the automotive area, Toyota
[76] develops an interactive virtual learning system using VR technology to train
their assembly line workers. The trainee is immersed in a virtual assembly line
environment wearing an HTC Vive headset. During the training session, the trainee
is guided by step-by-step assembly processes and interacts with the virtual object
using the controllers. The Volkswagen Group, collaborated with the VR studio,
Innoactive [96], has launched a plan of bringing VR training to 10,000 employees,
which includes over 30 VR training experiences covering tasks from vehicle
assembly, new team member training, to customer service [95]. Other companies
like Tesla [14], Mercedes-Benz [100], BMW [101], Volvo [13], Ford [20], and
Bentley [10, 11] also apply VR/AR in their factories to improve the assembly
workflows. Compared with traditional training methods using paper manuals, such
training systems not only can benefit the trainees with more motivational,
personalized and intuitive training experience but also can benefit the
 Manufacturing Assembly Simulations in Virtual and Augmented Reality 15

manufacturers with reduced training cost, improved training efficiency and

excellent scalability.
In the aerospace area, Boeing has been using VR/AR in various tasks, such as
wing assembly tasks, wiring tasks, and some specialized manufacturing tasks, to
help improve training efficiency, reduce design errors, and speed up maintenance
processes. With the help of a Microsoft HoloLens providing the assembly
instructions to the trainees, Boeing showed that this approach can shorten training
time by 75% per person [6, 12, 43]. Similarly, Airbus [3], Rolls-Royce [103] and
Lockheed Martin [102] also reported benefit from VR/AR utilization.
Almost all the leading industrial manufacturing companies, such as Siemens
[97], Bosch [98] and Dassault Systems [99] have been actively seeking
opportunities to apply the VR/AR technologies to industries.
A typical MAS application in VR is shown in Fig. 11, which is a MAS
scenario of seat mounting in VR developed by Bentley Motors and OPTIS [10, 11,
61]. The native CAD models are imported into the virtual environment and haptic
feedback is generated when an operator’s hand collides with a virtual object. The
haptic feedback gives car makers a way to simulate and analyze where clashes and
collisions might occur during the worker’s operation, and find out the optimal
possible manufacturing and installation protocols. Multiple markers are attached
onto the operator’s body for motion capturing and tracking.

Figure 11: A operator wearing an Oculus headset for immersive VR and attached
with multiple markers for motion tracking (Image courtesy of OPTIS) [11].

AR technology allows the operator to see the real environment with virtual
information superimposed upon the physical world. This has been popular in AR-
assisted assembly training and guidance [66, 89, 90, 84]. AR also has been applied
to other assembly related tasks, such as manual assembly station planning [63],
assembly workplace design [59], and assembly constraint analysis [60].
16 Augmented, Virtual, and Mixed Reality Applications in Advanced Manufacturing

A typical MAS in AR developed by DAQRI [23] for workforce training and

guidance is shown in Fig. 12, which is a control module in an assembly task.
During the assembly task, the instruction information, such as which item to pick
and which tool to use, is overlaid onto the operator’s view through an AR helmet.
The operator can follow the instructions step by step to get familiar with the
assembly sequence and acquire assembly skills. The virtual instructions can be
multimedia including text, image, video, voice, etc. The task completion status is
also computed and visualized.

Figure 12: An AR guidance scenario from DAQRI [23].

1.4 A Case Study

This section presents a case study to provide a practical guide to implementing an

AR assembly simulation application. In this case study, AR instructions are
developed to guide the user in a spindle carriage installation task [38]; see Fig. 13.
Hand interaction with virtual objects is explored using a Leap Motion Controller
[45]. The performance comparison between AR guidance and traditional manual
guidance is also presented.

Figure 13: The spindle subassembly of a CNC carving machine.

 Manufacturing Assembly Simulations in Virtual and Augmented Reality 17

1.4.1 Available tools

There are various tools that can be used for developing AR MAS applications.
Some commonly used ones are introduced as follows:
Unity. Unity is a game engine developed by Unity Technologies that allows
its users to develop games [77]. The engine can provide various visual effects
including 2D and 3D graphics, textures, lighting and shading. The users can create
a scene by importing customized CAD models with texture rendering. The engine
platform supports scripting via programming languages such as C# and JavaScript.
Unreal Engine. Unreal Engine is a game engine that was first brought to the
gaming platform in 1998 and has been widely used [78]. The engine is
programmed in C++ and is available for different operating systems.
ARToolKit. ARToolKit is a library that can be utilized for AR application
development and was originally developed by Dr. Hirokazu Kato. It is being
supported by multidisciplinary institutions across the country [42]. The library can
provide AR rendering through camera pose estimation and target tracking.
Vuforia. Vuforia is a Software Development Kit (SDK) for AR development
[82]. It provides the functionalities of target recognition and tracking, enabling its
users to superimpose 3D digital contents with respect to the world coordinates as
2D image registration for a composite view. The user can define the trackable
marker for maker-based AR by importing the desired image target for
augmentation with different configurations.
Wikitude. Wikitude SDK combines target recognition, tracking, and
simultaneous localization and mapping (SLAM), as well as Geo-location, for
various AR development engines and configurations across Android and iOS [86].
ARKit. ARKit from Apple is a platform for building AR iOS applications. It
provides object detection and mapping functions for marker-less tracking, enabling
the user to directly overlay information onto the environment with different
geometries [5].
ARCore. ARCore is an AR platform from Google. It integrates three main
top-notch elements, i.e. motion tracking, environmental understanding, and light
estimation [32]. ARCore provides a wide variety of features and functionalities
with different APIs and can be deployed on both Android and iOS devices.
In this demonstration project, the application is developed using Unity with
the support of Vuforia SDK due to its fundamental usability and accessibility for
AR implementation.
18 Augmented, Virtual, and Mixed Reality Applications in Advanced Manufacturing

1.4.2 Modeling

In this case study, NX is chosen as the CAD software to build the 3D digital model
for the spindle carriage. Figure 14 shows the created CAD model of the spindle
carriage in NX [104].

Figure 14: The CAD model of the spindle carriage.

Once modeling is finished, an AR assembly scene in Unity can be created by

importing CAD models into the scene as Unity game objects either through the
drop-down menu as shown in Fig.15 or using drag-and-drop to the “Assets” panel.
The imported CAD models would then be categorized as game objects and be
sorted on the hierarchy panel once they have been assigned to the scene. Multiple
static and dynamic features can then be added to each of the game objects with
different settings and parameters.
 Manufacturing Assembly Simulations in Virtual and Augmented Reality 19

Figure 15: Importing CAD model of spindle carriage to Unity.

1.4.3 AR realization

The AR guidance function is achieved using Vuforia SDK in the Unity

environment. The AR composite view can be realized by utilizing Vuforia prefabs
and the computed target database imported from the developer portal in which the
user can upload image patterns for target markers, as well as visualize the
augmented contents according to the quality of the computed features. Figure 16
shows the workflow of Vuforia prefabs in Unity.

Figure 16: The workflow of AR realization using Vuforia in Unity.

The database enables the marker-based AR by activating “ARCamera” and

“ImageTarget” prefabs to recognize and track objects for data registration with a
built-in algorithm, which can provide the coordinate transformation of camera pose
in a real-time manner. Once the data has been loaded, the user can activate the
20 Augmented, Virtual, and Mixed Reality Applications in Advanced Manufacturing

“Image Target Behavior” of the “ImageTarget” prefab by setting the dataset and
image target. After the setting is finished, the augmented view can be visualized
through the display while the marker is being captured by the camera. Figure 17
shows the AR scene with generated CAD models superimposed.

Figure 17: An assembly scene runs in real time with AR instructions. Multiple
types of instructions are rendered through a display including texts, graphics, and
3D animations.

1.4.4 Hand interaction

A Leap Motion Controller is applied in this study to realize simple hand

interactions with virtual objects. Leap Motion SDK is used to return the spatial
coordinates of the real hands and Vuforia is responsible for superimposing graphic
hands onto the real hands in an AR scene. To interact with virtual objects, the
scripted interaction behavior and physics features for graphic hands and objects
are required to simulate the real-world activities as if the user is interacting with
the physical objects. Figure 18 shows the interaction between real hands and virtual
objects with various geometries.
 Manufacturing Assembly Simulations in Virtual and Augmented Reality 21

Figure 18: Hand interaction with virtual objects in an AR scene.

1.4.5 Experiment and Evaluation

To evaluate the effectiveness of the AR guidance, we have applied it in an

experiment to instruct users to conduct an assembly task and its performance is
compared with the traditional manual guidance. Figure 19 shows the overall
experimental setup. There are 20 subjects recruited completing the task.

Webcam

AR onsite display

Subassembly
of the CNC
machine

Webcam

Toolkit

Marker

Figure 19: The workstation setup for the experiment.

22 Augmented, Virtual, and Mixed Reality Applications in Advanced Manufacturing

Figure 20: A comparison of two subjects performing the experiment with two
different methods: (left) manual and (right) AR.

The 20 subjects are divided into two groups with AR and manual instructions
provided, respectively. During the experiment, each subject is asked to perform
the assembly task by following the instructions step by step; see Fig. 20. To assess
the performance, the completion time and number of errors are recorded
throughout the experiment. The performance comparison is shown in Figure 21;
by using AR guidance, the completion time and number of errors show 33.2% and
32.4% of reduction, respectively. The considerable improvement demonstrates the
great potential of applying AR technology to industries for manufacturing
assembly training.

Figure 21: The comparison between manual and AR guidance: (left) completion
time and (right) number of errors.

1.5 Technology Limitations and Research Needs

This section discusses the limitations and challenges of current VR/AR and MAS
technologies and future research needs towards MAS realism improvement,
worker behavior understanding, and sharing and collaborative MAS.
 Manufacturing Assembly Simulations in Virtual and Augmented Reality 23

1.5.1 MAS realism improvement

To further improve the realism of MAS in VR/AR environment, research and

development efforts are needed to advance VR/AR technologies aimed at
accomplishing the following objectives:
(1) High-fidelity and low-latency graphic visual interfaces need to be
developed. For example, for head-mounted VR/AR systems, the optical system
design needs to be advanced [29] for increased display resolution, update rate, and
field of view, and for reduced size and weight of the headset.
(2) To enhance the visual interface for better immersive VR/AR experience,
the following efforts are needed: (a) Advance the technique of 3D auditory
feedback; (b) Develop multichannel haptic devices to provide users with richer
touch feedback and allow them to feel the surface texture, shape,
softness/hardness, and temperature of an object; (c) Further develop locomotion
interface [56, 71] to provide realistic walking/running experience in the virtual
environment; and (d) Explore the senses of smell and taste to augment the realism
of MAS applications.
(3) To realize better environment perception for AR systems, marker-less
tracking methods need to be improved. The SLAM technologies have provided
promising solutions for environment perception in AR. They need to be further
studied for improved efficiency and robustness. Object detection and recognition
techniques also need to be further developed for more advanced environment
awareness.
(4) More accurate, efficient, robust, and low-cost human pose estimation
approaches need to be investigated. Multi-modal motion tracking devices,
advanced data fusion techniques, and deep learning methods can be exploited in
this study. Motion sickness [44, 55] is another challenging issue affecting the
user’s experience, which needs to be solved with more accurate and efficient
motion tracking approaches.
(5) Currently most of the human-environment interactions are conducted with
some input/output devices such as a 3D mouse or a controller. More natural ways
such as voice, gaze, and gesture interaction [73] technologies need to be developed.

1.5.2 Worker behavior understanding

In a MAS task, understanding the worker’s behavior can be used for quantification
and evaluation of the worker’s performance, as well as to provide onsite
instructions in VR/AR [74]. Studies in the following aspects need to be conducted:
24 Augmented, Virtual, and Mixed Reality Applications in Advanced Manufacturing

(1) Methods for real-time worker activity recognition need to be researched

for understanding the worker’s current assembly activity during a MAS task. Then
the time taken for each assembly step can be analyzed for quantification and
evaluation purposes.
(2) Besides activity, the worker’s emotional states (e.g., stress, confusion and
fatigue) should be considered in MAS, which can be estimated using sensors such
as an Electroencephalography (EEG) headset, or through advanced facial
recognition techniques [40].
(3) On-demand virtual guidance for MAS should be developed based on the
worker’s states (e.g., activity and emotional states) to improve the interaction
efficiency with minimal guidance to the worker [105]. Also, methods for online
assembly quality evaluation need to be developed.

1.5.3 Sharing and collaborative MAS

Due to the increasing complexities of assembly products and prevalence of global

partnership in the design and manufacturing processes, sharing and collaborative
MAS are needed in such contexts. It allows team members located in different
geographical locations and time zones to share their knowledge and expertise and
conduct cloud-based collaboration. Research opportunities in the area of sharing
and collaborative MAS include the following:
(1) For MAS involving multiple users in the VR/AR environment, sharing the
same virtualized/augmented world and digital contents requires more accurate,
efficient and robust registration and tracking algorithms, in order to register
multiple users to the same coordinate system precisely and keep synchronizing
their shared contents promptly.
(2) The design interface and interaction in these situations will be more
challenging because it involves not only interactions between users and digital
contents, but also interactions between different users. How to enable users to have
natural, effective distance collaborations in the VR/AR scenarios as they do in the
real workplace is another challenge that needs to be addressed.
(3) To achieve simultaneous interfaces of VR/AR contents and concurrent
interactions among collaborators in different geographical locations, advanced
computing, networking, and communication architectures and methodologies,
along with the necessary hardware and software, need to be developed and
implemented.
 Manufacturing Assembly Simulations in Virtual and Augmented Reality 25

1.6 Conclusion

This chapter focuses on utilizing Virtual Reality/Augmented Reality (VR/AR)

technologies for assembly simulations in advanced manufacturing. First, some
basic terminologies and concepts have been clarified in terms of VR, AR, and
manufacturing assembly simulation (MAS). Then the state-of-the-art
methodologies for developing MAS applications including modeling, sensing, and
interaction that enable the VR/AR assembly simulations has been reviewed and
discussed. In particular, the discussion has included how to acquire digital data
from a real object and how to reconstruct a 3D model using the acquired data. This
chapter also has presented different methods for AR tracking and human pose
estimation. Moreover, human-computer interfaces for rendering and interaction,
including visual, auditory and haptic interfaces, have been discussed. Further, this
chapter has included some assembly application examples that incorporate VR/AR
technologies for MAS and has demonstrated the potential of MAS systems for
shortening the product design cycle, improving product quality, and enhancing
worker skills. A hands-on project has been used in a case study to provide a
practical guide to implementing a VR/AR assembly simulation application.
Finally, the limitations and challenges of current VR/AR technologies and future
research needs in terms of MAS realism improvement, worker behavior
understanding, and sharing and collaborative MAS have been discussed.

References

1. 3D Systems, 2018. Sense. https://www.3dsystems.com/shop/sense.

2. 3D Systems, 2018. Touch. https://www.3dsystems.com/haptics-
devices/touch.
3. Airbus, 2017. Virtual Reality with real benefits.
https://www.airbus.com/newsroom/news/en/2017/09/virtual-reality-with-
benefits.html.
4. Al-Amin, M., Tao, W., Qin, R. and Leu, M.C., 2018. Sensor data based
models for workforce management in smart manufacturing. Proceedings
of the 2018 IISE Annual Conference.
5. Apple, 2018. ARKit. https://developer.apple.com/arkit/.
6. AREA, 2015. Augmented Reality can increase productivity.
http://thearea.org/augmented-reality-can-increase-productivity/.
7. ART, 2018. Homepage. https://ar-tracking.com/.
8. Ascension Technology Corporation, 2018. Homepage.
https://www.ascension-tech.com.
26 Augmented, Virtual, and Mixed Reality Applications in Advanced Manufacturing

9. Aukstakalnis, S. and Blatner, D., 1992, “Silicon Mirage: the Art and
Science of Virtual Reality,” Peachpit Press, Inc., Berkeley, CA.
10. Bentley, 2017. Bentley reinvents the design of future vehicles with
Virtual Reality. https://www.optis-world.com/Showcase/News-
Release/Customer-focus/Bentley-reinvents-the-design-of-future-
vehicles-with-Virtual-Reality.
11. Bentley, 2017. Prelude to GTC 2017: Touch and feel the pixels in your
virtual car. http://www.digitaleng.news/virtual_desktop/2017/04/prelude-
gtc-2017-touch-feel-pixels-virtual-car/.
12. Boeing, 2018. Boeing tests Augmented Reality in the factory.
https://www.boeing.com/features/2018/01/augmented-reality-01-
18.page.
13. Bruce, C. 2015. Volvo and Microsoft partner on virtual reality and
autonomous tech. https://www.autoblog.com/2015/11/20/volvo-
microsoft-virtual-reality-autonomous-tech/.
14. Bruce, C., 2016. Tesla uses augmented reality to improve EV
manufacturing. https://www.autoblog.com/2016/03/22/tesla-augmented-
reality-factory/.
15. Burdea, G. and Coiffet, P., 1994, Virtual Reality Technology, John Willey
& Sons, INC., Chap. 1.
16. Cadena, C., Carlone, L., Carrillo, H., Latif, Y., Scaramuzza, D., Neira, J.,
Reid, I. and Leonard, J.J., 2016. Past, present, and future of simultaneous
localization and mapping: Toward the robust-perception age. IEEE
Transactions on Robotics, 32(6), pp.1309-1332.
17. Cao, Z., Simon, T., Wei, S.E. and Sheikh, Y., 2017, July. Realtime multi-
person 2d pose estimation using part affinity fields. In CVPR (Vol. 1,
No. 2, p. 7).
18. Cheung, G., Baker, S., Kanade, T., 2003a, Visual hull alignment and
refinement across time: a 3D reconstruction algorithm combining shape-
from-silhouette with stereo, Proceedings of IEEE conference on computer
vision and pattern recognition (S. II - 375-82, vol. 2 ).
19. Constine, J., 2014. Facebook’s $2 billion acquisition of Oculus closes,
now official, https://techcrunch.com/2014/07/21/facebooks-acquisition-
of-oculus-closes-now-official/.
20. Covington, P., 2017. Inside Ford’s Virtual Reality labs.
https://www.triplepundit.com/2017/01/ford-virtual-reality-labs/.
21. Cruz-Neira, C., Sandin, D.J. and DeFanti, T.A., 1993, September.
Surround-screen projection-based virtual reality: the design and
implementation of the CAVE. In Proceedings of the 20th annual
 Manufacturing Assembly Simulations in Virtual and Augmented Reality 27

conference on Computer graphics and interactive techniques (pp. 135-

142). ACM.
22. Cui, Y., Schuon, S., Chan, D., Thrun, S. and Theobalt, C., 2010, June. 3D
shape scanning with a time-of-flight camera. In Computer Vision and
Pattern Recognition (CVPR), 2010 IEEE Conference on (pp. 1173-
1180). IEEE.
23. DAQRI, 2017. Best practices for enterprises planning to adopt
augmented reality. https://medium.com/daqri-blog/best-practices-for-
enterprises-planning-to-adopt-augmented-reality-e81eec647dea.
24. Dassault System, 2018. CATIA. https://www.3ds.com/products-
services/catia/.
25. Dassault System, 2018. SOLIDWORKS. https://www.solidworks.com/.
26. Durrant-Whyte, H. and Bailey, T., 2006. Simultaneous localization and
mapping: part I. IEEE robotics & automation magazine, 13(2), pp.99-
110.
27. El-hakim, S., 1998, Theme issue on imaging modeling for virtual reality,
ISPRS journal of photogrammetry & remote sensing, 53(6): 309-310.
28. El-hakim, S., 2000, A practical approach to creating precise and detailed
3d models from single and multiple views. Proceedings of XIX congress
of the international society for photogrammety and remote sensing .
29. Elliott, E., Norton, K., and Humphreys, M., 2018, Freeform optics
design: Optical design challenges in virtual and augmented reality.
https://www.laserfocusworld.com/articles/print/volume-54/issue-
04/features/freeform-optics-design-optical-design-challenges-in-virtual-
and-augmented-reality.html.
30. Fusiello, A., V. R., 1997, Efficient stereo with multiple windowing,
Proceedings of IEEE computer society conference on computer vision and
pattern recognition, 858 - 863.
31. Gonzalez, C.G., Woods, R.E., 2002, Digital image processing, Prentice
Hall, Upper Saddle River, NJ, USA.
32. Google, 2018. ARCore. https://developers.google.com/ar/.
33. Google, 2018. Google Glasses. https://www.x.company/glass/.
34. Greenwald, W., 2018. The Best VR (Virtual Reality) headsets of 2018.
https://www.pcmag.com/article/342537/the-best-virtual-reality-vr-
headsets.
35. Heilig, M.L., 1962. Sensorama simulator. U.S. Patent 3,050,870.
36. Horn, B. K., Brooks, M. J., 1989, Shape from shading. MIT Press,
Cambridge, MA, USA.
28 Augmented, Virtual, and Mixed Reality Applications in Advanced Manufacturing

37. Intel, 2018. Intel RealSense technology. https://software.intel.com/en-

us/realsense?language=ru.
38. Inventables, 2018. X-Carve instructions. http://x-carve-
instructions.inventables.com/750mm/.
39. Jayaram, S., Vance, J., Gadh, R., Jayaram, U. and Srinivasan, H., 2001,
“Assessment of VR Technology and its Applications to Engineering
Problems,” Journal of Computing and Information Science in
Engineering, 1(1), pp. 72-83.
40. Kahou, S.E., Bouthillier, X., Lamblin, P., Gulcehre, C., Michalski, V.,
Konda, K., Jean, S., Froumenty, P., Dauphin, Y., Boulanger-
Lewandowski, N. and Ferrari, R.C., 2016. Emonets: Multimodal deep
learning approaches for emotion recognition in video. Journal on
Multimodal User Interfaces, 10(2), pp.99-111.
41. Kastrenakes, J., 2017. Google’s Project Tango is shutting down because
ARCore is already here.
https://www.theverge.com/2017/12/15/16782556/project-tango-google-
shutting-down-arcore-augmented-reality.
42. Kato, H., 1999. ARToolKit. https://www.hitl.washington.edu/artoolkit/.
43. Kishore, A. 2017. Boeing: productive VR cuts training time by 75%.
https://www.lightreading.com/video/video-services/boeing-productive-
vr-cuts-training-time-by-75-/d/d-id/733756.
44. Lang, DJ, 2016. For virtual reality creators, motion sickness a real issue.
http://phys.org/news/2016-03-virtual-reality-creatorsmotion-
sickness.html.
45. Leap Motion, 2018. Homepage. https://www.leapmotion.com/.
46. Leu, M.C., ElMaraghy, H.A., Nee, A.Y., Ong, S.K., Lanzetta, M., Putz,
M., Zhu, W. and Bernard, A., 2013. CAD model based virtual assembly
simulation, planning and training. CIRP Annals, 62(2), pp.799-822.
47. Leu, M.C., Tao, W., Ghazanfari, A. and Kolan, K., 2018. NX 12 for
Engineering Design, Missouri S&T.
48. Lima, J.P., Roberto, R., Simões, F., Almeida, M., Figueiredo, L.,
Teixeira, J.M. and Teichrieb, V., 2017. Markerless tracking system for
augmented reality in the automotive industry. Expert Systems with
Applications, 82, pp.100-114.
49. Magic Leap, 2018. Homepage. https://www.magicleap.com/.
50. MakerBot, 2018. Digitizer.
https://www.makerbot.com/stories/news/makerbot-digitizer-desktop-3d-
scanner-order-today/
 Manufacturing Assembly Simulations in Virtual and Augmented Reality 29

51. Martin, W. N., Aggarwal, J., 1983, Volumetric descriptions of objects

from multiple views. IEEE transactions on pattern analysis and machine
intelligence, 150-158.
52. Microsoft, 2018, HoloLens. https://www.microsoft.com/en-us/hololens.
53. Microsoft, 2018. Kinect. https://www.xbox.com/en-US/xbox-
one/accessories/kinect.
54. Milgram, P. and Kishino, F., 1994. A taxonomy of mixed reality visual
displays. IEICE TRANSACTIONS on Information and Systems, 77(12),
pp.1321-1329.
55. Munafo, J., Diedrick, M. and Stoffregen, T.A., 2017. The virtual reality
head-mounted display Oculus Rift induces motion sickness and is sexist
in its effects. Experimental brain research, 235(3), pp.889-901.
56. Nabiyouni, M., Saktheeswaran, A., Bowman, D.A. and Karanth, A.,
2015, March. Comparing the performance of natural, semi-natural, and
non-natural locomotion techniques in virtual reality. In 3D User
Interfaces (3DUI), 2015 IEEE Symposium on (pp. 3-10). IEEE.
57. Niu, Q., Chi, X., Leu, M.C., Ochoa, J., 2008, Image processing, geometric
modeling and data management for development of a virtual bone surgery
system, Computer aided surgery, 13 (1): 30-40.
58. Occipital, 2018. Structure Sensor. https://structure.io/
59. Ong, S.K., Pang, Y.,Nee, A.Y.C., 2007, Augmented reality aided
assembly design and planning, CIRP Annals – manufacturing
technology, 56(1):49-52.
60. Ong, S.K., Wang, Z.B., 2010, Augmented assembly technologies based
on 3d bare-hand interaction, CIRP Annals – manufacturing technology,
60(1): 1-4.
61. OPTIS, 2017. Improving production time with Virtual Reality.
https://marketing.optis-world.com/acton/attachment/17674/f-05a2/1/-/-/l-
0027/l-0027:4cb/quadro-bentley-customer-story.pdf.
62. OptiTrack, 2018. Homepage. http://optitrack.com/.
63. Reinhart, G., Patron, C., 2003, Integrating augmented reality in the
assembly domain – fundamentals, benefits and applications, CIRP
Annals – manufacturing technology, 52(1): 5-8.
64. Remondino, F., El-hakim, S., 2006, Image-based 3D modeling: a review,
Photogrammetric record, 21 (115): 269-291.
65. Resnick, J., 2016. Virtual assembly lines are making the auto industry
more flexible. https://arstechnica.com/cars/2016/09/virtual-assembly-
lines-are-making-the-auto-industry-more-flexible/.
30 Augmented, Virtual, and Mixed Reality Applications in Advanced Manufacturing

66. Saaski, J., Salonen, T., Hakkarainen, M., Siltanen, S., Woodward, C.,
Lempiainen, J., 2008, Integration of design and assembly using
augmented reality, Micro-assembly technologies and applications, 260:
395–404.
67. Salvi, J., Pages, J., Batlle, J., 2004, Pattern condification strategies in
structured light systems, Pattern recognition, 37(4): 827-849.
68. Sense Glove, 2018. Homepage. https://www.senseglove.com/.
69. Shapiro, L.S., Brady, J.M., 1992, Feature-based correspondence: an
eigenvector approach. Image and vision computing, 10(5): 283-288.
70. Simon, T., Joo, H., Matthews, I. and Sheikh, Y., 2017, July. Hand
keypoint detection in single images using multiview bootstrapping. In
The IEEE Conference on Computer Vision and Pattern Recognition
(CVPR) (Vol. 2).
71. Steinicke, F., Visell, Y., Campos, J. and Lécuyer, A., 2013. Human
walking in virtual environments (pp. 199-219). New York: Springer.
72. Taketomi, T., Uchiyama, H. and Ikeda, S., 2017. Visual SLAM
algorithms: a survey from 2010 to 2016. IPSJ Transactions on Computer
Vision and Applications, 9(1), p.16.
73. Tao, W., Lai, Z., Leu, M.C. and Yin Z., 2018. American Sign Language
Alphabet Recognition Using Leap Motion Controller. Proceedings of the
2018 IISE Annual Conference.
74. Tao, W., Lai, Z., Leu, M.C. and Yin, Z., 2018. Worker Activity
Recognition in Smart Manufacturing Using IMU and sEMG Signals with
Convolutional Neural Networks, 46th SME North American
Manufacturing Research Conference, NAMRC 46, Texas, USA
75. Teslasuit, 2018. Ultimate tech in smart clothing. https://teslasuit.io/.
76. Toyota, 2017. Toyota assembly line VR training – The training center of
the future. http://realitymatters.eu/project/virtual-assembly-line-training/.
77. Unity, 2018. Homepage. https://unity3d.com/.
78. Unreal Engine, 2018. What is unreal engine 4.
https://www.unrealengine.com/en-US/what-is-unreal-engine-4
79. Van Den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O.,
Graves, A., Kalchbrenner, N., Senior, A. and Kavukcuoglu, K., 2016.
Wavenet: A generative model for raw audio. arXiv preprint
arXiv:1609.03499.
80. Vicon, 2018. Homepage. https://www.vicon.com/.
81. Visbox, 2018. CAVE. http://www.visbox.com/products/cave/.
82. Vuforia, 2018. Homepage. https://www.vuforia.com/.
 Manufacturing Assembly Simulations in Virtual and Augmented Reality 31

83. Wang, X., Ong, S.K. and Nee, A.Y.C., 2016. A comprehensive survey of
augmented reality assembly research. Advances in Manufacturing, 4(1),
pp.1-22.
84. Webel, S., Bockholt, U., Engelke, T., Gavish, N., Olbrich, M. and
Preusche, C., 2013. An augmented reality training platform for assembly
and maintenance skills. Robotics and Autonomous Systems, 61(4),
pp.398-403.
85. Wei, S.E., Ramakrishna, V., Kanade, T. and Sheikh, Y., 2016.
Convolutional pose machines. In Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition (pp. 4724-4732).
86. Wikitude, 2018. Wikitude homepage. https://www.wikitude.com/
87. Wu, S., Tao, W., Leu, M.C. and Suzanna, L., 2018. Engine sound
simulation and generation in driving simulator. Proceedings of the 2018
IISE Annual Conference.
88. Xsens Corporation, 2018. Homepage. https://www.xsens.com/.
89. Yuan, M.L., Ong, S.K., Nee, A.Y.C., 2008, Augmented reality for
assembly guidance using a virtual interactive tool, International journal
of production research, 46(7):1745-1767.
90. Zhang, J., Ong S.K., Nee, A.Y.C., 2011, RFID-assisted assembly
guidance system in an augmented reality environment, International
journal of production research, 49(13): 3919-3938.
91. Zhu, W., Chadda, A., Leu, M.C., Liu, X.F., 2011a, Real-time automated
simulation generation based on CAD modeling and motion capture,
Journal of computer aided design and applications, PACE(1), 103-121.
92. Zhu, W., Daphalapurkar, C.P., Puthenveetil, S.C., Leu, M.C., Liu, X.F.,
Chang, A.M., Gilpin-Mcminn, J.K., Hu, P.H., Snodgrass, S.D., 2012,
Motion capture of fastening operation using Wiimotes for ergonomic
analysis, Proceedings of international symposium on flexible automation,
St. Louis, USA.
93. Zhu, W., Vader, A., Chadda, A., Leu, M.C., Liu, X.F, Vance, J., 2010,
Low-cost versatile motion tracking for assembly simulation, Proceedings
of international symposium on flexible automation, Tokyo, Japan.
94. Zhu, W., Vader, A., Chadda, A., Leu, M.C., Liu, X.F, Vance, J., 2011b,
Wii Remote based low-cost motion capture for automated assembly
simulation, Virtual Reality, 1-12.
95. Carlton, B., 2018. Volkswagen Will Train 10,000 Employees Using VR
This Year. https://vrscout.com/news/volkswagen-employee-training/
96. Innoactive GmbH, 2019. https://innoactive.de/
 32 Augmented, Virtual, and Mixed Reality Applications in Advanced Manufacturing

97. Siemens PLM, 2018. Digital Mockup, Virtual & Augmented Reality.
https://www.plm.automation.siemens.com/global/en/industries/automotiv
e-transportation/automotive-oems/digital-mockup-virtual-augmented-
reality.html
98. Bosch Sensortec GmbH, 2018. Augmented & Virtual Reality.
https://www.bosch-
sensortec.com/bst/applicationssolutions/ar_vr/overviewarvr#
99. Hills-Duty, R., 2018. Dassault Systems SOLIDWORKS 2019 Introduces
VR and AR Functionality. https://www.vrfocus.com/2018/10/dassault-
systemes-solidworks-2019-introduces-vr-and-ar-functionality/
100. Claassen, R., 2018. AR and VR in Manufacturing: Being There.
https://www.smart-industry.net/ar-and-vr-in-manufacturing-being-there/
101. Nica, G., 2018. BMW Used Virtual Reality to Set Up Workstations for
3 Series Assembly. https://www.bmwblog.com/2018/11/20/video-bmw-
used-virtual-reality-to-set-up-workstations-for-3-series-assembly/
102. Winick, E., 2018. NASA is using HoloLens AR headsets to build its
new spacecraft faster.
https://www.technologyreview.com/s/612247/nasa-is-using-hololens-ar-
headsets-to-build-its-new-spacecraft-faster/
103. Castellanos, S., 2017. Find out how Rolls-Royce is utilising virtual
reality to train engineers. https://blogs.wsj.com/cio/2017/09/21/rolls-
royce-enlists-virtual-reality-to-help-assemble-jet-engine-
parts/?mod=djemlogistics
104. Leu, M.C., Tao, W., Ghazanfari, A. and Kolan, K., 2017. NX 12 for
Engineering Design. Missouri University of Science and Technology.
105. Tao, W., Lai, Z., Leu, M.C., Yin, Z. and Qin, R., 2019. A Self-Aware
and Active-Guiding Training System for Worker-Centered Intelligent
Manufacturing. Manufacturing Letters.

View publication stats