Machine Translated by Google

"Mixed Reality": An overview with a focus on future

developments and
areas of application

Heregger Gabriel

Institute of Informatics-Systems
Alps-Adriatic University of Klagenfurt, Universitatsstrasse 65-67
9020 Klagenfurt, Austria {gabrielhe}@edu.aau.at

Abstracts. This work initially aims to define the term "Mixed Reality" more
closely, in order to then summarize the current market and application
situation and then devote the main part to the expected development in the
next few years . Finally, the results obtained are briefly explained again for
the reader.

Keywords: Mixed Reality, Virtual Reality, Augmendted Reality, Technologis che

Neuerungen, Zukunftsabsch¨atzung

1 Introduction
Public interest in the field of virtual reality (VR) and augmented reality (AR) has
increased significantly in recent years. Although these terms were already discussed
and dealt with in more detail in the early 1990s, most of the areas of application
were pure fiction at the time, limited by the technology, which was far too weak
compared to today. That's why the topic disappeared from the minds of the masses
relatively soon. But since then, a lot has improved technically, and we live in a time
when VR and AR are a reality and are slowly but surely being integrated more and
more into everyday life. Increasingly efficient and, above all, smaller computers
open up new possibilities in development, innovation and application. There are
already various devices, mostly in the form of VR/AR glasses, that allow the user to
interact with the virtual space without spatial restrictions.
Machine Translated by Google

2 definitions of terms
Since the areas of application for the various terms of VR and AR can often merge
into one another and sometimes cannot be clearly differentiated
are, the term Mixed Reality (MR) was chosen for this work to ¨

to cover the entire spectrum. In order to give the reader a better overview, the
various terms are briefly defined in the following points.

2.1 Virtual Reality - immersion in virtual worlds

The term virtual reality was coined in 1982 by the author Damien
Broderick in his sci-fi novel The Judas Mandala. He found use
for the first time by Jaron Lanier in 1989. Nowadays, VR stands for the possibility to
immerse yourself in a completely computer-generated world using suitable technology
and to interact with it. In connection with VR, the concept of immersion is also of
particular importance. This describes the degree with
which the user is integrated and locked into the virtual world.
The fewer stimuli the user perceives from the real world, the more he feels
themselves as part of the virtual reality. To guarantee high immersion, it is
important to make the virtual application as real and interactive as possible.
[1]

2.2 Augmented Reality - extension of the real

Based on the English term “augment” (in German: expand, enrich) AR stands for
the computer-supported expansion of the perceived reality. In the course of AR, a
user does not immerse himself completely in a computer-generated world, as is
the case with VR, but rather the world around him
expanded by additional elements, although these are nowadays
mostly around visual objects, but the term AR refers to any
Sensory perception generated for the user. AR has especially with
the smartphone revolution are finding their way into everyday life more and more.
[2]

2.3 Mixed Reality - From real to virtual

The term mixed reality was coined for the first time in 1994 in a
Publication by Paul Milgram and Fumio Kishino. [3] This is from you
Concept of the ”virtuality continuum”, i.e. the entire bandwidth between
the virtual and the reality. Thus, MR includes any merger
the real world with virtual content. Even today, the
The terms MR and AR are often used interchangeably, but the AR is part of the
MR. Most notably Microsoft uses the term MR around its VR and AR
To market products and to position oneself on the market. 1
Machine Translated by Google

Fig. 1. Mixed Reality Spektrum

3 types of devices - inventory

In general, the devices used today can be roughly divided into two categories
summarize. On the one hand the holographic and on the other hand the always
sive devices, both categories focus on the head-mounted displays
(HMD), i.e. devices that are worn fixed to the head for use. 2

Fig. 2. Mixed Reality Ger¨atetypen

3.1 Immersive devices - VR for everyone

These are the increasingly popular HMDs that

enclose the user's entire field of vision and thus all visual stimuli
shut off from the outside. Ideally, since the screen is very close to the user's body,
it appears to the wearer as if it were part of the
computer generated environment. Some devices protect the user from
any audio-visual stimuli in order to enable complete immersion in the virtual world
in the course of immersion. A distinction is also made between conventional video
glasses, which allow the user to observe video elements undisturbed, and VR
glasses, which have additional sensors
possess that can register and implement head movements. In connection
With data gloves or techniques of gesture recognition, the user has further options
for interacting with the virtual environment
required. As public interest in VR devices continues to grow, so is
the offer of providers with different variants and designs in
increased sharply in recent years. It is important to note that there is a distinction
between VR glasses with a display and VR headsets for smartphones
Machine Translated by Google

must become. There are a number of versions for the latter, but they hang
these always depend on the smartphones used and do not achieve the same quality
a standalone version. In the following section, the three leading products of the VR glasses
with integrated display, which have been on the market since 2016, are described
are, considered.

Oculus Rift: The HMD, designed in 2012, was crowdfunded by its developers on
Kickstarter.com within 4 hours
funded with 2.4 million USD. That alone tells you how asked
even then this technology was. Since then, the Oculus Rift has been released several times
revised and is after the first two versions, which are 2013 and 2014 as
Developer kits, i.e. designed for developers, have been released since
March 2016 consumer version available. Among other things, the takeover by Facebook
¨

and the initially advertised costs of 300-400 USD, which were then replaced by a sales
price of 599
USD have been replaced.[5] The HMD is now available for around USD 450.
Also with regard to the development for Oculus Rift has changed in recent years
done some things. The manufacturers are currently offering a free and open source one
Development Kit and engines like Unity, Unreal Engine and Source Engine
are already adjusted for development. WebVR, a programming interface for the Mozilla
Firefox and Google Chrome browsers, also supports it
Using Oculus Rift. [4] 3

Fig. 3. Oculus Rift

Machine Translated by Google

HTC Vive: In cooperation with the company Valve, HTC developed the HMD, which
¨

was presented to the public in 2015. Valve started researching VR in 2012 and
presented the Steam VR platform they developed for the first time in 2014. In addition
to the HMD, the HTC Vive also requires two controllers, one for each hand, and a so-
called base station, which uses laser beams that are detected by sensors on the
headset and the controllers to carry out the necessary calculations, the so-called
Lighthouse System. The entire package is currently available for just under 700 EUR.
Valve has also published the source code of their OpenVR programming interface on
GitHub, which can be used for other headsets and input devices besides Steam VR.[6]
4

Fig. 4. HTC Vive Lighthouse

PlayStation VR: The general interest in and supply of MR devices is currently strongest
in the gaming sector, so the fact that Sony is also involved with the Playstation VR,
initially known as Project Morpheus, is not surprising. Although a Playstation 4 is
required to use this HMD, the Playstation VR is comparatively relatively cheap at
around EUR 350. Since its launch in October 2016, 12.2 million VR titles have been
sold for the system. In the coming year, the number of available titles, currently almost
150, is expected to almost double. [7]

3.2 Microsoft HoloLens - AR of the future?

The flagship from Microsoft when it comes to Mixed Reality is the so-called HoloLens,
with which the fundamental difference to conventional VR systems lies in the fact that
on the one hand it does not require any additional devices and is therefore not tied to a
specific location and on the other hand On the other hand, the user does not completely
immerse himself in a virtual world, but rather expands the real environment by means of
3D projections. The special thing about this is that objects are detected using sensors
Machine Translated by Google

be projected in reality in a way that fits in perspective and is true to scale.

In addition, the user is able to firmly anchor the projections and thus bypass them
in space. Since the release of the developer version at the end of 2016, which
costs around 3000 USD, many companies have been trying to integrate the
HoloLens into their company and are looking for ways to use it in the best possible
way.[10] 5

Fig. 5. Microsoft HoloLens in the industry

4 Current Use and Developments

In addition to the currently most heavily marketed gaming area, MR offers a
number of advantages in many other areas of application. There is no doubt that
the greatest potential lies in the industrial, research and healthcare sectors.

4.1 MR in Industry 4.0

The possible uses in industry are far-reaching and have experienced an increasing
upswing in recent years. Like so many things in the market economy, mixed reality
has found its way into the industry because it helps to reduce costs. Personnel
who operate large machinery, for example, no longer have to be trained directly on
it, but can learn the basics by means of simulation on the digital image. The
company pool3 has already developed a number of MR applications for various
partners, including a VR tour through an object from the Hartl Haus company, as
well as an AR tool for easier operation of a combination forklift from Jungheinrich.
[11]6

MR can also help to optimally design and market a product from the design process
through production to product presentation. This makes it easier to create designs
in a VR environment and, for example, in 360ÿ
Present showrooms to customers interactively. By means of virtual business
premises, customers can be advised without geographical restrictions.
Machine Translated by Google

Fig. 6. Jungheinrich EKX combination truck

4.2 Research and Medicine

The possibilities that arise in the field of science are still

hardly exhausted, but there are already numerous areas of application today.
Among other things, certain experiments in research can be carried out in safe
Environments are simulated and practiced and certain processes in the laboratory
are explained and observed easily and without risk. The greatest potential is there
but in health care and medicine. So it is nascent and
¨

practicing physicians and specialists to practice possible difficult interventions and

operations and at the same time helpful information on an HMD
to be displayed.[12]

4.3 Education using interactive MR

MR also offers new forms of interaction and presentation of information in the

education sector. In this way, students can be given the opportunity to
Perceiving content interactively and, in the course of a Virtual Learning
Environment (VLE), i.e. a kind of virtual classroom, information on a
to process in a variety of ways. A practical example of this would be
synthetic computer-generated characters, such as that from USC/ISI (University
of Southern California, Information Science Institute) entwickelte p¨adagogische
Agent, named Steve [13]. This is able to demonstrate and teach students skills,
answer questions and provide feedback on theirs
to give exercise. [14]

5 The ”near future” and possible innovations

The limits of MR applications are far from being exhausted

in the coming years, some projects will follow that have potential
to revolutionize certain areas of everyday life.
Machine Translated by Google

5.1 Microsoft MR Headsets

The first MR headsets have been on the market since the end of 2017 and, according to Microsoft,
¨

their future lies in the seamless transition from VR and AR. However, the first versions currently
available are not much more than conventional ones
VR headsets, although they bypass the so-called base stations with the help of new ones
tracking capabilities, but they don't offer any fusion of VR and AR.
[15] Nevertheless, it is very likely that in the next few years more technologies will move in the direction
¨

of MR and realize this transition from the real to the virtual more and more effectively.

5.2 Innoactive Hub

The platform, marketed as HoloCloud in recent years, by the company

Innoactive is primarily intended to offer global companies the opportunity to employ their employees
an easy way to network and collaborate using virtual
To allow clearing easily and without local restrictions. The possible applications here are enormous,
from meetings to collaborative design,
to interactive support. Also in the direction of customer support there
Steps, with the help of VR showrooms, which will soon be available from
of MediaMarktSaturn GmbH as "Virtual SATURN".[16] 7

Fig. 7. VirtualSATURN showroom

5.3 Avatars as a learning aid

In many institutes certain skills are required in certain situations

of working life are trained by means of role play. Traditionally
if this happens as a group activity with other students or course participants, the effectiveness of these
¨

exercises always depends on the other participants and their ability to take on a role. MR applications will
soon offer the option of using so-called avatars, i.e

interact with computer-generated and human-controlled characters

and thereby better practice certain situations. [17]
Machine Translated by Google

5.4 Wireless

The trend in the next few years will undoubtedly be towards wireless use. Oculus
plans to launch a portable version of their VR glasses for just $199 in 2018 with
Oculus Go. In connection with their Santa Cruz project, two hand-held controllers,
users should in future receive a full 6DOF (Six Degrees Of Freedom) experience,
i.e. a direct mapping of their movements in virtual space. [18]

5.5 MR for military use

Of course, the advantages of additional information, which is displayed directly

and without a time delay in the field of vision, are also extremely interesting in the
area of planning and implementation for soldiers and generals in the military. For
this very reason, the Israeli army, for example, has been testing the Microsoft
HoloLens since the summer of 2016 in the area of strategy development and
control of drones for information gathering in foreign territory. There is no doubt
that MR headsets will become an integral part of soldiers' equipment in modern
warfare in the future. [19]

6 Conclusion

The applications for MR are already wide-ranging and continue to increase.

Examples of this include simulations and virtual rooms for various areas in
industry, research, medicine and education. By using these applications, costs
can be saved, for example, work and learning processes simplified and new ways
of interaction created. The devices that are currently mainly in use can be roughly
divided into VR headsets and holographic HMDs such as Microsoft's HoloLens.
Furthermore, it can be said that most real applications are still in the often early
development phase and the public will probably have to wait a few more years
before they are ready for the market. It is also important to note that although full
¨

MR is far from being achieved and there are no devices that would fall into this
category yet, there is already a lot of interest from both private and public sectors
in MR only these technologies exist. The potential for future developments is great
and the countless possibilities are only limited by the state of the art. It can
therefore be said that precisely because of the rapid technological progress of our
time, MR will become an ever-growing part of our everyday life within the next few
decades and will soon become indispensable.
Machine Translated by Google

References
1. C. Schwarzbauer (2005). Digital Media For Artists . http://www.dma.ufg.ac.at/ app/link/
Grundlagen%3AAllgemeine/module/13975 2. R. Azuma (1997). A Survey of Augmented Realit.y
http://www.cs.unc.edu/ ~azuma/ARpresence.pdf 3. P. Milgram, H. Takemura, A. Utsumi, F. Kishino
(1995). Augmented Reality: A class of displays on the reality-virtuality continuum. http://
citeseerx.ist.psu. edu/viewdoc/download?doi=10.1.1.83.6861&rep=rep1&type=pdf

4. P. W. Butcher, P. D. Ritsos Building (2017). Immersive Data Visualizations for the

Web. http://pdritsos.com/files/Butcher-Ritsos-CW2017.pdf
5. Oculus Rift: Step Into the Game. Kickstarter. (2015). https://www.kickstarter.
com/projects/1523379957/oculus-rift-step-into-the-game
6. R. M¨uller. HTC Vive - Virtual Reality raumf¨ullend. https://vrjump.de/artikel/
live
7. J. Feltham (2016) Project Morpheus. VR Focus. https://www.vrfocus.com/2015/
05/project-morpheus-the-story-so-far/
8. Mixed Reality Spectrum. Microsoft. https://developer.microsoft.com/en-us/ windows/mixed-reality/
mixed_reality 9. Dr.-Ing. Dipl.-Kfm. C. Runde (2014). Head Mounted Displays & Datenbrillen. https://
www.vdc-fellbach.de/files/Whitepaper/2014_VDC-Whitepaper_Head_ Mounted_Displays_&_Datenbrillen.pdf
10. B. Lange (2016). Augmented Reality: HoloLens in der Arbeitswelt. iX Magazin f¨ur professionelle
Informationstechnik. https://www.heise.de/ix/heft/ Realitaet-4-0-3866416.html 11/2017 11. pool3 -
Technische Animationen. www.augmented-reality.at 12. Birt, J., Moore, E., & Cowling, M. (2017).
Improving paramedic distance educa tion through mobile mixed reality simulation . Australasian
Journal of Educational Technology, 33 ( 6), 69- 83. https://doi.org/10.14742/ajet 13. Gratch J,
Marsella S. (2001). Tears and Fears: modeling emotions and emotional behaviors in synthetic agents.
In: Proceedings of the Fifth International Conference on Autonomous Agents.

14. Z. Pan, A. D. Cheok, H. Yang (2006). Virtual reality and learning environments. https://ac. els-
mixed reality for virtual cdn.com/S0097849305002025/1-s2.0-
S0097849305002025-main.pdf?
_tid=9ef95544-f468-11e7-a961-00000aab0f6c&acdnat=1515411764_
7a8fe3a4bf179f04d9091579681ab302

15. H. Dingman (2017). PCWorld, Windows Mixed Reality headsets are just VR headsets. https://
www.pcworld.com/article/3233247/virtual-reality/ dont-be-fooled-windows-mixed-reality-headsets-are-
just-vr-headsets.
html
16. Innoactive Hub (2018) https://innoactive.de/hub/ 17. J. Keengwe,
K. Murphy, J. Cash, J. Kellinger (2017). Handbook of Re search on Pedagogical Models for Next-
Generation Teaching and Learn ing. University of North Dakota, USA. https://www.igi-global.com/
book/ handbook-research-pedagogical-models-next/181765

18. Oculus Go, Project Santa Cruz (2017). https://www.oculus.com/blog/

pioneering-the-frontier-of-vr-introducing-oculus-go-plus-santa-cruz-updates/
19. S. J¨arkel (2017). Virtual fighting Israel's army is testing Microsoft's HoloLens.
https://heise.de/-3729642