Lecture 2:

AR/VR applications and historical perspectives

Course code: DES643
Course name: Design for Augmented Reality and Virtual Reality

Dr Amar Behera
Email: [email protected]
Extension: 2401

Design Studio 605H-A

Diamond Jubilee Academic Complex
Learning Outcomes

• Understand how AR/VR technologies took shape

• Identify AR/VR hardware equipment
• Appreciate the role of software tools in AR/VR
technology usage
 The boundaries of
our definition of VR

• Is listening to music using headphones VR?

• Watching a movie at a theatre?
• Both examples above use tech to provide
artificial sensory stimulation
• Think about a portrait or a painting on a wall – it
transports you to another realm altogether,
perhaps to another century in the past
• How about reading a novel?
The phenomenon of
“fooling” the brain

• In real life, when any living thing explores its

environment, neural structures known as “place
cells” are formed Place cells Grid cells
• These cells encode spatial information about the
surroundings
• The place cells get activated when the organism
returns to the same place
• In addition, grid cells even encode locations in a
manner similar to Cartesian coordinates
• Turns out these neural structures (place cells and
grid cells) may form even when undergoing a VR
experience
Competing terminologies
• Virtual environment: Considered synonymous with VR and preferred by
academic researchers
• Think about the difference between a photographically captured real world
vis-à-vis a purely synthetic world
• Augmented reality: An interactive experience that combines the real
world and computer-generated content
• Mixed reality: Merging of a real-world environment and a computer-
generated one
• Distinctions between these terms have kept changing with time and
eroded away
• All sorts of new terms: XR, X Reality, VR/AR, AR/VR, VR/AR/MR
What makes
a VR system?

• Tempting to think only of the software and hardware; the human is a key part of the system
• Brought together, these form a tightly knit field of “perception engineering”: using engineering
methods to design, develop, and deliver perceptual illusions to the user
Standard VR systems

• VR headsets or multi-projected environments to

generate realistic images, sounds, or other sensations
that simulate a user’s presence in a VR environment!
• VR headsets use a
• Head mount display (HMD) with screens in front of
eyes
• Rooms with large screens
• Audio and video feedback typically
• May also allow other sensory (force, vibration) feedback
through haptic technology or 3D touch (create an
experience of touch) HMD and wired gloves at NASA AMES
Applications
of VR
Historical development of VR
Ivan Sutherland’s
HMD (1968)
• Head mounted display was so heavy that
it had to be suspended from the ceiling,
and the formidable appearance of the
device inspired its name—the Sword of
Damocles.
• Head tracking
• See-through optics
• Rendering, interaction and model
generation
• Displays (2” x 1” CRTs)
• Displayed output from a computer
program in the stereoscopic display
Boeing (1992)

Researchers at Boing used a see-through HMD to guide the assembly

of wire bundles for aircraft
KARMA (1993)
• Printer maintenance
View inside mother’s
womb (1994)

• University of North Carolina at Chapel Hill

Nintendo Virtual Boy (1995)
• 32-bit tabletop portable video game
console
• First console capable of displaying
stereoscopic "3D" graphics
• Launch games:
• Mario's Tennis
• Red Alarm
• Teleroboxer and
• Galactic Pinball
AR: Pepper’s Ghost (1862)

Illusion technique, used in the theatre,

cinema, amusement parks, museums,
television, and concerts, in which an image
of an object off-stage is projected so that it
appears to be in front of the audience
Google Glass (2013)
• Touchpad
• Sliding backward shows
current events, such as
weather, and sliding
forward shows past
events, such as phone
calls, photos, and circle
updates.
• Camera
• 5 MP photos and
record 720p HD video
• Display
Let’s look
at
something
fun… • https://illinois.edu/about/quadcam.html
• Sort of immediately transports you to a new location:
makes you feel like you are somewhere else in the real
world: Telepresence
Teleoperation
Open loop VR v Closed loop VR

• Does the sensory stimulation depend on

actions taken by the organism?
• If the answer is “no”, then the VR system
is called open-loop
• In the case of closed-loop VR, the
organism has partial control over the
sensory stimulation, which could vary as a
result of body motions, including eyes,
head, hands, or legs
• Other possibilities include voice
commands, heart rate, body
temperature, and skin conductance
Source: NTU HCI Lab
First-person v Third-person
aspects

• Scientist v Engineer/Developer
• Scientist (third-person) designs VR system for
another organism (first-person)
• Engineer/Developer switches between being the
first-person and third-person
• This is dangerous as you will ignore the flaws of
your own design
• Your body adapts to the flaws with time and
makes them seem non-existent
• Similar to proof-reading your own writing
Mimicing reality
• As a creator of VR experiences,
think carefully about the task,
goals, or desired effect you want
to have on the user
• Can you make the experience
even better than reality?
• Example: Programming on a
beach with a virtual screen
floating in the air
AIRR Tablet: Floating Display with High-
Speed Gesture UI

Source: Ishikawa Group Laboratory

 Synthetic v Captured worlds
• A synthetic world can be completely invented
from geometric primitives and simulated
physics
• Synthetic worlds are common in video games
• A captured world is created using modern
imaging techniques
• Use of cameras and depth sensors
• Techniques such as Simultaneous Localization
and Mapping
• However, it is difficult to construct an accurate
and reliable representation, unless the
environment is explicitly engineered for such
capture (for example, a motion capture studio)
Turtlebot3 Automatic Parking Example with SLAM

Source: ROBOTIS emanual

Tracking human motion
in a VR system

• Facial expressions while wearing a VR headset

• Hand gestures
• Emotional state
• Eye gaze
• Avatars for anonymity
• Mapping human body motion to that of the
avatar
• Avoiding the “uncanny valley”: High degree of
realism that makes your uncomfortable
Health and safety
while using VR

• Fatigue - Brain works harder to

integrate the unusual stimuli being
presented to the senses
• Inconsistencies with prior
expectations
• Outputs from other senses
• Dizziness and nausea
• Problems in the VR hardware and
low-level software
First movie
ever made

Impact: This movie had people screaming and scampering to the back of
the room as they thought the train was going to run them over
Portal 2
• First-person perspective
puzzle game
• Players solve puzzles by
placing portals and
teleporting between
them
• People have dreamt of
entering their video game
worlds for long – VR has
made it a reality
Virtuix Omni
Treadmill
Lucky’s Tale
• Maintains a third-person
perspective as the player
floats above his character

• Go to menti.com
• Use code 6262 4916
 Dumpy
Game –
DePaul
University
Immersive cinema

• Henry: Emmy award winning movie

made using Unreal Engine
Cinema: Tricky questions to consider

• What should viewers be allowed to do?

• How do you make sure they do not miss part
of the story?
• Should the story be linear, or should it adapt to
the viewer’s actions?
• Should the viewer be a first-person character
in the film, or a third-person observer who in
invisible to the other characters?
• How can a group of friends experience a VR
film together?
• When are animations more appropriate versus
the capture of real scenes?
 The VR
Cinema App

Potential app extensions:

• Characters from the movie appear in the virtual theatre
• Theater rips apart in a movie about tornadoes
• Viewing together with friends from around the world
Telepresence using
panoramic views
• A panoramic view gives us a feeling of being
somewhere else
• Examples:
• Google’s Street View and Google Earth
• Jaunt VR captured a 360° panoramic
view of a Paul McCartney concert
• Video conferencing with a feeling of
presence in the meeting room
• By connecting panoramic cameras to robots,
the user is even allowed to move around in
the remote environment
• Telecommute to work instead of commuting
References/Bibliography
• Steven Lavalle, Virtual Reality, Cambridge University Press
 Before the next class…

• Identify 5 trends in AR/VR research and post on the

forum for the lecture with any relevant citations/links
• What trend attracts you the most?
• What contribution would you like to make to this field?
• Alternately, how would you like to use your knowledge
of AR/VR?