Lecture 3:

AR/VR historical perspectives, hardware and software

Course code: DES643
Course name: Design for Augmented Reality and Virtual Reality

Dr Amar Behera
Email: [email protected]
Extension: 2401

Design Studio 605H-A

Diamond Jubilee Academic Complex
Learning Outcomes

• Understand how AR/VR technologies took shape

• Identify AR/VR hardware equipment
• Appreciate the role of software tools in AR/VR
technology usage
Robotic avatars
• The DORA robot from the
University of Pennsylvania
mimics the users head
motions, allowing him to
look around in a remote
world while maintaining a
stereo view
• The Plexidrone, a flying
robot that is designed for
streaming panoramic video
Virtual societies
using VR
• Avatars connected to real
people
• People interact in a fantasy
world through avatars
• Groups of people could spend
time together in these spaces
for a variety of reasons
Empathy
using VR
• https://www.youtube.com/
watch?v=mUosdCQsMkM
• You can actually tilt and pan
the camera to see the entire
environment
• Film producer Chris Milk
offered a first person
perspective on the suffering
of Syrian refugees
Empathy for
others:
Body-
Swapping

• What if you were 10 cm shorter than everyone else?

• What if you teach your course with a different gender?
• What if you were the victim of racial discrimination by the police?
Use of VR in education
• Visualize geometric relationships in difficult concepts
• Visualize data that is hard to interpret
• Transfer learning from virtual environment to real
environment
• Particularly useful if real environment is too costly or poses
health risks
• Flight simulation
• Firefighting
• Nuclear power plant safety
• Search and rescue
• Military operations
• Medical procedures
VR in humanities education and digital heritage
Virtual prototyping using VR
Ty Hedfan – the
hovering house
VR in health care
• Distributed medicine: Doctors train people to
perform routine medical procedures in remote
communities around the world
• Guidance through telepresence
• Clinical training: Programmable robopatients
• Immerse in 3D organ models generated from
medical scan data
• Plan and prepare for medical procedures
• Explain medical procedures to patient or family:
Informed decisions
VR in health care

• Overcome phobias and stress disorders

using repeated exposure
• Improve and maintain cognitive skills
despite ageing
• Improve motor skills to overcome
balance, muscular, or nervous system
disorders
• Improve longevity using remote travel
opportunities and fun physical therapy
• Overcome loneliness by joining in
remote activities
Cool AR
HoloLens for
improved
productivity
Some key challenges in
developing these applications
• Use of advanced computer vision techniques
• Identify objects, reconstruct shapes, and identify lighting sources
in the real world
• Determine how to draw virtual objects that appear to be naturally
embedded
• Vision algorithms make frequent errors in unforeseen
environments
• Estimating real world lighting conditions to determine how to
draw the virtual objects
• Any shadows they might cast onto real parts of the environment
and other virtual objects
• Furthermore, the real and virtual objects may need to be perfectly
aligned in some use cases
• Strong burdens on both tracking and computer vision systems
Solving the challenges

• A fixed screen should show images that are

enhanced through 3D glasses.
• A digital projector could augment the
environment by shining light onto objects, giving
them new colors and textures, or by placing text
into the real world.
• A handheld screen, which is part of a
smartphone or tablet could be used as a window
into the augmented or mixed world
See-through v Pass-through
displays

• See-through
• Users see most of the real world by simply looking
through a transparent material, while the virtual
objects appear on the display to disrupt part of the
view.
• Achieving high resolution, wide field of view, and
the ability to block out incoming light
• Pass-through
• Sends images from an outward-facing camera to a
standard screen inside of the headset
• Latency, optical distortion, color distortion, and
limited dynamic range
CANON MREAL X1 – See-through experience
New human
experiences using VR

• Experiences through the eyes of robots or other people

• Giving people experiences that are impossible in the real
world
• Experience of flying
• Being 2 mm tall and looking at ants in their face
• Being 50m tall and towering over a city while people
scream and run from you
• Effect of drugs in the body
• Becoming your favorite animal
• Becoming a piece of food
3D Roller
Coaster

Try out the full 3D experience at:

https://www.youtube.com/watch?v=hNAbQYU0wpg
Brain Connected VR
Paintings: Staring at
rectangles

• Man and humped bulls:

Bhimbhetka, M.P. (10000 BCE)
Paintings: Staring at
rectangles

• The Peasants’ Revolt (1381)

• King is shown twice
• Perspective missing in soldiers’ assembly
Paintings: Staring at
rectangles

• Vredeman de Vries, Hans – 16th

century - Ideal Palace Architecture.
Paintings: Staring at
rectangles
• The Fighting Temeraire (1839)
• Impressionist painting
• Leaves the viewer to imagine
• Relatively small, thin, yet visible
brush strokes
• Open composition, emphasis on
accurate depiction of light in its
changing qualities (often
accentuating the effects of the
passage of time)
• Ordinary subject matter, unusual
visual angles, and inclusion of
movement as a crucial element of
human perception and experience
 From paintings to motion: Stroboscopic effect

• Visual phenomenon caused by aliasing that

occurs when continuous rotational or other
cyclic motion is represented by a series of short
or instantaneous samples (as opposed to a
continuous view) at a sampling rate close to the
period of the motion
• “Wagon-wheel effect“: In video, spoked wheels
(such as on horse-drawn wagons) sometimes
appear to be turning backwards
• Flipping quickly through a sequence of pictures
gives the illusion of motion, even at a rate as low Simon Stampfer, coined the term in his 1833 patent application
as two pictures per second. for his stroboscopische scheiben (better known as the
• Above ten pictures per second, the motion even "phenakistiscope")
appears to be continuous, rather than perceived
as individual pictures.
Horse in
motion (1878)
• Evenly spacing 24 cameras
along a track and triggering
them by trip wire as the
horse passes
A Trip to the
Moon (1902)
• 1902 French science-fiction
adventure film directed by
Georges Méliès.
• Group of astronomers who
travel to the Moon in a
cannon-propelled capsule,
explore the Moon's surface,
escape from an underground
group of Selenites (lunar
inhabitants), and return to
Earth with a captive Selenite.
Animation: Fantasmagorie (1908)

• Low levels of realism in animation

• Yet, popular

• Film was created by drawing each frame on

paper and then shooting each frame onto
negative film, which gave the picture a
blackboard look. It was made up of 700
drawings, each of which was exposed twice
(animated "on twos"), leading to a running
time of almost two minutes.
Animation movies kept improving:
Mickey Mouse – Steamboat Willie (1928)
Video games offer a
first-person
perspective

Early games:

• Pong

• Donkey Kong
Donkey Kong
(1981)

One of the earliest platform games

Creator: Nintendo
 First person
shooter (FPS)
games:
Doom

Advanced FPS game might require a powerful PC and graphics

card, whereas simpler games would run on a basic smartphone
Going from 2D to 3D
• Presenting a separate picture to each eye to
induce a “3D” effect
• Increasing the field of view so that the user is not
distracted by the stimulus boundary
• One way our brains infer the distance of objects
Wheatstone's mirror stereoscope (1838)
from our eyes is by stereopsis
• Information is gained by observing and matching Contributions to the Physiology of Vision. — Part
features in the world that are visible to both the the First.
left and right eyes On some remarkable, and hitherto unobserved,
Phenomena of Binocular Vision.
• The differences between their images on the Presented to the Royal Society of London on June
retina yield cues about distances 21st 1838,
https://stereosite.com/collecting/the-birth-of-
stereoscopy-wheatstone-on-binocular-vision-
1838-original-source/
ViewMaster (1930s)

Thin cardboard disks containing seven

Stereoscopic 3-D pairs of small transparent color
photographs on film
Sensorama (1962): Morton Heilig

One of the earliest known examples of immersive, multi-sensory technology

Improving on limitations
of early systems

• Viewpoint is fixed with respect to the picture

• If the device is too large, then the user’s head
also becomes fixed in the world
• Alternative: Special glasses that select a different
image for each eye using polarized light filters.
• This popularized 3D movies, which are
viewed the same way in the theaters today
• Increase the field of view
• Cinerama: Curved, wide field of view that is
similar to the curved, large LED (Light-
Emitting Diode) displays
CAVE system for VR

• A virtual reality (VR) environment consisting

of a cube-shaped VR room or a room-scale
area in which the walls, floors and ceilings are
projection screens.
• The user may wear a VR headset or heads-up
display and interacts through input devices
such as wands, joysticks or data gloves
Computer Assisted Virtual Environment
Before the next class…
• Browse through recent publications in the below
journals/conferences and share one article from the last
5 years that excited you the most on the forum:
• IEEE Virtual Reality (IEEE VR)
• IEEE International Conference on Mixed and Augmented Reality
(ISMAR)
• ACM SIGGRAPH Conference
• ACM Symposium on Applied Perception
• ACM SIGCHI Conference
• IEEE Symposium of 3D User Interfaces
• Journal of Vision
• Presence: Teleoperators and Virtual Environments.