CSPE51 – Augmented

& Virtual Reality

B.Tech. ( CSE) – III yr – A & B Sections
Faculty - Dr. M. Sridevi
Course Objectives
• To know basic concepts of virtual reality
• To understand visual computation in computer graphics
• To understand interaction between system and computer
• To know application of VR in Digital Entertainment
• To know basic concepts of augmented reality
Course Contents
• UNIT I Introduction of Virtual Reality: Fundamental Concept and Components of Virtual Reality -
Primary Features and Present Development on Virtual Reality - Multiple Models of Input and
Output Interface in Virtual Reality: Input - Tracker - Sensor - Digital Glove - Movement Capture -
Video-based Input - 3D Menus & 3D Scanner – Output - Visual /Auditory / Haptic Devices.

• UNIT II Visual Computation in Virtual Reality: Fundamentals of Computer Graphics - Software

and Hardware Technology on Stereoscopic Display - Advanced Techniques in CG: Management of
Large Scale Environments & Real Time Rendering.

• UNIT III Interactive Techniques in Virtual Reality: Body Track - Hand Gesture - 3D Menus -
Object Grasp. Development Tools and Frameworks in Virtual Reality: Frameworks of Software
Development Tools in VR. X3D Standard; Vega - MultiGen - Virtools.
Course Contents – Contd…
• UNIT IV Application of VR in Digital Entertainment: VR Technology in
Film & TV Production - VR Technology in Physical Exercises and Games -
Demonstration of Digital Entertainment by VR.
• UNIT V Augmented and Mixed Reality: Taxonomy - technology and
features of augmented reality - difference between AR and VR - Challenges
with AR - AR systems and functionality - Augmented reality methods -
visualization techniques for augmented reality - wireless displays in
educational augmented reality applications - mobile projection interfaces -
marker-less tracking for augmented reality - enhancing interactivity in AR
environments - evaluating AR systems.
 Upon completion of this course,
the students will be able to:
• Provide opportunity to
Course explore the research issues in
Augmented Reality and
Virtual Reality (AR & VR)
• Know the basic concept and
framework of virtual reality
• Understand fundamentals of
computer graphics
• Know the computer-human
interaction
• Develop simulator for real
time application using AR &
VR
 Text Books

1. Burdea, G. C., P. Coffet.,

Books “Virtual Reality Technology”,
Second Edition, Wiley-IEEE
Press, 2003/2006
2. Alan B. Craig, “Understanding
Augmented Reality, Concepts
and Applications”, Morgan
Kaufmann, 2013.
Reference Books
1. Alan Craig, William Sherman,
Jeffrey Will, “Developing
Virtual Reality Applications,
Foundations of Effective
Design”, Morgan Kaufmann,
2009.
• Image and Video Source -
from the intranet and text books
VR simulator
• https://www.youtube.com/watch?v=L5lo63YzAAU

• https://www.youtube.com/watch?v=p4vO64Y27JE
Introduction • Devices used
• What is AR, VR,
MR, XR?
• Applications
• Reality Vs Virtual
reality
Devices used in Reality Technology
 Augmented Reality (AR) is a live, direct

What is AR? or indirect view of a physical, real-

world environment whose elements
are augmented (or supplemented) by
computer-generated sensory input
such as sound, video, graphics or GPS
data.
AR exists on top of our own world it
provides as much freedom as you are
given within your normal life. AR
utilizes your existing reality and adds to
it utilizing a device of some sort.
Mobile and tablets are the most
popular mediums of AR now, through
the camera, the apps put an overlay of
digital content into the environment.
Custom headsets are also being used.
Popular AR examples : Pokemon Go
and Snapchat’s new AR bitmojis.
Video : AR Examples

https://www.youtube.com/watch?v=2sj2iQyBTQs
What is VR? • Virtual Reality (VR) is an immersive
experience also called a computer-
simulated reality.
• It refers to computer technologies using
reality headsets to generate the realistic
sounds, images and other sensations that
replicate a real environment or create an
imaginary world.
• VR is a way to immerse users in an
entirely virtual world. A true VR
environment will engage all five senses
(taste, sight, smell, touch, sound), but it
is important to say that this is not always
possible.
• Gaming industry uses this technology
into more practical applications.
• The market and the industry are still
excited about this tech trend and further
progress is expected in the near future.
Video : VR Example

https://www.youtube.com/watch?v=DfUUnp6Z_wc
What is MR? • Mixed Reality (MR), sometimes
referred to as hybrid reality, is the
merging of real and virtual worlds
to produce new environments
and visualizations where physical
and digital objects co-exist and
interact in real time.
• It means placing new imagery
within a real space in such a way
that the new imagery is able to
interact, to an extent, with what is
real in the physical world we
know.
• The key characteristic of MR is
that the synthetic content and the
real-world content are able to
react to each other in real time.
Video : MR Example

https://virsabi.com/mixed-reality/
 • Extended Reality (XR) refers to all

What is XR? real-and-virtual

environments and
combined
human-
machine interactions generated
by computer technology and
wearables.
• Extended Reality includes all its
descriptive forms like the
Augmented Reality (AR), Virtual
Reality (VR), Mixed Reality (MR).
• In other words, XR can be defined
as an umbrella, which brings all
three Reality (AR, VR, MR)
together under one term, leading
to less public confusion. Extended
reality provides a wide variety and
vast number of levels in the
Virtuality of partially sensor
inputs to Immersive Virtuality.
Video : AR, VR, MR, XR
Applications & Challenges

https://www.youtube.com/watch?v=2ZMg0mfEw-k
Challenges
Applications
• Military
• Health care
• Education
• Visualization
• Entertainment
 Advantages
and
Disadvantages
of VR
Tools
• Unity
• Unreal Engine 4
• Cryengine
• 3d Max
• Maya
• X3D
• Vega MultiGen
• Virtools
• Many cloud-based tools available
Japan - Photos

• JAXA Lab
 Olympic Museum -
Japan - Photos and https://www.youtube.com/watch?v=-
_t6Kt2VB_Y
Videos
 Unit – 1
Introduction of Virtual Reality: Fundamental Concept and Components
of Virtual Reality - Primary Features and Present Development on
Virtual Reality - Multiple Models of Input and Output Interface in
Virtual Reality: Input - Tracker - Sensor - Digital Glove - Movement
Capture - Video-based Input - 3D Menus & 3DScanner – Output -
Visual /Auditory / Haptic Devices.
 Introduction to VR
• Virtual reality (VR) - Computer simulation that creates an image of a world that
appears to our senses in much the same way we perceive the real world, or
“physical” reality.
• In order to convince the brain that the synthetic world is authentic, the computer
simulation monitors the movements of the participant and adjusts the sensory
display or displays in a manner that gives the feeling of being immersed or being
present in the simulation.
• The term “virtual world” does not only refer specifically to virtual reality worlds.
It can also be used to refer to the content of other media, such as novels, movies,
and other communication conventions.
Types of VR
• Fully Immersive
• Non- Immersive
• Semi - Immersive
 • Fully-immersive simulations give users the most realistic simulation
experience, complete with sight and sound.
• To experience and interact with fully-immersive virtual reality, the

Fully-
user needs the proper VR glasses or a head mount display (HMD). VR
headsets provide high-resolution content with a wide field of view.

immersive
The display typically splits between the user’s eyes, creating a
stereoscopic 3D effect, and combines with input tracking to establish
an immersive, believable experience.
• This type of VR has been commonly adapted for gaming and other
entertainment purposes, but usage in other sectors, namely
education, is increasing now as well. The possibilities for VR usage
are endless.
 • Non-immersive virtual experiences are often overlooked as a virtual
reality category because it’s already so commonly used in everyday

Non-
life. This technology provides a computer-generated environment, but
allows the user to stay aware of and keep control of their physical

immersive
environment. Non-immersive virtual reality systems rely on a
computer or video game console, display, and input devices like
keyboards, mice, and controller.
• A video game is a great example of a non-immersive VR experience.
 • Semi-immersive virtual experiences provide users with a partially
virtual environment. It will still give users the perception of being in a
different reality when they focus on the digital image, but also allows
Semi- users to remain connected to their physical surroundings. Semi-
immersive technology provides realism through 3D graphics, a term
immersive known as vertical reality depth. More detailed graphics result in a
more immersive feeling.
• This category of VR is used often for educational or training purposes
and relies on high-resolution displays, powerful computers, projectors
or hard simulators that partially replicate design and functionality of
functional real-world mechanisms.
Primary Features and Present
Development on Virtual Reality
• Virtual reality is not a new invention, but dates back more
than 60 years.
• In 1962, U.S. Patent #3,050,870 was issued to Morton Heilig
for his invention entitled Sensorama Simulator, which was
the first virtual reality video arcade.
• Figure shows early virtual reality workstation has three-
dimensional (3D) video feedback (obtained with a pair of
side-by-side 35-mm cameras), motion, color, stereo sound,
aromas, wind effects (using small fans placed near the
user's head), and a seat that vibrated. It was thus possible
to simulate a motorcycle ride through New York, where the
"rider" sensed the wind and felt the pot-holes of the road
as the seat vibrates. The rider could even smell food when
passing by a store.
• Sutherland's vision of an "ultimate display" to the virtual world was
not limited to graphics. In 1965 he predicted that the sense of touch
(or haptics) would be added in order to allow users to feel the virtual
objects. This idea was made reality by Frederick Brooks, Jr., and his
colleagues at the University of North Carolina at Chapel Hill.
• By 1971 the scientists demonstrated the ability to simulate two-
dimensional continuous force fields associated with molecular
docking forces [Batter and Brooks, 1971]. Later they simulated three-
dimensional collision forces using a surplus robotic arm normally used
in nuclear material handling.
• Most of today's haptic technology is based on miniature robotic arms. The
military was very eager to test the new digital simulators, since they hoped to
replace very expensive analog ones. Flight simulators were hardware designed for
a particular airplane model. When that airplane became obsolete, so did its
simulator, and this was a constant drain of funds. If the simulation could be done
in software on a general-purpose platform, then a change in airplane models
would only require software upgrades.
• Much research went on in the 1970s and early 1980s on flight helmets and
modem simulators for the military, but much of this work was classified and was
not published. This changed when funds for defense were cut and some
researchers migrated to the civilian sector.
• The National Aeronautics and Space Agency (NASA) was another agency of the American
government interested in modern simulators. It needed simulations for astronaut
training, as it was difficult or impossible to otherwise recreate conditions existing in
outer space or on distant planets.
• In 1981, on a very small budget, NASA created the prototype of a liquid crystal display
(LCD)-based HMD, which they named the Virtual Visual Environment Display (VIVED).
NASA scientists simply disassembled commercially available Sony Watchman TVs and put
the LCDs on special optics. These optics were needed to focus the image close to the
eyes without effort. The majority of today's virtual reality head-mounted displays still use
the same principle. NASA scientists then proceeded to create the first virtual reality
system by incorporating a DEC PDP 11-40 host computer, a Picture System 2 graphics
computer (from Evans and Sutherland), and a Polhemus noncontact tracker. The tracker
was used to measure the user's head motion and transmit it to the PDP 11-40. The host
computer then relayed these data to the graphics computer, which calculated new
images displayed in stereo on the VIVED.
• In 1985 the project was joined by Scott Fisher, who integrated a new kind of
sensing glove into the simulation. The glove was developed earlier by Thomas
Zimmerman and Jaron Lanier as a virtual programming interface for
nonprogrammers.
• By 1988 Fisher and Elizabeth Wenzel created the first hardware capable of
manipulating up to four 3D virtual sound sources. These are sounds that remain
localized in space even when the user turns his or her head. This represented a
very powerful addition to the simulation. The original VIVED project became
VIEW (for Virtual Interface Environment Workstation) and the original software
was ported to a newer HewlettPackard 9000, which had sufficient graphics
performance to replace the wireframe rendering used in VIVED with more
realistic flat-shaded surfaces.
• With all the afore mentioned technological developments, scientific exchange of
information among the small group of specialists of the time followed. France
was one of the first countries to organize a major international conference on the
subject, held in Montpellier in March 1992. The name of this conference was
Interfaces for Real and Virtual Worlds, and it drew hundreds of papers and many
vendors. Later the same year the United States organized the first conference on
Medicine Meets Virtual Reality. In San Diego, about 180 medical practitioners met
with 60 scientists and engineers to discuss the great potential of virtual reality as
a tool for medicine. In September 1993 the world's largest professional society,
the Institute of Electrical and Electronics Engineers (IEEE), organized its first VR
conference in Seattle. Virtual reality had become part of the mainstream
scientific and engineering community.
Present Development in VR
• VR has been used in different fields, as for gaming, military training, architectural
design, education, learning and social skills training, simulations of surgical
procedures, assistance to the elderly or psychological treatments are other fields
in which VR is bursting strongly .
• There are many possibilities that allow the use of VR as a stimulus, replacing real
stimuli, recreating experiences, which in the real world would be impossible, with
a high realism. This is why VR is widely used in research on new ways of applying
psychological treatment or training, for example, to problems arising from
phobias (agoraphobia, phobia to fly, etc.) . Or, simply, it is used like improvement
of the traditional systems of motor rehabilitation, developing games that
ameliorate the tasks. More in detail, in psychological treatment, Virtual Reality
Exposure Therapy (VRET) has showed its efficacy, allowing to patients to
gradually face fear stimuli or stressed situations in a safe environment where the
psychological and physiological reactions can be controlled by the therapist.
Video
• https://www.smlease.com/entries/technology/what-is-virtual-reality/

• https://www.youtube.com/watch?v=tXIcG3B7CsY

• https://www.youtube.com/watch?v=4APerjQYORk&t=39s
Fundamental Concept of VR
Milgram’s Reality-Virtuality Continuum

Mixed Reality (MR)

Real Augmented Augmented Virtual

Environment Reality (AR) Virtuality (AV) Environment

Milgram coined the term “Augmented Virtuality” to identify systems which are mostly synthetic
with some real world imagery added such as texture mapping video onto virtual objects.
Combining the Real and Virtual Worlds
• Precise models
• Locations and optical properties of the viewer (or camera) and the display
• Calibration of all devices
• To combine all local coordinate systems centered on the devices and the
objects in the scene in a global coordinate system
• Register models of all 3D objects of interest with their counterparts in the
scene
• Track the objects over time when the user moves and interacts with the
scene
Realistic Merging
Requires:
• Objects to behave in physically plausible manners when manipulated
• Occlusion
• Collision detection
• Shadows

Types of reality:
- AR, VR, MR, XR
Elements of
VR
3 I’s in VR
Components of Virtual Reality
Components
of VR
VR output
Devices
 Steps
involved in
VR
Video - Application
• https://www.youtube.com/watch?v=iofuy7tTiYE
Hardware Components used in VR System

Output /
Input Devices Display
& Devices
User Tracking Computation
Mouse Computer Engine Visual display

Data glove, etc.. Graphic Engine Aural display

Tracker Haptic display

Sensory display
VR Input
Devices
VR Input
Devices
 Computation
Computer engine:
• Responsible for calculating the physical behaviour of the Virtual World
• Render the state of the world into visual, aural, haptic, …

Graphics engine:
• Enough computation power to perform the virtual world physical
simulation calculations, graphic rendering platform.

OS:
• Perform Multithreaded operations
Field ofFoV
View (FoV)

https://www.youtube.com/watch?v=pUuAx_zFnEk
FoV & FoR
Field of View (FoV) and Field of Regard (FoR)

• https://www.youtube.com/watch?v=qgF2mZTPkYs

• https://www.youtube.com/watch?v=y0bOi3kVIBs
Videos - DoF
• https://www.youtube.com/watch?v=Hfzkfi_RMeI

• https://www.youtube.com/watch?v=DdvBrKl3SHg
Display Devices
 Visual display
• Stationary display
• Head based display
• Hand based display
Stationary display
 Stationary display –
Desktop display & CAVE
Head Based displays (HBD)
HMD
Hand based
VR display
• Pair of binoculars
• Palm size devices
Aural Display

• Loudspeaker
• Headphone
Video to experience aural display
• https://www.youtube.com/watch?v=__-AS_2KKtc
Haptics
• Haptics , is the technology of adding the sensation of touch and
feeling to computers.
• When virtual objects are touched, they seem real and tangible.
• Derived from greek word haptikos‟ meaning “ABLE TO
COME INTO CONTACT WITH”
Haptics = Touch = Connection
Touch is at the core of personal experience.
Of the five senses, touch is the most proficient, the only one
capable of simultaneous input and output Haptic senses links to
the brain's sensing position and movement of the body by means
of sensory nerves within the muscles and joints.
Haptic Display
Components:
• Tactile – Input through skin
• Proprioceptic – Input through the muscular
and skeletal systems

Display:
i. World grounded display
ii. Self grounded display
Haptic
information

Combination Of :
Tactile Information
Refers to the information
acquired by the sensors
connected to the body
Kinesthetics Information
Refers to the information
acquired by the sensors
inthe joints
 Basic
Configuration
https://www.youtube.com/watch?v=HDgFwRN7Frk&t=165s https://www.youtube.com/watch?v=OK2y4Z5IkZ0&t=400s
Commonly
used Haptic
device -
Phantom

https://www.youtube.com/watch?v=cDTp3BE33Eg
 Commonly
used Haptic
device –
Cyber Grasp

https://www.youtube.com/watch?v=UrhSno47B4o
Haptic Rendering

❑PRINCIPLE OF HAPTIC INTERFACE

❖Interaction occurs at an interaction tool that mechanically couples two controlled dynamical systems :
a) haptic interface with a computer
b) human user with a central nervous system

❑CHARACTERISTICS
❖Low back-drive inertia and friction
❖Balanced range,resolution and bandwidth of position sensing and force reflection, minimal constraints on motion
❖Symmetric inertia,friction,stiffness and resonant frequency properties, proper ergonomics

https://www.youtube.com/watch?v=HDgFwRN7Frk&t=165s
Video – Haptic

https://www.youtube.com/watch?v=XnDoej4mnHU

https://www.youtube.com/watch?v=4aMCJDOEi0k
 Other Sensory display
• Vestibular display (Sense of balance) - Motion platform & Bladder
equipped chair.
• Olfactory display (smell)
• Computer controlled display of gustation (Taste)
Input Devices & User Tracking

https://www.youtube.com/watch?v=u5L9Mvh7tAk
 Input Devices
https://www.youtube.com/watch?v=aOYHg8qdxTE
https://www.youtube.com/watch?v=Owma8HKEuso&t=104s

https://www.youtube.com/watch?v=xWeqrAw0QRw
VR Input
Devices

https://www.ultraleap.com/company/news/blog/vr-controllers-vs-hands/
VR Input
Devices
Tracking in
VR

https://www.youtube.com/watch?v=_q_8d0E3tDk&t=3s
User tracking – Monitor the user’s body
movement
Types:
• Cognitive input (Events specially triggered by the user)
• User monitoring (Tracking the body movements of the user)

Position Sensor – Tracking device of VR system

Disadvantage: 1. R/S of some fixed origin & sensor( Line of sight)
2. Factors (Accuracy, Precision, Interfering media &
Encumbrance)
https://www.youtube.com/watch?v=IWYsdHZDNJk&t=60s https://www.youtube.com/watch?v=2jY3B_F3GZk
Electromagnetic
Tracking system

https://www.youtube.com/watch?v=rN5-NDzZduE&t=43s
Ultrasonic
Tracking
system
Mechanical
Tracker

Data glove Power glove

Optical Tracker
Coordinate
Visual perception
Video : Tracking
https://www.youtube.com/watch?v=RL6YJd_11FM&t=29s

Video : Full body tracking

https://www.youtube.com/watch?v=X5POJ1NdRXg

Video : Cyberith demo

https://www.youtube.com/watch?v=q_rv-213IjI
Interaction Techniques
• Direct
• Physical
• Virtual
• Agent
Interaction Steps

1. Make Selection
2. Perform manipulation
3. Navigation
Travel Paradigms used in VR experiences
Paradigms (Contd…)

later, or this method can be totally

unnatural such as selecting a
destination from a menu and
popping there instantaneously.
3D Menus

https://www.youtube.com/watch?v=ZAaW00gcD78
3D Scanner

https://www.youtube.com/watch?v=lIL3vTjw1WY
Software
• Software components - integrated to enable cogent VR experiences.
• Software ranges from low-level libraries for
- simulating events, rendering display imagery, interfacing with I/O
devices,
- creating and altering object descriptions,
- to completely encapsulated “turnkey” systems that allow one to begin
running an immersive experience with no programming effort.
Software
• Simulation code
• Rendering libraries
• VR libraries
• Ancillary software
Few Applications
• VR:
https://www.youtube.com/watch?v=00uXQFtWdts&t=44s

• AR:
https://www.youtube.com/watch?v=ik79Sh5M-Bs
 Unit – 1 (Recap)
✓Introduction of Immersive technologies: VR , AR , MR , XR
✓Primary Features and Present Development on Virtual Reality
✓Fundamental Concept and Components of Virtual Reality
✓ DoF, FoV, FoR
✓Input devices & Tracker - Sensor - Digital Glove - Movement Capture -
Video-based Input
✓ Interaction - 3D Menus & 3D Scanner
✓Output devices- Visual /Auditory / Haptic Devices.
✓Software
✓Video demo of different applications
Reference
• Images and videos used in this presentation are referred from the
Internet source
• Burdea, G. C., P. Coffet., “Virtual Reality Technology”, Second
Edition, Wiley-IEEE Press, 2003/2006
• Alan Craig, William Sherman, Jeffrey Will, “Developing Virtual
Reality Applications, Foundations of Effective Design”, Morgan
Kaufmann, 2009.