Bharati Vidyapeeth Deemed to be University

College of Engineering, Pune

Department of Electronics & Communication Engineering

Sub: Augmented Reality & Virtual Reality

UNIT-III Virtual Environment

Content: Input/Output devices: Input (Tracker, Sensor, Digital gloves, movement capture, video-
based Input, 3D Menus & 3D Scanner, etc.), Output (Visual/Auditory/Haptic Devices) Generic VR
system: Introduction, Virtual environment, Computer environment, VR technology, Model of
interaction, VR Systems, Animating the Virtual Environment.

Input Devices:
1. Tracker:
• A tracker is a crucial input device in the VR technology. Its primary function is to precisely
monitor and record the position and orientation of physical objects or body parts within the
virtual environment. These tracked movements and orientations are then transmitted to the
computer or VR system in real-time, allowing for the accurate representation of the user's
interactions within the virtual world. In VR applications, trackers are commonly used to
monitor the movement of various elements, including the user's head, hands, or even external
objects like props or controllers.
Example- A head tracker is integrated into VR headsets, enabling the system to continuously
update the position and orientation of the user's head as they look around the virtual
environment. This information is crucial for rendering the correct perspective and ensuring a
realistic visual experience.
• Hand trackers or controller trackers, on the other hand, allow users to interact with virtual
objects by tracking the movement and gestures of their hands or handheld controllers. These
trackers provide the system with data about the user's hand movements, allowing them to
manipulate virtual objects, grab items, or perform complex gestures.
• Trackers can use various technologies such as infrared sensors, accelerometers, gyroscopes, or
optical tracking systems to achieve high precision in monitoring movements. This accuracy is
essential for creating a sense of presence and immersion in VR, as any lag or inaccuracies can
disrupt the illusion of being in a virtual environment.

1
2. Sensor:
o In VR technology, sensors are essential components that play a critical role in detecting
and capturing various physical movements and changes in orientation.
o These sensors are instrumental in tracking the user's actions and interactions within the
virtual environment, contributing to the overall sense of immersion and presence.
Types of sensors used in VR:
a) Accelerometers: Accelerometers are sensors that measure acceleration forces,
including changes in speed or direction of movement. They are often integrated into
VR headsets and controllers to detect head and hand movements accurately. For
example, when you tilt your head in VR, the accelerometer detects the change in
orientation and relays this information to the VR system for rendering.
b) Gyroscopes: Gyroscopes are sensors that measure the rate of rotation or angular
velocity. They work in conjunction with accelerometers to provide precise information
about the orientation of an object or body part in space. Gyroscopes are particularly
useful in tracking rotational movements, such as looking around or turning your hand
in VR.
c) Inertial Measurement Units (IMUs): IMUs combine accelerometers and gyroscopes
to provide comprehensive motion tracking. IMUs are commonly used in VR headsets
to monitor head movements accurately, ensuring that the virtual world aligns with the
user's perspective as they look around.

2
 d) Positional Tracking Sensors: These sensors use various technologies, such as optical
tracking, infrared sensors, or magnetic tracking, to monitor the absolute position of
objects or body parts in 3D space.

e) Eye-tracking Sensors: Eye-tracking sensors are designed to monitor the movement

and focus of a user's eyes within the VR headset. This technology enables rendering,
where the highest level of detail is rendered only in the area where the user is looking,
reducing the computational load and improving performance.

• Sensors in VR technology are essential for providing real-time data on the user's
movements and actions. This data is then processed by the VR system to render the
virtual environment, accordingly, ensuring that the user experiences a seamless and
immersive interaction. High-quality sensors are crucial for minimizing latency, accurately
representing the user's physical actions, and creating a convincing sense of presence in the
virtual world.

3. Digital Gloves:
Digital gloves are wearable devices equipped with sensors that accurately track hand and
finger movements. Users can manipulate virtual objects with their hands, making interactions
in VR feel natural and intuitive. Digital gloves are crucial for applications like surgical
simulations or virtual sculpting.

3
 • Digital gloves are embedded with various sensors, including motion sensors,
accelerometers, gyroscopes, and sometimes haptic feedback devices. These sensors track
the movements and positions of each finger and the hand.
• The sensors in the gloves work in conjunction with software algorithms to accurately
track the precise movements and positions of the user's hands in real-time. This allows for
highly accurate hand and finger tracking.
• Advanced digital gloves can track the movement of each individual finger, allowing for a
wide range of gestures and interactions. This level of articulation enables users to perform
tasks like picking up, manipulating, and releasing virtual objects.
• Haptic Feedback: Some digital gloves are equipped with haptic feedback technology. This
means that when the user interacts with a virtual object, they can feel a sensation that
simulates touch, pressure, or vibration. This enhances the sense of presence and realism in
VR.
• Immersion: By providing a more natural way to interact with virtual environments, digital
gloves enhance the sense of immersion and presence in VR.
Applications:
Digital gloves in VR have a wide range of applications. They are used in fields like
gaming, simulations, training, design, and more.
Example- In medical simulations, surgeons can practice procedures in a virtual
environment using their hands.
4. Movement Capture: Movement capture systems record and replicate the movements of users
within the virtual environment. These systems use specialized equipment, including multiple
cameras and markers, to track the user's body movements It relies on an array of sensors.
These sensors can include infrared cameras, accelerometers, gyroscopes, and sometimes
additional specialized equipment like electromagnetic trackers.

4
5. Movement capture is extensively used in the entertainment industry for creating lifelike
character animations in video games and movies.

• Movement capture systems can operate in real-time or for post-processing. Real-time systems
provide immediate feedback and interaction within the virtual environment, while post-
processing systems record data for later analysis and use in animation.
• The accuracy and precision of movement capture systems can vary depending on the
technology and setup.
• Some systems capture the entire body's movements, while others focus on specific body parts
like hands or head. Full-body capture provides the most immersive experience, but partial
capture can still be highly effective for certain applications.

Applications: Movement capture in VR has a wide range of applications. It is

extensively used in the entertainment industry for creating realistic animations in movies,
video games, and VR experiences. It's also utilized in areas like sports training, medical
simulations, rehabilitation.

6. Video-based Input:
• Video-based input involves using cameras to capture the user's movements and gestures. This
technology can track facial expressions, body language, and even eye movements, allowing
for a deeper level of interaction and immersion in VR. It is often used in applications like
social VR or training simulations.Video-based input relies on one or more cameras placed
in the physical environment. These cameras capture real-time video footage of the
surroundings.
• Specialized software processes the video feed using computer vision algorithms. These
algorithms can identify objects, track movements, and perform other visual recognition
tasks. Video-based input systems often operate without the need for markers or sensors
attached to the user or objects. This provides a more natural and unrestricted interaction.
• Using computer vision, the system can recognize specific gestures made by the user. For
example, it can detect hand movements, gestures, and poses, allowing for natural
interaction within the VR environment.

5
 • Depth Sensing: Some systems incorporate depth-sensing technology, such as infrared depth
cameras or structured light sensors. This allows for the creation of 3D depth maps of the
environment, enabling more accurate spatial recognition.
• The system can identify and track physical objects in the real world. This enables users to
interact with these objects in the virtual space. For example, users might pick up a real-
world object and see it represented in the VR world.
• Video-based input can be used in mixed reality, where the real world is combined with virtual
elements.
▪ For example, the system might place virtual objects on a real-world table, and
the user can interact with them as if they were physically present.
• Advanced video-based input systems can track the movements of the user's entire body or
specific body parts. This enables a higher degree of immersion and realism, allowing users
to move freely within the virtual environment.

7. 3D Menus & 3D Scanner:

a) 3D menus
• 3D menus enable users to interact with virtual objects and make selections in a three-
dimensional space. They offer a more intuitive way to navigate VR environments. 3D
menus in VR are interactive menus that exist within the virtual environment. Unlike
traditional 2D menus on a screen, 3D menus are spatially situated in the VR world, allowing
users to interact with them as if they were physical objects.
• Users can navigate 3D menus using hand gestures or controllers. For example, they might
point at a menu option or use hand movements to select and interact with different elements.
• 3D menus provide a more immersive selection experience. Users can physically reach out
and grab menu items, giving a tangible feel to the interaction.
• 3D menus find applications in gaming, simulations, architectural design, and other VR
experiences where users need to interact with a variety of options and setting.

6
 b) 3D Scanner:
• A 3D scanner in VR is a technology that captures the three-dimensional shape and texture
of real-world objects or environments. It converts physical objects into digital
representations that can be integrated into the virtual environment.
• 3D scanners can use various technologies, including laser-based scanners, structured light
scanners, or photogrammetry (using multiple photographs to create a 3D model). High-
end 3D scanners can capture objects with high precision. This allows for extremely
detailed and realistic representations in VR. The scanned data is typically converted into a
3D mesh, which is a digital representation of the object's surface. This mesh can then be
imported and interacted with in VR.
• Scanned objects can be exported for 3D printing, enabling users to bring virtual creations
into the physical world. Some 3D scanners also capture texture information, allowing for
the application of realistic colors and textures to the scanned objects in the virtual
environment. Few advanced systems enable real-time scanning, allowing users to scan
objects or environments on the fly and immediately see them in the VR environment.
• 3D scanning in VR has a wide range of applications, including product design,
archaeology, architecture, art, and in healthcare for creating custom prosthetics or
orthotics.

7
Output Devices:
Visual Devices in Virtual Reality:
Virtual reality (VR) is all about immersing users in a digital environment that feels real. Achieving
this requires high-quality visual output devices. These serve as the window through which users
perceive the virtual world.

1. VR Headsets:
• At the heart of visual output in VR are the headsets. These are specialized devices that
users wear on their heads. VR headsets typically consist of two high-resolution displays,
one for each eye. These displays render the 3D environment, providing depth and
perspective to the user.
• The lenses within the headset are crucial as they focus the images, ensuring they align
correctly with the user's eyes. This eliminates any blurriness or discomfort, and
contributes to a clear, immersive experience.
• Modern VR headsets also come equipped with tracking sensors. These sensors monitor
the movement and orientation of the user's head in real-time. This information is then
used to adjust the rendered images, accordingly, providing a seamless and responsive
experience.
• The refresh rate of a display refers to how many times per second it updates the image. In
VR, a high refresh rate is essential for reducing motion sickness and ensuring a smooth
experience. When the displayed image lags the user's movements, it can lead to
discomfort or disorientation. High refresh rates, often 90 Hz or higher, minimize this
discrepancy, resulting in a more comfortable experience.
• The field of view represents the extent of the virtual world that the user can see at any
given time. A wider field of view contributes to a more immersive experience, as it
creates a greater sense of peripheral vision. Modern VR headsets strive to provide as wide
a field of view as possible, often around 100 to 120 degrees. The combination of high-
resolution displays, accurate head tracking, and a wide field of view creates a sense of
presence - the feeling that you are truly inside the virtual environment

8
Auditory Devices in Virtual Reality:
Audio plays a critical role in immersing users in a virtual environment. Auditory devices in VR
are responsible for delivering realistic and spatially accurate sound cues, enhancing the user's
perception of the virtual world.
Headphones and Surround Sound:
• To create an immersive auditory experience, users typically wear headphones. These
headphones deliver high-quality audio directly to the user's ears, providing a more isolated
and direct sound experience compared to traditional speakers.
• In some VR setups, particularly those used in larger spaces, surround sound systems can be
employed. These systems use multiple speakers positioned around the user to create a 360-
degree audio experience.
• Spatial audio is a crucial feature in VR. It allows users to accurately perceive the direction
and distance of sounds within the virtual environment. This means that if an object is to the
right of the user in the virtual world, the sound associated with that object will seem to come
from the right side. This enhances the sense of presence and spatial awareness, making the
experience more convincing.
• Well-implemented audio in VR is not just about directional sound. It's also about creating a
realistic audio environment. This might involve ambient sounds, echoes, reverberations, and
other audio effects that match the virtual environment.

Haptic Devices in Virtual Reality:

Haptic devices in VR are all about touch. They provide tactile feedback to users, allowing them
to feel sensations corresponding to in-game events or interactions. This adds a whole new
dimension to the VR experience.

9
 • Haptic devices come in various forms, including gloves, vests, controllers, and even
specialized accessories. Each of these is equipped with haptic feedback technology to
simulate touch sensations.
• For example, haptic gloves have sensors or actuators embedded in the fingertips,
palms, and other areas. When users interact with virtual objects, these devices can
generate sensations like pressure, vibration, or even resistance.
• Texture and Impact Feedback - One of the most impressive aspects of haptic
feedback is its ability to convey texture. Users can feel the roughness of a surface, the
softness of fabric, or the hardness of a solid object. This adds a whole new level of
realism to virtual interactions. Impact feedback is another critical aspect. When a user,
for instance, grips a virtual object tightly, the haptic device can exert a resisting force,
simulating the sensation of grasping a physical object.
• Enhanced Immersion: Haptic feedback bridges the gap between the digital and
physical worlds. It brings a sense of touch to virtual experiences, making them feel
more real. When you reach out and touch an object in VR, the haptic feedback
provides confirmation that solidifies the illusion.
• In-Game Events and Interactions: Haptic feedback is synchronized with in-game
events. For example, if a user's virtual hand collides with a virtual wall, the haptic
device might generate a corresponding vibration or pressure sensation. This makes
interactions feel more dynamic and engaging.

Generic VR System:
• Virtual Reality is a computer-generated simulation of a three-dimensional environment that
users can interact with. It aims to create a sense of presence, immersing users in a digital
world that can be either realistic or entirely fantastical.
• Virtual Environment: The virtual environment is the digital space in which users interact
within the VR system. It serves as the canvas upon which the VR experience unfolds. This
environment can be designed to mimic the real world with high fidelity or take on entirely
abstract and imaginative forms.
• High-quality graphics ensure that the visuals are realistic, detailed, and visually appealing.
This involves rendering techniques that create lifelike textures, lighting, and shading.
Additionally, advanced techniques such as ray tracing can further enhance realism by
simulating how light interacts with objects in the virtual world.
• Audio plays a crucial role in shaping the virtual environment. Spatial audio, achieved
through specialized hardware or software, allows users to perceive sound from different
directions. This creates a more immersive experience, as users can locate objects or other
virtual entities based on the sounds they emit.
• Interactivity is another key aspect of the virtual environment. This involves the user's ability
to manipulate objects, navigate the space, and engage with the elements within the VR world.
The more interactive and responsive the environment is to user actions, the more convincing
and engaging the VR experience becomes.
• Computer Environment: The computer environment is the underlying infrastructure that
powers the VR experience. It encompasses both hardware and software components. To run
VR applications smoothly, a powerful computer is required, equipped with high-performance
graphics processors, sufficient RAM, and a robust central processing unit (CPU).

10
• Graphics processors (GPUs) are particularly critical for VR, as they handle the rendering of
complex 3D graphics in real-time. VR applications demand significant computational power
to maintain high frame rates and ensure a smooth, lag-free experience. Advanced rendering
techniques, like asynchronous reprojection and foveated rendering, optimize performance and
visual quality.
• Specialized VR software is responsible for rendering and simulating the virtual world. This
software manages the communication between the hardware components and ensures that the
user's actions are accurately reflected in the virtual environment. It also handles tasks like
tracking user movements, integrating input from controllers, and managing the display output
to the VR headset.

Virtual Reality Technology:

VR technology encompasses a broad spectrum of components that collectively enable the
creation of immersive virtual experiences. These components include:
Input and Output Devices:
Input devices like motion controllers, haptic gloves, and tracking systems allow users to
interact with the virtual environment. Output devices, such as VR headsets and spatial audio
systems, deliver sensory information to the user.
Tracking Systems:
These systems monitor the user's movements, enabling the VR system to accurately update
the virtual environment based on the user's position and orientation.
3D Modeling Software:
This software is used to create virtual objects, environments, and characters. It plays a crucial
role in designing the elements that populate the VR world.
Rendering Algorithms:
These algorithms are responsible for generating the visual representation of the virtual
environment. They handle tasks like lighting, shading, and texture mapping to create realistic
visuals.

Fig. Virtual Reality system

11
 Model of Interaction:
The model of interaction in Virtual Reality (VR) is the framework that defines how users
engage with the virtual environment. It encompasses the methods and technologies employed
to track user movements, gestures, and actions, providing a responsive and natural interaction
model. Tracking User Movements:
Accurately tracking the movements of the user is principal in VR. This is achieved through a
combination of hardware and software components. Sensors, often integrated into VR
headsets and controllers, monitor the position and orientation of the user's head and hands.
This data is then processed to update the virtual environment in real-time, ensuring that the
visuals align with the user's movements.
• Gestures and Actions: VR systems utilize various input devices, such as motion controllers
or haptic gloves, to capture user gestures and actions. These can include hand movements,
gestures like pointing or grabbing, and even more complex actions like drawing or typing in a
virtual space. The system interprets these inputs to trigger corresponding interactions within
the virtual environment.
• Responsive Interaction:The responsiveness of the interaction model is crucial for
maintaining immersion. A delay or discrepancy between a user's action and its representation
in the virtual world can disrupt the sense of presence. Therefore, VR systems aim for minimal
latency, ensuring that interactions feel immediate and natural.
• Natural Interaction: The goal of the interaction model is to mimic real-world interactions as
closely as possible. This includes how users reach out, pick up objects, move around, and
perform other actions. By replicating these interactions, VR enhances the feeling of being
physically present in the virtual space.
• Customization and Adaptability: VR developers often have the flexibility to design
interaction models that suit the specific context of their applications. For example, a VR
game might implement a unique set of gestures and actions tailored to the gameplay
mechanics. This customization allows for a more tailored and engaging user experience.

VR Systems:
• VR systems are comprehensive setups that encompass both hardware and software
components, creating an integrated environment for immersive virtual experiences. These
systems are designed to deliver a seamless and compelling VR for users.
• Hardware Components:
VR systems consist of a range of hardware elements. This includes VR headsets, controllers,
tracking systems, and computing hardware. The VR headset is perhaps the most prominent
component, as it serves as the primary interface between the user and the virtual environment.
It houses the displays, sensors, and often includes audio output.
• VR Headsets: VR headsets are specialized devices worn on the head that provide users with
a visual and auditory interface to the virtual environment. They typically feature high-
resolution displays that offer a wide field of view, immersing users in the digital world. Some
advanced headsets also incorporate spatial audio technology, enhancing the auditory
experience.

12
• Controllers: Controllers are handheld devices that users interact with in the virtual space.
They come equipped with various sensors that track their position and orientation. This
enables users to perform gestures and actions, which are then translated into the virtual
environment.
• Tracking Systems: Tracking systems are crucial for capturing the movements and positions
of the user's head, hands, and sometimes even the entire body. These systems utilize a
combination of sensors, cameras, and infrared technology to monitor the user's movements in
real-time.
• Computing Hardware: The computing hardware of a VR system is responsible for
processing and rendering the virtual environment. This requires significant computational
power, often provided by high-performance graphics processors (GPUs), ample memory
(RAM), and powerful central processing units (CPUs).
• Software Infrastructure: VR systems are supported by specialized software that manages
various aspects of the VR experience. This includes rendering the virtual environment,
integrating user input, and ensuring responsive interactions. The software also handles tasks
like managing tracking data and output to the VR headset.

VR Platforms:
• Different VR systems may be associated with specific platforms or ecosystems. For example,
Oculus, HTC Vive, and PlayStation VR are major VR platforms, each offering its unique
set of features, hardware compatibility, and content libraries.
1. Oculus: Oculus, owned by Facebook (now Meta Platforms, Inc.), is a prominent VR
platform known for its range of VR headsets, including Oculus Rift, Oculus Quest, and
Oculus Quest The Oculus ecosystem offers a variety of VR content, including games,
social experiences, and educational applications.
o Oculus Quest and Quest 2 are standalone VR headsets that do not require a PC,
making them more accessible to a broader audience. Oculus has a strong focus on
social VR, with platforms like Facebook Horizon, which allows users to socialize
and create in virtual spaces.

2. HTC Vive: HTC Vive is another well-known VR platform known for its high-quality VR
hardware, including the HTC Vive and HTC Vive Pro. HTC Vive's tracking technology,
called "Lighthouse," provides precise and room-scale tracking for a highly immersive
experience. It is popular among enthusiasts and professionals for applications like
architectural visualization and training simulations. SteamVR is often associated with
HTC Vive, giving users access to a vast library of VR content on the Steam platform.

3. PlayStation VR (PSVR): PlayStation VR is a VR platform developed by Sony for use

with the PlayStation 4 and PlayStation 5 gaming consoles. It offers a range of exclusive
VR games and experiences designed to work seamlessly with PlayStation hardware.
PSVR benefits from the established PlayStation gaming ecosystem, making it an
accessible option for console gamers. The platform leverages the PlayStation Move
controllers for VR interaction

13