Output Devices:

Graphical Displays

Rynson W.H. Lau

 The Main Topics

 The Human Visual System

 HMD Image Formation
 Personal Graphics Displays
 Large Volume Displays
 Volumetric Displays
 1. The Human Visual System

 Retina
 126 million photoreceptors
 uneven distribution

 Fovea - the central area of the retina

 high-resolution, color perception area
 representing the focused area

 Low-resolution, motion perception photoreceptors

covering the rest of the viewing field
 Color Perception
 The retina consists of rods and three kinds of cones
 Rods are for night vision
 Cones are for color vision
1. L-cone: most sensitive to red light
2. M-cone: most sensitive to green light
3. S-cone: most sensitive to blue light
Human visible light spectrum
 Human color sensitivity
 The eye is most sensitive to light in the middle of the visible spectrum,
i.e., green color.

 The response in each color channel in the eye is proportional to the

corresponding number of cones:
V() Note that these
 Red Receptor Sensitivity qR() curves are not
in scale.

 Green Receptor Sensitivity qG()

qG() q ()
 Blue Receptor Sensitivity qB() R

qB()

 Luminous-efficiency function V() represents the overall sensitivity, i.e.,

the sum of the red, green and blue response curves.
 Field of View (FOV)
 Each eye:
 Approximately 150o horizontally, 135o vertically.
 Both eyes:
 Approximately 180o horizontally (combined),135o vertically.
 Binocular overlap:
(Within this overlap region, the brain can use the horizontal shift of objects
between the two eyes to estimate depth.)
 Approximately 120o horizontally
 Depth Perception
 When objects are near:
 convergence (of the two eyes)
 accommodation (focusing of each eye)
 disparity (location difference of objects in the two eyes)
 parallax (when viewer moves his head horizontally, nearby objects
appear to move more than distant ones)

 When objects are far away, depth cue includes:

 occlusion (nearby objects tend to occlude distant objects)
 perspective (distant objects appear smaller)
 haze (distant objects tend to be grayer)
 surface texture (similar to perspective)
 Convergence allows us to rotate both eyes to focus on a
nearby point/object.

 Accommodation is the adjustment of the ciliary muscles to

allow us to focus on distant or nearby objects.
(Contraction of the ciliary muscle reduces the focusing distance, while
relaxation of the muscle increases it.)
 Our brain uses the disparity
of an object observed by the
two eyes to estimate the
depth of the object.

 Point F will appear shifted

horizontally between the left
and right eyes as it is
projected to different
positions in the two eyes.

 We may replicate this shift in image rendering to help the

brain perceive depth in the simulated world.
The pencil will appear shifted horizontally, relative to the church, as
observed by the right and left eyes.
 Parallax refers to the different amounts of movement
between nearby and distant objects, as observed by a
moving viewer.
(As the viewer moves his/her head to the left and right, distant objects
should appear moving less than nearby objects.)
 Visual Acuity
 In general, human eyes can resolve up to 1 arc min.
(1 arc min is a minute of an arc or 1/60o.)
 Our visual acuity changes at different locations of
the eye and with our age.

Relative Acuity

Horizontal Angular Distance

 Retina Display
 To produce a display that matches our visual acuity, for
each eye, we need a resolution of at least:
(60 * 150o) X (60 * 135o) = 9000 X 8100 pixels.

 Therefore, the display for each eye should be of angular

size of 150o X 135o and a resolution higher than 9000 X
8100 pixels in total.

 This is expensive, in terms of rendering and memory

costs, even with today’s technologies.

 Manufacturing a display to cover such a large field of

view (FOV) is also not easy.
 Visual System and Displays

Important factors that determine the immersiveness

of a VR system:
 depth perception (stereo displays, depth cues)
 large FOV (to cover the eyes)
 high-resolution images
2. HMD Image Formation
Although the LCD screen may be small, the lens in the HMD magnifies the
image to a virtual image, which is larger and further away from the viewer.
The magnification factor (M) is = h / h’.
Since d = f d’ / (d’ – f), M = f / (d’ – f).
In a real scene, if we
look at an object, our
eyes converge on this
object, and
accommodate the
ciliary muscles to
focus on it.
When viewing in an HMD, if
we look at the same object,
our eyes converge on this
object, but they accommodate
the ciliary muscles to remain
focusing on the virtual image
plane, resulting in the
convergence and
accommodation conflict.
3. Personal Graphics Displays

 Head-mounted displays (HMDs)

 Hand-held displays
 Autostereoscopic monitors
 3.1 Head-Mounted Displays
 Images are projected 1-5m in front of the user.
 Special optics
 let the eyes focus at short distances (LCDs are only a
couple of inches from the eyes) without getting tired.
 magnify the small displayed image to fill the user’s field
of view (FOV) as much as possible.

 The lower the resolution or the higher the FOV, the

greater the arc-minutes of each pixel.
 Older HMDs have low resolution and use diffuser to
blur the edges between pixels.
An HMD-based VR System
 There were a number of low-quality (relatively to
what we have today) and very expensive HMDs on
the market. Most of the companies that
manufactured them are gone now.

 The most popular HMDs today include:

 Oculus Quest 2
 Oculus Quest Pro
 PlayStation VR2
 Microsoft Hololens (for AR only)
Oculus Quest 2 Oculus Quest Pro

PlayStation VR2 Microsoft Hololens 2

 HMD Comparison
Oculus Quest Meta Quest PlayStation
Hololens 2
2 Pro VR2
Resolution 3664 x 1920 3600 x 1920 4096 x 1080 4000 x 2040
Resolution / eye 1832 x 1920 1800 x 1920 2048 x 1080 2000 x 2040
Horizontal FOV 106o ~110o
Vertical FOV 96o 96o ~110o
Tracking 6DOF inside-out 6DOF SLAM Eye-tracking: 2 IR 6DOF tracking
tracking; 4 front- (simultaneous cameras; head based on
Technology cameras for localization and tracking: 4 visible gyroscopes and
controllers; mapping) with 3 light cameras; accelerometers; 4
gyroscopes and camera sensors camera-based embedded cameras
accelerometers for per controller hand tracking for headset and
headset and controllers; IR
controllers. cameras for eye
tracking.
Price from US$400 from US$1,500 US$3,500 US$600
 AR glasses concern how to project computer
information on the glass.
 Microsoft Hololens (https://www.microsoft.com/en-us/hololens)
 Google Glass (https://www.youtube.com/watch?v=SQd394a4qEo)
 Apple Vision Pro (https://www.apple.com/apple-vision-pro/)
 Here, we take a look at HoloLens 2 from Microsoft,
which is a relatively new AR device in the market.

Summary: https://www.youtube.com/watch?v=6lxGU66w0NM

 Laser-based RGB
projectors: projecting
stereo images with
large field-of-view.
 Two main processors: Intel Cherry Trail SoC and
Holographic Processing Unit (HPU). SoC works like a
CPU and handles OS and applications. HPU works like
an I/O processor and handles all I/Os.

 Time-of-Flight depth sensing: projecting laser lights to

the scene and measuring how long it takes to get
reflections from them – 45fps for near-depth sensing in
hand tracking, and 1-5fps for far-depth sensing in
spatial mapping.

 IMU Sensor: accelerometers, gyroscopes,

magnetometers.
 Head tracking: based on 4 grayscale cameras.

 Hand tracking: based on time-of-flight depth sensor,

together with a two-handed articulated model.

 Eye-tracking: based on two IR cameras.

 Spatial mapping: generating real-time environment mesh

based on the time-of-flight depth sensor.

 Voice command.

 Open source Mixed Reality Toolkit (MRTK) for Unreal

Engine development.

 Technical talk: https://www.youtube.com/watch?v=S0fEh4UdtT8

 3.2 Hand-Held Displays (HHDs)

 These are personal graphics displays that the user

holds in one or both hands in order to periodically view
a synthetic scene.

 The user can go in and out of the simulation as

required by the application.

 The users can push buttons to interact with the virtual

scene.
 Virtual binoculars SX (NVICS Inc)

 Resemble the look and feel of regular binoculars.

 Incorporates two miniature LCOS displays and a

tracker.

 The computer updates the graphics based on the

direction and the interaction by the user.

 Resolution: 1280 x 1024

 Weight: 1kg
LCOS (liquid-crystal-on-silicon)
The liquid crystal layer is mounted directly on the control
circuitry silicon chip, allowing lower manufacturing cost,
reduced circuitry and power consumption.
 3.3 Autostereoscopic Displays
 Users do not need to wear any viewing apparatus in
order to perceive a stereo image.

 Special column-interlaced image format, which

alternates individual columns assigned to the left-eye
view and the right-eye view.

 A pair of stereo images simultaneously presented on a

single panel.

 Reduced horizontal image resolution (half of the

columns for the left eye and the other half for the right).
Passive Autostereoscopic Displays

 The user’s head is not tracked. Thus, it restricts

the user’s eyes to be within a small area in
order to perceive stereo vision.
 DTI 2018XL Virtual Window
 Uses a standard LCD panel.
 The pixels in the LCD panel are back-lit by a number of
very narrow column light sources from florescent lamps
or LEDs, via reflectors and plastic light guide.
 The back light passes through a variable diffuser and
the LCD panel, and forms stereo viewing zones in front
of the display.
 The resolution is 1280x1024 in monoscopic mode,
640x1024 in stereoscopic mode.
The relation between backlighting
distance, d, and distance to the
user, D, determines a stereo
viewing cone where the user has
to be, in order to see an image in
stereo.
The relation between backlighting
distance, d, and distance to the user,
D, determines a stereo viewing
cone where the user has to be, in
order to see an image in stereo.

If you want to know how LCD panels work, see:

https://www.youtube.com/watch?v=VamqtyatBss
 A stereo image is formed by combining the left and right
images on alternate columns of the LCD.
 The left image appears on the odd numbered columns
and the right image appears on the even numbered
columns.
 The left eye sees only the odd columns of the stereo
image while the right eyes sees the even columns.
 Active Autostereoscopic Displays

 Passive autostereoscopic displays have

fixed/predefined viewing locations.

 This problem is solved in the active one by

tracking the user’s head motion.
 Ecomo 4D Displays

 A pair of cameras placed at the top of the display track

the user’s motion.

 A prism mask is placed over the display.

 The position of the prism mask is translated by a

stepper motor, based on the position of the user’s head.

 As mechanical means are used to support

stereoscopic viewing, disadvantages may include the
potential for breakage and noisy operation.
Some other autostereoscope displays:
http://www.youtube.com/watch?v=ZzRq7GhBLRQ&feature=player_embedded#!
 4. Large-Volume Displays
 Graphics displays that allow several users located in
close proximity to simultaneously view a stereo or
monoscopic image of the virtual world are called large
volume displays.

 They can be further classified as monitor-based (placed

side-by-side) or projector-based (workbenches, CAVEs,
display walls, domes).

 Their larger work envelope improves users’ freedom of

motion and natural interaction capability, compared to
personal displays.
4.1 Monitor-based Large-Volume Displays

 Active glasses and a stereo-ready monitor.

 Each user wears a set of shutter glasses and looks at

the monitor.

 The stereo-ready monitor is of a special design that is

capable of refreshing the screen at double the normal
scan rate, or between 120 and 140 scans/sec.

 A computer sends two alternating, slightly offset images

to the monitor.
 An infrared (IR) emitter located at top of the display
synchronizes the display signals and the active glasses.

 The IR controller directs orthochromatic liquid crystal

shutters of the active glasses to close and occlude one
eye or the other alternatively.

 The brain registers a quick sequence of right and left

eye images and fuses the two to form a stereo image.
 The extremely short close/open time of the shutters
combined with the 120Hz refresh rate of the monitor
results in flicker-free images.

 The shutters filter out a portion of the light. So, the

images seen by the user appear darker than on a
normal screen.

 A direct line of sight between the IR emitter and IR

receiver is needed, for the computer to control the
active glasses.
 Combining with Head Trackers
 In some cases, it is better to change the image
according to the user’s viewing direction.
 A head tracker can be added to the system to do this.
 Multi-Panel Displays
 The FOV increases as the distance between the user
and the display decreases.
 There is a limit to this distance; it cannot be too small.
 An alternative solution is to use larger monitors or
several monitors placed side by side.
PV290 (Panorama Technologies)

 Three LCD panels integrated

 Image size: 111x29cm2, resolution: 3840x1024
 Alternative Approach

 Simply construct your own system by mounting multiple

monitors on a frame.
 May require multiple graphics/display cards.
 Latest Technology

 Size: 49” diagonal size

 Resolution: 5,120 x 1,440
 Require only one display card
 4.2 Projector-Based Displays

 Earlier CRT Projectors

 Use three tubes (RGB) to
produce image at
1280x1024,120Hz.
 Luminance is low.
 Enclosures must be
constructed around the tubes.
 Digital Projectors
 These replaces the older CRT ones.
 They have much higher luminance.
 There are two main types of digital projection
technology: LCD and DLP (digital light processing).
 LCD Projectors
 Contain 3 separate LCD glass panels (RGB).
 As light passes through them, individual pixels can be set
to allow light to pass or not.
 The output images from the 3 panels are then combined
to form the final color image.
+ LCDs are generally more light efficient than DLPs.
+ LCDs in general produce sharper images.
- Pixellation: pixels of an image can be easily seen.
- They are in general more bulky, due to more
internal components.
- Possibility of “dead pixels”.
 DLP Projectors
 It is based on an optical semiconductor, called digital
micromirror device (DMD), to recreate the image.
 Each pixel is a tiny mirror that can be set to tilt by an electric
current: reflecting light to the output in one orientation but
not the other.
 A rotating color filter can
be synchronized to
produce color images.
 Digital Micromirror Devices
 The DMD is a chip with an array of
very small aluminum mirror
elements.

 Each micromirror is supported by

two posts and torsion hinges to float
above the chip surface.

 A 5V potential applied at an address

in the chip will electrostatically
deflect the corresponding
micromirror by +/-10o.
 + DLPs produce smoother images, with less
pixellation.

+ They produce images with higher contrast.

A summary of the DMD:

https://www.youtube.com/watch?v=gmlvujD2oHI

A summary of LCD and DLP projection:

https://www.youtube.com/watch?v=qOsibeDX8jM
 Immersive Workbenches

 The image is output by a

projector, bounces off a mirror,
and is reflected upward through
a diffuser screen.

 Several viewers wearing active

glasses and sitting around the
display can simultaneously see
3D objects floating on top of the
workbench.
 CAVES

 A number of mirrors and projectors are used to

display images on the walls, including the floor.
 Walls and Domes
 Graphics displays that produce large enough
images to accommodate many more viewers are
walls and domes.
 In order to simultaneously satisfy the requirements
of large image size, image brightness, and high
resolution, it is necessary to use projector arrays.
 A projector array is a large-volume display formed
by a number of functionally identical projectors
calibrated to display a tiled composite image of the
virtual scene.
 PanoWall Projector Arrays
 Incorporate three projectors, each of 1280x1024, placed
side by side behind the screen to form the final image of
3200x1024 pixels.
 To preserve image quality, it is necessary to carefully
synchronize the three projectors and assure color
uniformity and brightness.
 The blending of the three images is done by overlapping
approximately 25% of adjacent images.
 To adjust the luminosity at the overlapping areas, the
images coming from the computer is preprocessed.
 A seamless matrix blending approach is used, which
adopts a two-dimensional look-up table to adjust the
intensity of each overlapping pixel.

This is an example projector array with four overlapping projectors.

 V-Dome
 Domes are large, spherical displays that produce 360o
FOV surrounding an audience as large as 400-500
viewers.
 To produce such imagery, domes rely on multiple
projectors arranged radially around a semispherical rear-
projection screen.
 The projectors need special optics to pre-distort the
image prior to projection on the curved screen.
 4.3 Polarization
 Use a pair of polarized projectors and provide each viewer
with a pair of inexpensive polarized glasses.
 These glasses have lenses polarized differently for the right
and left eyes.
 Each eye sees the image component of the stereo pair with
matching polarization.
 5. Volumetric Displays

 A display of 360°all-encompassing view of objects and

scenes.
 With a resolution of 768x768 (pixels) x 198 (slides).
 Composed of Texas Instruments’ DMD chip, rotating mirrors
and a translucent screen.
 The 2D projection engine images a time series of 5000
frames per second.
 The screen rotates 730 radians per minute.

 <video 1>

 <video 2>

 <video 3>