3422 Vol. 54, No.

11 / April 10 2015 / Applied Optics Research Article

Slim near-eye display using pinhole aperture

arrays
KAAN AKŞIT,* JAN KAUTZ, AND DAVID LUEBKE
NVIDIA Research, 2700 San Tomas Expy, Santa Clara, California 95050, USA
*Corresponding author: [email protected]

Received 4 December 2014; revised 12 March 2015; accepted 12 March 2015; posted 13 March 2015 (Doc. ID 228663); published 9 April 2015

We report a new technique for building a wide-angle, lightweight, thin-form-factor, cost-effective, easy-to-
manufacture near-eye head-mounted display (HMD) for virtual reality applications. Our approach adopts
an aperture mask containing an array of pinholes and a screen as a source of imagery. We demonstrate
proof-of-concept HMD prototypes with a binocular field of view (FOV) of 70° × 45°, or total diagonal FOV
of 83°. This FOV should increase with increasing display panel size. The optical angular resolution supported
in our prototype can go down to 1.4–2.1 arcmin by adopting a display with 20–30 μm pixel pitch. © 2015 Optical
Society of America

OCIS codes: (110.1220) Apertures; (110.1758) Computational imaging; (330.1400) Vision - binocular and stereopsis.

http://dx.doi.org/10.1364/AO.54.003422

1. INTRODUCTION introduces optical image distortion problems caused by the re-

Design of a near-eye head-mounted display (HMD) includes fractive elements.
multiple challenges such as achieving a wide field of view In this paper, we propose a novel method to build a light-
(FOV), high optical angular resolution, slim form factor, light weight, slim near-eye HMD for VR. Our approach pairs a
weight, and ease of replication. The use of any reflective, re- conventional display with an aperture array, or an array of
fractive, or diffractive components entails a trade-off in one pinhole apertures.
of these challenges. Pinholes have been used for various purposes in the display
Researchers have recently demonstrated various ideas to over- domain. Huang et al. demonstrated a visual aberration cor-
come these challenges. Lanman and Luebke developed an HMD recting display [4], which works with pinhole parallax barriers
[1] using microlenses in front of a microdisplay for virtual reality and microlenses. Akşit et al. used pinholes with wavelength se-
(VR) applications, which shares a similar principle with the lective filters in front of the eye to address the accommodation-
existing integral imaging displays/cameras. Their HMD also vergence conflict in stereoscopic displays [5]. Both cases have a
addressed the problem of the accommodation-vergence conflict large viewing distance. Sprague et al. embed a single pinhole
through a lightfield approach. Maimone and Fuchs demon- inside a contact lens to avoid bulky imaging optics in HMD
strated a unique see-through HMD [2], which uses a stack of design [6]. Song et al. propose a lightfield HMD [7] using
LCD panels. The image in this case is formed computationally free-form optics combined with a pinhole aperture array.
through modulating each panel with a set of optimized patterns. However, to the best of our knowledge, ours is the first system
Most recently, Maimone et al. introduced another see-through to use pinhole aperture arrays as a stand-alone image relay layer
HMD approach [3], which adopts a sparse array of point light in an HMD configuration.
sources with a single spatial light modulator (SLM). This ap- We believe our technique is a basis for a framework to build
proach achieves a wide binocular FOV of 110° in their prototype. simple HMDs with a small number of components. Our
A recent trend in HMD design is the use of mobile device proposal’s main contributions to the domain of HMD research
displays from phones and tablets, which provides very high are itemized as follows:
density, typically around ∼300–600 pixels per inch (ppi), • We provide a wide FOV, and high optical resolution
and large display dimensions up to around 6 in. diagonal. HMD design, almost matching the optical capabilities of a
Some HMD manufacturers take a classical magnifier approach human eye.
in their design to provide a practical VR experience with mobile • We provide the simplest optical design to build HMDs
devices, often using the same screen for both eyes. Although with a small number of optical components.
this design provides simplicity, it comes with a bulky form fac- • We provide a thin-form-factor, cost-effective, lightweight
tor due to the large focal length of the used lens(es), and it also solution, unlike many other designs in industry and research.

1559-128X/15/113422-06$15/0$15.00 © 2015 Optical Society of America

 Research Article Vol. 54, No. 11 / April 10 2015 / Applied Optics 3423

• We provide an easily replicable design for researchers in- A constant in our design is the eye pupil size d e , which is
terested in lightfield and VR research. equal to the size of the eyebox in our solution. We have chosen
d e as 8 mm, which varies between 2 and 8 mm in reality [8].
2. PROPOSAL Choosing an eyebox bigger than the actual size of an eye pupil
Our proposal is sketched in Fig. 1, in which a pinhole aperture will provide the freedom to compensate for change in gaze
array is placed at a distance to the cornea of an eye, with a without requiring a pupil tracker. For example, a 4 mm eye
display screen at a fixed distance beyond the pinhole aperture pupil size will have enough freedom to gaze at different parts
array. When designing such a HMD, we must first choose the of the screen, when an 8 mm eyebox is designed.
distance of the screen from the eye(s). Given the generally As highlighted earlier, we have chosen d ae
d ai to be
accepted 1 arcmin angular visual acuity threshold of a human 45–50 mm. Note that the choice of d ae and d ai directly affects
eye, we propose to place the screen at the tip of the nose, i.e., at the spacing of the pinholes d b as shown in Fig. 2(a). Using
45–50 mm distance from the eye. At this distance, the smallest Fig. 2(a), the spacing of the pinholes can be formalized as
resolvable feature corresponds to 10–15 μm—a pitch size the d e d ai
display industry should be able to provide in the near future as a db ≥ : (1)
d ai
d ae
standard.
A human eye cannot focus at a distance as small as 45– The size of a single elemental image d 1 can be calculated
50 mm. A pinhole can help in this case, as it will bound using Fig. 2(b) as
the cone of rays that enter the eye’s pupil as shown in d e m
d ai  d a d ae
Fig. 1. In other words, each screen pixel will send an almost d1  ; where m : (2)
d ae − m da
de
angularly bounded beam of light to the eye. The bundle of rays
from a single pinhole will form an image on the retina, which Equation (2) contains the pinhole size d a as a variable; d a is
are called retinal elemental images. In similar configurations, a directly correlated with the angular resolution of the system. d a
FOV of a single pinhole is ∼8°–12°, in which the eye pupil has to be selected in a way that the maximum possible angular
diameter plays a dominant role over the FOV. By increasing resolution is provided. According to Fourier optics, the angular
the number of pinholes in our system, the FOV can be wid-
ened. We predict a full FOV with a large enough screen and
with enough pinholes.

3. PINHOLE APERTURE ARRAY DESIGN

The mathematical representation of our proposed system
helped us to find the correct aperture designs for our proto-
types. There are two different variables in our system, which
are the cornea to pinhole aperture array distance d ae , and
the pinhole aperture array to screen distance d ai . The rest of
the variables are either a direct result of the change in these
variables or constant values.

Fig. 1. 2D sketch showing the image formation on the retina by

using a pinhole aperture array in front of a screen. Each elemental
image consists of N × N pixels. Depending on the pinhole aperture
array’s distance to the screen (d ai ), eye pupil size (d e ), cornea to pin-
hole aperture array distance (d ae ), pinhole diameter (d a ), and the spac- Fig. 2. 2D sketch showing (a) effect of the changing d ae and d ai
ing of the pinholes (d b ), the retinal elemental images can be made to over d b and (b) effect of changing d a over a single elemental image’s
abut or overlap (as shown here) on the retina. size (d 1 ) on screen. Both sketches are valid for a fixed pupil size (d e ).
 3424 Vol. 54, No. 11 / April 10 2015 / Applied Optics Research Article

resolution of such a system can be calculated using the Rayleigh

resolution formula (θ ≈ λ∕d a ); on the other hand, geometric
optics predicts a growing angular spot size on the cornea with
increasing d a (θ ≈ d a ∕2d ai ). The case in which two estima-
tions meet is the optimal pinhole size in terms of angular res-
olution, which is given as
pffiffiffiffiffiffiffiffiffiffi
d a ≈ 2λd ai : (3)
Note that Eq. (3) is valid as long as paraxial approximation
holds. It is beneficial to select d a accordingly in order to achieve
the best possible angular resolution. A sample design space is
scanned as in Fig. 3 using the derived set of equations
[Eqs. (1)–(3)]. Figure 3 provides an atlas for possible designs
with d e  8 mm and optimum d a derived using Eq. (3).

4. CONTENT CREATION
Our system requires a content creation routine similar to other
lightfield displays. The first task in content creation is to find
out how many pixels fit inside a single elemental image. The
pixel pitch of a display can be calculated by using the ppi (nppi )
value of the display. Using Eq. (2) or Fig. 3(b), a designer can
estimate size of a single elemental image (d 1 ). Once d 1 is cal-
culated, the amount of pixels in a single elemental image can be
calculated as N  nppi d 1 ∕25.4. The location and size of each
elemental image on the retina plane play an essential role in
finding how much each elemental image overlaps on the retina.
Overlap ratios on the retina can be found through ray tracing
the whole system. This information is useful for mapping any
content on each elemental image. The pseudo-code to map a
sample image to a single elemental image is provided as
i  5; j3
elemental  image:get_regioni  N  o; j  N  o; N ; N ;
where i and j are the numbers of elemental images at the x and y
axes of the screen, N is the size of a single elemental image, o is
the result of subtracting the overlap ratio in between retinal
elemental images from 100%, elemental is the result of the
mapping, and image is the source image, which is desired to
be projected on to the retina. In this pseudo-code, the fifth
in x axis, third in y axis elemental image is filled with an
N-by-N rectangular region cropped from a given image.
The cropping operation started with the offset coordinates
of i  N  o in the x axis, and j  N  o in the y axis.

5. EXPERIMENT AND DISCUSSION

Each retinal elemental image can be tiled on the retina by either
abutting each image with a minimum overlap, which provides
higher effective resolution but requires precise eye tracking, or
Fig. 3. Simulation results showing (a) minimum spacing between
heavily overlapping the retinal elemental images to sacrifice ef- pinholes d b in order to use screen without any crosstalk; (b) size of
fective resolution while providing a bigger self-repeating eyebox a single elemental image d 1 , when d a is chosen as in Eq. (3); and
without eye tracking [3]. We provide samples from each (c) optimum pinhole size with the changing screen to pinhole aperture
scenario with multiple prototypes. array distance d ai . Note that (a) and (b) are plotted using a constant eye
Prototype I: We first demonstrate a prototype with over- pupil size (d eye  8 mm).
lapping retinal elemental images as shown in Fig. 4. The proto-
type uses an active matrix organic light emitting display 45–50 mm from the viewer’s eye, and the distance varies
(AMOLED) found in a smartphone, with 386 ppi and a roughly with the varying facial structure from user to user.
5.7 in. diagonal screen size. The screen is positioned at The same screen is used for both eyes, with the overall system
 Research Article Vol. 54, No. 11 / April 10 2015 / Applied Optics 3425

Fig. 5. Sketch showing spot size on the retina with the changing
pinhole diameter. The system is simulated using a custom in-house
built ray optics and Fourier optics simulator (d ai  9 mm,
d ae  45 mm; see Fig. 1).

Fig. 4. Components of our system: (a) display, which is part of a aperture array at a distance of 45 mm from pinhole aperture
mobile device, with 386 ppi and 5.7 in. diagonal screen size. arrays; the observed image was blurry as expected as in
(b) Aperture array with 0.1 mm pinhole diameter and 1.4 mm Fig. 6(c). Later, the resultant image was observed with a pinhole
center-to-center spacing between pinholes. (c) Complete prototype aperture array as in Fig. 6(d). Both of the photographs were
with the thick spacers under the array, which has 9 mm thickness. captured with a Point Grey CCD (FL3-U3-32S2C) equipped
with an objective lens (YV2.8 × 2.8SA-SA2, focal length of
providing a binocular FOV of 70° × 45°. Our design uses 9 mm 2.8–8 mm and F-number of 1.2), chosen to roughly match
rubber spacers between the pinhole aperture arrays and the
front surface of a smartphone. Note that the protective cover
glass on such devices has some additional thickness, which
typically varies between 0.5 and 2 mm. The overall spacing
between the display and the aperture array is 9.5–11 mm.
We have manufactured our aperture arrays by photo-
plotting. The aperture array contains pinholes with 0.1 mm
diameter and 1.4 mm center-to-center spacing. This leads to
overlapping elemental images on the retina of about 76%,
which decreases the overall effective resolution of the screen
from 1920 × 1080 to 460 × 260.
We select the pinhole diameter by finding the angular
resolution limits predicted by geometric as well as diffraction
optics; the optimum pinhole diameter lies at their intersection
and should provide the highest angular resolution possible.
Simulations of our system using both ray tracing and
Fourier optics showed this optimum diameter to be ∼0.1 mm.
Figure 5 shows our initial estimations of retinal spot size with
this specific configuration. Retinal features (cones and rods) at
Fig. 6. (a) Sample desired image, (b) rendered content to be dis-
the optimum pinhole diameter approach the spot size levels, played on a screen, (c) blurry image observed on the screen without
which nearly matches with retinal features. any pinhole aperture array, shot to mimic the percept of an unaided
Images to be viewed through an aperture array display must human observer (camera focal length 2.8–8 mm and F-number 1.2),
first be processed computationally. The sample input content and (d) image observed through our HMD prototype with a close-up
in Fig. 6(a) was rendered as in Fig. 6(b). The rendered content region demonstrating the observability of PenTile OLED structure
displayed on the screen was first observed without a pinhole (386 ppi) (Media 1).
 3426 Vol. 54, No. 11 / April 10 2015 / Applied Optics Research Article

the aperture and FOV of the human eye. Figure 6(d) also pro-
vides a close-up view; note that the PenTile structure of the
OLED display can be observed. Each pixel in the display
has 65 μm pixel size, so resolving the PenTile structure requires
20–30 μm resolution on the screen plane or ∼1.4–2.1 arcmin
of angular resolution. This resolvable spot size on the screen
plane matches well with our initial resolvable spot size estimate
on the retina, which can be found in Fig. 5.
In informal subjective tests, viewers find our prototype to
have satisfactory resolution and FOV. The subjects also indi-
cate that the brightness level of the display is sufficient when
screen brightness is set to maximum. Fig. 8. Two photographs showing (a) smartwatch’s screen with 277
Prototype II: The overall screen resolution can be improved ppi observed from 10 mm distance through direct contact pinholes
by changing the center-to-center spacing between pinholes, and with 0.1 mm diameter and 1.4 mm center-to-center spacing, and
the distances between the cornea, display, and aperture. We (b) what a human observer sees through the pinholes (camera’s focal
length range, 2.8–8 mm; F-number, 1.2). A single pinhole’s image is
also demonstrate alternative designs. The second design has magnified in the lower left. The content shown on the display consists
a thickness of 35 mm, 0.3 mm pinhole diameter, and of sparsely placed white pixels.
20 mm cornea-to-aperture distance. Figure 7 shows the desired
target image, and the two images as observed through our two
different prototypes. Figure 7(c) shows the observed image
through another, thicker prototype with less overlap between correct sized pinholes, it is possible to build a very thin
retinal elemental images (30%). The overall effective resolution near-eye HMD.
is enhanced in this second sample design, since the content Prototype III: We improved the prototypes discussed
displayed on the screen contains fewer overlapping pixels previously with a new prototype using the same display.
representing the same image region. The general aim in this extended work is to provide a better
The toning artifacts caused by overlapping retinal elemental user experience overall, and to enhance the capabilities of
images become more visible as overlaps between regions are the previous prototypes.
decreased in Fig. 7(c). The dark edges are a result of vignetting; The new prototype has a different pinhole pattern as in
the brightness variation through the image is a result of over- Fig. 9(a), which contains pinholes with a diameter size of
lapping regions on the retina plane. It should be possible to
adjust the brightness level according to photometric values
to have a smoother observed image. Note that this would re-
quire precise eye pupil size and gaze information to render the
content correctly as proposed in [3].
Our proposed approach can support overall thickness
smaller than 9 mm. Existing mobile displays generally come
with a protective glass, which has a typical thickness around
0.55–2 mm. Such distances can be enough to get sharp images
at the final plane. Thus, a pinhole aperture array can also be
placed in direct contact with the front surface of a display
in many cases. Figure 8 shows a smartwatch screen with
277 ppi observed through a single pinhole from 10 mm dis-
tance. With such a thickness setting, high ppi display, and

Fig. 9. (a) Sketch showing the pinhole pattern of the new

prototype. (b) Sample content on display plane from previous dem-
Fig. 7. Picture and two photographs showing (a) desired target im- onstrators. (c) Sample content on display plane from the new proto-
age, (b) image observed through an HMD prototype with overlap ratio type. (d) Images showing housing design for the new prototype from
of 76% between retinal elemental images, and (c) image observed different perspectives. (e) Photograph showing an image captured from
through another HMD prototype with a lower overlap of 30%. the new prototype (Media 2).
 Research Article Vol. 54, No. 11 / April 10 2015 / Applied Optics 3427

0.15 mm. Thus, it provides ×2.25 brightness than the previous pinhole pattern can be improved in our future prototypes;
state. The center-to-center spacing of pinholes is 1.50 mm at further analysis on sampling as discussed in [9] can help us
the vertical axis, and the horizontal row-to-row spacing be- to design different pinhole patterns to improve sampling on
tween pinholes is 1.06 mm in this configuration. The rest the screen plane, and to overcome toning-related issues.
of the distances were kept the same with previous prototypes. Elliptical distortions (astigmatism) of a viewer’s eye can be cor-
The main intention in this type of design is to provide a rected by modifying the content accordingly as in [4]. Another
denser pinhole array pattern, which decreases the visible effect aspect of this type of display is dependency on the smartphone’s
of the pinhole pattern, and uses more pixels of the display. Such computational capability; our aim is to stream content to the
a design requires a different arrangement of the content: in the smartphone to have more processing power for smarter content
previous state, the visible part of the content on the display generation, and to address some of the highlighted issues. We
through pinholes was as in Fig. 9(b); however, the new design intend to tackle these issues in the near future with new designs
requires sample content as shown in Fig. 9(c). based on this work. We believe this is a basis of a framework to
Another improved aspect in this new prototype is a new build simple HMDs with too few optical components.
housing for the display as shown in Fig. 9(d). This design is
3D printed in-house. Figure 9(e) shows a sample image from
the new prototype, which is captured using the same camera 7. CONCLUSION
with the same settings as in Fig. 7. In this paper, we proposed a pinhole aperture array-based ap-
We have conducted an informal subjective experiment on proach to build a wide FOV and high optical angular resolution
the overall performance of the display with 31 participants; HMD using mobile displays. We also demonstrated multiple
the participants were both shown static and moving scenes. proof-of-concept prototypes using a photo-plotted mask on a
Below you can find a summary of the negative feedback from transparency film together with an existing mobile phone’s dis-
the participants of this experiment. play, or a smartwatch’s screen. We believe this is the simplest
• Overall resolution on retina found to be low as in the case HMD setting proposed so far in the domain of computational
of all other lightfield displays. near-eye display.
• People with different eye prescriptions detected image dis- The authors thank Andrew Maimone for fruitful discussions
tortions, due to the shape of their eyes’ point spread function and useful insights.
(PSF); however, the majority of people were able to perceive
clearly.
On the other hand, the positive feedback from the participants REFERENCES
of the experiment is as follows: 1. D. Lanman and D. Luebke, “Near-eye light field displays,” ACM Trans.
Graph. 32, 220 (2013).
• Large FOV was well received. 2. A. Maimone and H. Fuchs, “Computational augmented reality eye-
• People liked the idea of having a cost-effective, simple glasses,” in IEEE International Symposium on Mixed and
solution without requiring much optics. Augmented Reality (ISMAR) (IEEE, 2013), pp. 29–38.
• Nobody complained about the form factor or ergonomics 3. A. Maimone, D. Lanman, K. Rathinavel, K. Keller, D. Luebke, and H.
Fuchs, “Pinlight displays: wide field of view augmented reality eye-
during the experiments.
glasses using defocused point light sources,” in ACM SIGGRAPH
• Nobody complained about brightness during the experi- 2014 Emerging Technologies (ACM, 2014), p. 20.
ments. We believe housing helped in this case. 4. F.-C. Huang, G. Wetzstein, B. A. Barsky, and R. Raskar, “Eyeglasses-
free display: towards correcting visual aberrations with computational
6. FUTURE WORK light field displays,” ACM Trans. Graph. 33, 59 (2014).
5. K. Akşit, A. H. G. Niaki, E. Ulusoy, and H. Urey, “Super stereoscopy
A major trade-off in our proposal is the low light efficiency of technique for comfortable and realistic 3D displays,” Opt. Lett. 39,
the overall system. For our HMD prototypes, it is possible to 6903–6906 (2014).
have larger pinhole diameters and give up some angular reso- 6. R. Sprague, A. Zhang, L. Hendricks, T. O’Brien, J. Ford, E. Tremblay,
and T. Rutherford, “Novel HMD concepts from the DARPA SCENICC
lution, but overall system light efficiency would remain program,” Proc. SPIE 8383, 838302 (2012).
low. Our prototype also does not directly address the accom- 7. W. Song, Y. Wang, D. Cheng, and Y. Liu, “Design of light field head-
modation-vergence conflict. However, by overlapping retinal mounted display,” in International Optical Design Conference (Optical
elemental images with higher percentages, it is possible to ad- Society of America, 2014), paper ITh4A-3.
dress both the accommodation-vergence conflict and the ton- 8. S. De Groot and J. Gebhard, “Pupil size as determined by adapting
luminance,” J. Opt. Soc. Am. 42, 492–495 (1952).
ing effect on the image [1,3]. The amount of overlap could be 9. S. Dammertz and A. Keller, “Image synthesis by rank-1 lattices,” in
increased further using wavelength selective pinholes as in [5]; Monte Carlo and Quasi-Monte Carlo Methods (Springer, 2008),
thus a better representation of a lightfield can be achieved. The pp. 217–236.