Frequency Domain Normal Map Filtering

Charles Han Bo Sun Ravi Ramamoorthi Eitan Grinspun

Columbia University∗

Abstract zoomed in

Filtering is critical for representing detail, such as color textures or

normal maps, across a variety of scales. While MIP-mapping tex-
ture maps is commonplace, accurate normal map filtering remains (a) V-groove (b)
a challenging problem because of nonlinearities in shading—we
cannot simply average nearby surface normals. In this paper, we
show analytically that normal map filtering can be formalized as
a spherical convolution of the normal distribution function (NDF) (d) NDF NDF BRDF effective
BRDF
and the BRDF, for a large class of common BRDFs such as Lamber- (c) V-groove (f) convolution
tian, microfacet and factored measurements. This theoretical result (e) standard zoomed out
explains many previous filtering techniques as special cases, and
leads to a generalization to a broader class of measured and ana- Figure 1: Consider a simple V-groove. Initially in closeup (a), each face is
a single pixel. As we zoom out, and average into a single pixel (c), standard
lytic BRDFs. Our practical algorithms leverage a significant body
MIP-mapping averages the normal to an effectively flat surface (e). How-
of work that has studied lighting-BRDF convolution. We show how
ever, our method uses the full normal distribution function or NDF (d), that
spherical harmonics can be used to filter the NDF for Lambertian preserves the original normals. This NDF can be linearly convolved with
and low-frequency specular BRDFs, while spherical von Mises- the BRDF (f) to obtain an effective BRDF, accurate for shading.
Fisher distributions can be used for high-frequency materials.

1 Introduction
Representing surface detail at a variety of scales requires good fil-
tering algorithms. For texture mapping, there has been consider-
able effort at developing and analyzing filtering methods [Heck-
bert 1989]. A common, linear approach to reduce aliasing is MIP-
mapping [Williams 1983]. Normal mapping (also known as bump
mapping [Blinn 1978] or normal perturbation), a simple and widely
used analogue to color texture mapping, specifies the surface nor-
mal at each texel. Unfortunately, normal map filtering is very diffi-
cult because shading is not linear in the normal.
For example, consider the simple V-groove surface geometry in
Fig. 1a. In a closeup, this spans two pixels, each of which has
distinct normals (b). As we zoom out (c), the average normal of the
two sides (e) corresponds simply to a flat surface, where the shading
is likely very different. By contrast, our method preserves the full
normal distribution (d) in the spirit of [Fournier 1992], and shows
how to convolve it with the BRDF (f) to get an accurate result.
A more complex example is Fig. 2, which compares our method Figure 2: Top: Closeup of the base normal map; all other methods
with “ground truth”1 , standard MIP-mapping of normals, and the are identical at this scale and are not shown. Schematic (a) and dif-
recent normal map filtering technique of [Toksvig 2005]. At close fusely shaded (b) views are provided to aid in comparison/visualization.
range (top row), minimal filtering is required and all methods per- Middle: When we zoom out, differences emerge between our (6-lobe) spher-
form identically. However, as we zoom out (middle and especially ical vMF method, the Toksvig approach (rightmost), and normalized MIP-
bottom rows), we quickly obtain radically different results. mapping. (Unnormalized MIP-mapping of normals produces an essentially
It has long been known qualitatively that antialiasing involves black image.) Bottom: Zooming out even further, our method is clearly
convolution of the input signal (here, the distribution of surface more accurate than Toksvig’s model, and compares favorably with ground
truth. (The reader may zoom into the PDF to compare images.)
∗ {charhan,bosun,ravir,eitan}@cs.columbia.edu

URL: http://www.cs.columbia.edu/cg/normalmap normals) and an appropriate low-pass filter. Our most important
1 “Ground truth” images are obtained using jittered supersampling (on
contribution is theoretical, formalizing these ideas and developing
the order of hundreds of samples per pixel) and unfiltered normal maps. a comprehensive framework for normal map filtering.
In Sec. 4, we derive an analytic formula, showing that filtering
can be written as a spherical convolution of the BRDF of the ma-
terial with a function we define as the normal distribution function
(NDF)2 of the texel. This mathematical form holds for a large class
of common BRDFs including Lambertian, Blinn-Phong, microfacet
models like Torrance-Sparrow, and many measured BRDFs. How-
ever, the convolution does not apply exactly to BRDF models that
2 We define the NDF as a weighted mapping of surface normals onto the

unit sphere; more formally, it is the extended Gaussian Image [Horn 1984]
of the geometry within a texel.
 Acronym Definition Symbol convolution—it is easy to understand [Tan et al. 2005] as an im-
NDF normal distribution function γ (n) portant special case in this framework. Our formulation also al-
SH spherical harmonics Ylm lows spherical harmonic methods for low-frequency materials and
SRBF spherical radial basis function γ (n · µ ) Lambertian objects, which do not even require non-linear fitting
EM expectation maximization
of lobes. For high-frequency materials we use vMFs and spheri-
vMF von Mises-Fisher distribution γ (n · µ ; θ ),
θ = {κ , µ }
cal EM, which operate in the natural spherical domain of surface
movMF mixture of vMF lobes γ (n; Θ), normals; by contrast, planar Gaussian fits, as in [Tan et al. 2005],
Θ = {α j , θ j }Jj=1 have been shown to considerably reduce accuracy both in our work
(Fig. 7) and in other contexts [Strehl et al. 2000].
Table 1: Important abbreviations and acronyms used in the paper. Note that [Tan et al. 2005] treat the BRDF itself as a pre-baked
distribution of normals at fine-scale texels. We support and discuss
depend on the “reflected direction” between the light source and this multi-scale tradeoff between BRDF and geometry (Sec. 6.2).
viewer, e.g., the standard Phong model. Our analytic result imme- However, our spherical representation enables us to derive a for-
diately connects geometrical normal map filtering with the older mal convolution result of the NDF with the BRDF, and allows us to
lighting-BRDF convolution result for appearance [Basri and Jacobs separate or factor the two. The same NDF can be used with differ-
2001; Ramamoorthi and Hanrahan 2001]. This result also unifies ent, possibly non-Gaussian BRDFs, easily. We can also change the
many previous normal map filtering approaches, which can now be BRDF at runtime, and support dynamic, low-frequency complex
viewed as special cases. lighting.
Moreover, we can immediately apply a host of mathematical rep- Hierarchy of Representations: A hierarchy of scales, with
resentations originally developed for lighting-BRDF convolution. geometry transitioning to bump or normal maps, transitioning to
In particular, we develop two novel algorithms. Our first method BRDFs, was first proposed by Kajiya [1985]. This idea is explored
(Sec. 5) is a general framework that uses spherical harmonics (SH). in detail by [Becker and Max 1993], but they do not focus on nor-
Although general, this method is most suitable for low-frequency mal map filtering as in our work. Similarly, appearance-preserving
BRDFs, where it suffices to use a small number of SH coefficients. simplification methods replace fine-scale geometry with normal and
Our second method, intended for high-frequency materials, uses texture maps [Cohen et al. 1998]. It is likely that our approach could
an approximate function fitting technique known as spherical ex- enable continuous level of detail and antialiasing in these methods.
pectation maximization (EM) [Banerjee et al. 2005]. This method Separately, our formulation allows one to understand the tradeoff
approximates or fits the NDF with von Mises-Fisher (vMF) distri- between a normal distribution and the BRDF, since the final image
butions (Sec. 7). vMFs are spherical Gaussian-like functions, sym- is given by a convolution of the NDF and BRDF.
metric about a central direction (average normal). To our knowl-
Convolution and Precomputed Radiance Transfer (PRT):
edge, the use of vMFs and spherical EM is new in graphics, and
Many of our mathematical representations and ideas derive from
may have broader relevance. Finally, our framework enables real-
previous spherical convolution techniques [Basri and Jacobs 2001;
istic BRDFs, including measured materials (Fig. 4), as well as dy-
Ramamoorthi and Hanrahan 2001]. We also build on PRT meth-
namically changing reflectance, lighting and view (Fig. 8). We are
ods that have used spherical harmonics [Sloan et al. 2002]. Our
also able to incorporate low-frequency environment maps (Fig. 9).
spherical vMF method extends zonal harmonics [Sloan et al. 2005]
Although this paper presents an in-depth theoretical discussion, and spherical radial basis functions [Tsai and Shih 2006]. We
the final algorithms are relatively simple. Readers more interested also considered wavelet methods (introduced for reflectance in
in implementation may wish to pay special attention to Sec. 5.1, that [Lalonde and Fournier 1997]), but found the number of terms for an
describes the spherical harmonic method, and the full pseudocode artifact-free solution too large for practical use, even with smoother
provided in Algorithms 1 and 2 for the vMF method of Sec. 7. Ad- wavelets.3 A real-time method for wavelet-based BRDF represen-
ditionally, videos and example GLSL shader code for all of our tation and filtering on the GPU is given in [Claustres et al. 2007],
results are available for download from our website. which does not account for normal map filtering.
We emphasize, however, that ours is not a PRT algorithm; it re-
2 Previous Work quires minimal precomputation and works with conventional real-
time rendering techniques. Furthermore, our method rests on an
Normal Map Filtering: Many previous methods approximate explicit analytic convolution formula and uses the representations
the normal distribution function using a single symmetric or asym- above solely for normal map filtering, not for PRT.
metric lobe. [Schilling 1997] described the lobe using covariance
matrices, while [Olano and North 1997] mapped normal distribu-
tions consisting of a single 3D Gaussian. A simple GPU method is
3 Preliminaries
described in [Toksvig 2005]. In our framework, these methods can In this section, we introduce relevant notation for the reflectance
retrospectively be considered similar to using a single vMF lobe. equation, and BRDF and normal map representations. Note that
As seen in Figs. 2 and 7, one lobe is insufficient for complex NDFs. in this paper, we focus only on normal map filtering and do not
An early inspiration is [Fournier 1992], which uses up to seven consider displacement maps, occlusion, masking, geometry sim-
Phong lobes per texel (and up to 56 at the coarsest scales). This can plification or cast shadows—filtering these additional effects is an
be seen as a special case of our framework, with some similarities to important direction of future work (and discussed in Sec. 9).
our spherical vMF algorithm. Note that [Fournier 1992] uses non- The reflected light B at a spatial point x in direction ω o is
linear least-squares optimization to fit lobes. In our experience, this Z
is unstable and slow, especially considering the number of peaks B(x, ω o ) = L(x, ω i )ρ (ω ′i , ω ′o ) d ω i , (1)
and texels in a normal map. vMFs are very similar to Gaussian or S2
Phong lobes, but have many technical advantages, in allowing for where L is the lighting at x from incident direction ω i , and ρ is the
fast and robust fitting using existing algorithms for spherical EM. BRDF (actually, the transfer function including the cosine of the in-
The most recent and closest previous work is [Tan et al. 2005], cident angle). Most generally we must integrate over the sphere S2
which uses EM to fit Gaussian lobes to a planar projection of
the hemispherical NDF at each texel. We introduce a new theo- 3 PRT methods can use a coarse wavelet approximation of the lighting,
retical result in terms of the analysis of normal map filtering as since it is not visualized directly, but we directly visualize NDF and BRDF.
of incident directions, but in this paper L will usually be assumed to Normal Map Filtering: In screen space, the exitant radiance
be from a small number of point lights, making equation 1 a sum- or pixel color B(x, ω o ) at a surface location x should represent the
mation over discrete directions ω i . We will relax this restriction, average radiance at the N corresponding finer-level texels q:
and discuss extensions to environment maps, in Sec. 8.
1
Z
Equation 1 is the standard reflectance equation. However, we B(x, ω o ) = ∑ L(x, ω i )ρ eff (ω i , ω o ; n(q)) d ω i
have made explicit that ω ′i and ω ′o are the local directions in the N q∈x S
2

surface coordinate frame (in which the BRDF is defined). To find !

1
Z
them, we must project or rotate the global incident and outgoing
directions ω i and ω o to the local tangent frame. This local frame is
=
S2
L(x, ω i )
N ∑ ρ eff (ω i , ω o ; n(q)) d ω i .
q∈x
defined by the surface normal n and a tangent direction. In this pa-
per, we limit ourselves to isotropic BRDFs, so the tangent direction This formulation allows us to define a new effective BRDF,
is not important, and we can use any rigid rotation Rn mapping the 1  
normal to the z-axis [Ramamoorthi and Hanrahan 2001], ρ eff (ω i , ω o ; x) = ∑ ρ Rn(q) (ω i ), Rn(q) (ω o ) . (4)
N q∈x
ω ′i = Rn (ω i ) ω ′o = Rn (ω o ).
Note that the effective BRDF now depends implicitly on all the
3.1 BRDF Representation and Parameterization normals n(q) at x, rather than on a single normal.
This paper is about efficiently computing and representing ρ eff .
Effective BRDF: We define a new function, the effective BRDF
The next section shows how to explicitly represent ρ eff as a convo-
or transfer function that depends on the surface normal (that we
lution of the original BRDF and a new function we call the NDF.
denote as n(x) or simply n for clarity) as,
ρ eff (ω i , ω o ; n) = ρ (Rn (ω i ), Rn (ω o )) , 4 Theory of Normal Mapping as Convolution
allowing us to write equation 1 using the global directions,
Z In this section, we introduce our theoretical framework for normal
eff map filtering as convolution. The next sections describe mathemat-
B(x, ω o ) = L(x, ω i )ρ (ω i , ω o ; n(x)) d ω i . (2)
S2 ical representations that can be used for practical implementation.
BRDF Parameterizations: Many BRDFs can be written as
4.1 Normal Distribution Function (NDF)
ρ eff (ω i , ω o ; n) = f (n · ω (ω i , ω o )), (3) Our first step is to convert equation 4 into continuous form, defining
where the 1D function f is radially symmetric about the shading Z
normal n, and depends on the chosen parameterization ω (ω i , ω o ) ρ eff (ω i , ω o ; γ (·)) = ρ (Rn (ω i ), Rn (ω o )) γ (n) dn, (5)
S2
(henceforth ω ). In this paper, we focus most of our effort on these
types of BRDFs, which encompass Lambertian, Blinn-Phong or mi- where γ (n) is a new function that we introduce and define as the
crofacet half angle (like Torrance-Sparrow), and many factored and normal distribution function (NDF), and the integral is over the
measured BRDFs. sphere S2 of surface orientations. Note that a unique NDF γ (n)
A very common example is Lambertian reflectance, where the exists at each surface location x; for a discrete normal map, γ (n)
transfer function is simply the cosine of the incident angle, so that would simply be a sum of (spherical) delta distributions at n(q),
ω = ω i , and f (u) = max(u, 0). The Blinn-Phong specular model the fine-scale normals at x. Formally, γ (n) = N1 ∑q∈x δ (n − n(q)),
with exponent s uses a transfer function of the form f (u) = us , with as seen in Fig. 1d. For some procedurally generated normal maps,
the half-angle parameterization, ω = ω h = kω i +ω o
ω +ω k . Measured
γ (n) may be available analytically.
i o
BRDF functions f (ω h · n) can also be used.4
4.2 Frequency Domain Analysis in 2D
3.2 Normal Map Representation and Filtering Although we will not directly use the results of this section for ren-
Normal Map Input Representation: There are many equiv- dering, we can gain many insights by starting in the simpler 2D
alent normal map representations, including bump maps [Blinn case. This “flatland” analysis is easier because the rotation operator
1978] and normal offsets. For simplicity, we use normal maps, pa- in equation 5 is given simply by Rn (ω ) = ω + n, yielding
rameterized on a plane, that directly specify the normal in tangent Z 2π
space. In the actual implementation, we perform all computations ρ eff (ωi , ωo ; γ (·)) = ρ (ωi + n, ωo + n) γ (n) dn. (6)
in the local tangent frame of the geometric surface; lighting and 0
view are projected into this local frame, allowing the planar normal
Significant new insight is gained by analyzing equation 6 in the
map to be used directly without explicit rotation. For simplicity in
frequency domain. Specifically, we expand in Fourier series:
the proceeding discussion, the reader can therefore assume a planar
underlying surface while understanding that the extension to curved γ (n) = ∑ γk Fk (n)
3D geometry is straightforward. k
For memory and practicality reasons, normal maps do not typi-
cally use resolutions much higher than 512 × 512 or 1024 × 1024.
ρ (ωi + n, ωo + n) = ∑ ∑ ρlm Fl (ωi + n)Fm (ωo + n), (7)
l m
To obtain effectively higher or finer resolutions, we therefore often
tile the base normal map over the surface. where Fk (n) are the familiar Fourier basis functions √1 eikn . Not-
√ 2π
4 A number of recent papers have proposed factored BRDFs for mea- ing that Fk (ω + n) = 2π Fk (ω )Fk (n), equations 6 and 7 can be
sured reflectance. [Lawrence et al. 2006] uses a factorization f (θh )g(θd ), simplified to
in terms of half and difference angles. The f (θh ) term clearly fits into the
framework of equation 3, but the BRDF now also includes a product with ρ eff (ωi , ωo ; γ (·)) = 2π ∑ γk ρlm Fl (ωi )Fm (ωo ) ×
g(θd ). However, θd does not depend on n (and g does not need to be fil- k,l,m
tered). Thus, our framework also applies to general BRDFs of the form Z 2π
f (ω · n)g(ω i , ω o ), where the g factor does not depend directly on n. Fk (n)Fl (n)Fm (n)dn. (8)
0
 “Ground truth”
The integral above involves a triple integral of Fourier series,
and we denote the corresponding tripling coefficients Cklm . These
tripling coefficients have recently been studied in [Ng et al. 2004],
and for Fourier series they vanish unless k = −(l + m), where
Cklm = √1 . Since ρ eff above is already expressed in terms of
2π
Fl (ωi )Fm (ωo ), we can write a formula for its Fourier coefficients: Our method

eff
√
ρlm = 2πγ−(l+m) ρlm . (9)

Discussion and Analogy with Convolution: Equation 9 gives

a very simple product formula for the frequency coefficients of the
effective BRDF. This is much like a convolution, where the final Standard anisotropic
filtering
Fourier coefficients are a product of the Fourier coefficients of the
functions being convolved (here the NDF and BRDF). However,
the convolution analogy is not exact, since equation 8 involves a
triple integral and n appears thrice in equation 6. In 3D, the for-
mulae and sparsity for triple integrals in the frequency domain (es-
pecially those involving rotations) are much more complicated [Ng Figure 3: Spherical harmonic anisotropic filtering for Lambertian reflec-
et al. 2004]. Fortunately, many BRDFs are primarily single-variable tion. Note the behavior for far regions of the plane. With standard normal
functions f (ω · n) as in equation 3. In these cases, we will obtain a filtering, these regions are averaged to a nearly flat surface. By contrast,
spherical convolution of the NDF and BRDF. our method is quite accurate in distant regions.

4.3 Frequency Domain Analysis in 3D r

eff 4π
To proceed with analyzing equation 5 in the 3D case, we substitute ρlm (q) = ρ̂l γlm (q) ρ̂l = f , (11)
2l + 1 l
the form of the BRDF from equation 3. Recall in this case that the
BRDF only depends on the angle between ω and the surface normal where the NDF considers all normals covered by q. While q usu-
n, and is given by f (ω · n). The effective BRDF is now also only a ally corresponds to a given level and offset in a MIP-map, it can
function of ω , also consider more general “footprints”—we show an example with
Z anisotropic filtering in Fig. 3.
ρ eff (ω ; γ (·)) = f (ω · n)γ (n) dn. (10) Generality and Supported BRDFs: The form above is accu-
S2
rate for all BRDFs described by equation 3, including Lambertian,
Note that the initial BRDF ρ (·) = f (ω · n) is symmetric about n, Blinn-Phong and measured microfacet distributions. Moreover, our
but the final result ρ eff (ω ) is an arbitrary function on the sphere and results also apply when the BRDF has an additional Fresnel or
is generally not symmetric. g(θd ) multiplicative factor, since θd (and hence g) does not depend
We would like to analyze equation 10 in the frequency domain, on n and does not need to be filtered.
just as we did with equation 6. In 3D, we must use the spherical Note that for some specular BRDFs, we also need to multiply by
harmonic (SH) basis functions Ylm (·), which are the frequency do- the cosine of the incident angle for a full transfer function. For
main analog to Fourier series on the unit sphere. The l index is the the spherical vMF method in Sec. 7, we address this by simply
frequency with l ≥ 0, and −l ≤ m ≤ l, multiplying for each lobe by the cosine of the angle between light
and lobe center (or effective normal). For the spherical harmonic
∞ l ∞
method in Sec. 5, we simply use the MIP-mapped normals for the
γ (n) = ∑ ∑ γlmYlm (n) f (ω · n) = ∑ fl Yl0 (ω · n) cosine term, since it is a relatively low-frequency effect.
l=0 m=−l l=0
∞ l
ρ eff (ω ) = eff 5 Spherical Harmonics
∑ ∑ ρlm Ylm (ω ).
l=0 m=−l To recap, we have as input a normal map which provides a sin-
The above is a standard function expansion, as in Fourier series. gle normal n(q0 ) for each finest-level texel q0 . We also have a
Note that the symmetric function f (ω · n) is expanded only in terms BRDF ρ (·) = f (ω · n), with spherical harmonic coefficients ρ̂l .
of the zonal harmonics Yl0 (·) (m = 0), which are radially symmetric In this section, we develop a spherical harmonics-based algorithm
and thus depend only on the elevation angle. from the final formula in equation 11. Later, Sec. 7 will discuss
Equation 10 has been extensively studied in recent years, within an alternative algorithm better suited for higher-frequency effective
the context of lighting-BRDF convolution for Lambertian or radi- BRDFs. While the theory in the previous section is somewhat in-
ally symmetric BRDFs [Basri and Jacobs 2001; Ramamoorthi and volved, the practical algorithm in this section is relatively straight-
Hanrahan 2001]. In those works, the NDF γ (n) is replaced by the forward, involving two basic steps: (1) computing the NDF spher-
incident lighting environment map. Since the theory is mathemati- ical harmonic coefficients γlm (q) for each (coarse-level) texel q of
cally identical, we may directly use their results. Specifically, equa- the normal map, and (2) rendering the final color by directly imple-
tion 10 expresses a spherical convolution of the NDF γ (n) with the menting equation 11 in a GPU pixel shader.
BRDF filter f . In particular, there is a simple product formula in 5.1 Algorithm
spherical harmonic coefficients, similar to the way standard convo-
lution can be expressed as a product of Fourier coefficients, Computing NDF Coefficients: We compute a MIP-map of NDF
r coefficients5 , starting with the finest level normal map, and moving
eff 4π
ρlm = fγ . 5 As explained in Sec. 3.2, we are operating in the local tangent frame
2l + 1 l lm
of the geometric surface, with lighting and view projected into this frame.
Explicitly making the NDF and effective BRDF functions of a texel Thus, we do not need to explicitly consider rotations into the global frame.
q, we have Note that the overall geometric surface is assumed to be locally planar (a
to coarser levels. At the finest level (denoted by subscript 0), γ (q0 )
is a delta distribution at n(q0 ), i.e., γ (q0 ) = δ (n − n(q0 )) with cor-
responding spherical harmonic coefficients6
γlm (q0 ) = Ylm (n(q0 )).
An important insight is that, unlike the original normals, these
spherical harmonic NDF coefficients γlm (q0 ) can now correctly be
linearly filtered or averaged for coarser levels γlm (q). Hence, we
can simply MIP-map the spherical harmonic coefficients γlm (q0 ) in
the standard way, and no non-linear fitting is required.
Rendering: Rendering requires knowing the NDF coefficients
γlm (q), the BRDF coefficients ρ̂l , and then applying equation 11.
We have already computed a MIP-map of NDF coefficients. At
the time of rendering, we also know the BRDF. For many analytic
models, formulae for ρ̂l are known [Ramamoorthi and Hanrahan “Leather” “Violet Rubber”
2
2001]. For example, for Blinn-Phong, ρ̂l ∼ e−l /2s where s is the Figure 4: Our spherical harmonic algorithm for normal mapping, with two
Phong exponent. For measured reflectance, ρ̂l is obtained directly of the materials in the Matusik database—we can support general measured
by a spherical harmonic transform of f (ω · n). BRDFs and change reflectance or material in real time. Notice also the
Now, we can compute the spherical harmonic coefficients of the correct filtering of the zoomed out view, shown at the bottom right.
effective BRDF, per equation 11. Finally, to evaluate it, we must
expand in terms of spherical harmonics, Matusik database. Note the accuracy of our method (compare (b)
with the supersampled “ground truth” in (c)). Also note the smooth
l∗ l
transition between close (unfiltered) and distant (fully filtered) re-
ρ eff (ω , q) = ∑ ∑ ρ̂l γlm (q)Ylm (ω ), (12)
gions in (a) and (b), as well as the filtered zoomed out view in (d).
l=0 m=−l
Discussion and Limitations: Our spherical harmonic method
where ω (ω i , ω o ) depends on the BRDF as usual (such as incident is a practical approach for low-frequency materials. Unlike pre-
direction ω = ω i for Lambertian or halfway-vector ω = ω h for vious techniques, all operations are linear—no nonlinear fitting is
specular), and l ∗ is the maximum √ l used in the shader (accurate required, and we can handle arbitrary lobe shapes and functions
results generally require l ∗ ∼ 4s where s is the Blinn-Phong ex- f (ω h · n). Moreover, the BRDF is decoupled from the NDF, en-
ponent). For shading, assume a single point light source for now. abling simultaneous changes of BRDF, lighting and viewpoint.
At each surface location, we know the incident and outgoing direc- As with all low-frequency approaches, our spherical harmonic
tions, so it is easy to find the half-vector ω h or other parameteriza- method requires many terms for high-frequency specularities (a
tion ω , and then use the BRDF formula above for rendering.7 Blinn-Phong exponent of s = 50 needs about 200 coefficients). The
We implement equation 12 in a pixel shader using GLSL (see our following sections provide more practical solutions in these cases.
website for example code). The spherical harmonics Ylm are stored
in floating point textures, as are the MIP-mapped NDF coefficients
γlm (q). Real-time frame rates are achieved comfortably for up to 64
6 Spherically Symmetric Distributions
spherical harmonic terms (l ∗ ≤ 7, corresponding to a Blinn-Phong Spherical harmonics are a suitable basis for representing low-
exponent s ≤ 12 or a Torrance-Sparrow surface roughness σ ≥ 0.2). frequency functions, but are impractical for higher-frequency func-
tions due to the large number of coefficients required. For higher-
5.2 Results
frequency NDFs, then, we will instead use radially symmetric ba-
Lambertian Reflection: In the Lambertian case, using only nine sis functions, which are one-dimensional and therefore much more
spherical harmonic coefficients (l ≤ 2) suffices [Ramamoorthi and compactly represented. By performing an offline optimization, we
Hanrahan 2001]. An example is shown in Fig. 3. This figure also approximate the NDF at each texel as the sum of a small number of
shows the generality of our method in terms of the footprint for such lobes. Our approach is inspired by the symmetric Phong lobes
texel q, by using GPU-based anisotropic filtering, instead of MIP- used in [Fournier 1992], and effectively formalizes that method
mapping. Note that we preserve accuracy in far away regions of the within our convolution framework.
plane, while naïve averaging of the normal produces a nearly flat
surface that is much darker than the actual (as illustrated in Fig. 1e). 6.1 Basic Theoretical Framework for using SRBFs
Low-Frequency Specularities and Measured Reflectance: Consider a single basis function γ (n · µ ) for the NDF, symmetric
Specular materials with BRDF f (ω h · n) also fit within our frame- about some central direction µ . For now, γ is a general spherical
work. The BRDF can also be changed at run-time, since the NDF radial basis function (SRBF). Equation 10 now becomes
is independent of it. We have factored all of the materials in the Z
database of [Matusik et al. 2003], using the f (θh )g(θd ) factoriza- ρ eff (ω · µ ; γ (·)) = f (ω · n)γ (n · µ ) dn.
tion in [Lawrence et al. 2006]. Figure 4 shows two examples of S2
different materials, which we can switch between at runtime. It can be shown (for example, see [Tsai and Shih 2006]) that ρ eff
Figure 5 shows closeup views from an animation sequence of
is itself radially symmetric about µ (hence the form ρ eff (ω · µ )
cloth draping over a sphere, using the blue fabric material from the
above), and its spherical harmonic coefficients are given by
single “geometric normal”) over the region being filtered.
6 We use the real form of the spherical harmonics, rather than the com- ρleff = ρ̂l γl . (13)
plex form, to simplify implementation. Otherwise, γlm (q0 ) = Ylm∗ (n(q )).
0
7 Our spherical harmonic algorithm does not explicitly address color tex- Compared to equation 11, this is a simpler 1D convolution, since
tures; a simple approximation would be to MIP-map them separately, and all functions are radially symmetric and therefore one-dimensional.
then modulate the scalar result in equation 12. A more correct approach to To represent general functions, we can use a small number of repre-
filtering material properties is discussed for our vMF method in Sec. 7.3. sentative lobes γl, j . Note that the calculation of the lobe directions
 (a) Our method, frame 1 (b) Our method, frame 2 ( c ) “ G r o u n d t r u t h ”, f r a m e 2 (d) Our method, zoomed out
Figure 5: Stills from a sequence of cloth draping over a sphere, with closeups indicating correct normal filtering using our spherical harmonic algorithm (the
full movie is shown in the video). Note the smooth transition from the center (almost no filtering) to the corners (fully filtered) in (b)—compare also with ground
truth in (c). (d) is a zoomed out view that also filters correctly. We use a blue fabric material from the Matusik database as the BRDF.

is generally a nonlinear process; our particular implementation is

given in Sec. 7.
For rendering, we need to expand the effective BRDF in spheri-
cal harmonics, analogously to equation 12, but now using only the
m = 0 terms. Considering the summation of J lobes, we obtain
J ∞
ρ eff (ω , q) = ∑ ∑ ρ̂l γl, j (q)Yl0 (ω · µ j ), (14)
j=1 l=0

where we again make clear that the NDF γl, j is a function of the
texel q. This equation can be used directly for shading once we find
ω for the light source and view direction.

6.2 Discussion: Unifying Framework and Multiscale

Our theoretical framework in Sec. 6.1 unifies many normal filtering
algorithms. Previous lobe- or peak-fitting methods can be seen as
special cases. For instance, [Schilling 1997; Toksvig 2005] effec-
tively use a single lobe (J = 1), while [Fournier 1992] uses multiple
Phong lobes for γ (n · µ ). These methods have generally adopted
simple heuristics in terms of the BRDF. By developing a general
convolution framework, we show how to separate the NDF from the
BRDF. Since we properly account for general BRDFs ρ̂l , we can
even change BRDFs on the fly—in contrast, even [Tan et al. 2005]
is limited to predetermined Gaussian Torrance-Sparrow BRDFs.
Figure 6: Illustration of multiscale filtering of the BRDF (rendered sphere)
Equation 13 has an interesting multi-scale interpretation, as de-
and NDF (inset). (a) shows a closeup of the sphere, where we see the in-
picted in Fig. 6. At the finest scale (a), the geometry used is the
dividual facets and a sharp NDF/effective BRDF. In (b), we have zoomed
original highest-resolution normal map. Therefore, the NDF is a out to where the geometry now appears smoother, although roughness is
delta distribution at each texel, and the effective BRDF ρleff = ρ̂l . still clearly visible. The effective BRDF is now blurred, now incorporating
At coarser scales, the shading geometry used is effectively a fil- finer-scale geometry. As we zoom further out in (c) and (d), the geometry
tered version of the fine-scale normal map, with the NDF becoming appears even smoother, while the BRDF is further filtered.
smoother from (b)-(d). The effective BRDF is now filtered by the
smoothed NDF, essentially representing the complex fine-scale ge- or planar (and therefore distorted) RBFs, we consider NDFs repre-
ometry as a blurring of the BRDF. sented on their natural spherical domain, which also enables us to
Also note the symmetry between the BRDF and NDF in equa- derive a simple convolution formula.
tion 13. While the common fine-scale interpretation is for a delta Indeed, spherical Gaussian RBFs, such as in [Tsai and Shih
function NDF and the original BRDF, we can also view it as a delta 2006] or Phong lobes, as in [Fournier 1992], are most appropri-
function BRDF and an NDF given by ρ̂l . These interpretations are ate. However, the nonlinear minimization required for fitting these
consistent with most microfacet BRDF models, which start by as- models is inefficient, given that we need to do so at each texel. In-
suming a mirror-like BRDF (delta function) and complex NDF (mi- stead, we use a spherical variant [Banerjee et al. 2005] of EM, with
crofacet distribution), and derive a net glossy BRDF on a smooth the von Mises-Fisher8 (vMF) distribution [Fisher 1953]. Spheri-
macrosurface (delta function NDF). cal EM and vMFs have previously been used in other areas such
as computer vision [Hara et al. 2005] for approximating Torrance-
6.3 Choice of Radial Basis Function Sparrow BRDFs; here we introduce them for the first time in com-
puter graphics, to represent NDFs.
We now briefly discuss some possible approaches for approximat-
ing and representing our radial basis functions γ (n · µ ). One pos-
sible method is to use zonal harmonics [Sloan et al. 2005]; how- 7 Spherical vMF Algorithm
ever, our high-frequency NDFs lead to large orders l, making fit- We now describe our algorithms for fitting the NDF, and rendering
ting difficult and storage inefficient. An alternative is to use Gaus- with mixtures of vMF lobes. The fitting is done using a technique
sian RBFs, with parameters chosen using expectation maximization
(EM) [Dempster et al. 1977]. In this case, we simply need to store 8 For the unit 3D sphere, this function is also known as the Fisher dis-
3 parameters per SRBF: the amplitude, width and central direction. tribution. We use the more general term von Mises-Fisher distribution, that
Whereas [Tan et al. 2005] pursued this approach using Euclidean applies to n-dimensional hyperspheres.
known as spherical expectation maximization (EM) [Banerjee et al. Algorithm 1 The Spherical EM algorithm. Inputs are normals ni in
2005]. EM is a common algorithm for fitting in statistics, that finds a texel. Outputs are movMF parameters α , κ and µ for each lobe j.
“maximum likelihood” estimates of parameters [Dempster et al. 1: repeat
1977]. It is an iterative method, with each iteration consisting of 2: {The E-step}
two steps known as the E-step and the M-step. We use EM as op- 3: for all samples ni do
posed to other fitting and minimization techniques because of its 4: for j = 1 to J do
simplicity, efficiency, robustness, and ability to work with sparse 5:
γ (n ;θ )
hzi j i ← J j γ i(n j;θ ) {Expected likelihood of ni in lobe j}
data (the discrete normals in the NDF). We also show how to ex- ∑k=1 k i k
tend the basic spherical EM algorithm to handle color and different 6: end for
materials, create coherent lobes for hardware interpolation, and im- 7: end for
plement spherical harmonic convolution for rendering. Note that 8: {The M-step}
while the theoretical development of this section is somewhat com- 9: for j = 1 to J do
plicated, the actual implementation is quite simple, and full pseu- ∑Ni=1 hzi j i
10: αj ← N
docode is provided in Algorithms 1 and 2. ∑Ni=1 hzi j ini
11: rj ← N {Auxiliary variable for κ , µ in equation 16}
∑i=1 hzi j i
7.1 Fitting NDF with Mixtures of vMFs 3kr j k−kr j k3
12: κj ← 1−kr j k2
vMF distributions were introduced in statistics to model Gaussian- 13: µ j ← normalize(r j )
like distributions on the unit sphere (or hypersphere). An advantage 14: end for
of vMFs is that they are well suited to a spherical expectation max- 15: until convergence
imization algorithm to estimate their parameters. They are charac-
terized by two parameters θ = {κ , µ } corresponding to the inverse
width κ and central direction µ . vMFs are normalized to integrate The accuracy of our method is shown in Fig. 7, where we see that
to 1, as required by a probability distribution, and are given by about four lobes suffices in most cases, with excellent agreement
with six lobes. We also compare with the Gaussian EM fits of [Tan
κ
γ (n · µ ; θ ) = eκ (n·µ ) . (15) et al. 2005]. They work on a projection of the hemisphere onto the
4π sinh(κ ) plane, and use standard Euclidean (rather than spherical) EM. Be-
A mixture of vMFs (movMF) is defined as an affine combination cause this planar projection introduces distortions, they have a sig-
of vMF lobes θ j , with amplitude α j , where ∑ j α j = 1, nificant loss of accuracy near the boundaries (top row). Our method
(middle row) works on the natural spherical domain (hence the side
J view shown), and is able to fit undistorted lobes anywhere on the
γ (n; Θ) = ∑ α j γ j (n · µ j ; θ j ). sphere. Also note that [Tan et al. 2005] do not have an explicit
j=1 convolution formula, while our method can be combined with any
BRDF to produce accurate renderings (bottom row).
Here, θ j = {κ j , µ j } characterizes a single vMF lobe, and Θ stores
the parameters {α j , θ j }Jj=1 of all J vMFs in the movMF. 7.2 Spherical Harmonic Coefficients for Rendering
We use spherical EM (Algorithm 1) to fit a movMF to the nor- For rendering, we will need the spherical harmonic coefficients γl of
mals covered at each texel in the MIP-map. Line 5 of Algorithm 1 a normalized vMF lobe. To the best of our knowledge, these coeffi-
shows the E-step. For all normals ni in a given texel, we compute cients are not found in the literature, so we derive them here based
the expected likelihood hzi j i that ni corresponds to lobe j. Lines 9- on reasonable approximations. First, for large κ , we can assume
14 execute the M-step, which computes maximum likelihood esti- that sinh(κ ) ≈ eκ /2. In practice, this approximation is accurate as
mates of the parameters. In practice, we seldom need more than long as κ > 2, which is almost always the case. Hence, the vMF in
10 iterations, so the full EM algorithm for a 512 × 512 normal map equation 15 becomes
converges in under 2 minutes. Note that this is an offline computa- κ −κ (1−n·µ )
tion that needs to be done only once per normal map—unlike most γ (n · µ ; θ ) ≈ e .
previous work, it is also independent of the BRDF (and lighting). 2π
Note the use of auxiliary variable r j in line 11, which represents Let β be the angle between n and µ . Then, 1 − n · µ = 1 − cos β .
hx j i/α j , where hx j i is the expected value of a random vector gener- For moderate κ , β must be small for the exponential to be nonzero.
ated according to the scaled vMF distribution γ (x; θ j ). The central In these cases, 1 − cos β ≈ β 2 /2, and we get a Gaussian form,
normal µ j and the inverse width κ j are related to r j by κ − κ β2
γ (n · µ ; θ ) ≈ e 2 . (17)
r = A(κ )µ , 2π
1 In [Ramamoorthi and Hanrahan 2001], the spherical harmonic
where A(κ ) = coth(κ ) −
. (16) coefficients of a Torrance-Sparrow model of
κ qa similar form are com-
4π
The direction µ is found simply by normalizing r (line 13), while puted. For notational simplicity, let Λl = 2l+1 . Then,
κ is given by A−1 (krk); since no closed-form expression exists for 2 2
A−1 , we use the approximation in [Banerjee et al. 2005] (line 12). e−β /(4σ ) 2
γ= ⇒ Λl γl = e−(σ l) . (18)
Since EM is an iterative method, good initialization is important. 4πσ 2
For normal map filtering, we can proceed from the finest texels to Comparing with equation 17, we obtain σ 2 = 1
and
2κ
coarser levels. At the finest level, we have only a single normal at
each texel, so we need only a single lobe and directly set α = 1, 2 2 l2
Λl γl = e−σ l
= e− 2κ . (19)
µ = n, and κ to a large initial value. At coarser levels, a good
initialization is to choose the furthest-apart J lobes from among the This formula provides us the desired spherical harmonic coeffi-
4J µ ’s in the four finer-level texels; for this we use Hochbaum- cients γl for a vMF lobe, in terms of the inverse width κ .
Shmoys clustering [Hochbaum and Shmoys 1985]. Note that the Having obtained γl , we are now ready to proceed to rendering.
actual fitting uses all normals covered by a given texel in the MIP- Since each vMF lobe is treated independently, and the constants α j
map. and BRDF coefficients can be multiplied separately, we focus on
Figure 7: Fitting of a spherical NDF from one of the texels in the MIP-map with increasing numbers of vMF lobes (middle row, our method). With 3-4 lobes,
we already get excellent agreement in the rendered image. Each vMF lobe is symmetric about some central direction, and is fit on the natural spherical domain
(which is why we show both a top and side view in the middle row). By contrast, a Gaussian EM fit on a planar projection of the hemisphere (top row, Tan et al.
05), must remain symmetric in the distorted planar space, and has considerable errors at the boundaries of the hemisphere. Because no explicit convolution
formula exists in the planar case, we only show renderings with our method (bottom row), which accurately match a reference with a few vMF lobes.

convolving the normalized BRDF ρ̂l with a single normalized vMF For each vMF lobe, we would now like to find a y j that best
lobe γl . It is possible to directly use equation 19 for the vMF coef- describes the yi of all its underlying texels. In the appendix, we
ficients and equation 14 for rendering with general BRDFs. How- augment the EM likelihood function with an additional term whose
ever, a much simpler method is available for the important special maximization yields an extra line in the M-step,
forms of Blinn-Phong and Torrance-Sparrow like BRDFs. First,
consider a normalized Blinn-Phong model of the form, ∑N
i=1 hzi j iyi
yj ← (22)
s+1 ∑N
i=1 hzi j i
ρ = f (ω h · n) = (ω h · n)s ,
2π Note that, since y j does not affect the E-step, the preceding can be
where s is the specular exponent or shininess. It can be shown [Ra- run as a postprocess to the vanilla EM algorithm.
mamoorthi and Hanrahan 2001] that the spherical harmonic coeffi- This extension enables correct filtering of spatially-varying ma-
2
cients are ρ̂l ≈ e−l /2s . Therefore, the result after convolution with terials (as in Fig. 2). Note however that only linear blending of basis
the vMF is still approximately a Blinn-Phong shape: BRDFs (and not for example, freely varying specular exponents) is
allowed. Moreover, the result is a “best-fit” approximation, since
2 2
/2κ 2
/2s′ normals and colors are assumed decorrelated.
Λl ρleff = ρ̂l Λl γl = e−l /2s e−l = e−l ,
′ κs Coherent Lobes for Hardware Interpolation: In our case, ac-
s =
κ +s curate rendering involves shading the 8 neighboring MIP-map tex-
s′ + 1 ′ els (using the BRDF and respective movMFs), and then trilinearly
=⇒ ρ eff (ω h · µ ) = (ω h · µ )s . (20) interpolating them with appropriate weights. Greater efficiency
2π
(usually a 2× to 4× speedup) is obtained if we instead follow the
For a Torrance-Sparrow like BRDF of the form of equation 18, classic hardware approach of first trilinearly interpolating the pa-
we obtain a similar form for ρ eff , only with a new surface roughness rameters Θ of the movMFs. We can then simply run our GPU pixel
σ ′ in the Torrance-Sparrow model, given by shader once on the interpolated parameters Θ̃. For accurate inter-
q polation, this requires us to construct the movMFs in the MIP-map
σ ′ = σ 2 + (2κ )−1 . (21) such that lobe j of each texel be similarly aligned to the jth lobe
stored at each neighboring texel.
Equations 20 and 21 can easily be implemented in a GPU shader For alignment, we introduce a new term in our EM likelihood
for rendering (lines 12-13 in Algorithm 2 implement equation 20; function, and maximize (details are provided in the appendix). The
the full Algorithm 2 is explained at the end of Sec. 7.3). The sim- final result replaces line 13 in the M-step of Algorithm 1 with
plicity of these formulae allows us to change BRDF parameters on !
the fly, and also to consider very high-frequency BRDFs. K
µ j ← normalize r j +C ∑ α jk µ jk . (23)
7.3 Extensions k=1

Different Materials/Colors: It is often the case that one would C is a parameter that controls the strength of alignment (intuitively,
like to associate additional spatially varying properties (such as col- it seeks to move µ j closer to the central directions µ jk of the K
ors, material blending weights, etc.) to a normal map. For example, neighbors, favoring neighbors with larger amplitudes α jk .).
the normal map in Fig. 2 contains regions of different colors. We As in the Gaussian mixture model method of [Tan et al. 2005],
represent these properties in a feature vector yi associated with each we build our aligned movMFs starting at the topmost (that is, most
normal ni , and extend the EM algorithm accordingly. filtered) MIP-map level and proceed downward, following scan-
Algorithm 2 Pseudocode for the vMF GLSL fragment shader
1: {Setup: calculate half angle ω h and incident angle ω i }
2: ρ ← 0
3: for j = 1 to J do {Add up contributions for all J lobes}
4: {Look up vMF parameters stored in 2D texture map}
5: θ ← texture2D(vMFTexture[ j], s,t)
6: α y ← texture2D(colorTexture[ j], s,t)
7: α ← θ .x
8: r ← θ .yzw
α {θ .yzw stores α r}
3
9: κ ← 3krk−krk
1−krk2
10: µ ← normalize(r)
11: {Calculate shading per equation 20}
12: s′ ← κκ+s
s
{s is Blinn-Phong exponent}
s′ +1 ′
13: Bs ← 2π (ω h · µ )s {Equation 20}
14: ρ ← ρ + α y(Ks Bs + Kd )(ω i · µ )
15: end for
16: gl_FragColor ← L × ρ {L is light intensity}

line ordering within each individual level. In the interest of per-

formance, we use only previously visited texels as neighbors.
We next consider trilinear interpolation of the variables. Unfor-
tunately, the customary vMF parameters {κ , µ } control non-linear
aspects of the vMF lobe and therefore cannot be linearly interpo-
lated. To solve this problem, we recall from Sec. 7.1 that µ and
κ can be inferred from the scaled Euclidean mean r = hxi/α of a
given vMF distribution. By linearity of expectation, we can inter-
polate α r = hxi linearly, as well as the amplitude α , giving
α̃ j = T (α j ) r̃ j = T (α j r j )/T (α j ),
where T (·) denotes trilinear hardware interpolation. Finally, κ̃ j and
µ j can easily be found in-shader (lines 9 and 10 of Algorithm 2).
µ̃
Algorithm 2 shows pseudocode for our GLSL fragment shader.
Lines 5-10 look up α and α r, and then compute κ and µ . For im-
plementation, we store the jth lobe of each movMF in a standard
RGBA MIP-map (vMFTexture in Algorithm 2) using one channel
for α and one channel each for the three components of α r. Nor-
malized color/material properties α y are stored in corresponding
textures (colorTexture in line 6 of Algorithm 2). Line 5 reads the
parameters θ for a single vMF lobe as an RGBA value. Lines 12-13
compute the specular shading (assuming a Blinn-Phong model with
exponent s) using equation 20. The Torrance-Sparrow model can
be handled similarly, using equation 21. Line 14 computes the final
shading contribution by including the color parameters y, and scal- Figure 8: Our framework can handle complex scenes, allowing for general
reflectance, which can even be changed at run-time. Here, the BRDF be-
ing by the lobe amplitude α , specular coefficient Ks , and the cosine
comes less shiny over time. Note the correct filtering and overall changes in
of the incident angle, while adding the Lambertian component Kd .
appearance for further regions of the foreground dinosaur, and those in the
Note that this shader can be used equally with aligned or unaligned background. The bottom row shows closeups (when the material is shiny)
vMF lobes; the only difference is whether we manually compute with a different lighting condition. This example also shows that we can
and combine results from all 8 neighboring texels (unaligned) or combine filtered normal maps with standard color texture mapping.
use hardware interpolate to first obtain lobe parameters (aligned).
7.4 Results and normal maps for the dinosaur skins. It renders at 75 frames
Figure 2 shows the accuracy of our method, and makes comparisons per second at a resolution of 640x480 on an nVIDIA 8800 graphics
to ground truth and alternative techniques. It also shows our ability card. In this example, we used six vMF lobes, with both diffuse and
to use different materials for different parts of the normal map. specular shading implemented as a simple fragment shader. Please
Our formulation allows for general and even dynamically chang- see our website for videos of all of our examples.
ing BRDFs. Figure 8 shows a complex scene, where the reflectance
changes over time, decreasing in shininess (intended to simulate 8 Complex Lighting
drying using the model in [Sun et al. 2007]). Although not shown,
the lighting and view can also vary—the bottom row shows close- Our vMF-based normal map filtering technique can also be ex-
ups with different illumination. Note the correct filtering for di- tended to complex environment map lighting.9 Equation 2,
nosaurs in the background, and for further regions along the neck rephrased below, is a convolution (mathematically similar to equa-
and body of the foreground dinosaur. Even where individual bumps
are not visible, the overall change in appearance as the reflectance 9 The direct spherical harmonic method in Sec. 5 is more difficult to ap-
changes is clear. This complex scene has 14,898 triangles for the ply, since general spherical harmonics cannot be rotated as easily as radially
dinosaurs, 139,392 triangles for the terrain and 6 different textures symmetric functions between local and global frames.
 Figure 9 shows an image of an armadillo, with approximately
350,000 polygons and a normal map, rendered at real-time rates in
dynamic environment lighting. We are able to render interactively
with up to 6 vMF lobes and l ∗ = 8 in equation 26.

9 Conclusions and Future Work

We have developed a comprehensive theoretical framework for nor-
mal map filtering with many common types of reflectance models.
Our method is based on a new analytic formulation of normal map
filtering as a convolution of the NDF and BRDF. This leads to novel
practical algorithms using spherical harmonics and spherical vMFs.
The algorithms are simple enough to be implemented as GPU pixel
shaders, enabling real-time rendering on graphics hardware.
We believe this paper also makes broader contributions to many
areas of rendering, and beyond. The convolution result unifies a
geometric problem (normal mapping) with understanding of light-
ing and BRDF interaction in appearance. Moreover, we introduce
spherical EM and vMF distributions into computer graphics, where
they will likely find many other applications.
In [Kajiya 1985], a hierarchy of level-of-details was spelled out
including explicit 3D geometry, normal or bump maps, and BRDF
or reflectance. This paper has addressed filtering of normal maps
and to some extent, the transition to a BRDF at far distances. A
critical direction for future work is filtering of geometry or dis-
placement maps, where effects like local occlusions, shadowing,
masking and interreflections are important.
In summary, although normal mapping is an old technique, cor-
rect filtering has been challenging because shading is nonlinear in
the surface normal. In this paper, we have shown how frequency-
Figure 9: Armadillo model with 350,000 polygons rendered interactively domain analysis reveals important new insights, and taken a signif-
with normal maps in dynamic environment lighting. We use 6 vMF lobes, icant step towards addressing this long-standing problem.
and spherical harmonics up to order 8 for the specular component.
Acknowledgements: We express our deep appreciation to the
tion 10), that becomes a simple dot product in spherical harmonics, primary referee for his detailed comments to improve the exposi-
Z tion of this paper during “shepherding”. We are extremely grateful
B(µ ) = L(ω i )ρ eff (ω · µ ) d ω i , (24) to Tony Jebara for first pointing us towards vMF representations and
S2
spherical EM. Normal map filtering and multiscale representations
where the effective BRDF ρ eff is the convolution of the vMF lobe are a long-standing problem, and we have had many discussions
with the BRDF, and µ is the central direction of the vMF lobe (and initial efforts at a solution) with a number of researchers over
(effective “normal”) as usual. For the diffuse or Lambertian com- the years including Aner Ben-Artzi, Peter Belhumeur, Pat Hanra-
ponent of the BRDF ω (ω i , ω o ) = ω i , and the spherical harmonic han, Shree Nayar, Evgueni Parilov, Makiko Yasui and Denis Zorin.
coefficients can simply be multiplied according to the convolution This work was funded in part by NSF grants #0305322, #0446916,
formula, Blm = Λl ρleff Llm , so that #0430258, #0528402, #0614770, #0643268, a Sloan Research Fel-
lowship, a Columbia University Presidential Fellowship, and an
l∗ l ONR Young Investigator award N00014-07-1-0900. We also wish
B= ∑ ∑ Λl ρleff LlmYlm (µ ). (25) to thank NVIDIA for a generous donation of their latest graphics
l=0 m=−l
cards during the deadline crunch.
However, the specular component of the BRDF is expressed in
terms of ω (ω i , ω o ) = ω h , and we need to change the variable of
integration in equation 24 to ω h (which leads to a factor 4(ω i · ω h )), References
Z BANERJEE , A., D HILLON , I., G HOSH , J., AND S RA , S. 2005.
B(µ ) = [L(ω i (ω h , ω o )) · 4(ω i · ω h )] ρ eff (ω h · µ ) d ω h Clustering on the unit hypersphere using von Mises-Fisher dis-
S2
Z tributions. Journal of Machine Learning Research 6, 1345–1382.
= L′ (ω h )ρ eff (ω h · µ ) d ω h . BASRI , R., AND JACOBS , D. 2001. Lambertian reflectance and
S2
linear subspaces. In International Conference on Computer Vi-
Thus, we simply need to consider a new reparameterized lighting sion, 383–390.
L′ (ω h ) = L(ω i (ω h , ω o )) · 4(ω i · ω h ). As the half angle depends on B ECKER , B., AND M AX , N. 1993. Smooth transitions between
both viewing and lighting angles (ω o and ω i ), the above integration bump rendering algorithms. In SIGGRAPH 93, 183–190.
implicitly limits us to a fixed view with respect to the lighting. To
interactively rotate the lighting, we precompute a sparse set (typi- B LINN , J. 1978. Simulation of wrinkled surfaces. In SIGGRAPH
cally, about 16 × 16) of rotated lighting coefficients and interpolate 78, 286–292.
the shading. C LAUSTRES , L., BARTHE , L., AND PAULIN , M. 2007. Wavelet
Finally, in analogy with equation 25, Encoding of BRDFs for Real-Time Rendering. In Graphics In-
terface 07.
l∗ l
′
Λl ρleff Llm C OHEN , J., O LANO , M., AND M ANOCHA , D. 1998. Appearance
B= ∑ ∑ Ylm (µ ). (26)
preserving simplification. In SIGGRAPH 98, 115–122.
l=0 m=−l
D EMPSTER , A., L AIRD , N., AND RUBIN , D. 1977. Maximum- T SAI , Y., AND S HIH , Z. 2006. All-frequency precomputed radi-
likelihood from incomplete data via the EM algorithm. Journal ance transfer using spherical radial basis functions and clustered
of the Royal Statistical Society, Series B 39, 1–38. tensor approximation. ACM Transactions on Graphics (SIG-
F ISHER , R. 1953. Dispersion on a sphere. Proceedings of the GRAPH 2006) 25, 3, 967–976.
Royal Society of London, Series A 217, 295–305. W ILLIAMS , L. 1983. Pyramidal parametrics. In SIGGRAPH 83,
F OURNIER , A. 1992. Normal distribution functions and multiple 1–11.
surfaces. In Graphics Interface Workshop on Local Illumination,
45–52. Appendix: Spherical EM Extensions
H ARA , K., N ISHINO , K., AND I KEUCHI , K. 2005. Multiple light In this appendix, we briefly describe the likelihood function for
sources and reflectance property estimation based on a mixture spherical EM, and how we augment it for colors/materials and co-
of spherical distributions. In ICCV ’05: Proceedings of the Tenth herent lobes. The net likelihood function is a product of 3 terms,
IEEE International Conference on Computer Vision, 1627–1634.
H ECKBERT, P. 1989. Fundamentals of texture mapping and image P(X, Z|Θ)P(Y, Z|Θ)P(Θ|N(Θ)),
warping. Master’s thesis, UC Berkeley UCB/CSD 89/516. where X are the samples (in this case input normals), Y are the col-
H OCHBAUM , D., AND S HMOYS , D. 1985. A best possible heuris- ors/materials, Z are the hidden variables (in this case which vMF
tic for the k-center problem. Mathematics of Operations Re- lobe a sample X is drawn from), Θ are parameters for all vMF lobes
search. and N(Θ) are parameters for neighbors. The first factor corresponds
H ORN , B. K. P. 1984. Extended gaussian images. Proceedings of to standard spherical EM, the second factor corresponds to the col-
the IEEE 72, 1671–1686. ors/materials Y ,
K AJIYA , J. 1985. Anisotropic reflection models. In SIGGRAPH N 2
85, 15–21. P(Y, Z|Θ) = ∏ e−kyzi −yi k ,
i=1
L ALONDE , P., AND F OURNIER , A. 1997. A wavelet representation
of reflectance functions. IEEE TVCG 3, 4, 329–336. and the final factor to coherent lobes for interpolation,
L AWRENCE , J., B ENA RTZI , A., D ECORO , C., M ATUSIK , W., J K ′
P FISTER , H., R AMAMOORTHI , R., AND RUSINKIEWICZ , S. P(N(Θ)|Θ) = ∏ ∏ eC α jk ( µ j · µ jk )
.
2006. Inverse shade trees for non-parametric material represen- j=1 k=1
tation and editing. ACM Transactions on Graphics (SIGGRAPH
We use C′ above as a constant weighting factor (it will be related to
2006) 25, 3, 735–745.
the weight C used in the main text as discussed below).
M ATUSIK , W., P FISTER , H., B RAND , M., AND M C M ILLAN , L. In EM, we seek to maximize the log likelihood
2003. A data-driven reflectance model. ACM Transactions on
Graphics (SIGGRAPH 03 proceedings) 22, 3, 759–769. ln [P(X, Z|Θ)P(Y, Z|Θ)P(Θ|N(Θ))] =
N G , R., R AMAMOORTHI , R., AND H ANRAHAN , P. 2004. Triple N N J K
product wavelet integrals for all-frequency relighting. ACM ∑ ln γ (ni |θz ) + ∑ −kyi − yz k2 + ∑ ∑ C′ α jk (µ j · µ jk ) ,
i i

Transactions on Graphics (SIGGRAPH 2004) 23, 3, 475–485. i=1 i=1 j=1 k=1

O LANO , M., AND N ORTH , M. 1997. Nor- which, considering all J lobes and hidden variables hzi j i, becomes
mal distribution mapping. Tech. Rep. 97-041 " #
J N N K
http://www.cs.unc.edu/~olano/papers/ndm/ndm.pdf, UNC. 2 ′
R AMAMOORTHI , R., AND H ANRAHAN , P. 2001. A signal-
∑ ∑ ln γ (ni |θ j )hzi j i + ∑ −kyi − yz k i hzi j i + ∑ C α jk (µ j · µ jk ) .
j=1 i=1 i=1 k=1
processing framework for inverse rendering. In SIGGRAPH 01,
117–128. Maximizing with respect to y j , we directly obtain equation 22. The
maximization with respect to µ j is more complex,
S CHILLING , A. 1997. Toward real-time photorealistic rendering:
Challenges and solutions. In SIGGRAPH/Eurographics Work- N
!
K
shop on Graphics Hardware, 7–16. µ j = normalize κ j ∑ ni hzi j i +C′ ∑ α jk µ jk .
S LOAN , P., K AUTZ , J., AND S NYDER , J. 2002. Precomputed radi- i=1 k=1
ance transfer for real-time rendering in dynamic, low-frequency Finally, redefining C = C′ /κ j , we obtain equation 23.
lighting environments. ACM Transactions on Graphics (SIG-
GRAPH 02 proceedings) 21, 3, 527–536.
S LOAN , P., L UNA , B., AND S NYDER , J. 2005. Local, deformable
precomputed radiance transfer. ACM Transactions on Graphics
(SIGGRAPH 05 proceedings) 24, 3, 1216–1224.
S TREHL , A., G HOSH , J., AND M OONEY, R. 2000. Impact of
similarity measures on web-page clustering. In Proc Natl Conf
on Artificial Intelligence : Workshop of AI for Web Search (AAAI
2000), 58–64.
S UN , B., S UNKAVALLI , K., R AMAMOORTHI , R., B ELHUMEUR ,
P., AND NAYAR , S. 2007. Time-Varying BRDFs. IEEE Trans-
actions on Visualization and Computer Graphics 13, 3, 595–609.
TAN , P., L IN , S., Q UAN , L., G UO , B., AND S HUM , H. 2005.
Multiresolution reflectance filtering. In EuroGraphics Sympo-
sium on Rendering 2005, 111–116.
T OKSVIG , M. 2005. Mipmapping normal maps. Journal of Graph-
ics Tools 10, 3, 65–71.