Eigen-Harmonics Faces: Face Recognition under Generic Lighting

Laiyun Qing1,2, Shiguang Shan2, Wen Gao1,2

1
Graduate School, CAS, Beijing, China, 100080
2
ICT-ISVISION Joint R&D Laboratory for Face Recognition, CAS, Beijing, China, 100080
Emails: {lyqing, sgshan, wgao}jdl.ac.cn

coefficients of the spherical harmonic images of a novel

Abstract a face from just one image taken under arbitrary
illumination conditions. The eigen based methods can
The performances of face recognition systems are represent the common and differences of the subclasses
heavily subject to the variations in lighting. We in a larger class, such as human face. The eigen based
propose a novel approach for face recognition under methods have been applied successfully in many class
generic illumination conditions, named as Eigen- based vision system, such as Eigenfaces [4, 9, 19], 3D
harmonics faces in this paper. First, using bootstrap Morphable Model [6] and SSFS (statistical shape from
set consisting of 3D face models with texture, we shading) [1]. The spherical harmonic images are very
render the spherical harmonic images for every face convenient for processing images under generic
and train the PCA harmonics faces model. During lighting [2, 3, 14, 15, 21, 22] and they capture the
registration, given a novel face image under arbitrary intrinsic shape and texture of the facial surface.
illumination, we estimate the lighting of the image and According how to deal with the extrinsic imaging
recover the PCA coefficients of the spherical parameters, the methods in face recognition can be
harmonics images for this face. During testing, we classified into two fundamental strategies: model-based
recognize the face using the PCA coefficients. The approach and statistics-based approach. The model-
experimental results on the images under a wide range based approaches treat the extrinsic parameters as
of illumination conditions in the public CMU-PIE separate variables and model their functional role
database are promising. explicitly. The statistics based approaches analyze
images directly using statistical methods and does not
formally distinguish between intrinsic and extrinsic
1. Introduction
parameters. The former includes SSFS (statistical shape
from shading) [17], Symmetric Shape-from-Shading
Much progress in face recognition has been made in the
[24], Illumination Cone [8], Quotient Image [16] and
past few years [23]. However, face recognition remains
3D Morphable Model [6]. The latter includes
a difficult, unsolved problem in general [11, 12]. The
Eigenfaces [19] and FisherFaces [4, 9]. Our method is a
images of face depend not only on the identity of the
model-based approach.
person, but also on parameters such as head pose and
Many of the earlier model-based methods assume
illumination. Variations in pose and illumination, which
simple light models. Using spherical harmonics and
may produce changes larger than the differences
signal-processing techniques, Basri et al [2] and
between different people's images, are the main
Ramamoorthi [14] have shown that the set of images of
challenge for face recognition. FRVT test 2002 shows
a convex Lambertian object obtained under generic
that even for the best face recognition systems, the
lighting can be approximated accurately by a nine
recognition rate for faces captured outdoors is still very
dimensional linear subspace. Furthermore, a simple
low [12]. One of the characters of the outdoor images is
scheme for face recognition with excellent results was
the changing directional sunlight illumination.
described in [2]. However, to use this recognition
The goal of recognition algorithms is to separate the
scheme, the basis images spanning the illumination
characteristics of a face, which are determined by the
space for each face are required. These images can be
intrinsic shape and texture of the facial surface, from
rendered from a 3D scan of the face or can be estimated
the extrinsic imaging conditions of image generation,
by applying PCA to a number of images of the same
such as lighting and pose. In this paper we propose an
subject under different illuminations [3]. An effective
eigen-harmonics faces method, which recovers the PCA
approximation of this basis by 9 single light source

Proceedings of the Sixth IEEE International Conference on Automatic Face and Gesture Recognition (FGR’04)
0-7695-2122-3/04 $ 20.00 © 2004 IEEE
 images of a face was reported in [10]. The above- Ylm is the spherical harmonic function. The real forms
mentioned methods need a number of images and/or 3D of the first nine harmonic faces in are:
scans of each subject in the database, thus requiring ( x, y, z , ) = (sin θ cos φ , sin θ sin φ , cos θ )
specialized equipment and procedures for the capture
of the gallery set, thus limiting their applicability. π
boo ( p i ) = λi ,
In this paper we propose a method that recovers the 4
nine spherical harmonic images of the illumination π
space from just one image taken under arbitrary (b1−1 , b10 , b11 )( p i ) = λi ( y, z , x ) ,
3
illumination conditions. Our method computes a
15π (2)
statistical model of the harmonic images during a (b2− 2 , b2−1 , b21 ) = λi ( xy, yz, xz ) ,
bootstrap phase, which encapsulates texture and shape 64
information, similar with 3D Morphable Model [6]. 5π
b20 ( p i ) = λi (3 z 2 − 1) ,
The two methods need 3D scans of the subjects in 256
training set. But for test, only a face image is needed. 15π 2
Compared with the 3D Morphable Model [6], Our b22 ( p i ) = λi (x − y 2 ) .
256
method has following differences: First, we treat the
shape and texture together as harmonic images to Note that b 00 is an image obtained under constant,
consider the correlation between shape and texture data ambient light, and so it contains simply the surface
rather than to recover its texture and shape parameters albedo (up to a scaling factor). The other harmonic
separately; Second, Our method can process arbitrary images b lm contain both information of surface normal
illumination environment rather than illumination and the albedo.
environments with a ambient light and a directional We select 130 aligned 3D face scans with texture
light; Third, we speed the system by simplifying the information from USF Human ID 3D database [5] as
model (no specular reflection, no optimization). Of our bootstrap set. We render nine harmonic images per
course, the performance of the system will be a little face model using Equation set (2).
lower. But we see from the experimental results that the Let H j = (b 00 , b1−1 , b 10 , b11 , b 2− 2 , b 2−1 , b 20 , b 21 , b 22 )
performance is still applicable.
denote the vector composed by the nine harmonics
This paper is organized as follows. In the next
section, we will show how to construct the eigen- images of the j th person. We perform Principal
harmonic faces. In section 3, we explain how to recover Component Analysis [7] on the bootstrap set of H j ,
the parameters of the spherical harmonic images from a j = 1,2, , m, m = 130 . We subtract the average
novel face image. In section 4, we describe our
1 m
experiments and the results. The final Section presents h = ¦ j =1 H j from each harmonic
the conclusions and the future work. m
vector, a j = H j − h , and define a data matrix
2. Eigen-harmonic faces A = (a1 , a 2 , , a m ) . Then we compute the eigenvector
h 1 , h 2 , of the covariance matrix
Assumed the surface of human faces is convex
1 1 m
Lambertian surface, the set of images of face under C = AA T = ¦ j =1 a j a Tj , which can be achieved by
varying lighting can be approximated by a 9D linear m m
subspace spanned by harmonic images [2, 14]. The a Singular Value Decomposition [13] of A . The
harmonic images of face, here called as harmonic faces, eigenvalues of C , σ s21 ≥ σ s22 ,  , are the variances of
are images of the face seen under harmonic lights. the data along each eigenvetor. We select the first
Harmonic light is a virtual light in which only one n(n ≤ m) eigenvectors form an orthogonal basis,
harmonic component is included. Let λi denote the n
H = h + ¦i =1α i h i . (3)
albedo of a point pi on the face surface and (θ , φ )
n = 60 is used in our experiments, in which covers
denote its normal, the harmonic faces are constructed as:
90% of the variances. The average vector, the first
blm ( pi ) = λi ρ l Ylm (θ , φ ) , (1)
eigenvector and the last eigenvector are visualized as
2π π
where ρ l ( ρ 0 = π , ρ1 = , ρ1 = ) is the spherical four harmonic images in Figure 1. The first
3 4 eigenvectors have the shape of face, while the values of
harmonic coefficients of Lambertian reflectance [2, 14], the last eigenvectors are random.

Proceedings of the Sixth IEEE International Conference on Automatic Face and Gesture Recognition (FGR’04)
0-7695-2122-3/04 $ 20.00 © 2004 IEEE
 Then the rest of the vertices are aligned with image
warping technique.
The feature points on the 2D image are marked with
an enhanced Active Shape Model [20]. An example is
illustrated in Figure 2.

(a) (b)
Figure 2. The input face image (a) and its aligned image
(b) (the white pixels is invisible). The face image is from
CMU-PIE database.

3.2. Harmonic images recovering

Figure 1. The visualized harmonic images of the
eigenvectors. The first row is the average vector. The Given an aligned face image I , let h and h j denote the
second row is the first eigenvector and the third row is the
last eigenvector. Positive values and red indicates negative matrix form of the average spherical harmonic images
values. Lighter gray indicates positive values and darker gray and the PCA base harmonic images in this subsection
indicates negative values (for positive values, the range of (every column is spherical harmonic images), then by
pixel intensities is from 140 to 240 and for negative values, solving the least squares problem
the range is from 20 to 120). For page limit, only the first min h L − I , (4)
four harmonic images are given.
we get the vector of the nine illumination coefficients
L ˈ which approximates the Lambertian part of the
3. Eigen-harmonics faces based image illumination.
analysis After L is known, we recover the PCA coefficients
α i of the given face. These PCA coefficients α i can
The goal of the eigen-harmonic faces based image
be achieved by solving another least squares problem
analysis is to represent a novel face in an image by n
model the PCA coefficients α i in Equation (3). min h L + ¦i =1 α i (h i L) − I . (5)
The experiments in psychology have shown that Once the PCA coefficients of the harmonic images
human know the lighting direction of a face image are recovering, the harmonic images can be
before he recognizes the face. This indicates that we reconstructed with Equation (3). Figure 3 shows the
can use the average face model rather than the specific spherical harmonic images recovered from face images
face model in estimating lighting. This assumption has under two different illuminations of two persons (not
also been used in many vision systems [16, 21, 24]. As the face in bootstrap set). We can see that the harmonic
long as the lighting is known, we can recover the images recovered for the same person from different
parameters of the specified spherical harmonic images images are almost the same while the differences
of the given face. between different persons are still preserved.

3.1. Face Alignment of the frontal face images 4. Experimental Results

Before the lighting estimation, we need to align the face In the real world, illumination usually consists of an
in the input image to the model. The alignment is ambient light with one or more possible point lighting
achieved by morphing based on feature points. sources. To obtain representative images of such cases,
Given a 2D image, to create the correspondence CMU-PIE database [18] includes face images both with
between the vertices of the average face model and the the ambient lights on and with them off. There are
2D image, we first create correspondence between the totally 43 different lighting conditions, 21 flashes with
feature points on the average face and the 2D image. ambient light on or off. The images of 68 persons are

Proceedings of the Sixth IEEE International Conference on Automatic Face and Gesture Recognition (FGR’04)
0-7695-2122-3/04 $ 20.00 © 2004 IEEE
 4.1. Results of lighting estimation

We compute the irradiance environment map with light

spherical harmonic coefficients L as:
2 l
E (θ , φ ) = ¦ ¦ρ L l Y (θ , φ ) ,
lm lm (6)
l =0 m=−l

where ρ l and Ylm (θ , φ ) are the same as in Equation

(1). Some examples of the estimated irradiance
environment maps are illustrated in Figure 5.

Figure 5. The irradiance environment maps of estimated

Figure 3. The results of reconstructed spherical harmonic lighting. The first row is the input images and the second row
images recovered from images of the different persons under is the irradiance environment maps.
various lighting. The first column is the images we used for
the recovery followed by the set of spherical harmonic
images.
By selecting the irradiance maps of one person under
every light as gallery and all the other irradiance
included in PIE face database. For more details about spheres as probe, we classify the estimated lighting to
the CMU-PIE database, please refer to [18]. evaluate the results of lighting estimation. The
The images are divided into two sets, the images in normalized correlation is exploited as the image
set a are images with ambient lights off and the images similarity between two irradiance maps. Classification
in set b are images with ambient lights on. The images is achieved by finding a nearest neighbor based on the
in each set are divided into 4 subsets according to the image similarity.
greater of the longitudinal and latitudinal angles of the The classification rate is given in Figure 6. The
flash direction from the frontal face axis—Subset results of classification with feature points
1(f06~f09, f11, f12, f20), Subset 2(f05, f10, f13, f14, automatically labeled and manually labeled are little
f19, f21), Subset 3(f04, f15, f18, f22), and Subset 4(f02, different, which indicate the light estimation is stable
f03, f16, f17). Figure 4 shows some of the images we even if the feature points are not labeled exactly. The
used in our experiments. results of classification are not very nice because the
Because the current ASM model can only handle difference between neighboring light are little,
frontal face image, we select only the frontal images for especially after blurred as irradiance. Some other
experiments. metrics for evaluating the lighting results are welcome.

4.2. Results of face recognition

To compare two faces, we use cosine of the angle

between two PCA coefficients vectors of the spherical
c ,c
harmonic images 1 2 as face similarity,
c1 , c 2
dA = . (7)
c1 ⋅ c 2
Face recognition is achieved by finding a nearest
Figure 4. The real face image under various lighting used in
the experiments. neighbor based on the face similarity. The experimental
results are given in Table 1. To test the effect of
lighting variation only, the gallery and the probe are the
same images set (there are little differences in glasses in

Proceedings of the Sixth IEEE International Conference on Automatic Face and Gesture Recognition (FGR’04)
0-7695-2122-3/04 $ 20.00 © 2004 IEEE
 set a and set b). We have not tried the LDA metric of images under generic lighting and they capture the
the PCA coefficients yet, which is declared better than intrinsic shape and texture of the facial surface. The
angle metric in [6]. We can expect its performance will eigen based methods can represent the common and
be better than the angle metric in our experiments. differences of human faces. Tested on CMU-PIE
Since we have already estimated the PCA database of images covering large variations in
coefficients of the spherical harmonic images, we can illumination (the illuminations range point light to
reconstruct the spherical harmonic with Equation (3). ambient light, from extreme direction to frontal
Then faces can be recognized using the 9D linear direction), our algorithm achieved promising results.
subspace in [2]. So far, we have not addressed non-frontal view
images. If the pose of a non-frontal face image is
5. Conclusion known, the average 3D face mesh can be rotate to that
pose. Then we can detect every pixel whether it is
With the discovery that the effect of illumination on visible. The visible pixels are used in recovering the
diffuse object is low dimension with analytic analysis, PCA coefficients of the harmonic images. For the
it will not be more difficult to deal with generic extreme pose, i.e., profile, only half of the face is
illumination than to deal with simple light source visible. As the number of PCA coefficients is much less
model. In this paper, we propose an eigen-harmonic than the visible pixels, the half face is still enough for
faces technique for face recognition under generic reconstructing the harmonic images of the face. Face
illumination based on this discovery. The spherical recognition with various lights and poses are one of our
harmonic images are very convenient for processing next works.
Classi fi cati on Rate

0. 95
0. 85
Auto
ASM
0. 75
0. 65
0. 55 Manual
0. 45
2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
Flash No.

(a)
Classi fi cati on Rate

0. 95
0. 85
Auto
ASM
0. 75
0. 65
0. 55 Manua
0. 45 l
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
Flash No.

(b)
Figure 6. The results of lighting estimation. (a) Classification rate of on image set a;
(b) Classification rate of on image set b;

Table 1. Recognition rate comparisons using various metric on CMU-PIE face database
Performance of Subset No. (%)
Gallery Probe Metric
1 2 3 4
Co-rrelation 96 58 24 9
a(11) a
Angle distance 98 93 87 65
Co-rrelation 100 97 85 45
b(11) b
Angle distance 99 97 93 85

Proceedings of the Sixth IEEE International Conference on Automatic Face and Gesture Recognition (FGR’04)
0-7695-2122-3/04 $ 20.00 © 2004 IEEE
 http://www.frvt.org/DLs/FRVT_2002_Evaluation_Rep
ort.pdf
6. Acknowledgements [13] W.H. Press, S.A. Teukolsky, W.T. Vetterling, and B.P.
Flannery, Numerical Recipes in C, Cambridge Univ.
This research is partly sponsored by Natural Science Press, 1992.
Foundation of China (under contract No. 60332010), [14] R. Ramamoorthi and P. Hanrahan, “An efficient
National Hi-Tech Program of China representation for irradiance environment maps”, Proc.
(No.2001AA114190 and No. 2002AA118010), and SIGGRAPH’2001, pp.497-500, 2001.
ISVISION Technologies Co., Lt. Portions of the [15] R. Ramamoorthi, “Analytic PCA Construction for
research in this paper use the USF Human ID 3-D Theoretical Analysis of Lighting Variability in Images
database and the CMU-PIE database. The authors wish of a Lambertian Object,” IEEE Trans. on PAMI, vol.24,
no. 10, pp. 1-12, 2002.
to thanks everyone involved in collecting these data.
[16] A. Shashua and T. Riklin-Raviv, “The Quotient Image:
Class-Based Re-Rendering and Recognition with
References Varying Illuminations”, IEEE Trans. on PAMI, vol.23,
no.2, pp.129-139, 2001.
[1] J. Atick, P. Griffin, and A. Redlich, “Statistical [17] T. Sim, T. Kanade, “Combining Models and Exemplars
Approach to Shape from Shading: Reconstruction of for Face Recognition: An Illuminating Example,” Proc.
3D Face Surfaces from Single 2D Images”, Journal of CVPR Workshop on Models versus Exemplars in Comp.
Neural Computation, vol. 8, no. 6, pp. 1321-340, 1996. Vision, December, 2001.
[2] R. Basri and D. Jacobs, “Lamberian reflectance and [18] T. Sim, S. Baker, and M. Bsat, “The CMU Pose,
linear subspaces”, Proc.ICCV’2001, vol. II: pp.384- Illumination, and Expression (PIE) Database”, Proc.
390, 2001. FG’2002, 2002
[3] R. Basri and D. Jacobs, “Photometric Stereo with [19] M. Turk and A. Pentland, “Eigenfaces for
General, Unknown Lighting”, Proc. CVPR’2001, vol. Recognition,” Cognitive Neuroscience, vol.3, no.1,
II: pp. 374-381, 2001. pp.71-96, 1991
[20] Left Blank for Blind Review
[4] P.N. Belhumeur and J.P. Hespanha and D.J. Kriegman,
[21] Z. Wen, Z. Liu and T. S. Huang, "Face Relighting with
“Eigenfaces vs. Fisherfaces: Recognition using class
Radiance Environment Maps", Proc. CVPR’2003, vol.
specific linear projection”, IEEE Trans. on PAMI,
II, pp.158-165, June 2003.
vol.19, no.7, pp.711-720, July, 1997.
[22] L. Zhang, D. Samaras, "Face Recognition Under
[5] V. Blanz, V. and T. A. Vetter, “orphable Model for the
Variable Lighting using Harmonic Image Exemplars",
Synthesis of 3D Faces”, Proc. SIGGRAPH'99, pp. 187-
Proc. CVPR’2003, vol. I, pp.19-25, June 2003.
194, 1999.
[23] W. Y. Zhao, R. Chellappa, A. Rosenfeld, and P. J.
[6] V. Blanz, T. Vetter, “Face Recognition Based on Phillips, “Face recognition: A literature survey”, UMD
Fitting a 3D Morphable Model”, IEEE Trans. on PAMI, CfAR Technical Report CAR-TR-948, 2000.
vol.25, no.9, pp.1-12, September, 2003.
[24] W. Zhao and R. Chellappa, “Illumination-
[7] R.O. Duda, P.E. Hart, and D.G. Stork, Pattern
Classification, second edition, John Wiley & Sons, insensitive Face Recognition using Symmetric
2001. Shape-from-Shading”, Proc. CVPR’2000, 2000
[8] A. S. Georghiades, P. N. Belhumeur and D. J.
Kriegman, “From Few to Many: Illumination Cone
Models for Face Recognition under Variable Lighting
and Pose,” IEEE Trans. on PAMI, vol.23, no.6, pp.643-
660, June 2001.
[9] R. Gross, I. Matthews and S. Baker, “Eigen Light-
Fields and Face Recognition Across Pose”, Proc.
FG’2002, 2002.
[10] K. C. Lee, J. Ho and D. Kreigman, “Nine Points of
Light: Acquiring Subspaces for Face Recognition under
Variable Lighting,” Proc. CVPR’2001, vol. I, pp.519-
526, 2001.
[11] P. J. Phillips, H. Moon, D. M. Blackburn, E. Tabassi,
and J. M. Bone, “The FERET Evaluation Methodology
for Face-Recognition Algorithms,” IEEE Trans. on
PAMI, vol.22, no.10, pp.1090-1104, 2000.
[12] P. J. Phillips, P. Grother, R. J. Micheals, D. M.
Blackburn, E. Tabassi, and J. M. Bone, “FRVT 2002:
Evaluation Report,” March 2003,

Proceedings of the Sixth IEEE International Conference on Automatic Face and Gesture Recognition (FGR’04)
0-7695-2122-3/04 $ 20.00 © 2004 IEEE