AUTOMATIC CONSTRUCTION OF 3D HUMAN FACE MODELS

BASED ON 2D IMAGES

Li-an Tang Thomas S.Huang

Beckman Institute and Coordinated Science Laboratory

University of Illinois at Urbana-Champaign
405 N. Mathews Avenue, Urbana, IL 61801, U.S.A
E-mail: 1tangQifp.uiuc.edu

.................................................................................................................................
ABSTRACT
We propose an approach to the automatic construction of Generic
3D human face models using a generic face model and sev- face model
eral 2D face images. A template matching based algorithm
is developed to automatically extract all necessary facial Feature
features from the front and side profile face images. Then
the generic face model is fitted to these feature points by ge-
ometric transforms. Finally, texture mapping is performed
to achieve realistic results.
.................................................................................................................... 1111.1.1.

1. MOTIVATION
Figure 1: General face modeling process
Recently, in order to achieve low bit-rate video coding,
model-based coding systems have attracted great interests
in visual telecommunications, e.g., videophone and telecon- 2. AUTOMATIC FACIAL FEATURE
ferencing where the human faces are the major part in the EXTRACTION
scenes. The main idea of this approach is to construct a 3D
model for human face. Only the moving parts on the face Facial features are the salient appearances on the human
are analyzed and the motion parameters are transmitted, face, such as the eyes, mouth, etc.. For the purpose of face
finally the original facial expressions could be synthesized modeling, we also need to exact locations around the key
by deforming the face model using the facial motion pa- points of the facial features. Fig. 2 shows all feature points
rameters. Thus, a very high compression factor could be which are extracted by this algorithm. These feature points
reached in this way. are a subset of the facial landmarks defined on the human
One of the basic problems related to model-based cod- face for use in anthropometry of the human head [5].
ing systems is how to create a 3D face model for a partic-
ular person using several 2D face images, in particular, the
front and side profile view images. Usually this can be done
by fitting a generic face model to face images according to
several features on the face. Several automatic modeling
algorithms have been developed [1],[2], provided that some
f
I

rather stringent constraints are satisfied. Other algorithms

try to find facial features using energy-minimizing splines,
such as snakes [3] and deformable templates [4]. They usu-
ally assume known initial positions of the facial features.
We strongly believe that the facial feature can be well-
defined only when it is viewed as a whole. An image pattern
is considered a mouth only when it satisfies the global ar-
rangement of the mouth. It is this idea that motivates us
to develop an automatic facial feature extraction and model
mapping algorithm to build 3D face models. The feature Figure 2: Feature points on the human face
extraction algorithm automatically locates all facial feature
points and a set of geometric transformations are applied The algorithm is an integration of “coarse-to-fine search-
to map the generic model to these feature points to obtain ing” (face detection) and “global-to-local matching”(feature
a specific face model. Fig. 1 outlines our automatic face extraction). A set of multiresolution templates is built for
modeling procedure. the whole face and individual facial features. A resolution

0-7803-3258-X/96/$5.000 1996 IEEE 467

pyramid structure is also established for the input face im-
age. This algorithm first tries to find the rough face location
in the image at the lowest resolution by globally matching
it with the face templates. The higher resolution images
and templates are used to refine the face location. Then
each facial feature is located using a combination of tech-
niques, including image processing, template matching and
deformable templates. Finally, a feedback procedure is pro-
vided to verify extracted features using the anthropometry
of human faces and if necessary, the features will be re-
matched. Fig. 3 gives a systematic view of this algorithm.

Figure 4: Face templates

will serve as the initial estimation of facial features once the

face is detected from an image.
4
I .
Feature
Verification
I

1 - 1 1I
1
A set of multiresolution face templates is created by
pixel consolidation [6]. This set of face templates are used
U
to extract feature points for all incoming face images in our
experiment.
Figure 3: Automatic facial feature extraction system In the mean time, a resolution pyramid is also created
for the face images.
The same procedure is applied to both front and side
profile views. 2.2. Face detection
We impose the following constraints to make this algo-
rithm work well: Often faces can be more easily detected in images that have
We assume that face images have known orientation, lower resolutions. There are two reasons for this. First,
either front view or side profile view. However, we the computations are much faster due to the dimensionality
make no assumption about the scale and the location reduction. Second, confusing details presented in higher
of the face in the image. resolution images may not appear at reduced resolution.
Although the sizes and proportions among facial features
The facial features should not be occluded by other differ significantly in high resolution images, all faces look
objects, e g., long hairs, thick moustache and beard almost the same at very low resolution. However, accurate
0 The person should not wear eyeglasses. Actually, this extraction of facial features requires details which are only
is also a basic requirement for face modeling since we revealed in high resolution images. Therefore, a "coarse-
are not using a model with eyeglasses for the face. to-fine searching" strategy is introduced to detect the face
area by multiresolution template matching.
The face should have a neutral expression.
The search starts from the highest level (with the low-
0 As for the background, there is no necessary assump- est resolution) of the resolution pyramid of the face image.
tion of uniformity for detecting face area and extract- Every face template with the size smaller than that of the
ing facial features. However, the algorithm does re- face image is matched within the whole face image to get
quire that the background around the face bound- the maximum correlation coefficient, and its location is also
ary(in both front and side views) be in sharp contrast recorded. The templatepossessing the largest correlation
to the face area so that the feature points on the face coefficient is the best match. In this way, the rough posi-
boundary could be determined. tion and the approximate size of the face can be found in
the image. Since at the lowest resolution, both the image
2.1 Mult iresolut ion templates and the template are small, the search can be done quickly
even though a full search along the whole face image is re-
An area which mainly covers the eyebrows, eyes, nose and quired at this stage. Then the search is implemented at
mouth is selected as the face template as shown in Fig. 4 the next level of the resolution pyramid, i.e., the size of the
where the gradient version of the face image is used. image is doubled. We also choose the template that has the
Smaller areas around facial features are chosen as the resolution roughly increased by a factor of 2 from the pre-
feature templates. Templates covering larger areas are used vious one. Together with two neighbor templates, the three
for globally locating the features then smaller templates are templates are again correlated with the higher resolution
used to determine the feature points. The locations of these image. At this step, the correlations are only implemented
subtemplates are also shown in Fig. 4. All feature points in a small search area around the rough face location. The
shown in Fig. 4 can be predefined in the face template and one with the maximum correlation coefficient is chosen to

468
refine the face location and the size. This procedure is re- Fig. 5 shows the results of the feature extraction. The
peated until the full resolution image has been processed. vertices of the white polygons in Fig. 5(b) denote the ex-
tracted feature points. The arrows in Fig. 5(a) shows the
best matched template at each search level.
2.3. Facial feature extraction
Normally, Finding all facial feature points in an image
Similarly, the differences exhibited by facial components on with size of 512 x 480 pixels costs 4 seconds on an SGI
different faces are also diminished when we compare very Crimson workstation.
small areas of these features. Since the local areas are usu-
ally very small, searching these kinds of templates along 3. MAPPING GENERIC FACE MODEL
a relatively large area will result in many false alarms of
possible locations. A possible location of the whole fea- Having extracted all facial feature points, a personalized
ture should be determined before the feature points can 3D face model can be created by deforming the generic face
be extracted. This leads to the “global-to-local matching” model using global-to-local mapping procedure which is a
scheme which is adopted to extract all feature points. Since series of geometric transformations, including affine, bilin-
the local feature areas involve more details, this matching ear and rigid transformations. The control vertices on the
is usually implemented on the original face image. generic face model will be exactly mapped to the corre-
After the face location has been detected, the initial sponding feature points by such transformations. Finally,
position of all facial features can be estimated immediately a realistic face model can be created by texture mapping
according to the face template. The estimated facial fea- using the 2D face images. Fig. 6 outlines the face modeling
tures should not be far away from their actual positions. process.
Therefore, for each facial feature, we first try to find its
global location by matching a template which encloses the
whole feature with the face image in a designated search Generic face Face
area around the initial position. The search area for each model images
feature is predefined based on the proportions among facial
features. After some facial features have been determined,
the search areas for the rest of the features can be more
accurately decided. Smaller area templates are used to find
more accurate locations of the feture points.
Finally, a feedback strategy is introduced to verify the
extracted feature points to ensure that they are all correctly
identified. Two criteria, the symmetry and the anthropom-
Figure 6: Face model mapping process
etry measures are used to test the extracted feature points
and, if necessary, the features will be re-matched.

3.1. Global-to-local model mapping

The generic model is first globally mapped to fit the face
boundaries. Then each facial features are mapped locally
to fit the feature points. Finally, face models mapped to
different views of the face images are combined to get a
complete 3D face model. Fig. 7 shows one example of map-
ping generic face model to two views of a person’s face im-
_ _ ~
~ ~
ages. The resulting face model can be viewed from different
directions as we can see from Fig. 8.
(a) multiresolution face temdates

(b) resolution pyramid of face image

Figure 7: Mapped face models
Figure 5: Extracted facial features

469
 One problem arises from the generic model itself. The
mesh used in our generic face model is not dense enough
to accurately represent the face shape. Even the feature
extraction algorithm can find more feature points on the
face image, they can not be used to make the face model
better because there is no correspondence on the generic
model.
One of future directions of face modeling is to use more
2D views. Every view provides additional information about
the face shape which is not exhibited by other views. Of
course, this will pose a challenge for automatic facial feature
extraction.
Figure 8: Multiple views of a mapped face model

Acknowledgements
3.2. Texture mapping
This work was supported in part by U. S. Army Research
The geometric mapping of the face model results in a 3D Laboratory under Cooperative Agreement No. DAALO1-
wire-frame model which represents the shape of a particular 96-2-0003, and in part by Joint Services Electronics Pro-
person’s face. To view this model as a realistic human face, gram Grant ONR N00014-96-1-0129.
some texture must be used to render the polygon surfaces.
A subdivision algorithm similar to that of Catmull 171 5. REFERENCES
is developed for texture mapping in our system. The main
idea is to iteratively subdivide the triangles on the 3D face [l] T. Akimoto and Y. Suenaga, “3d facial model cre-
model and corresponding triangles on the 2D texture map, ation using generic model and front and side views of
until each triangle reaches some designated size. The tex- face,” IEICE Bansactaons on Informataon 8 System,
ture value at each end point of the triangle in 2D face vol. E75-D, pp. 191-197, March 1992.
image is assigned to the corresponding vertex on the 3D
face model. The inside of each triangle is rendered us- [2] G. Xu, H. Agawa, Y. Nagashima, F. Kishino, and
ing Gouraud shading. Fig. 9 shows an example of texture Y. Kobayashi, “Three-dimensional face modeling for
mapped face model. virtual space teleconferencing systems,” Bansactions of
the IEICE, vol. E73, no. 10, pp. 1753-1761, 1990.
[31 A. W. M. Kass and D. Terzopoulos, “Snakes: Active
contour models,” in Proceedings of Internatzonal Con-
ference on Computer Vzsion, pp. 259-269, 1987.
[4]A. L. Yuille, “Deformable templates for face recogni-
tion,” Journal of Cognitive Neuroscaence, vol. 3, no. 1,
pp. 59-70, 1991.
[5] L. G. Farkas, Anthropometry of the Head and Face an
Medacane. Elsevier Science Publishers, New York, 1981.
[6] D. H. Ballard and C. M. Brown, Computer Vasion. New
Jersey: Prentice-Hall, Inc., 1982.
[7] E. Catmull, “Computer display of curved surfaces,” in
Figure 9: A texture mapped face model Proceedangs of IEEE Conference on Computer Graph-
ics, Pattern Recognatzon and Data Structures, pp. 11-
17, May 1975.
4. CONCLUSIONS

We have described an algorithm which automatically con-

structs a specific 3D face model based on several 2D face
images and a generic face model. A robust facial feature
extraction algorithm is developed to find all feature points
necessary to the model mapping procedure. While the fea-
ture extraction part might be sensitive to the face orienta-
tion, the face detection procedure makes less assumptions
about the image status. The most significant part of the
facial feature extraction algorithm is that we have devel-
oped a technique which can be called “double resolution
pyramids” which makes this algorithm scale-invariant and
the computations much faster than for a single resolution
pyramid.

470