IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, TBME-00849-2012 1

A New 3D Tool for Planning Plastic Surgery

A. Bottino, Member, IEEE, M. De Simone, A. Laurentini, Member, IEEE, C. Sforza

Abstract— Face plastic surgery (PS) plays a major role in A. Previous work
today medicine. Both for reconstructive and cosmetic surgery, Planning PS requires taking into account a number of
achieving harmony of facial features is an important, if not the
major goal. Several systems have been proposed for presenting physiological and psychological constraints. The frequency of
to patient and surgeon possible outcomes of the surgical secondary rhinoplasties, due to unsatisfactory outcomes from
procedure. In this paper, we present a new 3D system able to previous procedures, and estimated around 8%-15% of the
automatically suggest, for selected facial features as nose, chin, cases [5], shows that planning PS is far from simple. Fully
etc, shapes that aesthetically match the patient’s face. The basic planning PS would require considering several factors, as the
idea is suggesting shape changes aimed to approach similar but exterior shape to obtain, its interaction with soft tissues and
more harmonious faces. To this goal, our system compares the bones, and the evolution of tissues with time. This work deals
3D scan of the patient with a database of scans of harmonious
faces, excluding the feature to be corrected. Then, the with the problem of supporting the aesthetic judgment of
corresponding features of the k most similar harmonious faces, surgeon and patient for choosing more harmonious facial
as well as their average, are suitably pasted onto the patient’s shapes.
face, producing k+1 aesthetically effective surgery simulations. To this purpose, a number of computer tools have been
The system has been fully implemented and tested. To proposed, aimed at presenting possible outcomes of the
demonstrate the system, a 3D database of harmonious faces has surgical procedure. Ozkul and Ozkul [6] presented a system
been collected and a number of PS treatments have been for simulating rhinoplasty in 2D profile images. Image
simulated. The ratings of the outcomes of the simulations, interaction tools emulating real surgical procedures were
provided by panels of human judges, show that the system and
the underlying idea are effective. supplied. Rabi and Aarabi [7] emulated the outcomes of PS in
2D frontal images by suitably pasting on the patient’s face
Index Terms— Computer supported surgery, Plastic surgery, facial features of other subjects. Liu et al. [8] attempted to
face scanning, planning surgery outcomes, anisotropic ICP predict the PS outcome on the basis of a number of results of

P
I. INTRODUCTION former cases. Commercial 2D simulation packages are also
available [9]. These packages essentially allow manipulating
LASTIC surgical procedures enjoy increasing popularity 2D images with morphing operators.
in different countries as well as among different social Since human face is a complex 3D object, several systems
groups. According to recent statistics by The American for manipulating 3D faces have been studied. Gao et al.
Society for Aesthetic Plastic Surgery, in 2008 more than one presented a technique for warping the 3D B-spline surface
million facial plastic surgeries were performed, with a growth obtained from patient’s face scan [10]. Lee et al. [11]
of 162% in ten years [1]. According to an estimate, in the US constructed a patient specific 3D model from a generic model
more money is spent annually on beauty related items or based on CT data and a 2D image of the patient and used
services than on both education and social services [2]. The morphing operators for emulating surgical procedures. A 3D
popularity of plastic surgery (PS) also triggered new studies in PS planning system specific of nose surgery was proposed by
face recognition aimed at taking into account facial differences Lee, Lin and Lin in [12]. The system works on CT data of the
due to these procedures ([3], [4]). Facial PS is defined head of the patient from which the face surface is obtained for
reconstructive when aimed at correcting deformities congenital realistic rendering purposes. Wang et al. [13] proposed a 3D
or due to accidents or diseases, and cosmetic otherwise. simulator of augmentation nose surgery, based on a surface
Actually, there is not a well defined boundary between the two scan and a lateral X-ray image. Deformation tools were
fields. In any case, achieving as far as possible an harmonious provided, as well as the evaluation of the size of the required
or at least regular face shape is a main goal for both kinds of prosthesis. Commercial 3D systems have also been recently
surgery. proposed [14].
B. Our approach and its rationale
The quoted planning systems supply manual interfaces for
Manuscript received May 11, 2012; revised 24 July, 2012 A. Bottino, M. changing the shape of 2D or 3D patient’s images. All choices
De Simone and A. Laurentini are with the Dipartimento di Automatica e in manipulating faces and evaluating results are left to the
Informatica, Politecnico di Torino, Corso Duca degli Abruzzi, 24, 10129 surgeon's and patient’s judgment.
Torino Italy (corresponding author: A. Bottino
phone: Our approach is aimed at automatically suggesting, for the
+390115647175; fax: +390115647099; e-mail: particular feature object of surgery and for the particular
[email protected]). patient’s face, the shapes most suitable to enhance facial
C. Sforza is with the Dipartimento di Scienze Biomediche per la Salute, harmony. To explain the rationale of our approach, we will
Universita degli Studi di Milano, Milano, Italy
mail: review some results of the research on facial attractiveness.
[email protected]). Recently this research has been a major issue in psychology,
IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, TBME-00849-2012 2

psychobiology, anthropology, evolutionary biology, behavioral features (TRF).

and cognitive sciences, and in medical areas such as plastic We also use the average feature since, as previously
surgery and orthodontics. Empirical rating studies have mentioned, averaging faces has been found to improve
demonstrated high beauty rating congruence over ethnicity, attractiveness.
social class, age, and sex (see for instance [15] [16] [17] [18]). Two of the authors presented a first implementation of this
Ratings are in particular very similar for beautiful and ugly idea in [29] focused on 2D face profiles, and choosing as target
faces. This means that the human perception of facial feature the nose. In this paper, we extend the idea to 3D face
attractiveness is essentially data-driven, and largely scans, and to any part of the face.
irrespective of the perceiver. Many papers aimed at analyzing The content of the paper is as follows. In Section II, we
and assessing beauty using Computer Vision and Image describe the 3D scans database used and we detail the various
Processing techniques have been recently presented. Several of steps of the algorithm. In Section III, we present examples of
them have been surveyed in [19]. PS simulation. Processing times are reported in Section IV and
Although several results have been obtained, the question the ratings of the results of PS simulations supplied by a panel
whether facial beauty can be synthesized into some relatively of human raters are discussed in Section V. Concluding
simple quantitative rules is far from being answered. remarks are reported in Section VI.
Traditional rules, as the classic canons, have been found not to
apply to many faces rated beautiful (see [20] [21] [22]).
Several empirical results support the idea that there is not a
unique beauty prototype. Although averaging facial images
produces faces more attractive than each component image,
attractive faces are mostly not average ([23] [24]). Beauty
ratings have been found largely independent on ethnicity, but Fig. 1. Different shapes could be more or less attractive, depending on the
faces rated beautiful can be rather different in various ethnic perceived general harmony
groups ([25] [26]). Cognitive theories support the idea that
humans define and use several category prototypes for face II. THE PROPOSED TECHNIQUE
image analysis [27]. According to this idea, several beauty In this section, we describe in detail the reference database
rating programs provide results close to those of human raters,
based on classifiers trained with a set of already rated faces used and the various steps performed by the system, that is:
[19]. Using around 100 samples, the quality of the ratings • Normalizing face scans
increases without saturation with the number of samples in the • Selecting and removing on the patient’s face the
training set [28], highlighting the needs for large DBs of rated target feature
faces. • Determining and removing the corresponding
Concluding, simple rules based on measurable face features on the harmonious faces of the reference
features, able to assess beauty or suggest how to improve it, DB
are still lacking, and we must learn attractiveness by examples. • Retrieving the k DB faces most close to the
Therefore, our general idea for patient-specific beautification patient’s face
is to consider several harmonious face prototypes, and to • Suitably pasting on the patient’s face the TRF of
suggest surgical procedures able to approach the samples the k closer DB faces and their average.
closer to the patient’s face. Observe that the idea of many Underlying our technique is the assumption that facial
different prototypes of harmonic faces implies that there is not harmony is unaffected by (slightly) non-isotropic 3D global
a unique harmonic prototype of a particular facial feature (e.g. scaling.
mouth, nose). Different shapes could be attractive, depending A. Creating the reference DB
on the integration with the rest of the face. An example of The outlined beautification process requires a set of
rather different noses belonging to movie actors is shown in harmonious faces (the reference database). Unfortunately, not
Fig. 1. many 3D face databases are available, and those existing,
In more details, our approach to the face beautification being essentially aimed at supporting face recognition research,
works as follows: contain attractively average faces, and only a very few scans
1. consider a particular patient’s face feature useful for our purposes. As mentioned before, a dense
(target feature), and compare the face of the sampling of the manifold of beautiful faces in the face space is
patient with a set of faces rated harmonic likely to require several hundreds of samples. To build such
(reference database), excluding the target large database is beyond the scope of this paper, and we have
feature; constructed a database sufficient to demonstrate
2. minimize some convenient metric in the face
space and find the nearest k attractive faces;
3. blend the corresponding target feature of the k
nearest attractive faces, as well as their average,
with the original face, in order to produce k+1
surgery simulations; in the following these
features will be referred to as target reference
IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, TBME-00849-2012 3

the PS simulation software

implemented and the
effectiveness of the general
approach.
To construct the reference DB, we collected scans from two
sources. First, we selected, with the help of a human raters
panel, the most attractive faces (actually very few) from the Fig. 2. Examples of face scans (rendered view and wireframe detail) from
available 3D databases, such as [30] and [31]. Second, we different databases
acquired high-resolution 3D attractive face scans of identification of landmarks on face profile
professional male and female models. Data have been obtained Then, the sagittal plane and the two landmarks are made
with the VECTRA-CR-two-pod system [32]. Each 3D scan coincident for all the 3D scans by translation, rotation and
has approximately 90.000 vertices with an associated 10 scaling. Finally, a further finer registration with an
Megapixel resolution texture.
In total, we selected as convenient to our purposes 53 face
scans of both male and female multiethnic subjects. Due to the
lack of a sufficient amount of male samples we finally decided
to create a reference DB containing only female scans (36
samples) and, therefore, to use merely females as subjects of
the PS simulation procedure.
B. Scan normalization
The 3D face scans used for constructing the current and
future reference database come from different sources, and are intermediate face (a face of the reference DB chosen at
different for size, orientation, resolution, type of meshes, etc. random, see section II.D for details) is performed using the
(Fig. 2). Therefore, for obtaining meaningful comparisons, Iterative Closest Point algorithm (ICP, [36]) using the isotropic
mesh resolution and geometric normalization are applied. scaling software of the VTK library [37]. At the end of the
First, since many 3D scans present various imperfections, registration process, all scans lie in a common reference
as missing polygons or spikes, mesh regularization is system, and have common sagittal plane and reference axis on
performed. In addition, faces have been cropped eliminating the sagittal plane, passing through the centroid of the
hairs and clothes (Fig. 3). Both cleaning and cropping have intermediate face and oriented as described in [34].
been performed manually with Blender [33].
Mesh resolution normalization
in [34], exploiting the Extended Gaussian Image (EGI). The Reference DB samples and patient’s scan could have
sagittal plane is intersected with the face scan in order to different resolution. Mesh normalization is required by one of
determine its profile, where two facial landmarks, the nasion the following steps of the algorithm, where a dense point-to-
and subnasale are identified (see Fig. 4). These landmarks are point correspondence between different face scans is
computed with the algorithm described in [35]. determined. Different resolutions can be used in the resampling
(a) (b) process in order to balance the precision of the reconstructions
Fig. 4. Intersection of a face scan with its sagittal plane and automatic and the computational costs.
(a)
axis
(b)
Fig. 3. An example of scan before (a) and after (b)
cleaning and cropping
Our normalization algorithm performs
the following steps. Orientation and size
normalization
Geometric normalization of 2D face
images is mostly obtained by making
coincident the centers of the eyes using
suitable translation, rotation and 2D
isotropic scaling. In our 3D case,
normalization involves isotropic 3D scaling
after a first rough registration using the
sagittal plane and two feature points.
The sagittal plane is determined by the
algorithm described Fig. 5. Re-sampling process: (a) the mesh is sliced with horizontal planes obtaining a number of polylines; (b) the polylines are
intersected with vertical planes rotated around the vertical axis. (c) The final point cloud; (d) reconstruction of the mesh connectivity
IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, TBME-00849-2012 4

To normalize resolution, face scans are re-sampled. As shown in Fig. the polyline selected and parallel to a direction evaluated from the polyline
5(a), each face mesh is first intersected with H horizontal planes equally by the Newell's Method [38].
spaced within a reference bounding box. Then, the obtained polylines are Removing the same feature from the scans of the reference database
intersected with a set of V vertical planes (Fig. 5 (b)), passing through the requires identifying a corresponding polyline on each face. To do this, we
reference axis, common for all aligned faces, and equally spaced angularly. must first establish a dense point-to-point correspondence between the
The resolution of the final meshes can be adjusted changing the H and V patient’s face and each reference face. To this purpose, we use a technique
values. For the experiments described in Section III, and submitted to human similar to that proposed in [39]. First, faces are registered with a thin plate
raters (Section V), both variables were assigned the value 776, which spline transformation (TPS), a non-rigid smooth surface transformation
produced mesh resolutions near to 300K vertices. For the results presented in particularly suited for morphing or animating faces, based on the
Section IV (Processing times), we also used two lower resolutions (160K correspondence of a number of keypoints. We used 37 pairs of
and 40K). corresponding landmarks manually selected on both the source and
The result of re-sampling is the point cloud shown in Fig. 5 (c). The final destination scans (see Fig. 8). These keypoints are a subset of the standard
triangular mesh is determined, as shown in Fig. 5(d), by forming quads and landmarks used in Anthropometry [40] with additional points in eyes area.
splitting each of them in two triangles. Texture coordinates for each vertex TPS provides a smoothly interpolated mapping between source and
of the new 3D mesh are computed with bilinear interpolation of those of the destination scans, which makes the landmarks coincident and brings the
neighbouring points in the original mesh. other points of the deformed source face very close to the destination face.
C. Selecting the patient’s target feature This allows an accurate alignment of points in smooth regions, like cheeks
Any area of the scanned patient’s face can be selected by the user as and forehead. Then, after TPS morphing, for each vertex of the first scan a
target of the PS simulation. An interactive tool allows drawing on the correspondence with a vertex of the second scan is derived with a closest
scanned face a 3D polyline, with vertices located at the re-sampled mesh point searching algorithm that exploits a KD- tree data structure to ensure
points. The polyline can be more or less detailed according to the particular computational efficiency.
target feature and to the judgment of the user. In Fig. 6 an example is shown Consequently, a new polyline, equivalent to the polyline drawn on the
relative to the nose area. A detail view of an example of polyline and facial patient’s scan, is determined for each reference face. An example of the
triangulation is shown in Fig. 7. transfer of the feature boundary on the reference DB faces is shown in Fig. 9.
Once the boundary polyline has been determined on all reference faces, the
target feature is easily removed from them, as done for the patient’s face.

(a) (b)
Fig. 6. Interactive feature selection: contour drawing (a); removal of the feature (b).

Fig. 8. The keypoints used for establishing dense point-to-point correspondence between faces

Fig. 7. A detail view of the polyline delimitating the area subject of the PS procedure (red dots are vertices of both the polyline and the face mesh)

[a] fb)
D. Removing the target feature from all the reference faces Our algorithm Fig. 9. An example of transferring a feature boundary from the patient’s face (a) to the scans in the reference DB (
requires removing the surface corresponding to the target feature both from
the patient’s face scan and from all the scans of the reference database.
The first operation is easily performed by intersecting the facial mesh with a Finding the correspondences between two scans is a time consuming
cylinder whose generatrices pass through process. In principle, each new patient’s scan would require computing the
correspondences with all the
IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, TBME-00849-2012 5

reference faces. As mentioned before, fully sampling the attractive a) the surface of the TRF is registered with respect to the
face manifold could require several hundreds of samples, resulting in patient’s scan;
long computation times. To avoid this, we pre-computed the b) the original target feature (surface and texture) is
correspondences between an intermediate face (a face randomly conveniently morphed toward the TRF surface. A feature
chosen) and all the reference faces. In this way, for each new patient’s of this particular morphing is that it produces a smooth
face, we have only to compute the correspondences with the blending of patient’s scan and TRF surface;
intermediate face (see Fig. 10). c) the original texture intensity is corrected according to the
different new shape.
In more detail:
a) Registration. An initial registration of the TRF on the
patient’s face is obtained applying the same transformations used to
match patient and corresponding face of the reference DB. This
registration is refined to reduce the differences between the contours
(a) (b) (c) of the removed target feature and the TRF, by applying a second
Fig. 10. A point of the input face (a) corresponds to a point on the intermediate face (b) anisotropic ICP only to the points of their contours. The resulting
that, in turn, corresponds to a point on each face in the reference DB (c) transformation is applied to the entire mesh of TRF.
b) Morphing. The morphing process is schematically shown
in (Fig. 11). The general idea is to use the dense correspondence
Observe that, given the discrete nature of the correspondence, two computed between target feature and TRF, excluding a strip close to
vertices of the first scan could collapse in only one vertex in the the contour, where the original surface and texture is morphed to an
second and there might be vertices of the destination scan which have intermediate position, depending on the distance from the feature
no correspondences in the source scan. Therefore, the correspondence contour.
function is not bijective. However, bidirectional correspondences are
necessary for our algorithm, since in the last step we also construct a
mean feature, obtained from those of a set of reference faces by
averaging their corresponding points. Therefore, computing the mean
feature requires correspondences between all reference faces, which (a)
can be obtained evaluating the backward correspondences from each Fig. 11. The reference feature (a, in(b)green) is used to morph the(c)target feature (b) to
reference face to the intermediate face. These correspondences can be obtain the final result (c)
pre-computed as well.
E. Retrieving the most similar reference faces In order to describe the details of the process, let us refer to Fig. 12,
After the target feature removal, reference samples are sorted in where we show in (a) the target feature (red, on the left), the TRF
order of similarity with the patient’s scan. The similarity measure (gray, on the right) and in Fig. 12(b) their registration. For each point
between two faces is given by their RMS distance, which is the square Pi of the target feature:
root of the average of the sum of squares of the closest point distances, 1. we compute ci, its distance from the closest point on its
after their alignment with ICP using anisotropic scaling. contour, and Di, its vector difference with its
As mentioned in the introduction, average faces have often been corresponding point on the transformed TRF (Fig. 12c);
found to be more attractive then each component face. For this reason, 2. we transform Pt (Fig. 12d) according to the following
we also computed an average feature from the first k-nearest neighbors equation:
in the following way. Starting from F1, the closest face, we evaluate P — P + W (P )* DI
the (weighted) mean of each vertex of its TRF and its corresponding where W(Pi) is a scalar weight in the interval [0,1]
points on the remaining k-1 faces. The correspondences between the obtained as:
target and the average features that are required for pasting a TRF on f c /( * t) if c / c t
the patient’s face, are easily obtained, being equal to those between
c <
W( ) \ P i
max / max

target feature and TRF of F1. 1

I 1 otherwise
As for the weights, we experimented with both a uniform and a cmax is the maximal c value for all target feature points
weighted mean, based on the inverse RMS distances. However, since a and t is a threshold (experimentally set to 0.1) that
panel of human raters did not express significant preferences for either controls the width of the blending strip.
option, we used equal weights. Fig. 12(e) shows, in false colours, a map of the weights applied to
F. Blending the TRFs the target feature points, ranging from 1 (blue) to 0 (red). The green is
This final step produces the simulated surgery outcome, that is the the area of blending. As it can be seen, points closer to the patient
3D textured model of the patient’s face where the surface of the target feature boundaries are smoothly interpolated with that of the reference
feature has been modified according to the surface of the TRF feature. An example of transformation of a point is shown in (c) and
retrieved, and the original patient’s texture has been conveniently (d). In this case, the distance c- is such that the point is outside the
applied on the new surface. This process should produce a smooth, blending strip, and the point sticks directly on the corresponding point.
realistic and seamless junction. For computing W(P} we used the Euclidean distance between P, and
The process consists of three steps: the feature contour. A better option could have been the geodetic
IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, TBME-00849-2012 6

distance, however too expensive to compute. (b) we have highlighted the points where the differences of normal
directions are significant, that is the area where the texture correction
takes place.

k **

(a) (b)

V *> V f*
v

(c) (d)
Fig. 12. a) target feature (left) and TRF (right); b) features alignment; c, d) evaluation of Fig. 13. Patient’s face subject to mouth
the weight of point P, (with W(Pi)=1) and its transformation; e) a false color map of the reshaping (a); map of the differences of the
weights W(P) (red = 0, blue = 1) normal directions (b); planning outcome
without (c) and with (d) texture adjustment
c) Texture correction. The texture applied on the new shape
corresponds to the original skin of the target features, but in general is III. EXAMPLES OF PS SIMULATION
not in full agreement with the new geometry, since the reflected light The beautification algorithm has been applied to 26 test cases, and k
depends on the surface normals, more or less changed. Exactly has been set to 4, which appears in keeping with the cardinality (36) of
reconstructing the illumination for the new surfaces is not easy, since it the reference DB. Then, for each case, we produced 4+1= 5 PS
would require detailed information about lights and surface materials, simulations, including that related to the average feature, and overall
which is not available in general. 26x5=130 simulations
Actually, in most images of simulated PS the effects of this problem The test cases have been chosen from the faces discarded when
are not conspicuous. Anyway, we implemented an approximate constructing the reference DB. Some of them were professional
algorithm providing a texture correction that appears effective in our models, but presenting, in our opinion, some slight defect. Different
case. target features have been considered in the various cases: nose, mouth,
Assuming that the reflective features of the surface material (skin) is chin and chin and mouth together. The majority of the subjects were
the same for all the feature points and a Lambertian reflection model, involved into nose reshaping.
I

the intensity of a surface point depends approximately only on the Some simulations are shown in this section. All have been used for
direction of its normal. evaluating the average computation times (Section IV) and rated by
Therefore, we assign to the texture of the morphed target feature the human panels (Section V).
intensity of the texture of the original feature where the surface had the Examples related to various face features are shown in Fig. 14 and
same normal. In more detail, this is Fig. 15. To make the 3D shape changes more evident,
performed in the following way.
First, we convert the feature texture from RGB to HSV space and
we construct a look-up table (LUT) relating normal direction and
average texture intensity for the target feature. The LUT is constructed
by averaging, for 218 equally spaced directions of the normals, the V
values of the mesh vertices whose normals are closer to each direction.
Then, for each vertex of the morphed mesh, we find the closest
normal in the LUT and retrieve the intensity. Then, the intensities of
the pixels of the texture triangle corresponding to a mesh triangle are
computed by interpolating that of its vertices. Some smoothing of the
intensities is applied to reduce irregularities related to the initial
normal space sampling. Finally, the adjusted intensities are combined
with the original HS values and the texture is converted back to RGB.
An example of texture correction is shown in Fig. 13. A surgery
simulation of a patient subject to mouth and chin reshaping (a) is
shown in (c) without, and in (d), with the adjusted texture. Finally, in
I There are cases where the feature material is not unique, e.g. the mouth area where
lips and surrounding skin can be considered as two different materials; in this case we
segment these different regions on the texture plane on the basis of their chromaticity
values and we process separately each material
IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, TBME-00849-2012 7

for each case we present three different views of the original patient’s scan
and of the PS simulation. The simulations shown are the best among 4+1 (a)
simulations for each face according to a panel of human raters (see Section
V). In Fig. 16 we also show an example where all the 4+1 outcomes of nose original
reshaping simulations are present. The leftmost image in the top row is the
original face. From left to right and top to bottom follow the results obtained
pasting the mean feature and the feature of the closest DB samples in
decreasing similarity order.
simulation
outcome

(b)

original
original

s
i
simulation m
outcome
Fig. 15.uExamples of simulated surgeries related to a) mouth and b) mouth and chin reshaping
l
a
t
i
o
original n

o
u
t
c
simulation
o
outcome
m
e

original

Fig. 16. In this case, all the 4+1 simulated nose reshaping surgeries are shown. Only one of them had been shown

s
i
m
IV. PROCESSING TIMES
Fig. 14.
u Examples of simulated surgeries related nose reshaping The entire PS simulation process requires a pre-processing phase, in part
l manually performed by a trained operator, and a totally computer-performed
a computation of the possible surgery outcomes. Computation times have been
t
i
determined by running our system on an Intel Core i7 3.7 GHz processor,
o 12GB RAM, and a 64-bit Linux operative system.
n The times reported in this section are relative to processing the 3D scan of
a single patient. We suppose that the construction of the reference DB and all
o
u
the related operations, as finding suitable face scans, mesh re-sampling,
t computing correspondences with the intermediate face, have already been
c performed.
o
m
A. Pre-processing
e Three manual pre-processing operations are required: 1) scan cleaning
and cropping, 2) facial landmark identification
IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, TBME-00849-2012 8

and 3) patient’s target feature selection. Clearly, scans’ cleaning We recall that a dense sampling of the manifold of harmonious
strongly depends on their quality. Anyway, a trained operator (one of (female) faces is likely to require some hundreds of samples, when our
the authors) was able to perform operation 1), on the average, in about reference DB only contains 36 samples. Therefore, the effectiveness of
6 minutes per mesh (Snorm). As for facial landmarks placement, in the system could considerably improve with a larger DB, and the
order to speed up execution times, a rough initialization of their examples presented should essentially be considered a demonstration
position, obtained from that on the intermediate face, allows the of the system developed.
trained operator to perform the task in about 30 sec. (Landm). Finally, However, we also arranged the rating of the simulation results
operation 3) requires to the trained operator about 10 sec. (Fsel). through a public website of the Politecnico di Torino. For all the 26
The average times required by the remaining computer- performed test cases, the original scan and the five surgical simulations were
phases of the pre-processing algorithm are reported in Table 1. In shown on the same page in random order. Raters were allowed to
addition to the top resolution class for resampled mesh (300K), used rotate interactively original and modified faces. Then they were asked
for the images of the previous Section and presented to human raters, 2 to evaluate, for each PS simulation, if there was an improvement
additional classes of resolution have been used namely 160K and 40K (positive answer), a worsening (negative answer) or no change (neutral
vertices. answer) in the perceived attractiveness. A total of 15,485 votes, with
For each resolution class, the actual average number of vertices an average of 119 votes per simulated result, were obtained from 312
resulting from the re-sampling process is in column AvgPt, and the judges. The ratings showed a substantial congruence between gender
number of horizontal and vertical planes used for the re-sampling is and age range of the raters, which is consistent with congruence of
indicated in column H and V. For each class, the table reports the human panels judgments of attractiveness reported in many papers.
average time required for finding the sagittal plane and normalizing A first result, actually rather obvious, is that in almost all cases the
faces for size and orientation (Fnorm), the average execution times for results were perceived as positive or neutral. Globally, the number of
resampling the mesh (Re-samp) and evaluating its bi-directional simulations scoring more than 50% of positive grades is 68.4%, while
point-to-point correspondences with the intermediate face (Corresp). those obtaining more than 50% of not negative grades were 99.2%.
To compute Fnorm and Re-Samp, actually depending also on A more interesting result is the connection between closeness of
resolution and orientation of the initial scans, we used the set of faces scans in the face space and perceived attractiveness improvement. In
in the reference DB, re-sampled at the three class resolutions. These Table 3, column Best+ shows, for the four closer scans of the
faces were randomly scaled, translated and rotated before applying the reference DB, the number and percentage of test patients for which
normalization process. that corresponding PS simulation obtained the highest number of
Pre-processing
Fnor
positive grades. It can be seen that the closest scan outperforms (11
CLAS
S
AvgP
t H V m Re-samp Corresp times and 42% of the cases) the others. In the table, the sum of test
300K 304,962 776 776
(se (sec)
152.64 366.85 8.27
(sec)
patients gives 28 instead of 26, since there are two cases where two
160 160, 5 5 81. 170.38 7.15 different scans provide the same highest grade. Although relative to a
40K 40,3 2 2 48. 126.54 6.25 small DB, and thus affected by noise due to under-sampling, this is a
Table 1. All values are in seconds, except H and V (the number of planes confirmation of the general assumption at the basis of this paper.
used during re-sampling) and AvgPt Another interesting result is the high rating of the average TRF.
Table 2 reports the average execution times required by each step of Actually, it received the highest percentage of positive answers in 18
the simulation process. Similarity is the average time for removing the cases. However, if we compare the percentage of positive answers of
target feature from a database sample and matching it with the the average TRF and the highest of the four closer scan, their absolute
PS simulation
C. Total processing times Overall time for obtaining k simulation difference is very small (2.3%). Also this result is probably affected by
Similarit MorphPt
CLAS
outcomes, withS a y
Morph
the under-sampling. Anyway, it is in keeping with the fact already
reference DB containing 300Kn faces,
3.03 3.13is given by: 6,277
(sec) mentioned that the result of averaging several facial images is usually
Snorm + Landm
160K + Fnorm
2.42 + Re-samp
2.35 + Corresp
3,256 + Fsel + n rated more attractive than each component image.
*Similarity + (k+1)
40K *Morph
1.66 1.49 830
For instance,
Tablewith the required
2. Times currentbysettings (36ofsamples
each phase in the reference
the simulation Closer simulations proposed appear in general
Summarizing, the surgical
process.
DB, 5 different MorphPt is total
simulations), the average
time isnumber
1052 of points
sec. for per
resolution 300k
category and per target feature. more attractive than thescanoriginal.nrThe Best +
simulation obtained using the
and 647 sec. for resolution 40k. 1st
most similar reference face is 11
several 42.3%
times the more appropriate to
Execution times for larger reference DBs are provided by the 2nd
select and the average TRF provides 7
a good26.9%
pick as well.
following examples. For n=100 and k=5,execution time is 1249 sec. 3rd 6 23.1%
for resolution 300k and 755 sec. for resolution 40k. For n=200 and k=7 VI. Table 4th AND 4FUTURE WORK
SUMMARY 15.4%
it is 1558 sec. for resolution 300k and 924 sec. for resolution 40k. 3. Number of times (and
These numbers shows that the approach is computationally feasible We have presentedpercentage)
a new 3Din system
which theforTRFtheofaesthetic planning of
even with practical DBs. Anyway, improving and parallelizing code face plastic surgery. the
The closer reference
system provides the
can suggest,
highest percentage of positivefor any selected facial
could shorten considerably computation times. area, new 3D shapes answers
able to(Best
improve
+) the general harmony of the face.
Observe that he most expensive part of the computation required for Since until now the research in the field has not been able to quantify
setting up the system, that is pre-processing all the faces of the rules for evaluating and improving general facial feature harmony, we
reference DB, is executed once for all, and adding a new sample proposed a “learning by examples” approach. In our case, it consists in
requires only pre-processing this sample. suggesting changes able to approach the shape of similar, but more
harmonious faces. For doing this, a database of harmonious faces has
V. HUMAN RATINGS OF PS SIMULATIONS been constructed. The feature to be improved (target feature) is deleted
IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, TBME-00849-2012 9

from a patient’s face scan, and the residual surface is compared with a [8] J.Liu, X.Yang, T.Xi, L.Gu, and Z. Yu, ”A novel method for computer
set of harmonious face scans, where the same feature has been deleted. aided plastic surgery prediction,”IEEE Proc. BMEI ’09, pp. 1-5, 2009
[9] http://www.seattlesoftwaredes
Retrieving and suitably pasting onto to patient’s face the target feature ign.com/, last accessed May 2012
of the k more similar harmonious faces, as well as the average feature, [10] J. Gao, M. Zhou, H. Wang and C. Zhang, “Three dimensional surface warping
provides k+1 surgery suggestions. for plastic surgery planning”, IEEE proc.Int. Conf. On Systems, Man and
Cybernetics, pp.2016-2021, 2001.
Implementing this general idea required to face and solve a number [11] T. Lee, Y. Sun, Y. Lin, L. Lin and C.Lee, “Three-dimensional facial model
of problems, including scan normalization, finding dense point-to- reconstruction and plastic surgery simulation”, IEEE Trans. On Info. Tech. In
point correspondences between different scans, performing smooth Biomed., Vol.3, No.3, pp.214-220, 1999.
and seamless blending of scans, correcting the final textures of the [12] T. Lee, C. Lin and H. Lin, “Computer aided prototype system for nose
surgery”, IEEE Trans. On Info. Tech. In Biomed., Vol.5, No.4, pp.271- 278,
morphed target feature to account for differences in shape and thus in 2001.
intensity of reflected light. All the corresponding sub-systems have [13] J. Wang, S. Liao, X. Zhu, Y. Wang, C.Ling, X.Ding, Y. Fang, and X.
been implemented and tested. Several examples of surgery simulation Zhang,“ Real time 3D simulation for nose surgery and individual prosthesis
design,” Comp. Methods and Programs in Biomd, in press
have been provided, showing how the system works for key facial [14]
areas as nose, mouth and chin. http://www.axisthree.com/profes
The tests show that the general idea at the basis of our approach is sionals/home, last accessed May 2012
[15] Cunningham, M.R., Roberts, A.R., Barbee, A.P., Wu, C.H. and Druen, P.B.:
effective. Although the database of harmonious faces used is rather Their ideas of beauty are, on the whole, the same as ours: consistency and
small, the ratings provided by a human panel demonstrate that the variability in the crosscultural perception of female physical attractiveness. J.
closer is the scan of the harmonious face, the better is the aesthetical Pers. Soc. Psychol., Vol.68, pp. 261-279, 1995
result of pasting the correspondent target feature. [16] J. N. Thakera, and S. Iwawaki, “Cross-cultural comparisons in interpersonal
attraction of females towards males,” Journal of Social Psychology, Vol.108,
Further work for improving the system will be carried on along pp.121-122, 1979
three directions. It has been already observed that a dense sampling of [17] S. M. Maret, “Attractiveness ratings of photographs of blacks by Cruzans and
the space of harmonious faces is likely to require several hundreds of Americans, “Journal of Psychology, Vol. 115, pp.113—116, 1983
[18] E. Wagatsuma and C. Kleinke,”Ratings of facial beauty by asian- american and
samples. In addition, an estimated 30% of today face surgeries are caucasian females,”J. Of Soc. Psychology, Vol.109, pp. 299-300, 1979
relative to male patients. Thus, improving both system performances [19] A.Bottino, A. Laurentini, “The analysis of facial beauty: an emerging area of
and application areas requires a) increasing maybe five or ten times the research in pattern analysis”, LECTURE NOTES IN COMPUTER SCIENCE, pp.
11, 2010, Vol. 6111/2010,
cardinality of the female harmonious faces database, with particular [20] K. Schmid, D. Marx, and A.Samal, “Computation of face attractiveness index
attention to covering several ethnic groups; b) constructing a database based on neoclassic canons, symmetry and golden ratio, “ Pattern Recognition,
of harmonious male scans. With regard to the last point, according to Vol.41, pp.2710-2717, 2008
[21] B.W. Baker and M.G.Woods, “The role of the divine proportions in the esthetic
most scholars of human attractiveness in human science and medicine, improvement of patients undergoing combined orthodontic/orthognathic surgical
the elements of human face beauty are very different in male and treatment,” Int. J. Adult. Orthod. Orthognath. Surg., Vol.16, pp. 108-120, 2001
female subjects, and thus the harmonious female database is not likely [22] J.P.Moss, A.D. Linney, and M.N. Lowey,” The use of three-dimensional
techniques in facial aesthetics”, Semin. Orthod., Vol.1, pp. 94-104, 1995
to be effective for beautifying males. [23] T.R. Alley, and M.R. Cunningham, “Average faces are attractive, but very
A second important area of improvement is extending the system attractive faces are not average”, Psychological science, Vol.2, pp.123-125, 1991
described, which deals with the aesthetic planning of surgery, to also [24] D. Perret, K. May, and S. Yoshikawa, “Facial shape and judgements of female
attractiveness”, Nature, Vol.368, pp.239-242, 1994
deal with the surgical constraints, combining the surface model [25] T. Le, L.Farkas, R. Ngim, L. Levin, and C. Forrest,: “Proportionality in asian and
described with a volumetric scan of soft and hard tissues. Finally, in north American Caucasian faces using neoclassic facial canons as criteria”,
order to deal with large reference DBs, we are planning to improve the Aesthetic Plastic Surgery, Vol.26, pp.64-69, 2002 .
computational performances, for instance rewriting the code, which is [26] M. Bashour, “History and Current Concepts in the Analysis of Facial
Attractiveness”, Plastic and Reconstructive Surgery, Vol.118, No.3, pp.741-756,
highly parallelizable, addressing a multi-thread architecture, or porting 2006
some of the algorithms in GPu. [27] E. Rosch, “Principles of categorization”, In Rosch, E. and Lloyd B., (1978):
“Eds. Cognition and Categorization”, Erlbaum, 1978.
REFERENCES [28] Y. Eisenthal, G. Dror, and E. Ruppin,“ Facial Attractiveness: beauty and the
machine,“ Neural Computation, Vol. 18, pp. 119-142, 2006
[1] American Society for Aesthetic Plastic Surgery, “Statistics 2008”. 2009. [29] A. Bottino, A, Laurentini, and L Rosano, “A New Computer-aided Technique for
Available online at: Planning the Aesthetic Outcome of Plastic Surgery”, Proceedings of 16th Int.
http://www.surgery.org/sites/ Conf. on Computer Graphic, Visualization and Computer Vision WSCG 2008,
default/files/2008stats.pdf, last Plzen (Cz)
accessed May 2012 [30] BFM (3D Basel Face Model), available online
[2] P. Adamson and S.Doud Galli, ”Modern concepts of beauty,” Current Opinion in at
Otolaryngology& Head and Neck Surgery,Vol.11, pp. 295300, 2003 http://faces.cs.unibas.
[3] R.Singh, M. Vatsa, H.S. Bhatt, S. Bharadwaj, A. Noore, and S.S. ch, last accessed May 2012
Nooreyezdan,” Plastic Surgery: A New Dimension to Face Recognition,” [31] USF Human ID, available online at
IEEE Trans. On Info. Forensics And Security, Vol.5, http://figment.csee.usf.edu/dat
No.3, pp. 441-448, 2010 aandcode.php, last accessed May 2012
[4] H. S. Bhatt, S. Bharadwaj, RSingh, M. Vatsa, and A. Noore “Evolutionary [32] Canfield Imaging System,
Granular Approach for Recognizing Faces Altered Due to Plastic Surgery,”IEEE http://www.canfieldsci.com , last accessed
Conf. on Auto. Face and Gesture Reco, 2011 ,pp. 720-725 May 2012
[5] R. Bracaglia, R. Fortunato and S.Gentileschi, “Secondary Rhinoplasty”, [33] Blender 3D. Available online at:
Aesthetic Plastic Surgery, Vol.29, pp.230-239, 2005. http://www.blender.org. Last accessed: May
[6] T. Ozkul, M.H. Ozkul, “Computer simulation tool for rhinoplasty planning”, 2012
Comput. In Biol. And Med., Vol.34, pp.697-718, 2004 [34] G.Pan, Y. Wang, Y.Qi, and Z. Wu, “Finding symmetry plane of 3D face shape,”
[7] S. Rabi and P. Aarabi, “Face Fusion: an automatic method for virtual plastic IEEE Proc. ICPR 2006
surgery”, Proc. Int. Conf. On Information Fusion, July 2006. [35] Bottino, A. and Cumani, S., “A fast and robust method for the identification of
IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, TBME-00849-2012 10

face landmarks in profile images”, WSEAS Trans. on Computers, 7(8), 2008.

[36] Z. Zhang. “Iterative point matching for registration of free-form curves and
surfaces”. Int. J. Comput. Vision 13, 2. 1994
[37] The Visualization Toolkit Library. Available online at:
www.vtk.org/. Last accessed: May 2012
[38] Filippo Tampieri. 1992. Newell's method for computing the plane equation of a
polygon. In Graphics Gems III, David Kirk (Ed.).
Academic Press Professional, Inc., San Diego, CA, USA 231-232.
[39] Y. Hu, M. Zhou, and Z. Wu, “A Dense Point-to-Point Alignment Method for
Realistic 3D Face Morphing and Animation,” International Journal of Computer
Games Technology, vol. 2009
[40] Leslie G. Farkas. 1994. Anthropometry of the head and face. Raven,
New York
A. Bottino was born in Torino, Italy, in 1971. He received his master degree
in Computer Science Engineering and his Ph.D. from Politecnico di Torino in
1995 and 2000. He is currently a researcher in Computer Science at the
Dipartimento di Automatica e Informatica of the same university. He is author of
several journal and conference papers. His current research interests include computer
graphics, computer vision and virtual heritage.
M. De Simone was born in Lanciano, Italy, in 1975. He received his master degree in
Computer Science Engineering from Politecnico di Torino in 2005. After an experience
as researcher on web technologies and knowledge management in a private company,
he is currently PhD student in Computer Science at the Dipartimento di Automatica e
Informatica of Politecnico di Torino. He is author of several journal and conference
papers. His current research interests include computer graphics, computer vision and
game developing.
A. Laurentini was born in Genova, Italy, and received the degree of Ingegneria
Elettronica from the Politecnico di Milano in 1963. In 1965 he joined the Politecnico di
Torino, where he has been professor of Computer Science at the Dipartimento di
Automatica e Informatica until November 2011. He is member of IEEE and ACM, and
author of more than eighty scientific papers. His current research interests include
computer vision, computer graphics and computational geometry.
C. Sforza was born in Monza, Italy, and received her M.D. degree from the Universita
di Milano, Italy in 1986 and her Master degree in Sports Medicine in 1989 from the
same University. She is a professor of human anatomy at the Medical and Dental
School of the Universita di Milano, and Director of the Laboratory of Functional
Anatomy of the Stomatognathic System. She has been involved in research on the
functional anatomy of the stomatognathic apparatus and on the anatomical bases of
stomatognathic function and dysfunction. She has authored/coauthored more than 230
publications indexed in PubMed