262 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL.

16, 2021

SecureFace: Face Template Protection

Guangcan Mai , Member, IEEE, Kai Cao , Xiangyuan Lan ,
and Pong C. Yuen , Senior Member, IEEE

Abstract— It has been shown that face images can be

reconstructed from their representations (templates). We propose
a randomized CNN to generate protected face biometric
templates given the input face image and a user-specific key. The
use of user-specific keys introduces randomness to the secure
template and hence strengthens the template security. To further
enhance the security of the templates, instead of storing the key,
we store a secure sketch that can be decoded to generate the key
with genuine queries submitted to the system. We have evaluated
the proposed protected template generation method using three
Fig. 1. Security and privacy issues introduced by inverting biometric
benchmarking datasets for the face (FRGC v2.0, CFP, and templates. Raw biometric data (e.g., face and fingerprint images) can be
IJB-A). The experimental results justify that the protected reconstructed from the corresponding templates stored in the system. The
template generated by the proposed method are non-invertible reconstructed data can then be presented to authentication systems to get
and cancellable, while preserving the verification performance. access (security issues) and identified by a biometric search engine (privacy
issues).
Index Terms— Biometric, template security, deep templates,
template protection, randomized CNN, protected templates.
to the intra-subject variation in biometric templates and the
I. I NTRODUCTION avalanche effect [6] of standard ciphers, biometric templates
must be decrypted before comparison.5 This is different
B IOMETRIC recognition systems are being widely
deployed in security-sensitive (e.g., personal devices1 and
border security2) and privacy-aware applications (banking3
from traditional passwords that can be compared in their
encrypted (hash) form. Another possible cipher for the pro-
and patient ID4 ). It is critical to protecting the biometric data tection of biometric templates is homomorphic encryption [9],
stored in the system because biometrics is unique and irrevo- [10]. Such encryption approaches compare templates in their
cable once it is compromised [1]. To ensure the security and encrypted form to give the encrypted results, which are
privacy, biometric vendors generally store biometric templates, then decrypted to yield the decision. However, homomorphic
representations of raw biometric data (e.g., face images), encryption is computationally expensive, especially for high-
instead of raw biometric data in the systems. However, recent dimensional biometric templates. It also suffers from the
studies show that, raw biometric data can be reconstructed same key management issue as most homomorphic encryption
from plaintext biometric templates (Fig. 1) and then used to construction [9], in which the decryption key of the encrypted
access the target system and identify the target users (e.g., results can be used to decrypt the encrypted templates. An
face [2], [3], fingerprint [4], and iris [5]). Thus, biometric alternative approach is the use of biometric template protection
templates stored in the systems have to be protected. schemes [11]–[13], which generates a pseudonymous identifier
One straightforward approach to protect biometric templates (PI)6 and auxiliary data (AD)7 from the plaintext enrollment
is to use standard ciphers [6] such as SHA-3. However, due template and store them in the systems. During authentication,
the stored PI is compared directly with the PI* generated from
Manuscript received December 28, 2019; revised April 19, 2020 and the query template and the stored AD. The generally agreed
June 9, 2020; accepted July 6, 2020. Date of publication July 15, 2020;
date of current version July 31, 2020. This work was supported in part by
criteria for template protection schemes [11]–[14] are:
the Hong Kong RGC under Grant HKBU 12200820. The associate editor • Non-invertibility (security): It should be computationally
coordinating the review of this manuscript and approving it for publication infeasible8 for the protected templates (PI and AD) to be
was Prof. Raymond Veldhuis. (Corresponding author: Pong C. Yuen.)
Guangcan Mai was with the Department of Computer Science, Hong Kong inverted into the plaintext biometric template or recon-
Baptist University, Hong Kong. He is now with the Lenovo Machine Intelli- structed into the raw biometric data.
gence Center, Hong Kong (e-mail: [email protected]).
Kai Cao was with the Department of Computer Science and Engineer- 5 To our knowledge, there are two available methods to directly compare
ing, Michigan State University, East Lansing, MI 48824 USA (e-mail: the biometric templates in their hash form [7], [8]. However, only constrained
[email protected]). datasets are used in their evaluation, and the samples of all enrolled subjects
Xiangyuan Lan and Pong C. Yuen are with the Department of Computer are used to train the template extractor.
Science, Hong Kong Baptist University, Hong Kong (e-mail: xiangyuanlan@ 6 Unless specified otherwise, the variables with a superscript ‘∗’ denote data
life.hkbu.edu.hk; [email protected]). processed at the query stage and correspond to the data processed at the
Digital Object Identifier 10.1109/TIFS.2020.3009590 enrollment stage (e.g., a query raw biometric data x ∗ and an enrollment raw
1 https://www.apple.com/iphone-x/#face-id
biometric data x.)
2 http://fortune.com/2016/09/12/border-security-biometrics/ 7 We use the terms ‘PI and AD’ and ‘protected template’ interchangeably,
3 http://www.citibank.com.hk/english/ways-to-bank/voice-biometrics.htm because they are the secure form of plaintext biometric templates.
4 https://www.kairos.com/human-analytics/healthcare 8 Cannot be solved using an algorithm with polynomial complexity.

1556-6013 © 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.

Authorized licensed use limited to: WASHINGTON UNIVERSITY LIBRARIES. Downloaded on August 25,2021 at 17:44:37 UTC from IEEE Xplore. Restrictions apply.
MAI et al.: SECUREFACE: FACE TEMPLATE PROTECTION 263

• Cancellability (revocability and unlinkability): A new key storage, we store the secure sketches generated from the
protected template can be generated for a subject whose keys and binary intermediate features of the randomized CNN.
template is compromised. Besides, different templates of With this construction, the keys can be decoded from the
a subject can be generated for different applications. secure sketch at the query stage only if the query biometric
• Verification performance: The protected templates must data is sufficiently similar to the enrollment biometric data.
be discriminative enough to satisfy the requirements for To improve the decoding success rate of the secure sketches
authenticating a person. for genuine queries and strengthens the security of the secure
This work aims at generating protected or secure biometric sketches, we propose an orthogonal triplet loss for optimizing
templates from raw biometric data. State-of-the-art approaches the binary intermediate features. In a nutshell, this paper makes
for generating protected templates are based on two-stage the following contributions:
approaches (i.e., template extraction followed by template pro- • A randomized CNN to generate protected deep biometric
tection). For such an approach, the biometric templates are first templates, which depend on both raw biometric data (e.g.,
extracted from raw biometric data using a template extractor face images) and user-specific keys.
(e.g., Eigenface [15], deep templates [16]) which is generally • A randomized triplet loss to train the randomized CNN.
optimized for verification performance only. The template The protected deep biometric templates are similar for
protection schemes (e.g., feature transformation [17]–[19], biometric data of the same subject with the same user-
biometric cryptosystems [20]–[24] and hybrid approaches specific key, and different sufficiently otherwise.
[14]) are then applied on the extracted templates. There are • A secure system construction using the randomized CNN
two limitations with such two-stage approaches: (a) the two without storing the keys. We store the secure sketches
stages (i.e., template extraction and template protection) can generated from the keys and binary intermediate features
be attacked individually by adversaries with knowledge of the of the randomized CNN.
template extractor and the template protection methods. (b) the • An orthogonal triplet loss to extract the binary intermedi-
template extractors used to extract biometric templates are ate features, which are used to generate the secure sketch.
generally optimized to improve the verification performance, • Evaluation and analysis based on three face benchmark-
whereas the security-related objectives are often neglected and ing datasets (FRGC v2.0 [25], CFP [26] and IJB-A [27]
can only be improved at the stage of template protection. demonstrate that the proposed method satisfies the criteria
This usually causes a significant trade-off issue between ver- for template protection schemes [11], [13], i.e., non-
ification performance and template security because they are invertibility (security), cancellability (unlinkability and
not optimized jointly. Besides, the entropy (security) of the revocability), while maintaining verification performance.
protected templates is upper bounded by the entropy of the
extracted templates. Otherwise, adversaries can directly guess II. R ELATED W ORK
the extracted templates for an easier attack. A. Template Reconstruction Attack
To address these limitations, we propose an end-to-end9 It was believed that templates extracted from raw biometric
approach for generating protected biometric templates. Specif- data are not possible to be inverted back. However, it has been
ically, the proposed approach includes a randomized CNN and demonstrated that such inversion can be done on face [2],
a secure sketch construction components. With these two com- [3], fingerprint [4], and iris [5]. In general, a realistic threat
ponents, we formulate two loss functions (i.e., a randomized to biometric systems under template reconstruction attack is
triplet loss and an orthogonal triplet loss) to jointly optimize analogous to a chosen-plaintext attack in cryptography, where
the verification performance and the security of the resultant the input biometric data and templates are regarded as plaintext
protected biometric templates. Different from typical template and ciphertext, respectively. To achieve the attack, the adver-
extractors which generate templates based on raw biometric sary typically requires the following knowledge: (a) The
data only, the proposed randomized CNN generates templates templates of the target subject (This is usually caused by either
additionally depending on user-specific keys, to introduce database leakage or insider attack). (b) The template of any
randomness into the protected biometric templates. input biometric data (This can be introduced by subscribing a
To train the randomized CNN, we propose a randomized (black-box) template extractor (biometric SDK) of the target
triplet loss, aiming to make the protected templates sufficiently system or standardized templates (e.g., fingerprint minutiae
similar given the same user-specific key, and the input raw [28])). With the above knowledge, a typical strategy for an
biometric data coming from the same subject. A typical system adversary is to first learn an inversion function of the template
with the randomized CNN is required to store the enrollment extractor by collecting biometric input data and then inverting
templates extracted from the raw biometric data, and the the templates of the target [3].
corresponding keys in the enrollment stage, where the stored A straightforward countermeasure for the template recon-
keys are needed for extracting query templates from the raw struction attack is to strengthen the security of template
biometric data in the query stage. However, the stored keys databases to avoid the database leakage and the insider attack.
would make it easy to invert the enrollment template once However, even with strict security measures, it is impossible to
the keys are compromised, which is not secure. To avoid avoid the database leakage completely10, .11 To further protect
9 The ‘end-to-end’ refers that the model for extracting protected templates 10 https://goo.gl/QUMHpv
in this paper can be optimized in an end-to-end way. 11 https://goo.gl/KdxzqT

Authorized licensed use limited to: WASHINGTON UNIVERSITY LIBRARIES. Downloaded on August 25,2021 at 17:44:37 UTC from IEEE Xplore. Restrictions apply.
264 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 16, 2021

Fig. 2. An overview of the proposed secure system construction with the proposed randomized CNN. The protected deep templates {S S, prpt , y p } stored in the
system satisfy the criteria for template protection, i.e., non-invertibility (security), cancellability (unlinkability and revocability), and verification performance.

biometric systems from template reconstruction attack, it is neglected and can only be improved in the module of tem-
suggested to secure the templates using template protection plate protection. Besides, the two-stage process is vulnerable
schemes [11]–[13]. because the modules of template extraction and protection can
be attacked individually.

B. Biometric Template Protection

III. T HE P ROPOSED A PPROACH
Unlike traditional passwords that can be compared in their
encrypted or hash form with standard ciphers (e.g., AES and An overview of a secure system constructed with the
SHA-3), biometric templates cannot be compared in such form proposed randomized CNN is shown in Fig. 2. In the training
due to typical intra-user variations [11], [13] and the avalanche stage, the neural network is jointly optimized by the random-
effect of standard ciphers [6]. Biometric template protection ized triplet loss Lrt and the orthogonal triplet loss Lot in an
[11]–[13] aims to create a secure form of the templates, and end-to-end manner. The processes of enrollment and query are
then the comparison can be performed in the secure form. shown with blue and red lines, respectively.
Specifically, in the enrollment stage, biometric template pro-
tection schemes generate PI and AD from plaintext enrollment A. Secure System Overview
template and store them in the systems. In the query stage,
PI* is first generated from the query template with the stored This section first mathematically describes how the
AD and then directly compared with the stored PI, without enrollment and query of a biometric verification system can be
revealing the plaintext enrollment template. performed with the proposed randomized CNN (Section III-B)
Biometric template protection schemes that are designed and the secure sketch construction (Section III-C). Then,
for compact binary or real-valued vectors can be categorized the requirements for the protected templates to satisfy the
into feature transformation [17]–[19], biometric cryptosystems criteria for template protection are also discussed.
[20]–[24] and hybrid approaches [14]. Please refer to [11] for Enrollment: Given an enrollment image x, a user-specific
detailed analysis about the security and cancellability of the key k ∈ {0, 1}m , and a user-specific permutation vector pr pt ,
biometric template protection schemes. our system’s enrollment process E(·) generates and stores a
A trade-off exists between verification performance and randomized template y p and a secure sketch SS,
template security because of the two-stage process that uses y p , SS = E(x, k, pr pt ) (1)
template protection schemes after extraction of biometric
templates. This trade-off exists because the biometric template where the secure sketch SS ∈ {0, 1}n , (n ≥ m). Note that the
extractors are generally optimized for improving template key k is not stored in our system and is decoded from the
discriminability, whereas the security-related objective is often stored secure sketch SS at the query stage. The permutation

Authorized licensed use limited to: WASHINGTON UNIVERSITY LIBRARIES. Downloaded on August 25,2021 at 17:44:37 UTC from IEEE Xplore. Restrictions apply.
MAI et al.: SECUREFACE: FACE TEMPLATE PROTECTION 265

vector pr pt = { pr1pt , · · · prr pt , · · · prRpt }(R > n) and satisfy that The feature extraction network f ext (·) is a convolutional
∀r  ∈ {1 · · · R}, ∃r  = prr pt . Here, the randomized template network with at least one fully connected layer for extraction
y p refer to the PI, the secure sketch SS and the permutation of intermediate features. It can be constructed using the
vector pr pt refer to AD, in a system with a template protection convolutional part of a popular CNN such as ResNet [29].
scheme [11]–[13]. Let b denote the extracted intermediate feature to be sent to
Query: Given a query image x ∗ , the secure sketch SS as the random partition, we have
well as the permutation vector pr pt stored in the system, our
system’s query process Q(·) first generates a query template b = f ext (x) (3)
y∗p , The random partition fr pt (·, pr pt ) separates the intermediate
y∗p ∗
= Q(x , SS, pr pt ) (2) feature b into two parts, b A and bB ,

The decision of accepting or rejecting the query image x ∗ b A , bB = fr pt (b, pr pt )

is then made on the basis of the similarity scor e( y p , y∗p ) s.t.b A = {b pr1pt , · · · , b p R−n }
r pt
between the enrollment and query templates. To ensure that bB = {b p R−n+1 , · · · , b p R } (4)
r pt
the protected templates (PI: y p and AD: SS, pr pt ) stored in r pt

the constructed secure system satisfy the criteria for template where b A would be sent to the RandNet for extraction of
protection [11]–[13], it is required to achieve the following: the randomized template y p , and bB is used to construct the
• Non-invertibility (security): It is not computationally secure sketch SS. Note that to avoid the linkability between the
feasible to reconstruct (synthesize) the enrollment image protected template y p and the secure sketch SS, the elements
x from the stored randomized template y p , the secure in b A and bB are designed to be mutually exclusive. Besides,
sketch SS and the permutation vector pr pt . the permutation vector pr pt for random partition are designed
• Cancellability (revocability and unlinkability): A new to be specific to both the subject and the application to further
set of a randomized template y p , a secure sketch SS, and enhance the security and privacy of the resulting templates.
a permutation vector pr pt can be generated for the target The RandNet uses an intermediate feature partition b A and
subject whose template is compromised. There is no a subject-specific key k as input to produce the protected
method to determine whether two randomized templates template,
(e.g., y1p and y2p ) or two secure sketches (e.g., SS 1 and
y p = frnd (b A , k) (5)
SS 2 ), or two permutation vectors (e.g., pr1pt and pr2pt ))
are derived from the same subject or not, given the The RandNet introduces the key k-based randomness and
different subject-specific keys. is the key component in the randomized CNN. We have
• Verification performance: The similarity between the introduced two types of randomness in the RandNet: ran-
enrollment template and the genuine query template dom activation and random permutation-flip. In the RandNet,
should be maximized if the same key is given. Otherwise, we first create a different subnetwork from a father network
the similarity should be minimized. via random activation and deactivation of its neurons according
to the key k, where the template y with partial randomness is
B. Randomized CNN produced. Then, with our random permutation-flip, the ele-
ments in the template y are randomly permuted, and the
The randomized CNN is obtained by embedding random-
signs of randomly selected elements are flipped, depending
ness into a CNN. The randomized CNN generates a random- on the key k. The output template is the final randomized
ized template y p and an intermediate feature bB using an template y p .
input image x and a key k, which indicates the random-
1) Random Activation: Given a neural network with all
ness embedded within the deep network. The randomized neurons activated, various subnetworks can be created by
template y p is then used as the PI in the system, and the
random deactivation of some neurons. An example is shown
intermediate feature bB will be used to construct the secure
in Fig. 3, in which the networks in Figs. 3(b), 3(c), and 3(d)
sketch SS (AD in the system, see section III-C). To satisfy are subnetworks created from the father network in Fig. 3(a)
the criteria for template protection, we introduce the RandNet
by random deactivation of half of the neurons in each layer.
into the CNN for producing the randomized template y p .
With random activation, an L-layer father neural network
With the discriminability preserved, the randomized templates with h l (1 ≤ l ≤ L) neurons at each layer will have N L
y p extracted from the same images x with different keys
subnetworks,
k differ significantly and cannot be matched to each other.
Besides, there is no way to invert the randomized templates 
L
h 
y p back into the input image x without the corresponding NL = l
dl (6)
keys k, which is assumed here to be secure and is discussed l=1
in sections III-C. where dl denotes the number of the neurons at layer l to
The randomized CNN consists of three components: be deactivated. The random activation can be regarded as
a feature extraction network f ext (·), a random partition randomly assigning a subnetwork to an enrollment subject, for
fr pt (·, pr pt ), and the RandNet frnd (·, k), which is a fully con- which the assignment is indicated by the key k. Specifically,
nected neural network with key k-based randomness (Fig. 2). the activation of the neurons are indicated by a randomly

Authorized licensed use limited to: WASHINGTON UNIVERSITY LIBRARIES. Downloaded on August 25,2021 at 17:44:37 UTC from IEEE Xplore. Restrictions apply.
266 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 16, 2021

Fig. 3. Subnetworks produced by a standard network with random activation, in which the black and white circles denote ‘activated’ and ‘deactivated’
neurons, respectively. (a) Standard network with all neurons activated; (b), (c), and (d) are different subnetworks obtained by random deactivation of some
neurons.

generated binary string, where the corresponding random seed where s k ∈ {+1, −1} A is a randomly (uniformly distrib-
depends on the key k. uted) generated sign vector, based on the key k. Note that
There are two requirements for the subnetworks: (a) the information on the order and sign of the elements ym in the
templates extracted using the subnetworks should be discrim- enrollment template y is necessary to invert the template [2],
inative, and (b) the templates extracted from the same subject [3], analyze the linkability, and perform comparison. This is
should be sufficiently different while different keys are given. because that in general, each element of a template vector,
To these ends, motivated by the Dropout technique [30] and represents a different semantic meaning. Examples like the
the triplet loss [31], we formulate a randomized triplet loss to projection on the different basis for most component analysis,
directly train the father network, whose neurons are randomly e.g., PCA [15]. Suppose that the cosine similarity is used
deactivated depending on keys. Thus, the subnetworks are for calculating the comparison score between templates when
implicitly trained. Specifically, the randomized triplet loss Lrt applying the random permutation-flip, it can be shown that:
aims to minimize the distance of positive pairs given the same (a) Score-Preserving: if the same key is given, the similarity
key and maximize the distance of positive pairs given the score between two templates is preserved. Therefore, as long
different keys and negative pairs. Mathematically, the Lrt can as the key is known, the discriminability of the randomized
be expressed as template is well-preserved. (b) Zero-Expectation: if different
keys are given, the expectation of the similarity score between
1  
Q
pos 2 two templates is zero, where the range of the cosine similarity
Lrt = D pos − yq,k
anc
− yq,kq 2 + αrt
Q q
+ is [-1,1]. Note that the cosine similarity of zero would not
q=1
  be classified as the same subject in typical face verification
neg
+ D pos − yq,k
anc
− yq,kq 22 + αrt systems.12
+
q
 The cosine similarity between the enrollment template y
anc neg 2
+ D pos − yq,kq − yq,k  2 + αrt (7) and the query template y∗ can be expressed as
q +
pos y T y∗ 1 
A
where D pos = yq,k anc − y
q
2
q,kq 2 denotes the distance of the scor e( y, y∗ ) = = ya ya∗ (9)
positive pairs given the same key. The size of a mini-batch y · y∗ y · y∗
a=1
for optimization is Q, given that the randomized CNN is
Score-Preserving: Given the same key k, it can be shown
optimized using a mini-batch based stochastic gradient descent
anc , y pos , and yneg denote the templates of that the cosine similarity between the enrollment and query
(SGD). The yq,k q q,kq q,kq randomized template equal to scor e( y, y∗ ).
anchor, positive, and negative samples in the q-th triplet,
pos neg
where the key kq is given. The yq,k  , and yq,k  denote the  ( y kp )T y∗,k
p
q q
corresponding templates of positive and negative samples, with scor e y kp , y∗,k =
y kp · y∗,k
p
a different key kq . Note that kq and kq are two different keys
p

which are uniformly sampled for every triplet in the training 1 

A

stage. The αrt is a margin that is enforced between positive = (sak )2 y pak y ∗p k
y · y∗ a
a=1
pairs given the same key and the other pairs.
2) Random Permutation-Flip: Given an enrollment template 1 
A
= ya ya∗ = scor e( y, y∗ )
y = {y1 , · · · , ya , · · · , y A } extracted from the deep networks y · y∗
a=1
with random activation, we further embed the randomness by
(10)
random permutation-flip. The objective is to further enhance
∗,k ∗
the non-invertibility and cancellability of the final randomized Zero-Expectation: Let y kp ( y p ) denote the randomized
template y p . Specifically, let pk = { p1k , · · · , pak , · · · , p kA } template obtained by applying randomized permutation-flip
denote a permutation vector that depends on k and satisfies on templates y ( y∗ ) with keys k (k∗ ). It can be shown that
∀a  ∈ {1 · · · A}, ∃a  = pak , the randomized template given by 12 As reported in [3], with FaceNet [31] and BLUFR [32] as feature extractor
the random permutation-flip can be expressed as and protocol, resp., the mean verification thresholds on LFW [33] and FRGC
v2.0 [25] were 0.51 and 0.80, respectively, at FAR=0.1%, and 0.38 and 0.64,
y kp = {s1k y p k , · · · , sak y pak , · · · , s Ak y p k } (8) respectively, at FAR=1.0%.
1 A

Authorized licensed use limited to: WASHINGTON UNIVERSITY LIBRARIES. Downloaded on August 25,2021 at 17:44:37 UTC from IEEE Xplore. Restrictions apply.
MAI et al.: SECUREFACE: FACE TEMPLATE PROTECTION 267

∗
the expectation of the scores between templates y kp and y∗,k
p ,
∗,k ∗
E[scor e( y p , y p )] is zero. Please refer to the Supplemen-
k

tary A for the proof.

C. Secure Sketch Construction

To extract a randomized template y∗p that is similar to
y p from a genuine query image x ∗ , the user-specific key k
is required in the query stage. One of the straightforward
methods for providing the key k in the query stage is to
store it in the system in the enrollment stage. However, for
smart adversaries, the availability of the k would significantly
reduce the difficulties of inverting the enrollment template y p Fig. 4. Given a ECC with sufficiently large code length n and the average
and linking the enrollment templates across systems. Note that error tolerance τecc /n, the lower bound and an upper bound of the code
the study of the template protection generally assumes that rate [34], [35], i.e., m/n, where m denotes the message length.
adversaries can access to the templates stored in the system.
To solve this problem, we propose to store a secure sketch
SS generated from the key k, instead of the key itself in that |bB | for genuine queries is less than the error
the system. The stored secure sketch SS can be successfully tolerance τecc of the chosen ECC.
decoded if the query image x ∗ is sufficiently similar to the 2) Large |bB | for Impostor Queries: The SS should not
corresponding enrollment image x. The similarity can be mea- be correctly decoded by impostor queries to prevent the
sured using the hamming distance between their corresponding false accept attack and therefore the |bB | for impostor
binary intermediate features bB . A successful decoding implies queries is greater than the error tolerance τecc of the
that the decoded key k∗ is identical to the corresponding chosen ECC.
key k. 3) High Entropy of Both bB and k: It should be difficult to
We now present how the secure sketch SS is constructed obtain (guess) the key k from the SS without genuine
from a key k and an intermediate feature bB that is generated query images x ∗ (or the corresponding genuine features
from the randomized CNN in the enrollment stage. Then b∗B ); This requires that the entropy of both the key k
we describe how the key k∗ is decoded from SS with the and the entropy of the feature bB are high, because
corresponding intermediate feature b∗B in the query stage. the adversary can obtain the key k via either directly
Enrollment: Motivated by the fuzzy commitment [20], guessing or guessing the binary feature bB with Eq. (13).
the secure sketch SS is generated with the error correcting To fulfill the above requirements, one should first understand
code (ECC) that the entropy m of the message (key k in our construction)
which can be encoded in an ECC codeword c is decreasing
SS = c ⊕ bB (11) while the corresponding error tolerance τecc increases, accord-
where ⊕ denotes a modulo-2 addition. The codeword c has ing to the coding theory [34], [35]. Besides, given a ECC
length n (same as the length of the bB ) and is obtained by whose code length n (the size of the codeword c in Eq.(12))
encoding the key k using a ECC encoder E NCecc (·): is sufficiently large, there are an upper bound and a lower
bound for the code rate m/n with the average error tolerance
c = E NCecc (k) (12) τecc /n [34], [35] (Fig. 4). Therefore, to allow a small error
Query: In the query stage, the key k∗ can be decoded with tolerance τecc and high key entropy m, the minimization of
the intra-subject variations for bB should be weighted more.
k∗ = D ECecc (b∗B ⊕ SS) (13) Given that bB consists of part of the elements of b, assuming
that the elements of b equally contribute to the satisfication
where D ECecc (·) denotes the decoder of the ECC employed
of the requirements for bB , we optimize the b in the training
in the system. The decoded key k∗ is identical to k only if the
stage with the requirements for bB .
distance |bB | between features b B and b∗B less than the error
To optimize the binary intermediate feature b with desired
tolerance τecc of the chosen ECC, accordingly to the properties
properties, i.e., high entropy, minimum intra-subject variations,
of ECC [34], [35]. This is because
and maximum inter-subject variations, we have formulated an
b∗B ⊕ SS = c ⊕ b B ⊕ b∗B orthogonal triplet loss Lot . Note that instead of optimizing the
= c ⊕ (b B ⊕ b∗B ) binary intermediate feature b in the binary form, we optimize
b by optimizing its relaxed form h,
= c ⊕ b B (14)
b = sgn (h) , h ∈ (−1, 1)|b| (15)
Requirements: The requirements related to the construction
of the secure sketch SS are listed as below: where sgn(h i ) = 1 if h i >= 0 and sgn(h i ) = 0 otherwise,
pos neg
1) Small |bB | for Genuine Queries: For genuine queries, i ∈ {1, · · · , |b|}. Let hanc
q , hq and hq are column vectors
the SS can be correctly decoded, that is, the decoded key that denote the intermediate features of anchor, positive and
k∗ = k. According to Eqs.(13) and (14), this requires negative samples in the q-th triplet of a mini-batch, we have

Authorized licensed use limited to: WASHINGTON UNIVERSITY LIBRARIES. Downloaded on August 25,2021 at 17:44:37 UTC from IEEE Xplore. Restrictions apply.
268 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 16, 2021

formulated the Lot as below:

1  
Q
pos neg 2
Lot = λ hanc
q − hq
2
2 − hanc
q − hq 2 + αot
Q +
q=1
+ μLort h + ωLbin (16)
where αot is a margin that is enforced between positive and
negative pairs. Different from the original triplet loss [31],
we introduce a hyper-parameter λ > 1 to control the fac-
tor of optimizing intra and inter subject variations. This is
because that, to allow an ECC with smaller error tolerance,
it is required to weight more on optimizing the intra-subject
variations. To improve the entropy of the resultant templates,
we introduce the orthogonal term Lot with factor μ to mini-
mize the correlation of binary intermediate features of different
subjects. Mathematically,
Fig. 5. Example images from training datasets: (a) VGG-Face2, (b) MS-
1   anc T neg 2
Q Celeb-1M; and testing datasets: (c) FRGC v2.0, (d) CFP, and (e) IJB-A.
Lort h = (hq ) hq (17)
Q
q=1
A. Experiment Setting
To reduce the binarization loss in the testing stage, we intro-
duce the binarization term Lbin with factor ω, where Lbin can 1) Datasets: We use two large scale face datasets (i.e.,
be expressed as VGG-Face2 [36] and MS-Celeb-1M [37]) for training the
proposed randomized CNN. Three benchmarking face datasets
1 
Q
pos neg (i.e., FRGC v2.0 [25], CFP [26], and IJB-A [27]) are used for
Lbin = |hanc
q |−1 + |hq |−1 + |hq |−1
Q 1 1 1 testing. Example images of these datasets are shown in Fig. 5.
q=1
In the following, we briefly describe these datasets.
(18)
• VGG-Face2 [36] comprises of 3.31 million images
D. Network Architecture of 9,131 subjects downloaded from Google Image
Search. The training partition with 3.15 million images
Input: We use RGB images of size 112 × 112 × 3 as input. of 8631 subjects is used.
Feature Extraction Network: We first employ the ResNet- • MS-Celeb-1M [37] originally contains 10 million images
50 [29] without the fully connected (FC) layers as our con- of 100K subjects. We used the refined MS-Celeb-1M
volutional layers to extract 512 feature maps, whose size are [38] where the images that far from the subject center
3 × 3. The 512 feature maps are then flattened and connected has been removed. The refined MS-Celeb-1M consists
to a fully connected layer of 4096 hidden units, where tanh of 3.8 million images of 85K subjects.
activation is applied. • FRGC v2.0 [25] is a constrained dataset that contains
RandNet: The output of the feature extraction network is frontal face images taken under different illuminations.
then connected to FC layers, i.e., two fully connected layers There are 50K images of 4,003 subjects in FRGC v2.0,
of 512 hidden units, where the ReLU activation is applied on where 16,028 images of 466 subjects (as specified in the
the first fully connected layer. target set of Experiment 1 of FRGC v2.0 [25]) are used
Output: The loss layers for the intermediate features and in this study.
output template are connected on the output of feature extrac- • CFP [26] is an unconstrained dataset, which consist of
tion network (Lot ) and FC layers (Lrt ), respectively (Fig. 2). images from 500 subjects. There are 10 frontal face and
4 profile images for each subject.
IV. P ERFORMANCE E VALUATION AND A NALYSIS
• IJB-A (IARPA Janus Benchmark A) [27] is an
This section evaluates the proposed protected template unconstrained benchmarking dataset, which comprises
generation method according to the criteria of the tem- of 5,712 still images and 2,085 videos from 500 subjects.
plate protection schemes [11]–[14], verification performance,
Note that the images from both VGG-Face2 and MS-Celeb-1M
non-invertibility, and cancellability. Specifically, the trade-off
are preprocessed by [38].13 The images from FRGC v2.0 and
between the verification performance and the non-invertibility
CFP are aligned with landmarks detected by MTCNN [39].
(security) is first analyzed (Section IV-B). Then, the evaluation
The images from IJB-A are aligned with the provided land-
on unlinkability (cancellability) is reported (Section IV-C).
marks. All of the images are cropped to 112 × 112 before
Section IV-D discusses the verification performance of the pro-
being used. Besides, each pixel (in [0,255]) in RGB images
posed method, the accuracy of the state-of-the-art secure face
is normalized to [-1,1] by first subtracting 127.5 and then
templates, and the face templates without protection. Finally,
dividing by 128.
the security of the proposed protected template under various
attacks are analyzed (IV-E). In the following, we describe the
13 Provided with https://github.com/deepinsight/insightface
experimental settings and report the evaluation results.

Authorized licensed use limited to: WASHINGTON UNIVERSITY LIBRARIES. Downloaded on August 25,2021 at 17:44:37 UTC from IEEE Xplore. Restrictions apply.
MAI et al.: SECUREFACE: FACE TEMPLATE PROTECTION 269

2) Verification Protocols: The evaluation in this paper bases biometric system. This is because that both the verification
on verification tasks of FRGC v2.0 [25], CFP [26], and IJB- performance and non-invertibility (security) depends on the
A [27]. For FRGC v2.0 and CFP, we report the results based on error tolerance τecc of the chosen ECC for constructing the
our constructed FVC2004 [40] like protocol with 10-fold vali- secure sketches. We analyze this trade-off using the curve
dation. Specifically, in each validation, we enroll 10% subjects of GAR @ (FAR = 0.1%) versus entropy, by varying the
with one image in the system. The genuine comparisons are τecc , where the GAR (FAR) denotes the genuine (false) accept
constructed by comparing all images (excluding the enrollment rate. In this section, how the verification performance and
image) of the enrolled subjects against the corresponding non-invertibility depend on the error tolerance τecc is first
enrollment image. The impostor comparison is constructed by elaborated. Then, the trade-off analysis of the proposed method
comparing each enrollment subject against one image of all is reported.
non-enrolled subjects. On average, for FRGC v2.0, we have 1) Verification Performance and Error Tolerance τecc :
1,556 and 19,544 genuine and impostor comparisons in each In terms of verification performance, the constructed system
fold. Note that only the frontal faces in CFP are used and based on the proposed randomized CNN can be viewed as a
there are 450 and 22,500 genuine and impostor comparisons ‘and’-based fusion of the decisions made by both the binary
in each fold. For IJB-A, we report the results based on the intermediate feature bB and the randomized template y p .
1:1 verification protocols. Different from typical verification Specifically, a genuine query image x ∗ that can be accepted
tasks that the comparison is image-to-image, the comparison by the system requires that: (a) the randomized templates for
in IJB-A is template-to-template. A template in IJB-A is either enrollment y p and query y∗p are sufficiently similar; and (b) the
a still image or a sequence of video frames. For the template distance |bB | between the intermediate features for enrollment
of video frames, we fuse them as they can be processed as b B and query b∗B is less than the error tolerance τecc (Eq. (14)).
a single image by averaging the corresponding output of the This implies that the GAR given by the intermediate feature bB
feature extraction network (h). All of the protocols used in dominates the GAR of the overall system, where the threshold
our evaluation are based on 10-fold validation. We report the is given by the error tolerance τecc . For the FAR, an impostor
average results over the 10 folds. query image can be rejected based on the intermediate feature
3) Implementation Details: We implemented the pro- bB with the comparison score threshold τecc . If the rejection
posed randomized CNN with deep learning framework is not successful, the impostor query image can be further
MXNet14 [41]. The parameters of the neural network were rejected based on the randomized template y p .
initialized using ‘Xavier’ with Gaussian random variables in 2) Security and Error Tolerance τecc : The security level
the range of [−2, 2] normalized by the number of input neu- indicates the difficulties of inverting the enrollment template
rons. The SGD with a momentum of 0.9 and weight decay of (both randomized template y p and secure sketch SS) back
5 × 10−4 is used for the optimization. To train the randomized to the input image x  . The successfully inverted input image
CNN, we first initialize the neural network parameters by x  can be used to access the system as the corresponding
pre-training on VGG-Face2 [36] with SoftMax loss. Then enrolled subject. The most straightforward way to synthesize
the randomized CNN is optimized by fine-tuning MS-Celeb- the image x  is the brute-force attack that directly guesses
1M [37] with the proposed randomized and orthogonal triplet the pixel values of the image x  . However, this is infeasible
loss. The pre-training is done with 400K batches and the because the possible combinations of the pixel values are huge,
batch size is set to 64, where the momentum is set to 0 and (112 × 112)256 for an image with size 112 × 112 as used in
the learning rate is 0.1. The finetuning is trained with 400K this work.
batches and each batch has 40 triplets, where the learning To the best of our knowledge, perhaps the most effective
rate is initialized with 0.005 and is divided by 10 at the 40K inverting strategy is to synthesize the image x  by learn-
iteration. ing a reconstruction model [3]–[5] which takes randomized
The parameters αrt and αot in Eqs.(7) and (16) are set to templates y p and secures sketches SS as input. However,
0.35. In Eq.(16), we set the λ = 2 to focus more on minimizing such reconstruction models cannot be learned directly because
the intra-subject variations and set the binarization factor ω = the randomized templates y p not only depend on the input
0.01. To demonstrate the availability of different accuracy- images x, but also the subject-specific keys k. To learn the
security trade-off, we train the proposed randomized CNN reconstruction model, the adversaries have to obtain the key k
with different orthogonal factors, μ ∈ {0.01, 0.02, 0.03, 0.04}. first. As mentioned in the second requirement in section III-C,
In the testing stage, we used the BCH [34] codes with the one could guess the key k by directly guessing k or alterna-
code length of 2047 as the ECC for generating the secure tively guessing the intermediate feature bB [42]. Therefore,
sketch. Thus, in each evaluation, we randomly partition binary the difficulties for obtaining the key k depends on the easier
intermediate feature b with 4096 elements into 2049 and way and can be expressed as
2047 elements to construct the b A and bB , respectively.
Hsys = min{m, H } (19)
B. Verification Performance Versus Non-Invertibility where m denotes the message length of the chosen ECC with
In general, there is a trade-off between the verification given error tolerance τecc and the H denotes the entropy of the
performance and the non-invertibility of the deployed secure intermediate feature bB . Assuming that the average impostor
Hamming distance (aIHD) generates from the intermediate
14 Version 0.10.0 from https://github.com/dmlc/mxnet/ feature bB obeys a binomial distribution with expectation

Authorized licensed use limited to: WASHINGTON UNIVERSITY LIBRARIES. Downloaded on August 25,2021 at 17:44:37 UTC from IEEE Xplore. Restrictions apply.
270 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 16, 2021

Fig. 6. Trade-off analysis of the verification performance and non-invertibility on FRGC v2.0, where the networks trained with different loss functions are
compared. ‘SoftMax’ (‘Triplet’) corresponds to the setting that the both loss functions (one for the binary intermediate feature and one of the final template)
are SoftMax (triplet) loss. The final template of the remaining models are trained with the proposed randomized triplet loss. The binary intermediate feature
of the ‘RandTri’ is trained with the triplet loss. For (a), the binary intermediate feature of the model named ‘μ = x’ is trained by the proposed orthogonal
triplet loss with parameter μ = x. For (b), the ‘Combined’ denotes the combined results of models named μ = 0.01, 0.02, 0.03, and 0.04.

Fig. 7. Trade-off analysis of the verification performance and non-invertibility on CFP, where the networks trained with different loss functions are compared.
‘SoftMax’ (‘Triplet’) corresponds to the setting that the both loss functions (one for the binary intermediate feature and one of the final template) are
SoftMax (triplet) loss. The final template of the remaining models are trained with the proposed randomized triplet loss. The binary intermediate feature of
the ‘RandTri’ is trained with the triplet loss. For (a), the binary intermediate feature of the model named ‘μ = x’ is trained by the proposed orthogonal triplet
loss with parameter μ = x. For (b), the ‘Combined’ denotes the combined results of models named μ = 0.01, 0.02, 0.03, and 0.04.

E H D and standard variation V H D , then the entropy H can denote the best results among the four models trained with the
be measured using the degree of freedom (DOF) [43] randomized triplet loss and the orthogonal triplet loss with the
E H D (1 − E H D ) parameter μ which is set to 0.01, 0.02, 0.03 and 0.04.
H= (20) It is observed that, given the system security Hsys , the pro-
V H2 D
posed method achieved the highest verification performance,
3) The Trade-off Analysis: According to the above analysis, GAR@(FAR = 0.1%). Note that the error tolerance τecc of
both the verification performance and security of the con- the chosen ECC and hence different security strengths is pre-
structed system depend on the error tolerance τecc of the set before deployment. Therefore, the curves of ‘Combined’
chosen ECC (BCH [34], [35] with code length 2047). For in Figs. 6(b), 7(b) and 8(b) show the verification performance
each model, we have summarized the DOF (H ), and the of the proposed method given different security strengths. At
GAR @ (FAR = 0.1%) versus the message length m of a security level of 56 bits,15 the proposed method are able
the chosen ECC by vary the τecc into Figs. 6, 7 and 8. In to achieve a GAR@(FAR = 0.1%) of 99.81%, 85.36%, and
these results, the ‘SoftMax’ (‘Triplet’) corresponds to both 78.19% on FRGC v2.0, CFP, and IJB-A, respectively. The
loss functions (one for the binary intermediate feature and one best performer of the other compared methods, ‘RandTri’, can
of the final template) for training the randomized CNN are only achieve the corresponding accuracy of 96.99%, 49.07%,
SoftMax (triplet) loss. For the remaining models (‘RandTri’ and 49.63%. Besides, it is observed that, the entropy H of
and μ = x), the loss function for training their final template the b B increases with the increase of orthogonal factor, μ.
is the proposed randomized triplet loss. The loss function This validates the effectiveness of the orthogonal terms Lort h
for training the binary intermediate feature of the ‘RandTri’ (Eq. (17)) for increasing the entropy of bB .
is the triplet loss. For Figs. 6(a), 7(a) and 8(a), the loss
function for training the models corresponded to ‘μ = x’ is
the proposed orthogonal triplet loss function with parameter 15 A security level of 53 bits is equivalent to the security strength a system
μ = x. The curves of ‘Combined’ in Figs. 6(b), 7(b) and 8(b) with an 8-character password (94-character alphabet) [23]

Authorized licensed use limited to: WASHINGTON UNIVERSITY LIBRARIES. Downloaded on August 25,2021 at 17:44:37 UTC from IEEE Xplore. Restrictions apply.
MAI et al.: SECUREFACE: FACE TEMPLATE PROTECTION 271

Fig. 8. Trade-off analysis of the verification performance and non-invertibility on IJB-A, where the networks trained with different loss functions are
compared. ‘SoftMax’ (‘Triplet’) corresponds to the setting that the both loss functions (one for the binary intermediate feature and one of the final template)
are SoftMax (triplet) loss. The final template of the remaining models are trained with the proposed randomized triplet loss. The binary intermediate feature
of the ‘RandTri’ is trained with the triplet loss. For (a), the binary intermediate feature of the model named ‘μ = x’ is trained by the proposed orthogonal
triplet loss with parameter μ = x. For (b), the ‘Combined’ denotes the combined results of models named μ = 0.01, 0.02, 0.03, and 0.04.

C. Unlinkability Analysis Therefore, we report the linkability of different settings using

This section evaluates the linkability of the templates two different metrics, i.e., A and B. Specifically, the metric A
extracted from the same subject using the proposed is for the comparison scores calculated by the cos similarity,
randomized CNN with different keys k (in different which is the comparison function of the system used in this
applications). The linkability measures how likely that two study. The metric B corresponds to the comparison scores
templates are extracted from the same subject instead of calculated by first removing sign of elements of the protected
two different subjects [48]. Note that the constructed secure templates and then sorting the elements according to the
system stores a randomized template y p (PI) and a secure numeric value. This is because that the sign and the order of
sketch SS (AD) for an enrolled subject. There are two the elements has been randomly flipped and permuted, resp.,
possible ways for the adversaries to link the subjects across by the random permutation-flip (Rand-PF). Therefore, higher
systems: either link the y p or the SS. The secure sketches SS linkability can be derived using the metric B if the random
in our construction are unlinkable because the features bB for permutation-flip is applied.
constructing the SS are formed by elements randomly selected It is observed from the Figs. 9-11 that, the linkability
from b, which is the output of the feature extraction network. of templates are fairly high while no protection is applied.
sys
The property of the linear ECC16 [11], [34] for analyzing the Specifically, the D↔ on the FRGC v2.0, CFP, and IJB-A
linkability of the typical fuzzy commitment construction [20] are 0.9987, 0.9821, and 0.9079, respectively. The linkability
is therefore not applicable to SS in our construction. of the templates is partially reduced on all three datasets if
For the linkability of the randomized templates y p , we ana- only the random activation is applied, shown in Figs. 9-11(b),
lyze it using the linkability benchmarking metrics D↔ (s) and or random permutation-flip, as shown in Figs. 9-11(c).
sys
D↔ defined in [48]. D↔ (s) is a local score-wise measure Finally, the linkability can be further reduced by using both
based on the likelihood ratio between the ‘Mated’ and ‘Non- the random activation, and the random permutation-flip.
sys
sys
Mated’ sample score distributions, and D↔ is a global Specifically, no matter which metric is used, the highest D↔
measure for benchmarking system-wise linkability. A ‘Mated’ on the FRGC v2.0, CFP, and IJB-A are 0.0139, 0.0214, and
(‘Non-Mated’) sample is a pair of enrollment and query 0.0163, respectively, as shown in Figs. 9(e), 10(d), and 11(d).
templates extracted from the same (different) subject(s). We This implies that the proposed protected template satisfies the
have analyzed the unlinkability of the templates extracted criteria of unlinkability.
from the randomized CNN with [48],17 where the results for
FRGC v2.0, CFP and IJB-A are shown in Figs. 9-11, respec- D. Discussion of the Verification Performance
tively. The ‘Unprotected’ shows the linkability of the templates
extracted without the random activation as well as the random 1) Secure Face Templates: We have summarized state-of-
permutation-flip, where the keys k are not applicable. The the-art secure face templates [8], [44]–[47] in TABLE I. Due
’RandAct’ (’RandPF’) shows the linkability of the templates to the severe tradeoff between the verification performance and
extracted with random activation (random permutation-flip) security, state-of-the-art secure face templates [8], [46], [47]
only. The ’RandAct + RandPF’ shows the linkability of are mainly evaluated using the constrained face benchmarking
the templates extracted with both randomizations. Ideally, datasets (e.g., CMU-PIE, FEI, Color FERET and FRGC).
the linkability of the randomized templates with different keys However, state-of-the-art face verification systems [49]–[55]
should be the highest one given different comparison metrics. are seldom evaluated on these constrained datasets, but on
unconstrained datasets (e.g., CFP and IJB-A). To date, a few
16 In a linear ECC, any linear combination of codewords is also a codeword. of secure templates are evaluated using unconstrained datasets
17 Code from https://github.com/dasec/unlinkability-metric.git (e.g, the homomorphic encryption based method [44] and

Authorized licensed use limited to: WASHINGTON UNIVERSITY LIBRARIES. Downloaded on August 25,2021 at 17:44:37 UTC from IEEE Xplore. Restrictions apply.
272 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 16, 2021

Fig. 9. Unlinkability analysis for the templates extracted by the proposed randomized CNN on FRGC v2.0 [25].

Fig. 10. Unlinkability analysis for the templates extracted by the proposed randomized CNN on CFP [26].

Fig. 11. Unlinkability analysis for the templates extracted by the proposed randomized CNN on IJB-A [27].

the retrieval based method [56]. However, as mentioned in a challenging key-management problem. Besides, the retrieval
Section I, the homomorphic encryption based method [44] based method [56] did not follow a popularly used evaluation
requires to store a decryption key and therefore introduces protocol that the training and testing images share no subject.

Authorized licensed use limited to: WASHINGTON UNIVERSITY LIBRARIES. Downloaded on August 25,2021 at 17:44:37 UTC from IEEE Xplore. Restrictions apply.
MAI et al.: SECUREFACE: FACE TEMPLATE PROTECTION 273

TABLE I
S TATE - OF - THE -A RT S ECURE FACE T EMPLATES

Note that the proposed method does not require to store keys TABLE II
in the system and is evaluated using practical unconstrained GAR ON IJB-A 1:1 V ERIFICATION P ROTOCOL @(FAR = 0.1%)
face benchmarking datasets with popularly used evaluation
protocol.
To compare the verification performance with state-of-
the-art secure face templates on the same dataset with the
GAR@(FAR=0.1%), we have evaluated the proposed method
on Color FERET [57] and CMU-PIE [58]. The key results
have been summarized in TABLE I, where the detail results
are included in Supplementary B. It is observed that the
proposed method outperforms Homomorphic Encryption [44]
on IJB-A, outperforms LSSC [45], [46] on Color FERET,
outperforms Deep CNN Based [8] on CMU-PIE. Besides,
the proposed method is comparable with Deep LDPC [47]
on CMU-PIE.
Note that the proposed method not only works well on
the constrained face benchmarking datasets (Color FERET,
ResNet-50 with ResNet-100 in the Feature Extraction
CMU-PIE, and FRGC v2.0) compared with the state-of-the-
Network and retrained the proposed randomized CNN,
art secure face templates [8], [45]–[47], but also works well
which corresponds to the row of Proposed (ResNet-100) in
on the unconstrained face benchmarking datasets (CFP and
TABLE II. It is observed that the Proposed (ResNet-100)
IJB-A), compared with the Homomorphic Encryption [44].
achieves a GAR of 81.9% at FAR = 0.1%. The verification
The unconstrained face recognition is getting more and
performance is well-preserved by the proposed method, with
more popular in the face recognition community. In con-
a drop of 6% compared to the state-of-the-art face templates
trast, the performance of the secure face templates [8],
without protection [55].
[45]–[47] on unconstrained face benchmarking datasets is
not known.
2) Face Templates Without Protection: TABLE II sum- E. Security of the System Under Attacks
marizes the verification performance, GAR@(FAR = 0.1%), We have quantified the security of the proposed secure
of state-of-the-art face verification templates, which were templates as Hsys in Eq. (19). However, there are attacks
evaluated using 1:1 verification protocol on the IJB-A [27]. (e.g., decodability attacks [59]–[61], attacks via record mul-
Note that the objective of this work is not the feature fusion for tiplicity [62], and similarity attacks [2], [63]) claimed to be
video-based face verification, therefore, we summarized the able to violate either unlinkability or the non-invertibility of
state-of-the-art methods which fuse the video-based templates the secure templates with reduced attacking complexity. This
using average pooling and calculated the comparison section analyzes the security of proposed secure templates
score using the cosine (L2) similarity in TABLE II for a under these attacks.
fair comparison. It is observed that the proposed method 1) Decodability Attack via Secure Sketch SS: The decod-
outperforms the methods [49]–[53] and achieves a GAR ability attack can be categorized into (conventional) decod-
of 78.2% at FAR = 0.1%. To demonstrate that the accuracy of ability attack [59] and generalized decodability attack [60],
the proposed method can be further improved by increasing the [61]. Conventional decodability attack [59] links two secure
complexity of the Feature Extraction Network, we replaced sketches SS 1 and SS 2 based on whether SS 1 ⊕ SS 2 = c1 ⊕

Authorized licensed use limited to: WASHINGTON UNIVERSITY LIBRARIES. Downloaded on August 25,2021 at 17:44:37 UTC from IEEE Xplore. Restrictions apply.
274 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 16, 2021

Fig. 12. Mated/Non-Mated hamming distance distribution on IJB-A with the model trained with μ = 0.2, where 2047 bits are chosen from b (contains
4096 bits) to form the bB . (a) direct comparison, b B , (b) Comparison with reversing bit-permutation, b B ∗ .

c2 ⊕ bB can be decoded, where SS = bB ⊕ c. It has been random bit-permutation. The generalized decodability attack
shown in [59], [60] that, with a high probability, SS 1 ⊕ SS 2 can eliminate the effect of random bit-permutation, while the
is decodable if bB ≤ τecc and vice versa. The work in [59] effect of random bit-selection cannot be eliminated. Therefore,
has shown that the conventional decodability attack [59] can perhaps the calculation of the minimum bB ∗ corresponds
be prevented by bit-permutation randomization. The gener- to the generalized decodability attack is: (a) first to reverse
alized decodability attack [61] is proposed against the bit- the bit-permutation process: reorder elements in bB,1 and b B,2
permutation based prevention. The basic idea of the gener- such that the bits come from the same indices in b are placed
alized decodability attack is to construct a new ECC based in corresponding indices in bB,1∗ and bB,2∗ ; (b) calculate
on the chosen ECC of two different applications and their bB ∗ = bB,1∗ ⊕ bB,2∗ . Note that due to the random bit-
corresponding bit-permutation matrices. Let bB,1 = P 1 bB,1∗, selection process, part of elements in bB,1 and bB,2 do not
and bB,2 = P 2 bB,2∗ , the generalized decodability attack can share the bit indices in b. The distribution of bB ∗ for both
determine whether bB,1 and bB,2 come from the same subject, mated and non-mated samples are shown in Figure 12(b),
if bB ∗ = bB,1∗ ⊕ bB,2∗ ≤ τecc with attacking complexity where the minimum bB for mate samples is 527. The
corresponding complexity
  of generalized decodability attack is
b
  
B ∗
n C (2047, 527) > 2047527 ≈ 2 1679 , which is computationally dif-
C n, bB ∗ = (21) ficult. Furthermore, considering an extreme case with reduced
r
r=1 attacking complexity that the bB,1 and bB,2 are extracted
where P 1 (P 2 ) denotes a permutation matrix to permute from the same input image, the elements in bB,1 and bB,2
bB,1∗(bB,2∗) to be bB,1(bB,2). share the bit indices in b are identical and therefore the error
The proposed secure face templates are secure under both occurs between the elements in bB,1 and bB,2 do not share
the conventional decodability attack [59] and generalized the bit indices in b. Let n share denotes the number of the bits
decodability attack [60], [61]. This is because that the binary with shared indices, then the corresponding attack complexity
feature bB for constructing our secure sketches SS is obtained is reduced to C (2047 − n share , 527). In our experiments,
by a user and application specific random partition (as the maximum n share is 1090 and C (2047 − 1090, 527) ≥
described in Section III-C). The proposed random partition 2956, which is much higher than the attacking complexity of
can be viewed as that the bB is obtained by a cascade of guessing the key k or the feature bB . Therefore, the gen-
bit-selection and bit-permutation on the binary intermediate eralized decodability attack could not reduce the attacking
feature b. complexity of the proposed secure templates.
With the random partition, the conventional decodability 2) Attacks via Record Multiplicity (ARM): Attacks via
attack [59] is not able to tell whether two secure sketches record multiplicity (ARM) [62] aims to construct images x̂
come from the same subject or not. This is because that the that can be used to access the system as the target user,
corresponding bB are large enough for both mate and non- given multiple compromised protected templates. In our cases,
mated samples (mated sample denotes that two bB come from the protected template stored in the system is the PI ( y p )
the same subject). Besides, the distribution of bB for both and AD ( pr pt and SS). Perhaps the most effective way for
mated and non-mated samples are similar (see Figure 12(a)), attacking the proposed method is first to estimate binary
where the minimum bB for mate samples is 942. Note feature b A and bB from the y p and SS. Then the b can be
that the maximum error tolerance τecc is 511 for BCH codes reconstructed with pr pt and the input image x can be recon-
with n = 2047. It has also been justified in [59] that the structed using [3]. However, it is computationally difficult
conventional decodability attack [59] can be prevented by bit- to estimate the b A from the y p because the y p is extracted
permutation randomization. from b A with randomness specified by a randomly generated
The generalized decodability attack [60], [61] is not able to key k, which is not stored in the system. Besides, the SS is
tell whether two secure sketches come from the same subject constructed using the bB and the key k and it is difficult to
or not because of the random partition, which, as mentioned obtain bB without k. Therefore, the availability of the multiple
above, can be viewed as a cascade of random bit-selection and compromised protected templates cannot help in reducing the

Authorized licensed use limited to: WASHINGTON UNIVERSITY LIBRARIES. Downloaded on August 25,2021 at 17:44:37 UTC from IEEE Xplore. Restrictions apply.
MAI et al.: SECUREFACE: FACE TEMPLATE PROTECTION 275

attacking complexity of the proposed method. The attacking of 0.1%. The corresponding GAR on CFP and IJB-A are
complexity of the proposed method under ARM is, therefore, 85.36% and 78.19%, respectively.
lower bounded by the effort of exhaustively searching the key
k. This implies that the ARM could not reduce the attacking ACKNOWLEDGMENT
complexity of the proposed secure templates.
3) Similarity Attack via Protected Templates y p : Similarity The authors would like to thank Dr. Anil K. Jain,
attack [2], [63] aims to estimate a pre-image x̂ which is Dr. Jiawei Li, and Dr. Mang Ye for their suggestions. The
similar to the enrollment image x such that majority of this work was done when Guangcan Mai was a
 the comparison
score of the corresponding PIs, scor e y p yˆp , y p , less than Ph.D. candidate at Hong Kong Baptist University.
the system decision threshold, given the AD (i.e., SS and
pr pt ) of the system. The key of a success similarity attack R EFERENCES
is the similarity preserving
  property of theinput and output [1] G. Mai, M.-H. Lim, and P. C. Yuen, “Binary feature fusion for
space, i.e., scor e y p yˆp , y p ≈ scor e x x̂, x . In the proposed discriminative and secure multi-biometric cryptosystems,” Image Vis.
method, the similarity preserving property is not satisfied if the Comput., vol. 58, pp. 254–265, Feb. 2017.
estimated image x̂ is not sufficiently similar to the enrollment [2] Y. C. Feng, M.-H. Lim, and P. C. Yuen, “Masquerade attack on
image x. Thanks to the random activation and the random transform-based binary-template protection based on perceptron learn-
ing,” Pattern Recognit., vol. 47, no. 9, pp. 3019–3033, Sep. 2014.
permutation-flip (as mentioned in Section III-B), the extracted [3] G. Mai, K. Cao, P. C. Yuen, and A. K. Jain, “On the reconstruction
protection template y p not only depends on the biometric of face images from deep face templates,” IEEE Trans. Pattern Anal.
input x, but also a randomly generated user specific key k, Mach. Intell., vol. 41, no. 5, pp. 1188–1202, May 2019.
[4] K. Cao and A. K. Jain, “Learning fingerprint reconstruction: From
which is not stored in the system but encoded into the secure minutiae to image,” IEEE Trans. Inf. Forensics Security, vol. 10, no. 1,
sketch SS. Note that the key k can be successfully decoded pp. 104–117, Jan. 2015.
from SS only if the query estimated image x̂ is sufficiently [5] J. Galbally, A. Ross, M. Gomez-Barrero, J. Fierrez, and
J. Ortega-Garcia, “Iris image reconstruction from binary templates:
similar to the enrollment x. If the key k cannot be successfully An efficient probabilistic approach based on genetic algorithms,”
decoded, there is no advantage to tell whether two protection Comput. Vis. Image Understand., vol. 117, no. 10, pp. 1512–1525,
templates ( y p and yˆp ) come from the same subject or not. This Oct. 2013.
[6] W. Stallings, Cryptography and Network Security: Principles and Prac-
can also be justified by the “Mated” (in blue dash line) and tice, vol. 7. London, U.K.: Pearson, 2016.
“Non-Mated” (in blue line) sample score distribution for the [7] R. K. Pandey, Y. Zhou, B. U. Kota, and V. Govindaraju, “Deep secure
protection templates, as shown in Figs. 9-11(d). In a nutshell, encoding for face template protection,” in Proc. IEEE Conf. Comput.
Vis. Pattern Recognit. Workshops (CVPRW), Jun. 2016, pp. 77–83.
the similarity attack can break the security of the proposed [8] A. K. Jindal, S. Chalamala, and S. K. Jami, “Face template protec-
scheme only if (i) the initial guess of the input image xˆ0 is tion using deep convolutional neural network,” in Proc. IEEE/CVF
sufficiently similar to the enrollment image x or (ii) the key k Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW), Jun. 2018,
pp. 462–470.
is known. These two scenarios can be viewed as a brute-force [9] C. Gentry, A Fully Homomorphic Encryption Scheme. Stanford, CA,
attack on the enrollment image x, or the attack of the key k. USA: Stanford Univ., 2009.
The former is infeasible due to the large search space, and the [10] M. Albrecht et al., “Homomorphic encryption security standard,” Homo-
morphicEncryption.org, Toronto, ON, Canada, Tech. Rep., Nov. 2018.
security of the latter is described in Eq. (19). This implies that [11] K. Nandakumar and A. K. Jain, “Biometric template protection: Bridg-
the similarity attack could not reduce the attacking complexity ing the performance gap between theory and practice,” IEEE Signal
of the proposed secure templates. Process. Mag., vol. 32, no. 5, pp. 88–100, Sep. 2015.
[12] Information Technology—Security Techniques—Biometric Information
V. C ONCLUSIONS Protection, Standard ISO/IEC 24745:2011, 2011.
[13] A. K. Jain, K. Nandakumar, and A. Ross, “50 years of biomet-
In this paper, we propose a novel method which can ric research: Accomplishments, challenges, and opportunities,” Pattern
construct protected biometric systems whose stored deep Recognit. Lett., vol. 79, pp. 80–105, Aug. 2016.
[14] Y. C. Feng, P. C. Yuen, and A. K. Jain, “A hybrid approach for generating
templates are non-invertible, cancellable and discriminative. secure and discriminating face template,” IEEE Trans. Inf. Forensics
A randomized CNN is proposed to generates secure deep Security, vol. 5, no. 1, pp. 103–117, Mar. 2010.
biometric templates based on both the input biometric data [15] M. A. Turk and A. P. Pentland, “Face recognition using Eigenfaces,” in
Proc. CVPR, 1991, pp. 302–306.
(e.g., face image) and user-specific keys. In our construction, [16] K. Cao and A. K. Jain, “Automated latent fingerprint recognition,”
no user-specific key is stored in the system, where a secure IEEE Trans. Pattern Anal. Mach. Intell., vol. 41, no. 4, pp. 788–800,
sketch generated from both a user-specific key and an inter- Apr. 2019.
[17] Y.-L. Lai et al., “Cancellable iris template generation based on indexing-
mediate feature which is stored in the enrollment stage. In first-one hashing,” Pattern Recognit., vol. 64, pp. 105–117, Apr. 2017.
the query stage, the user-specific key can be decoded from a [18] Z. Jin, J. Y. Hwang, Y.-L. Lai, S. Kim, and A. B. J. Teoh, “Ranking-
stored secure sketch if the query image is sufficiently close to based locality sensitive hashing-enabled cancelable biometrics: Index-
of-Max hashing,” IEEE Trans. Inf. Forensics Security, vol. 13, no. 2,
the corresponding enrollment image. Experimental results and pp. 393–407, Feb. 2018.
analysis on three face benchmark datasets (FRGC v2.0, CFP, [19] H. Kaur and P. Khanna, “Random distance method for generating
and IJB-A) show that the protected templates in the proposed unimodal and multimodal cancelable biometric features,” IEEE Trans.
Inf. Forensics Security, vol. 14, no. 3, pp. 709–719, Mar. 2019.
construction are non-invertible and unlinkable. Furthermore, [20] A. Juels and M. Wattenberg, “A fuzzy commitment scheme,” in Proc.
the verification performance of our protected templates is well 6th ACM Conf. Comput. Commun. Secur. (CCS). New York, NY,
preserved. Specifically, at a security level of 56-bits (stronger USA: Association for Computing Machinery, 1999, pp. 28–36, doi:
10.1145/319709.319714.
than an 8-character password system), we achieve state-of- [21] A. Juels and M. Sudan, “A fuzzy vault scheme,” Designs, Codes
the-art accuracy (GAR) on FRGC v2.0 0,of 99.81% at a FAR Cryptography, vol. 38, no. 2, pp. 237–257, Feb. 2006.

Authorized licensed use limited to: WASHINGTON UNIVERSITY LIBRARIES. Downloaded on August 25,2021 at 17:44:37 UTC from IEEE Xplore. Restrictions apply.
276 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 16, 2021

[22] Y. Dodis, R. Ostrovsky, L. Reyzin, and A. Smith, “Fuzzy extractors: [48] M. Gomez-Barrero, J. Galbally, C. Rathgeb, and C. Busch, “Gen-
How to generate strong keys from biometrics and other noisy data,” eral framework to evaluate unlinkability in biometric template pro-
SIAM J. Comput., vol. 38, no. 1, pp. 97–139, Jan. 2008. tection systems,” IEEE Trans. Inf. Forensics Security, vol. 13, no. 6,
[23] A. Nagar, K. Nandakumar, and A. K. Jain, “Multibiometric cryptosys- pp. 1406–1420, Jun. 2018.
tems based on feature-level fusion,” IEEE Trans. Inf. Forensics Security, [49] X. Yin and X. Liu, “Multi-task convolutional neural network for pose-
vol. 7, no. 1, pp. 255–268, Feb. 2012. invariant face recognition,” IEEE Trans. Image Process., vol. 27, no. 2,
[24] T. Stanko, F. Nur Andini, and B. Skoric, “Optimized quantization in pp. 964–975, Feb. 2018.
zero leakage helper data systems,” IEEE Trans. Inf. Forensics Security, [50] X. Yin, X. Yu, K. Sohn, X. Liu, and M. Chandraker, “Towards large-
vol. 12, no. 8, pp. 1957–1966, Aug. 2017. pose face frontalization in the wild,” in Proc. IEEE Int. Conf. Comput.
[25] P. J. Phillips et al., “Overview of the face recognition grand challenge,” Vis. (ICCV), Oct. 2017, pp. 3990–3999.
in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. [51] S. Sankaranarayanan, A. Alavi, C. D. Castillo, and R. Chellappa, “Triplet
(CVPR), 2005, pp. 947–954. probabilistic embedding for face verification and clustering,” in Proc.
[26] S. Sengupta, J.-C. Chen, C. Castillo, V. M. Patel, R. Chellappa, and IEEE 8th Int. Conf. Biometrics Theory, Appl. Syst. (BTAS), Sep. 2016,
D. W. Jacobs, “Frontal to profile face verification in the wild,” in Proc. pp. 1–8.
IEEE Winter Conf. Appl. Comput. Vis. (WACV), Mar. 2016, pp. 1–9. [52] J. Yang et al., “Neural aggregation network for video face recognition,”
[27] B. F. Klare et al., “Pushing the frontiers of unconstrained face detection in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017,
and recognition: IARPA janus benchmark a,” in Proc. IEEE Conf. pp. 4362–4371.
Comput. Vis. Pattern Recognit. (CVPR), Jun. 2015, pp. 1931–1939. [53] L. Tran, X. Yin, and X. Liu, “Representation learning by rotating
[28] Information Technology—Biometric Data Interchange Formats—Part 2: your faces,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 41, no. 12,
Finger Minutiae Data, Standard ISO/IEC 19794-2:2011, 2011. pp. 3007–3021, Dec. 2019.
[29] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for [54] X. Yin, X. Yu, K. Sohn, X. Liu, and M. Chandraker, “Feature transfer
image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. learning for face recognition with under-represented data,” in Proc.
(CVPR), Jun. 2016, pp. 770–778. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2019,
[30] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and pp. 5704–5713.
R. Salakhutdinov, “Dropout: A simple way to prevent neural networks [55] B. Yin, L. Tran, H. Li, X. Shen, and X. Liu, “Towards interpretable
from overfitting,” J. Mach. Learn. Res., vol. 15, no. 1, pp. 1929–1958, face recognition,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV),
2014. Oct. 2019, pp. 9347–9356.
[31] F. Schroff, D. Kalenichenko, and J. Philbin, “FaceNet: A unified [56] Y. K. Jang and N. I. Cho, “Deep face image retrieval for cancelable
embedding for face recognition and clustering,” in Proc. IEEE Conf. biometric authentication,” in Proc. 16th IEEE Int. Conf. Adv. Video
Comput. Vis. Pattern Recognit. (CVPR), Jun. 2015, pp. 815–823. Signal Based Surveill. (AVSS), Sep. 2019, pp. 1–8.
[32] S. Liao, Z. Lei, D. Yi, and S. Z. Li, “A benchmark study of large- [57] P. J. Phillips, H. Moon, S. A. Rizvi, and P. J. Rauss, “The FERET
scale unconstrained face recognition,” in Proc. IEEE Int. Joint Conf. evaluation methodology for face-recognition algorithms,” IEEE Trans.
Biometrics, Sep. 2014, pp. 1–8. Pattern Anal. Mach. Intell., vol. 22, no. 10, pp. 1090–1104, Oct. 2000.
[33] E. Learned-Miller, G. B. Huang, A. RoyChowdhury, H. Li, and G. Hua, [58] T. Sim, S. Baker, and M. Bsat, “The CMU pose, illumination, and
“Labeled faces in the wild: A survey,” in Proc. Adv. Face Detection expression (PIE) database,” in Proc. FG, 2002, pp. 53–58.
Facial Image Anal., 2016, pp. 189–248. [59] E. J. C. Kelkboom, J. Breebaart, T. A. M. Kevenaar, I. Buhan, and
R. N. J. Veldhuis, “Preventing the decodability attack based cross-
[34] R. Roth, Introduction to Coding Theory. Cambridge, U.K.:
matching in a fuzzy commitment scheme,” IEEE Trans. Inf. Forensics
Cambridge Univ. Press, 2006.
Security, vol. 6, no. 1, pp. 107–121, Mar. 2011.
[35] J. H. Van Lint, Introduction to Coding Theory, vol. 86. Berlin,
[60] K. Simoens, P. Tuyls, and B. Preneel, “Privacy weaknesses in biomet-
Germany: Springer-Verlag, 2012. [Online]. Available: https://www.
ric sketches,” in Proc. 30th IEEE Symp. Secur. Privacy, May 2009,
springer.com/gp/book/9783662001745
pp. 188–203.
[36] Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman, “Vggface2: [61] B. Tams, “Decodability attack against the fuzzy commitment scheme
A dataset for recognising faces across pose and age,” in Proc. FG, 2018, with public feature transforms,” 2014, arXiv:1406.1154. [Online]. Avail-
pp. 67–74. able: http://arxiv.org/abs/1406.1154
[37] Y. Guo, L. Zhang, Y. Hu, X. He, and J. Gao, “MS-Celeb-1M: A dataset [62] W. J. Scheirer and T. E. Boult, “Cracking fuzzy vaults and biometric
and benchmark for large-scale face recognition,” in Proc. ECCV, 2016, encryption,” in Proc. Biometrics Symp., Sep. 2007, pp. 1–6.
pp. 87–102. [63] Y. Chen, Y. Wo, R. Xie, C. Wu, and G. Han, “Deep secure quantization:
[38] J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “ArcFace: Additive angu- On secure biometric hashing against similarity-based attacks,” Signal
lar margin loss for deep face recognition,” 2018, arXiv:1801.07698. Process., vol. 154, pp. 314–323, Jan. 2019.
[Online]. Available: http://arxiv.org/abs/1801.07698
[39] K. Zhang, Z. Zhang, Z. Li, and Y. Qiao, “Joint face detection and
alignment using multitask cascaded convolutional networks,” IEEE
Signal Process. Lett., vol. 23, no. 10, pp. 1499–1503, Oct. 2016.
[40] D. Maltoni, D. Maio, A. K. Jain, and S. Prabhakar, Handbook of Guangcan Mai (Member, IEEE) received the
Fingerprint Recognition. London, U.K.: Springer-Verlag, 2009. [Online]. B.Eng. degree from the South China University
Available: https://www.springer.com/gp/book/9781848822535 of Technology, China, in 2013, and the Ph.D.
[41] T. Chen et al., “MXNet: A flexible and efficient machine learning degree from the Department of Computer Science,
library for heterogeneous distributed systems,” 2015, arXiv:1512.01274. Hong Kong Baptist University, Hong Kong, in 2018.
[Online]. Available: http://arxiv.org/abs/1512.01274 In 2016, he was a Visiting Scholar with Michi-
[42] G. Mai, M.-H. Lim, and P. C. Yuen, “On the guessability of binary gan State University, East Lansing, MI, USA. His
biometric templates: A practical guessing entropy based approach,” in research interests include biometric security, com-
Proc. IEEE Int. Joint Conf. Biometrics (IJCB), Oct. 2017, pp. 367–374. puter vision, and machine learning.
[43] J. Daugman, “The importance of being random: Statistical principles
of iris recognition,” Pattern Recognit., vol. 36, no. 2, pp. 279–291,
Feb. 2003.
[44] V. N. Boddeti, “Secure face matching using fully homomorphic encryp- Kai Cao received the Ph.D. degree from the
tion,” in Proc. BTAS, 2018, pp. 1–10. Key Laboratory of Complex Systems and Intel-
[45] M.-H. Lim and A. B. J. Teoh, “A novel encoding scheme for effec- ligence Science, Institute of Automation, Chinese
tive biometric discretization: Linearly separable subcode,” IEEE Trans. Academy of Sciences, Beijing, China, in 2010.
Pattern Anal. Mach. Intell., vol. 35, no. 2, pp. 300–313, Feb. 2013. His research interests include biometric recognition,
[46] P. Drozdowski, F. Struck, C. Rathgeb, and C. Busch, “Benchmarking image processing, and machine learning.
binarisation schemes for deep face templates,” in Proc. 25th IEEE Int.
Conf. Image Process. (ICIP), Oct. 2018, pp. 191–195.
[47] L. Chen, G. Zhao, J. Zhou, A. T. S. Ho, and L.-M. Cheng, “Face template
protection using deep LDPC codes learning,” IET Biometrics, vol. 8,
no. 3, pp. 190–197, May 2019.

Authorized licensed use limited to: WASHINGTON UNIVERSITY LIBRARIES. Downloaded on August 25,2021 at 17:44:37 UTC from IEEE Xplore. Restrictions apply.
MAI et al.: SECUREFACE: FACE TEMPLATE PROTECTION 277

Xiangyuan Lan received the B.Eng. degree from Pong C. Yuen (Senior Member, IEEE) is currently
the South China University of Technology, China, the Chair Professor of the Department of Computer
in 2012, and the Ph.D. degree from the Hong Kong Science, Hong Kong Baptist University. He is also
Baptist University, Hong Kong, in 2016. In 2015, the Vice President (Technical Activities) of the IEEE
he was a Visiting Scholar with the University of Biometrics Council, an Editorial Board Member
Maryland, College Park, MD, USA. He is currently of Pattern Recognition and a Senior Editor of the
a Research Assistant Professor with Hong Kong Journal of Electronic Imaging (SPIE). He also serves
Baptist University. His current research interests as a Hong Kong Research Grant Council Engineer-
include sparse representation and deep learning for ing Panel Member. His current research interests
computer vision and pattern recognition problems. include video surveillance, human face recognition,
biometric security, and privacy.

Authorized licensed use limited to: WASHINGTON UNIVERSITY LIBRARIES. Downloaded on August 25,2021 at 17:44:37 UTC from IEEE Xplore. Restrictions apply.