On sequence kernels for SVM

classification of sets of vectors:

application to speaker verification

Jérôme Louradour

CNRS-IRIT (SAMOVA Group)

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 1/29
Introduction

Text independent speaker verification

A binary classification task
Determine if a speech sequence has been uttered by a target speaker

Classical approach
“target speaker”
DECISION
test sequence
? SCORING MAKING
or
“impostor”
Classifier

FRONT-END
target speaker
GMM

TRAINING

impostors, UBM
“background”

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 2/29
Introduction

Text independent speaker verification

A binary classification task
Determine if a speech sequence has been uttered by a target speaker

Classical approach : UBM-GMM system

“target speaker”
DECISION
test sequence
? SCORING MAKING
or
“impostor”
Classifier

FRONT-END
target speaker
GMM
GMM

TRAINING

impostors, UBM
UBM
“background”

Front-end processing: acoustic vectors on frames

Probabilistic GMM modeling (generative)

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 2/29
Introduction

Text independent speaker verification

A binary classification task
Determine if a speech sequence has been uttered by a target speaker

Classical approach : UBM-GMM system

“target speaker”
DECISION
test sequence
? SCORING MAKING
or
“impostor”
Classifier

FRONT-END
target speaker
GMM
GMM

TRAINING

impostors, UBM
UBM
“background”

Front-end processing: acoustic vectors on frames

Probabilistic GMM modeling (generative)
y Motivation : apply SVM, powerful for binary classification
Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 2/29
Introduction

Support Vector Machines (SVM)

+ Theoretical power
+ Core algorithm well mastered
+ Good performance for binary classification

For speech processing . . .

– How to handle dynamic data?
– Large training database ⇒ Time & Memory consuming
– Bad results of SVM applied at frame-level

y Conception and study of kernels between sequences

for speaker verification

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 3/29
Sequence kernels

Outline

1 Sequence kernels

2 A novel sequence kernel

3 Experimental evaluation on a speaker verification task

4 Kernel between pairs of sequences for speaker verification

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 4/29
Sequence kernels
Principle

Basics on kernels
Similarity measure
Mercer property: symetric, positive definite
=⇒ k(x, y) = φ(x)> φ(y)
φ : expansion in a Feature Space RD (dimension D ≤ +∞)
Example for vectors: noyau Gaussien
kx−yk2
− 2
k(x, y) = e 2ρ
small ρ

Vecteurs d’apprentissage SVM model

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 5/29
Sequence kernels
Principle

Basics on kernels
Similarity measure
Mercer property: symetric, positive definite
=⇒ k(x, y) = φ(x)> φ(y)
φ : expansion in a Feature Space RD (dimension D ≤ +∞)
Example for vectors: noyau Gaussien
kx−yk2
− 2
k(x, y) = e 2ρ
high ρ

Vecteurs d’apprentissage SVM model

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 5/29
Sequence kernels
Principle

Basics on kernels
Similarity measure
Mercer property: symetric, positive definite
=⇒ k(x, y) = φ(x)> φ(y)
φ : expansion in a Feature Space RD (dimension D ≤ +∞)
Example for vectors: noyau Gaussien
kx−yk2
− 2
k(x, y) = e 2ρ
√
ρ= dσ

Vecteurs d’apprentissage SVM model (small C )

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 5/29
Sequence kernels
Principle

Basics on kernels
Similarity measure
Mercer property: symetric, positive definite
=⇒ k(x, y) = φ(x)> φ(y)
φ : expansion in a Feature Space RD (dimension D ≤ +∞)
Example for vectors: noyau Gaussien
kx−yk2
− 2
k(x, y) = e 2ρ
√
ρ= dσ

Vecteurs d’apprentissage SVM model (middle C )

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 5/29
Sequence kernels
Principle

Basics on kernels
Similarity measure
Mercer property: symetric, positive definite
=⇒ k(x, y) = φ(x)> φ(y)
φ : expansion in a Feature Space RD (dimension D ≤ +∞)
Example for vectors: noyau Gaussien
kx−yk2
− 2
k(x, y) = e 2ρ
√
ρ= dσ

Vecteurs d’apprentissage SVM model (high C )

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 5/29
Sequence kernels
Principle

Kernels between sets of vectors

Text-independent speaker recognition

Audio sequence = “Variable-length set of acoustic vectors
- same speaker,
- same recording session”

Three famillies of kernels

1 Mutual Information kernels
Based on a priori data distribution
2 Kernels between probability densities
Sequence 7→ distribution
3 Combination of vector kernels
Function of kernel values between inter(intra)-sequence elements

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 6/29
Sequence kernels
Principle

Mutual Information kernels

Exploit a prior generative model (UBM
with parameters θo estimated on unlabeled data

Fisher kernel [Jaakkola and Haussler, 1998]

Approximation of a Mutual Information kernel [Seeger, 2002]

> −1
κ(X, Y) = φ(X) S φ(Y)
φ(X) = ∇θ log p(X|θ)|θ=θo (Fisher expansion)
S : second moments of φ (Fisher information matrix)

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 7/29
Sequence kernels
Principle

Mutual Information kernels

Exploit a prior generative model (UBM
with parameters θo estimated on unlabeled data

Fisher kernel [Jaakkola and Haussler, 1998]

Approximation of a Mutual Information kernel [Seeger, 2002]

> −1
κ(X, Y) = φ(X) S φ(Y)
φ(X) = ∇θ log p(X|θ)|θ=θo (Fisher expansion)
S : second moments of φ (Fisher information matrix)

2 Gaussians GMM
Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 7/29
Sequence kernels
Principle

Mutual Information kernels

Exploit a prior generative model (UBM
with parameters θo estimated on unlabeled data

Fisher kernel [Jaakkola and Haussler, 1998]

Approximation of a Mutual Information kernel [Seeger, 2002]

> −1
κ(X, Y) = φ(X) S φ(Y)
φ(X) = ∇θ log p(X|θ)|θ=θo (Fisher expansion)
S : second moments of φ (Fisher information matrix)

Distance between Fisher expansions

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 7/29
Sequence kernels
Principle

Kernels between probability densities

GGGGGGGGGGGGGGGGGA p
Apprentissage
X, Y X , pY
learning

probability product kernels [Jebara and Kondor, 2003]

Z
κ(X, Y) = pX (z)q pY (z)q dz

Analytic form a GMM with degree q = 1 [Lyu, 2005]

“spherical” normalization for more robustness :
κ(X,Y)
κ̊(X,Y)= √
κ(X,X)κ(Y,Y)

Exponential embedding of divergences

D(pX ,pY )2
2ρ2 −
Analogous of the Gaussien kernel: κ(X, Y) = e
D : Distance between GMMs, approximation of KL divergence [Do,

y “supervectors GMM” Approach

2003]
[Campbell et al., 2006]

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 8/29
Sequence kernels
Principle

Combination of “static” kernels

X = { xt | t = 1 · · · TX }
Sequences of vector
Y = { yt 0 | t 0 = 1 · · · TY }

Similarity Similarity between

w
between vectors sets of vectors
>
k(x, y) = φ(x) φ(y) κ(X, Y)
Mercer kernel in terms of k(xt , yt 0 ),
k(xt , xt 0 ), k(yt , yt 0 ), . . .

Simple example: TX X
TY
X
1
κ(X, Y) = TX TY k(xt , yt 0 )
t=1 t 0 =1
complexity O(T 2 ) for each kernel computation

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 9/29
Sequence kernels
Principle

Combination of “static” kernels: GLDS kernel

 >  P 
1
P −1 1
κ(X, Y) = TX t φq (xt ) S B TY t 0 φq (yt 0)

φq : Polynomial expansion (Rd → RD )

D = (q+d)!
q!d! monoms of degrees ≤ q
SB : Second moment matrix of φq estimated on B

Normalizing with SB :
- GLDS kernel ∼ train on X (discriminant model) & test on Y
- Same amplitude of each feature

Explicit map:
+ High efficiency for testing (linear SVM model)
– Impossibility of using an expansion φq with high or infinite dimension
(in practice, max degree q = 3)

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 10/29
Sequence kernels
Principle

Combination of “static” kernels: GLDS kernel

 >  P 
1
P −1 1
κ(X, Y) = TX t φq (xt ) S B TY t 0 φq (yt 0)

φq : Polynomial expansion (Rd → RD )

D = (q+d)!
q!d! monoms of degrees ≤ q
SB : Second moment matrix of φq estimated on B

Normalizing with SB :
- GLDS kernel ∼ train on X (discriminant model) & test on Y
- Same amplitude of each feature

Explicit map:
+ High efficiency for testing (linear SVM model)
– Impossibility of using an expansion φq with high or infinite dimension
(in practice, max degree q = 3)

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 10/29
A novel sequence kernel

Outline

1 Sequence kernels

2 A novel sequence kernel

3 Experimental evaluation on a speaker verification task

4 Kernel between pairs of sequences for speaker verification

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 11/29
A novel sequence kernel
Definition

Extension of the GLDS kernel: FSNS kernels

FSNS kernels (Feature Space Normalized Sequence kernels)
 P >
−1  1 P 
κ(X, Y) = T1X t φ(xt ) SB + εI TY t 0 φ(yt 0 )

φ : any Mercer expansion

SB : second moments matrix of φ
Regularization ε > 0 : necessary if φ is of high dimension

y

avoid to compute φ
Objectif :
rewrite using the Mercer kernel k = φ> φ

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 12/29
A novel sequence kernel
Definition

Extension of the GLDS kernel: FSNS kernels

FSNS kernels (Feature Space Normalized Sequence kernels)
 P >
−1  1 P 
κ(X, Y) = T1X t φ(xt ) SB + εI TY t 0 φ(yt 0 )

φ : any Mercer expansion

SB : second moments matrix of φ
Regularization ε > 0 : necessary if φ is of high dimension

y

avoid to compute φ
Objectif :
rewrite using the Mercer kernel k = φ> φ
FSMS kernels (Feature Space Mahalanobis Sequence kernels)
 P >
−1  1 P 
κ(X, Y) = T1X t φ(xt ) ΣB + εI TY t 0 φ(yt 0 )
ΣB : covariance matrix of φ
SVM are invariant to translations in the Feature Space
⇒ same as FSNS with centering of φ
kernel ∼ KL divergence between Gaussians in the Feature Space
Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 12/29
A novel sequence kernel
Dual Form

Dual form of FSNS kernels

Background (unlabeled) data: B = { bi | i = 1 · · · N}

. &
Gram matrix Empirical map
   
k(b1.,b1 ) ··· k(b1.,bN ) k(b1 , x)
.. .. ..
K= k(bi ,bj )  ψB (x) =  .

k(bN ,b1 ) ··· k(bN ,bN ) k(bN , x)

Proposition (without regularization, ε = 0)

 P >  P 
κ(X, Y) = T1X t φ(xt ) SB −1 T1Y t 0 φ(yt 0 )
 P >
 1 P 
1 2 −1
= T1X t ψB (xt ) NK TY t 0 ψB (yt )
0

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 13/29
A novel sequence kernel
Dual Form

Dual form of FSNS kernels

Background (unlabeled) data: B = { bi | i = 1 · · · N}

. &
Gram matrix Empirical map
   
k(b1.,b1 ) ··· k(b1.,bN ) k(b1 , x)
.. .. ..
K= k(bi ,bj )  ψB (x) =  .

k(bN ,b1 ) ··· k(bN ,bN ) k(bN , x)

Proposition
 >
−1  1 P 
1
P
κ(X, Y) = t φ(xt ) SB + εI t 0 φ(yt )
0
TX TY
 >
−1  1 P 
1 1 2
P
= t ψB (xt ) N K + εK t 0 ψB (yt )
0
TX TY

Hypothesis : the φ(bi ) span the training “φ(x)”

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 13/29
A novel sequence kernel
Dual Form

Dual form of FSNS kernels

Background (unlabeled) data: B = { bi | i = 1 · · · N}

. &
Gram matrix Empirical map
   
k(b1.,b1 ) ··· k(b1.,bN ) k(b1 , x)
.. .. ..
K= k(bi ,bj )  ψB (x) =  .

k(bN ,b1 ) ··· k(bN ,bN ) k(bN , x)

Proposition (with centering)

 P >
−1  1 P 
κ(X, Y) = T1X t φ(xt ) ΣB + εI TY t 0 φ(yt )
0

 P >
−1  1 P 
= T1X t ψB (xt ) 1
N KΠK + εK TY t 0 ψB (yt )
0

Hypothesis : the φ(bi ) span the training “φ(x)”

1
Centering : Π = I − N1 (instead of I)

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 13/29
A novel sequence kernel
Dual Form

Computational complexity
Dot product of normalized expansions
>
κ(X, Y) = φ(X) MB φ(Y) = Uφ(X) , Uφ(Y) (1)
>
= ψB (X) RB ψB (Y) = VψB (X) , VψB (Y) (2)

form (1) form (2)

3
Pre-computation of U/V O(D ) O(N 3 )
Sequence expansion Uφ/VψB O(TD 2 ) O(T N 2 )
Dot product comput. O(D) O(N)
D : dimension of the Feature Space (size of φ)
N : number of background vectors (size of ψ)

+ Possibility to use expansions φ of infinite dim (Gaussian kernel)

– Complexity problem for large databases

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 14/29
A novel sequence kernel
Dual Form

Computational complexity
Dot product of normalized expansions
>
κ(X, Y) = φ(X) MB φ(Y) = Uφ(X) , Uφ(Y) (1)
>
= ψB (X) RB ψB (Y) = VψB (X) , VψB (Y) (2)

form (1) form (2)

3
Pre-computation of U/V O(D ) O(N 3 )
Sequence expansion Uφ/VψB O(TD 2 ) O(T N 2 )
Dot product comput. O(D) O(N)
D : dimension of the Feature Space (size of φ)
N : number of background vectors (size of ψ)

+ Possibility to use expansions φ of infinite dim (Gaussian kernel)

– Complexity problem for large databases
y Objective : find an appropriate approximation
Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 14/29
A novel sequence kernel
Approximation

Kernel Approximation

Goal
1 Reduce the size of the empirical map ψ
2 Keep a maximum of information

ICD : Incomplete Cholesky Decomposition [Fine and Scheinberg, 2001]

1 Selection of a sub-population
 of backround vectors:
codebook C = bp1 · · · bpi · · · bpm ⊂ B
(taille m)
index I = { p1 · · · pi · · · pm } ⊂ {1···N}
2 Low-rank approximation of the Gram matrix:
−1 >
K ≈ LI = K(:, I)K(I, I) K(:, I) (rang m)

P 2
min tr K − LI ≡ min kφ(bi ) − φC (bi )k
I C

+ Memory and CPU

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 15/29
A novel sequence kernel
Approximation

Kernel Approximation

Goal
1 Reduce the size of the empirical map ψ
2 Keep a maximum of information

ICD : Incomplete Cholesky Decomposition

Background data
Codebook

ICD, Gaussian kernel

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 16/29
A novel sequence kernel
Approximation

Approximate Form of FSNS kernels

Proposition

w
ICD
−1 >
K K(:, I)K(I, I) K(:, I)

κ(X, Y) = ψB (X) RB ψB (Y)

>
w κ(X, Y) ≈ ψC (X) RB×C ψC (Y)
>

Expansion of size m  N :  
k(bp1 , xt )
..
ψC (X) = T1X t 
P
.

k(bpm , xt )
´−1
RB×C = 1
N
K(:, I)ΠK(:, I)> + εK(I, I)

Complexity form (1) form (2) approx form

2 2
Expansion norm. O(TD ) O(TN ) O(Tm2 )
Dot product O(D) O(N) O(m)

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 17/29
Experimental evaluation on a speaker verification task

Outline

1 Sequence kernels

2 A novel sequence kernel

3 Experimental evaluation on a speaker verification task

4 Kernel between pairs of sequences for speaker verification

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 18/29
Experimental evaluation on a speaker verification task

Data
Speech corpus: NIST Speaker Recognition Evaluation
Development :
Background corpus (1/2)
Validation corpus (2/2)
- Hyper-parameters of kernels
- Parameter C (SVM learning)
- Decision threshold
Evaluation :
∼ 18000 tests, 400 target speakers
DCF = τFR Ploc FR% + τFA Pimp FA%

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 19/29
Experimental evaluation on a speaker verification task

Data
Speech corpus: NIST Speaker Recognition Evaluation
Development :
Background corpus (1/2)
Validation corpus (2/2)
- Hyper-parameters of kernels
- Parameter C (SVM learning)
- Decision threshold
Evaluation :
∼ 18000 tests, 400 target speakers
DCF = τFR Ploc FR% + τFA Pimp FA%

Front-end processing
SVM GMM
acoustic vectors MFCC +∆MFCC +∆logE LFCC +∆LFCC +∆logE
silence removal unsupervised clustering of the energy
normalization feature warping centring-reduction
Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 19/29
Experimental evaluation on a speaker verification task

Development of FSNS kernels

* No gain in performance when centering / regularizing

kx−yk2
− 2
* Best performance with a Gaussian RBF kernel: k(x, y) = e 2ρ
√
. . . and a good tuning: ρ = dσ

* Increasing the codebook size improves the performance

. . . up to a certain point (m ≈ 5000)

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 20/29
Experimental evaluation on a speaker verification task

Evaluation: FSNS kernels vs. State-of-the-art

Validation (NIST 2003/2004) Evaluation (NIST 2005)

EER(%) DCFmin EER DCF

(1) GLDS SVM 12.80 51.9 (1) GLDS SVM 12.54 48.8
(2) FSNS SVM 10.55 47.5 (2) FSNS SVM 11.91 41.6
(3) UBM-GMM 11.48 49.1 (3) UBM-GMM 12.06 40.6

* FSNS kernels: improvement w.r.t. GLDS kernel

* SVM systems competitive with GMM systems

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 21/29
Experimental evaluation on a speaker verification task

Evaluation: Sequence kernels

EER DCF
Probability product
13.92 52.1
kernel SVM
GLDS kernel SVM 12.54 48.8
Fisher kernel SVM 11.90 44.0
FSNS kernel SVM 11.91 41.6
UBM-GMM (ref) 12.06 40.6
GMM supervectors
10.40 37.7
SVM

* Best results: exploiting generative GMM within discriminative SVM

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 22/29
Experimental evaluation on a speaker verification task

Evaluation: Fusion

EER DCF
(1) FSNS kernel
11.91 41.6
SVM
(2) UBM-GMM 12.06 40.6
(3) GMM
10.40 37.7
supervectors SVM
(2+3) fusion no improvement
(1+2) fusion 9.71 37.0
(1+3) fusion 10.28 36.1

* Improvement when fusing GMM-based & “pure” SVM approaches

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 23/29
Kernel between pairs of sequences for speaker verification

Outline

1 Sequence kernels

2 A novel sequence kernel

3 Experimental evaluation on a speaker verification task

4 Kernel between pairs of sequences for speaker verification

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 24/29
Kernel between pairs of sequences for speaker verification
Principle

Another classification approach

TEST
? Pair of
sequences
?

FRONT-END
+1
same speaker ?
-1
Pair-of-
Sequences
= +1
MODEL
TRAINING
≠ -1

A single “universal” SVM system (efficiency?)

New kind of expertise (complementary to traditional approaches?)
SVM training: no disbalance +1/ − 1
y Conceiving a kernel between pairs of sequences
Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 25/29
Kernel between pairs of sequences for speaker verification
Principle

A kernel between pairs of sequences

Principe
pairs of
dissimilar GMMs
1 Map pairs of GMM in a suitable
vectorial space
2 Use a vectorial kernel pairs of
similar GMMs

w Learn a distance between GMM appropriate for speaker recognition

Each mapping component =q distance between Gaussian components
>
φc (GMMX , GMMY ) = (µX − µY ) Σ−1 (µX − µY )
D-MAP normalization of GMM [Ben et Bimbot, 2003]
kφ (GMM, UBM) k = 1/2
⇒ kφ (GMMX , GMMY ) k ∈ [0, 1]
Gaussian kernel
Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 26/29
Kernel between pairs of sequences for speaker verification
Principle

A kernel between pairs of sequences

Principe
pairs of
1 dissimilar GMMs
1 Map pairs of GMM in a suitable
vectorial space
2 Use a vectorial kernel pairs of
similar GMMs

w Learn a distance between GMM appropriate for speaker recognition

1

Each mapping component =q distance between Gaussian components

>
φc (GMMX , GMMY ) = (µX − µY ) Σ−1 (µX − µY )
D-MAP normalization of GMM [Ben et Bimbot, 2003]
kφ (GMM, UBM) k = 1/2
⇒ kφ (GMMX , GMMY ) k ∈ [0, 1]
Gaussian kernel
Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 26/29
Kernel between pairs of sequences for speaker verification
Experiments

Evaluation

EER DCF
Probability product
13.92 52.1
kernel SVM
GLDS kernel SVM 12.54 48.8
Pair-of-Sequences
11.58 46.0
kernel SVM
Fisher kernel SVM 11.90 44.0
FSNS kernel SVM 11.91 41.6
UBM-GMM (ref) 12.06 40.6
GMM supervectors
10.40 37.7
SVM

* Not so bad for a first try. . .

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 27/29
Kernel between pairs of sequences for speaker verification
Experiments

Evaluation

EER DCF
Pair-of-Sequences
11.58 46.0
kernel SVM
FSNS kernel SVM 11.91 41.6
Fusion 10.93 40.5

* Complementary to traditional approaches

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 28/29
Conclusion

Perspectives

SVM with sequence kernels

Model adaptation
Kernel for high-level features, kernel combination
How to handle at best SVM scores

Kernel between Pairs-of-Sequences

Other kernels?
Extension when several training sequence per target speaker

Kernels for sequence applied to Speaker Verification with SVM Jérôme Louradour 29/29