SimpleMKKM: Simple Multiple Kernel K-means

刘新旺

国防科技大学 计算机学院

https://xinwangliu.github.io/

2021 年 10 月 13 日

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 1 / 51

 Introduction

Content

1 Introduction

2 SimpleMKKM: Simple Multiple Kernel K-means

3 Localized SimpleMKKM

4 Parameter-free Localized SimpleMKKM

5 Conclusion and Future Work

6 Related References

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 2 / 51

 Introduction

Clustering

Clustering has been intensively studied during the last few decades,
and has been widely used in various applications.
Widely used clustering algorithms: (Kernel) K-means, Spectral
Clustering, Gaussian Mixture Model (GMM) Clustering, density-based
clustering, · · ·
The crucial factor affecting clustering performance is similarity among
data!
Similarity is dependent on features describing samples. One kind of
features (such as HOG, SIFT, et.) may not sufficiently describe
samples.
Why not try multiple kinds of features? Termed as multiple kernel
(multi-view) clustering in machine learning community.

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 3 / 51

 Introduction

Multiple Kernel Clustering (MKC)

Samples are represented by multiple groups (kinds) of features.

For example, color, shape and texture are extracted to distinguish one
flower from another (Nilsback, M-E and Zisserman, A., CVPR 2008).
These different sources (kinds of features) usually provide
complementary information, and it is better to let learning algorithms
optimally combine them in order to obtain excellent clustering
performance.
This line of research is known as multiple kernel (view) clustering in
the literature.

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 4 / 51

 Introduction

Application

Medical
Care

Auto
Pilot

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 5 / 51

 Introduction

Multiple Kernel K-means (MKKM)

The optimization objective of MKKM is
 
min min Tr Kγ (I − HH ) s.t. H⊤ H = Ik ,
⊤
(1)
γ∈∆ H
P Pm
where ∆ = {γ ∈ Rm | m p=1 γp = 1, γp ≥ 0, ∀p}, Kγ =
2
p=1 γp Kp and
m
{Kp }p=1 are m pre-specified kernel matrices.
Alternate Optimization Framework
Optimizing H given γ. With the kernel coefficients γ fixed, H is
obtained by solving a kernel k-means clustering optimization problem;
Optimizing γ given H. With H fixed, γ is optimized via solving the
following quadratic programming with linear constraints,
Xm  
min γp2 Tr Kp (I − HH⊤ ) , (2)
γ∈∆ p=1

which has a closed-form solution.

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 6 / 51
 Introduction

Recent Work on MKKM

Many efforts have been devoted to improving MKKM in Eq. (1). For
example,
Optimized Data Fusion for Kernel k-Means Clustering (Yu, Shi et. al
TPAMI 2012)
Localized multiple kernel k-means (LMKKM) (Gonen, Mehmet et. al
NIPS 2014)
Multiple Kernel K-means with Incomplete Kernels (Xinwang Liu. al
TPAMI 2019)
Late Fusion Incomplete Multi-view Clustering (Xinwang Liu. al
TPAMI 2019)
Efficient and Effective Incomplete Multi-view Clustering (Xinwang
Liu. al TPAMI 2020)
···

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 7 / 51

 SimpleMKKM: Simple Multiple Kernel K-means

Content

1 Introduction

2 SimpleMKKM: Simple Multiple Kernel K-means

3 Localized SimpleMKKM

4 Parameter-free Localized SimpleMKKM

5 Conclusion and Future Work

6 Related References

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 8 / 51

 SimpleMKKM: Simple Multiple Kernel K-means New Formulation

Content

1 Introduction

2 SimpleMKKM: Simple Multiple Kernel K-means

New Formulation
Generalization Error Analysis
Experimental Results

3 Localized SimpleMKKM

4 Parameter-free Localized SimpleMKKM

5 Conclusion and Future Work

6 Related References

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 9 / 51

 SimpleMKKM: Simple Multiple Kernel K-means New Formulation

The New Formulation

 
min max Tr Kγ HH⊤ s.t. H⊤ H = Ik , (3)
γ∈∆ H
P Pm
where ∆ = {γ ∈ Rm | m p=1 γp = 1, γp ≥ 0, ∀p} and Kγ =
2
p=1 γp Kp .
The new formulation is a minimize-maximize optimization, which can be
rewritten as a minimization problem,

min J (γ), (4)

γ∈∆

with    
J (γ) = max Tr Kγ HH⊤ s.t. H⊤ H = Ik . (5)
H

Eq. (14) can be efficiently solved with reduced sub-gradient descent

algorithm (Rakotomamonjy et. al, JMLR 2008).
刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 10 / 51
 SimpleMKKM: Simple Multiple Kernel K-means New Formulation

The Differentiability and its Computation

The Differentiability of J (γ)


J (γ) in Eq. (4) is differentiable. Furthermore, ∂J∂γ(γ) = 2γp Tr Kp H∗ H∗ ⊤ ,

p

where H∗ = arg maxH Tr Kγ HH⊤ s.t. H⊤ H = Ik .

5J (γ): The Reduced Gradient of J (γ)

The p-th element of 5J (γ) is [5J (γ)]p = ∂J (γ)
− ∂J (γ)
(∀p 6= u), and
Pm   ∂γp ∂γu
∂J (γ) ∂J (γ)
[5J (γ)]u = p=1,p̸=u ∂γu − ∂γp .

The descent direction for updating γ is



0 if γp = 0 and [5J (γ)]p > 0
dp = − [5J (γ)]p if γp > 0 and p = 6 u (6)


− [5J (γ)]u if p = u.
γ ← γ + αd, where α is the optimal step size.
刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 11 / 51
 SimpleMKKM: Simple Multiple Kernel K-means New Formulation

The Overall Algorithm Framework

Algorithm 1 SimpleMKKM
1: Input: {Kp }m p=1 , k, t = 1.
2: Initialize γ (1) = 1/m, flag = 1.
3: while flag do ∑ ( (t) )2
4: compute H by solving a kernel k-means with Kγ (t) = mp=1 γp Kp .
5: compute ∂J (γ)
∂γp
(p = 1, · · · , m) and the descent direction d(t) in Eq. (6).
6: update γ (t+1)
← γ (t) + αd(t) .
7: if max |γ (t+1)
− γ (t) | ≤ 1e − 4 then
8: flag=0.
9: end if
10: t ← t + 1.
11: end while

定理 (The Convexity of J (γ))

J (γ) in Eq. (4) is convex. As a result, the solution obtained by the

proposed SimpleMKKM in Algorithm 1 is the global optimum.
刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 12 / 51
 SimpleMKKM: Simple Multiple Kernel K-means Generalization Error Analysis

Content

1 Introduction

2 SimpleMKKM: Simple Multiple Kernel K-means

New Formulation
Generalization Error Analysis
Experimental Results

3 Localized SimpleMKKM

4 Parameter-free Localized SimpleMKKM

5 Conclusion and Future Work

6 Related References

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 13 / 51

 SimpleMKKM: Simple Multiple Kernel K-means Generalization Error Analysis

The Generalization Analysis

Let us define a function class first:

n 

F = f : x 7→ 1 − maxy∈Θ hϕγ (x), CyiHk γ ⊤ 1m = 1, γp ≥ 0,
o (7)
C ∈ Hk , |Kp (x, x̃)| ≤ b, ∀p, ∀x ∈ X ,

where Hk stands for the multiple kernel Hilbert space.

The Generalization Error Bound of the Learned Cluster
For any δ > 0, with probability at least 1 − δ, the following holds for all
f ∈ F:
p r
1X
n
π/2bk log 1/δ
E [f(x)] ≤ f(xi ) + √ + (1 + b) . (8)
n n 2n
i=1

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 14 / 51

 SimpleMKKM: Simple Multiple Kernel K-means Experimental Results

Content

1 Introduction

2 SimpleMKKM: Simple Multiple Kernel K-means

New Formulation
Generalization Error Analysis
Experimental Results

3 Localized SimpleMKKM

4 Parameter-free Localized SimpleMKKM

5 Conclusion and Future Work

6 Related References

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 15 / 51

 SimpleMKKM: Simple Multiple Kernel K-means Experimental Results

The Overall Clustering Performance

Datasets AvgKKM MKKM LMKKM ONKC MKKM-MiR LKAM LF-MVC MKKM-MM SimpleMKKM
ACC

Flo17 51.0± 1.3 43.6± 1.7 42.7± 1.5 43.4± 2.1 58.0± 1.2 48.9± 0.9 57.2± 1.3 51.0±1.3 59.1± 1.2
Flo102 27.1± 0.8 22.4± 0.5 - 39.2± 0.9 39.1± 1.3 40.4± 1.0 29.0± 1.0 27.1±0.8 42.5± 0.8
PFold 29.0± 1.6 27.0± 1.1 22.4± 0.7 35.3± 1.3 34.3± 1.7 33.8± 1.7 31.6± 1.7 29.0±1.6 34.7± 1.9
CCV 19.6± 0.6 18.0± 0.5 18.6± 0.1 22.1± 0.6 20.9± 0.9 18.9± 0.3 23.1± 0.9 19.6±0.6 22.2± 0.7
Digit 88.8± 0.1 47.3± 0.7 47.3± 0.7 89.5± 0.1 87.4± 0.1 95.0± 0.1 89.1± 0.1 88.8±0.7 90.3± 0.1
Avg. 43.1 31.7 - 45.9 47.9 47.4 46.0 43.1 49.8
NMI

Flo17 49.6± 0.8 44.3± 1.3 43.8± 1.0 43.1± 1.3 56.2± 0.6 48.2± 0.6 54.6± 0.9 49.7±0.8 57.5± 0.8
Flo102 46.0± 0.5 42.7± 0.2 - 55.7± 0.4 55.9± 0.6 55.8± 0.3 47.5± 0.3 46.0±0.5 58.6± 0.5
PFold 40.3± 1.2 38.0± 0.6 34.7± 0.6 44.0± 0.8 43.1± 1.0 43.6± 1.0 41.8± 0.9 40.3±1.3 44.4± 1.1
CCV 16.8± 0.4 15.1± 0.5 14.4± 0.1 18.4± 0.3 17.9± 0.4 16.8± 0.2 19.3± 0.3 16.8±0.4 18.2± 0.3
Digit 80.8± 0.2 48.8± 0.7 48.7± 0.7 81.7± 0.1 79.6± 0.1 89.4± 0.1 81.1± 0.2 80.8±0.2 83.3± 0.1
Avg. 46.7 37.8 - 48.6 50.5 50.8 48.9 46.7 52.4
Purity

Flo17 52.0± 1.0 45.1± 1.4 44.5± 1.4 45.2± 1.9 59.4± 0.9 50.1± 0.6 58.1± 1.4 52.0±1.0 60.5± 1.4
Flo102 32.3± 0.6 27.8± 0.4 - 45.1± 0.9 45.2± 1.0 46.7± 0.6 34.5± 0.5 32.3±0.6 48.6± 0.7
PFold 37.4± 1.7 33.7± 1.1 31.2± 1.0 41.9± 1.0 41.2± 1.4 41.6± 1.3 38.9± 1.5 37.4±1.7 41.8± 1.5
CCV 23.8± 0.5 22.2± 0.5 22.0± 0.1 24.3± 0.5 23.4± 0.7 22.2± 0.3 26.1± 0.5 23.8±0.5 25.3± 0.5
Digit 88.8± 0.1 50.1± 0.7 50.1± 0.7 89.5± 0.1 87.4± 0.1 95.0± 0.1 89.1± 0.1 88.8± 0.1 90.3± 0.1
Avg. 46.9 35.8 - 49.2 51.3 51.1 49.3 46.9 53.3
Rand Index

Flo17 32.3± 1.0 26.4± 1.3 26.0± 1.1 24.3± 1.6 39.6± 0.8 30.2± 0.8 38.6± 1.0 32.3±1.3 41.3± 1.1
Flo102 15.5± 0.5 12.1± 0.4 - 24.5± 0.6 24.9± 1.0 26.3± 0.6 17.2± 0.8 15.5±0.5 28.5± 0.8
PFold 14.4± 1.8 12.1± 0.7 7.8± 0.4 17.6± 1.3 17.4± 1.6 17.3± 1.7 16.2± 1.7 14.4±1.8 17.6± 1.9
CCV 6.6± 0.2 5.8± 0.2 5.6± 0.1 7.5 ± 0.3 7.0± 0.4 6.2± 0.1 8.4± 0.5 6.6 ±0.2 7.5± 0.2
Digit 77.5± 0.2 31.4± 0.6 31.3± 0.6 78.7± 0.1 75.4± 0.1 89.2± 0.1 78.2± 0.2 77.5±0.2 80.3± 0.1
Avg. 29.3 17.6 - 30.5 32.9 33.8 31.7 29.3 35.0

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 16 / 51

 SimpleMKKM: Simple Multiple Kernel K-means Experimental Results

What Factors Lead to Improved Performance?

Differ from Formulations and Solving Algorithms.

MKKM SimpleMKKM
Min-Min: Min-Max:
   
min min Tr Kγ (I − HH⊤ ) min max Tr Kγ HH⊤
γ∈∆ H γ∈∆ H
⊤
s.t. H H = Ik , s.t. H⊤ H = Ik ,
(9) (10)

Coordinate Descent Reduced Gradient Descent

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 17 / 51

 SimpleMKKM: Simple Multiple Kernel K-means Experimental Results

Ablation Study
Datasets MKKM+C MKKM+R SimpleMKKM+C SimpleMKKM+R
ACC
Flo17 43.6 43.7 54.2 59.1
Flo102 22.4 22.4 41.8 42.5
UCI-Dig 47.3 47.3 90.4 90.3
ProFold 27.0 26.6 29.0 34.7
CCV 18.0 17.9 22.1 22.2
Cal102-30 16.6 16.7 30.4 30.6

SimpleMKKM+C v.s. MKKM+C (SimpleMKKM+R v.s.

MKKM+R): The proposed Min-Max formulation is much better than
Min-Min.
SimpleMKKM+R v.s. SimpleMKKM+C: Reduced gradient descent
optimization is usually better than the widely used coordinate descent.
Novel Formulation + New Optimization!

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 18 / 51

 SimpleMKKM: Simple Multiple Kernel K-means Experimental Results

Clustering Performance with Number of Samples

Caltech102-30 Caltech102-30
0.32 0.32

0.3 0.3

0.28 0.28

0.26 0.26
ACC

NMI
0.24 0.24

0.22 0.22
LKAM
LKAM LF-MVC
Avg-KKM Avg-KKM
0.2 LF-MVC 0.2 MKKM-MM
MKKM MKKM
MKKM-MM SimpleMKKM
0.18 ONKC 0.18 ONKC
SimpleMKKM
0.16 MKKM-MiR 0.16 MKKM-MiR
10 15 20 25 30 35 40 45 10 15 20 25 30 35 40 45

Number of Kernels Number of Kernels

Caltech102-30 Caltech102-30
0.32 0.32

0.3 0.3

Rand Index
0.28 0.28
Purity

0.26 0.26

0.24 0.24
LKAM LKAM
0.22 0.22
LF-MVC LF-MVC
0.2 Avg-KKM MKKM-MM 0.2 Avg-KKM MKKM-MM
MKKM SimpleMKKM MKKM SimpleMKKM
0.18 0.18
ONKC ONKC
0.16 MKKM-MiR 0.16 MKKM-MiR
10 15 20 25 30 35 40 45 10 15 20 25 30 35 40 45

Number of Kernels Number of Kernels

图 1: The clustering performance with the variation of number of samples on

Caltech102.
刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 19 / 51
 SimpleMKKM: Simple Multiple Kernel K-means Experimental Results

The Learned Kernel Weights

Flower17 Flower102 CCV

1 1

0.8
0.8 0.8 0.7
Kernel Weights

Kernel Weights

Kernel Weights
0.6
0.6 0.6
0.5

0.4
0.4 0.4
0.3

0.2 0.2 0.2

0.1

0 0 0
KM KC -M
iR AM VC -M
M
KK
M KM KC -M
iR AM VC -M
M
KK
M KM KC -M
iR AM VC -M
M
KK
M
MK ON KM LK LF
-M KM pleM MK ON KM LK LF
-M KM pleM MK ON KM LK LF
-M KM pleM
MK MK MK MK MK MK
Sim Sim Sim

Caltech102-30 UCI_DIGIT Protein Fold

0.7 1

0.4
0.6
0.8 0.35
0.5
Kernel Weights

Kernel Weights

Kernel Weights
0.3
0.6
0.4 0.25

0.3 0.2
0.4
0.15
0.2
0.2 0.1
0.1
0.05

0 0 0
KM KC -M
iR AM VC -M
M
KK
M KM KC -M
iR AM VC -M
M
KK
M KM KC -M
iR AM VC -M
M
KK
M
MK ON KM LK LF
-M KM pleM MK ON KM LK LF
-M KM pleM MK ON KM LK LF
-M KM pleM
MK MK MK MK MK MK
Sim Sim Sim

图 2: The kernel weights learned by different algorithms.

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 20 / 51

 SimpleMKKM: Simple Multiple Kernel K-means Experimental Results

The Evolution of the Learned H

Flower17 Flower17 Flower17
0.6 0.58
0.6
0.58
0.56

0.56
0.54 MKKM
MKKM SimpleMKKM
0.54 MKKM 0.55
SimpleMKKM
SimpleMKKM

Purity
0.52
ACC

NMI
0.52

0.5 0.5

0.5
0.48 0.48

0.46
0.46
0.44
0.45
0 5 10 15 20 25 30 0 5 10 15 20 25 30 0 5 10 15 20 25 30

Number of Iterations Number of Iterations Number of Iterations

Flower102 Flower102 Flower102
0.42 0.58 0.48

0.4 0.46
0.56 MKKM
0.38 MKKM 0.44 MKKM
SimpleMKKM 0.54 SimpleMKKM SimpleMKKM
0.36 0.42

Purity
0.52 0.4
ACC

0.34
NMI

0.32 0.38
0.5

0.3 0.36
0.48
0.28 0.34

0.46 0.32
0.26

0.44 0.3
0.24
0.28
0 5 10 15 20 25 30 0 5 10 15 20 25 30 0 5 10 15 20 25 30

Number of Iterations Number of Iterations Number of Iterations

图 3: The clustering comparison of the learned H by MKKM and the proposed

SimpleMKKM with iterations.

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 21 / 51

 SimpleMKKM: Simple Multiple Kernel K-means Experimental Results

The Convergence Curves

Flower17 Flower102 CCV

37 520 1162
SimpleMKKM
36 SimpleMKKM 1160 SimpleMKKM
500
35
1158
Objective Value

Objective Value

Objective Value
34 480
1156
33
460 1154
32
1152
31 440
1150
30
420 1148
29

28 1146
400

0 5 10 15 20 25 30 0 5 10 15 20 25 30 0 5 10 15 20 25 30

Number of Iterations Number of Iterations Number of Iterations

Caltech102-30 UCI_DIGIT Protein Fold
21.5

21
SimpleMKKM 405
19
SimpleMKKM
400 SimpleMKKM
20.5
395 18.5
Objective Value

Objective Value

Objective Value
20
390
19.5 18
385
19
380 17.5
18.5
375
18 17
370
17.5
365 16.5
17

0 5 10 15 20 25 30 0 5 10 15 20 25 30 0 5 10 15 20 25 30

Number of Iterations Number of Iterations Number of Iterations

图 4: The objective of SimpleMKKM decreases with iterations.

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 22 / 51

 Localized SimpleMKKM

Content

1 Introduction

2 SimpleMKKM: Simple Multiple Kernel K-means

3 Localized SimpleMKKM

4 Parameter-free Localized SimpleMKKM

5 Conclusion and Future Work

6 Related References

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 23 / 51

 Localized SimpleMKKM Motivation

Content

1 Introduction

2 SimpleMKKM: Simple Multiple Kernel K-means

3 Localized SimpleMKKM
Motivation
Formulation
Experimental Results

4 Parameter-free Localized SimpleMKKM

5 Conclusion and Future Work

6 Related References

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 24 / 51

 Localized SimpleMKKM Motivation

Global and Local Kernel Alignment

SimpleMKKM applies “Global Kernel Alignment” which

indiscriminately force all sample pairs to be equally aligned with the
same ideal similarity.
Improve similarity matrices by “Local kernel Alignment” which
aligns with k-nearest neighbours.

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 25 / 51

 Localized SimpleMKKM Formulation

Content

1 Introduction

2 SimpleMKKM: Simple Multiple Kernel K-means

3 Localized SimpleMKKM
Motivation
Formulation
Experimental Results

4 Parameter-free Localized SimpleMKKM

5 Conclusion and Future Work

6 Related References

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 26 / 51

 Localized SimpleMKKM Formulation

Formulation of the Proposed Localized SimpleMKKM

The local alignment for the i-th sample is defined as follows,

D ⊤ ⊤
E
S(i) Kγ S(i) , S(i) H⊤ HS(i) , (11)
F

⊤
where S(i) Kγ S(i) denotes taking elements from Kγ according to the
neighborhood of the i-th sample.
Then we obtain the objective function of the proposed localized
SimpleMKKM as follow:
 Xn 
minγ∈∆ maxH∈Rn×k Tr H⊤ (A(i) Kγ A(i) )H
i=1 (12)
⊤
s.t. H H = Ik ,
Pm Pm
where ∆ = {γ ∈ Rm | p=1 γp = 1, γp ≥ 0, ∀p}, Kγ = 2
p=1 γp Kp and
⊤
A(i) = S(i) S(i) is the neighborhood mask matrix of the i-th sample.
刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 27 / 51
 Localized SimpleMKKM Formulation

Theoretical Observation
定理 (Connection Between SimpleMKKM and Localized SimpleMKKM)
The objective of SimpleMKKM is a special case of Eq. (12).
证明.
The objective in Eq. (12) can be written as
∑n ( ) ∑n ⟨ ⟩
Tr H⊤ (A(i) Kγ A(i) )H = A(i) ⊗ Kγ , A(i) ⊗ (HH⊤ )
i=1 i=1 F
∑n ⟨ (i) ⊤
⟩ ⟨(∑n )
⊤
⟩
= A ⊗ Kγ , HH = A (i)
⊗ Kγ , HH
i=1 i=1
∑m ⟨(∑n )F ⟩ F
(13)
= γp2 A(i) ⊗ Kp , HH⊤
p=1 i=1 F
∑m ⟨ ⟩ ( )
= γp2 K̃p , HH⊤ = Tr H⊤ K̃γ H ,
p=1


F
Pn
where K̃p = A(i) ⊗ Kp can be treated as a normalized Kp , and
P i=1
K̃γ = m γ 2 K̃ .This completes the proof.
p=1 p p

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 28 / 51

 Localized SimpleMKKM Formulation

Proposed Localized SimpleMKKM

Based on Theorem 2, our formulation in Eq. (12) can be equivalently

rewritten as,
minγ∈∆ J (γ), (14)
with
   
⊤ ⊤
J (γ) = max Tr H K̃γ H , s.t. H H = Ik . (15)
H

Eq. (14) can be readily solved via SimpleMKKM in Algorithm 1 with

normalized kernel matrices {K̃p }m
p=1 , where K̃p = M ⊗ Kp with
Xn
M= A(i) . (16)
i=1

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 29 / 51

 Localized SimpleMKKM Experimental Results

Content

1 Introduction

2 SimpleMKKM: Simple Multiple Kernel K-means

3 Localized SimpleMKKM
Motivation
Formulation
Experimental Results

4 Parameter-free Localized SimpleMKKM

5 Conclusion and Future Work

6 Related References

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 30 / 51

 Localized SimpleMKKM Experimental Results

Clustering Performance

Datasets Avg-KKM MKKM LMKKM ONKC MKKM-MiR LKAM LF-MVC MKKM-MM SimpleMKKM LSMKKM
ACC
MSRA 83.3 ± 0.8 81.3 ± 3.1 81.9 ± 0.7 77.4 ± 1.6 87.9 ± 0.6 72.5 ± 0.5 84.2 ± 0.5 83.3 ± 0.8 86.5 ± 0.2 91.2 ± 0.3
Still 31.3 ± 0.7 31.3 ± 0.6 31.1 ± 0.5 31.2 ± 1.3 31.5 ± 1.0 31.6 ± 0.2 31.7 ± 0.4 31.3 ± 0.7 31.3 ± 0.6 36.5 ± 0.7
CAL-7 59.2 ± 4.9 52.2 ± 4.3 53.9 ± 1.0 60.2 ± 1.6 68.3 ± 0.0 63.9 ± 3.1 65.2 ± 7.1 59.2 ± 4.9 68.2 ± 1.5 76.5 ± 1.8
PFD 29.0 ± 1.5 27.0 ± 1.1 22.4 ± 0.7 35.3 ± 1.3 34.3 ± 1.7 31.7 ± 2.1 32.6 ± 1.9 29.0 ± 1.5 34.7 ± 1.9 35.9 ± 1.3
Nonpl 49.7 ± 0.2 49.3 ± 0.2 - 49.9 ± 0.0 50.3 ± 0.2 53.8 ± 0.0 46.7 ± 0.6 49.7 ± 0.2 52.0 ± 0.0 69.3 ± 0.2
FLO17 50.8 ± 1.5 44.9 ± 2.4 37.5 ± 1.6 42.9 ± 2.1 57.7 ± 1.2 39.6 ± 1.6 54.3 ± 1.8 50.8 ± 1.5 59.5 ± 1.3 61.3 ± 1.4
FLO102 27.1 ± 0.8 22.4 ± 0.5 - 39.2 ± 0.9 39.1 ± 1.3 40.0 ± 1.0 43.2 ± 1.3 27.1 ± 0.8 42.5 ± 0.8 44.0 ± 1.0
Reuters 45.5 ± 1.5 45.4 ± 1.5 - 40.9 ± 2.1 39.7 ± 1.5 40.0 ± 2.2 45.4 ± 1.7 45.5 ± 1.5 45.5 ± 0.7 46.6 ± 0.6
NMI
MSRA 74.0 ± 1.0 73.2 ± 1.7 75.0 ± 1.4 62.4 ± 2.2 77.3 ± 1.1 62.7 ± 0.9 73.2 ± 0.3 74.0 ± 1.0 75.2 ± 0.5 82.6 ± 0.5
Still 12.8 ± 0.8 13.0 ± 0.8 13.2 ± 0.5 12.7 ± 0.4 12.9 ± 0.3 13.6 ± 0.2 11.4 ± 0.3 12.8 ± 0.8 12.8 ± 1.0 14.2 ± 0.3
CAL-7 59.1 ± 2.9 51.6 ± 4.1 52.1 ± 1.3 53.7 ± 1.5 63.8 ± 0.0 59.2 ± 1.5 65.1 ± 2.6 59.1 ± 2.9 63.7 ± 0.3 74.6 ± 1.2
PFD 40.3 ± 1.3 38.0 ± 0.6 34.7 ± 0.6 44.0 ± 0.8 43.1 ± 1.0 42.5 ± 1.2 41.7 ± 1.1 40.3 ± 1.3 44.4 ± 1.1 45.2 ± 0.9
Nonpl 17.2 ± 0.5 15.0 ± 0.5 - 14.4 ± 0.0 14.9 ± 0.0 9.2 ± 0.0 9.8 ± 0.2 17.2 ± 0.5 11.2 ± 0.0 22.6 ± 0.1
FLO17 49.7 ± 1.0 44.9 ± 1.5 38.8 ± 1.1 43.0 ± 1.4 57.2 ± 0.8 42.4 ± 0.8 54.1 ± 0.9 49.7 ± 1.0 57.8 ± 0.9 59.4 ± 0.7
FLO102 46.0 ± 0.5 42.7 ± 0.2 - 55.7 ± 0.4 55.9 ± 0.6 55.4 ± 0.4 57.8 ± 0.6 46.0 ± 0.5 58.6 ± 0.5 60.0 ± 0.4
Reuters 27.4 ± 0.4 27.3 ± 0.4 - 21.0 ± 1.8 21.3 ± 1.3 21.5 ± 2.3 27.2 ± 0.2 27.4 ± 0.4 27.7 ± 0.2 28.3 ± 0.6
Purity
MSRA 83.3 ± 0.8 81.5 ± 2.7 81.9 ± 0.7 77.4 ± 1.6 87.9 ± 0.6 72.5 ± 0.5 84.2 ± 0.5 83.3 ± 0.8 86.5 ± 0.2 91.2 ± 0.3
Still 33.8 ± 0.8 33.8 ± 0.7 33.3 ± 0.5 33.9 ± 0.9 34.0 ± 0.8 35.7 ± 0.3 34.8 ± 0.4 33.8 ± 0.8 33.8 ± 0.7 38.2 ± 0.6
CAL-7 68.0 ± 3.2 63.8 ± 3.7 66.4 ± 0.6 67.1 ± 1.3 72.1 ± 0.0 70.7 ± 2.0 76.7 ± 3.2 68.0 ± 3.2 72.3 ± 0.2 82.3 ± 1.4
PFD 37.4 ± 1.7 33.7 ± 1.1 31.2 ± 1.0 41.9 ± 1.0 41.2 ± 1.4 39.6 ± 1.5 39.2 ± 1.4 37.4 ± 1.7 41.8 ± 1.5 42.5 ± 1.3
Nonpl 72.5 ± 0.2 71.2 ± 0.2 - 60.4 ± 0.0 60.4 ± 0.0 60.4 ± 0.0 66.5 ± 0.2 72.5 ± 0.2 60.4 ± 0.0 71.6 ± 0.2
FLO17 51.9 ± 1.5 46.2 ± 2.0 39.2 ± 1.3 44.7 ± 2.0 59.2 ± 1.2 42.8 ± 1.2 55.6 ± 1.5 51.9 ± 1.5 60.9 ± 1.2 62.4 ± 1.3
FLO102 32.3 ± 0.6 27.8 ± 0.4 - 45.1 ± 0.9 45.2 ± 1.0 46.5 ± 0.6 49.7 ± 1.0 32.3 ± 0.6 48.6 ± 0.7 50.3 ± 0.8
Reuters 53.0 ± 0.4 52.9 ± 0.5 - 51.8 ± 1.5 50.9 ± 1.4 51.9 ± 1.0 52.9 ± 0.3 53.0 ± 0.4 53.3 ± 0.0 53.4 ± 0.4

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 31 / 51

 Localized SimpleMKKM Experimental Results

Running Time

10

Avg-KKM
Logarithm of Running Time(in Second)

8 MKKM
LMKKM
ONKC
6
MKKM-MiR
LKAM
4 LF-MVC
MKKM-MM
SimpleMKKM
2 LSMKKM

-2

-4
CAL-7 FLO102 FLO17 MSRA Nonpl PFD Reuters Still

图 5: Running time comparison of different algorithms on all benchmark datasets

(logarithm in seconds). The experiments are conducted on a PC with Intel(R)
Core(TM)-i9-10900X 3.7GHz CPU and 64G RAM in MATLAB R2020b
environment.

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 32 / 51

 Localized SimpleMKKM Experimental Results

Clustering Evolution

MSRA Still PFD

90 35
Performance (%)

Performance (%)

Performance (%)
30 40
85
25
80 20 30

15
75
10 20
70
5 10 15 20 25 30 5 10 15 20 25 30 5 10 15 20 25 30
Iteration Iteration Iteration

FLO17 FLO102 Reuters

60
60 50
Performance (%)

Performance (%)

Performance (%)
50
ACC
40 NMI
50 40 Purity
Rand Index
30 30
40
20
20
5 10 15 20 25 30 5 10 15 20 25 30 5 10 15 20 25 30
Iteration Iteration Iteration

图 6: The clustering performance of the learned H by localized SimpleMKKM

with iterations on six benchmark datasets.

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 33 / 51

 Localized SimpleMKKM Experimental Results

Objective Value

MSRA Still 10 4 PFD

1.15
600 2990
Objective value

Objective value

Objective value
1.1
2985

550 1.05
2980

2975 1

500 0.95
5 10 15 20 25 30 5 10 15 20 25 30 5 10 15 20 25 30
Iteration Iteration Iteration

10 4 FLO17 10 6 FLO102 10 5 Reuters

9.1
1.9
4
Objective value

Objective value

Objective value
1.8 9

1.7
3.5 8.9
1.6
8.8
1.5
3
1.4
5 10 15 20 25 30 5 10 15 20 25 30 5 10 15 20 25 30
Iteration Iteration Iteration

图 7: The objective of localized SimpleMKKM varies with iterations.

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 34 / 51

 Localized SimpleMKKM Experimental Results

Parameter Analysis

MSRA Still PFD

40
90
Performance (%)

Performance (%)

Performance (%)
40
85 30

80 30
20
75

10 20
70

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

FLO17 FLO102 Reuters

60
60 50
Performance (%)

Performance (%)

Performance (%)
55 50 ACC
40 NMI
50 Purity
40 30 RI
45
30 20
40

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

图 8: The effect of the size of neighborhood τ on the clustering performance on

six representative datasets.

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 35 / 51

 Parameter-free Localized SimpleMKKM

Content

1 Introduction

2 SimpleMKKM: Simple Multiple Kernel K-means

3 Localized SimpleMKKM

4 Parameter-free Localized SimpleMKKM

5 Conclusion and Future Work

6 Related References

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 36 / 51

 Parameter-free Localized SimpleMKKM Formulation

Content

1 Introduction

2 SimpleMKKM: Simple Multiple Kernel K-means

3 Localized SimpleMKKM

4 Parameter-free Localized SimpleMKKM

Formulation
Optimization
Experimental Results

5 Conclusion and Future Work

6 Related References

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 37 / 51

 Parameter-free Localized SimpleMKKM Formulation

Formulation of Parameter-free Localized SimpleMKKM

By substituting M in Eq. (16) with Mµ , we obtain the objective of the

proposed parameter-free localized SimpleMKKM as follows,
 
minγ∈∆ minµ∈Θ maxH∈Rn×k Tr H⊤ (Mµ ⊗ Kγ )H s.t. H⊤ H = Ik ,
(17)
Pl
where Mµ = 2
p=1 µp Mp and Θ = {µ ∈ Rl |µ⊤ el = 1, µp ≥ 0, ∀p}.

Merits of Eq. (17)

It calculates the kernel alignment in a local manner, which enables it
to capture the variation among base kernel matrices, leading to
improved clustering performance.
The optimal hyper-parameter can be automatically learned from data.

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 38 / 51

 Parameter-free Localized SimpleMKKM Optimization

Content

1 Introduction

2 SimpleMKKM: Simple Multiple Kernel K-means

3 Localized SimpleMKKM

4 Parameter-free Localized SimpleMKKM

Formulation
Optimization
Experimental Results

5 Conclusion and Future Work

6 Related References

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 39 / 51

 Parameter-free Localized SimpleMKKM Optimization

Optimization
To optimize Eq. (17), we first rewrite it as follows,
min T (γ) (18)
γ∈∆
with
n   o
T (γ) = minµ∈Θ maxH∈Rn×k Tr H⊤ (Mµ ⊗ Kγ )H s.t. H⊤ H = Ik .
(19)
According Theorem 1, given γ, the optimization in Eq. (19) can be readily
solved by SimpleMKKM in Algorithm 1, generating the global optimum.
Therefore, we have the following Theorem 3.
定理 (Differentiability of T (γ))

T (γ) in Eq. (18)

 is differentiable w.r.t
 γ. Further,
⊤
∂T (γ)
= 2γp Tr H∗ (Mµ∗ ⊗ Kp ) H∗ , where (H∗ , µ∗ ) =
∂γp

arg minµ∈Θ maxH∈Rn×k Tr H⊤ (Mµ ⊗ Kγ )H s.t. H⊤ H = Ik .
刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 40 / 51
 Parameter-free Localized SimpleMKKM Optimization

The Overall Algorithm Framework of Parameter-free

Localized SimpleMKKM

Algorithm 2 Parameter-free Localized SimpleMKKM

1: Input: {Kp }m l
p=1 , {Mp }p=1 and ϵ0 .
2: Output: H and γ, µ.
3: Initialize γ (0) = em /m, µ(0) = el /l and t = 1.
4: repeat
∑ ( )2
Kγ (t) = m
(t−1)
5: p=1 γp Kp .
6: compute H and µ by SimpleMKKM in Algorithm 1 with Kγ (t) .
7: compute ∂T (γ)
∂γp
(p = 1, · · · , m) according to Theorem 3 and the descent direction
(t)
d .
8: update γ (t+1) ← γ (t) + αd(t) .
9: if max |γ (t+1) − γ (t) | ≤ 1e − 4 then
10: flag=0.
11: end if
12: t ← t + 1.
13: end while
刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 41 / 51
 Parameter-free Localized SimpleMKKM Experimental Results

Content

1 Introduction

2 SimpleMKKM: Simple Multiple Kernel K-means

3 Localized SimpleMKKM

4 Parameter-free Localized SimpleMKKM

Formulation
Optimization
Experimental Results

5 Conclusion and Future Work

6 Related References

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 42 / 51

 Parameter-free Localized SimpleMKKM Experimental Results

Clustering Performance
Dataset Avg-MKKM MKKM LMKKM MKKM-MiR LKAM LF-MVC MKKM-MM SMKKM Proposed
ACC
Wdbc 91.0 ± 0.0 91.0 ± 0.0 91.0 ± 0.0 81.5 ± 0.0 79.4 ± 0.0 91.0 ± 0.0 91.0 ± 0.0 90.5 ± 0.0 93.0 ± 0.0
ProteinFold 29.0 ± 1.5 27.0 ± 1.1 22.4 ± 0.7 34.7 ± 1.8 37.7 ± 1.2 33.0 ± 1.4 29.0 ± 1.5 34.7 ± 1.9 37.1 ± 1.6
Flower17 50.8 ± 1.5 44.9 ± 2.4 37.5 ± 1.6 58.5 ± 1.5 50.0 ± 0.8 61.0 ± 0.7 50.8 ± 1.5 59.5 ± 1.3 62.1 ± 0.7
Caltech102 34.2 ± 1.0 32.8 ± 0.9 27.9 ± 0.8 34.8 ± 1.0 32.3 ± 1.0 34.4 ± 1.3 34.2 ± 1.0 35.8 ± 0.7 37.8 ± 0.7
Handwritten 96.0 ± 0.0 64.9 ± 2.4 65.0 ± 1.4 88.7 ± 0.1 95.4 ± 3.5 95.8 ± 0.0 96.0 ± 0.0 93.6 ± 0.0 95.9 ± 3.0
Flower102 27.1 ± 0.8 22.4 ± 0.5 - 40.2 ± 0.9 41.4 ± 0.8 38.4 ± 1.2 27.1 ± 0.8 42.5 ± 0.8 42.7 ± 1.0
NMI
Wdbc 55.2 ± 0.0 55.0 ± 0.0 55.0 ± 0.0 36.3 ± 0.0 34.2 ± 0.0 55.3 ± 0.0 55.2 ± 0.0 54.3 ± 0.0 62.5± 0.0
ProteinFold 40.3 ± 1.3 38.0 ± 0.6 34.7 ± 0.6 43.7 ± 1.2 46.2 ± 0.6 41.7 ± 1.1 40.3 ± 1.3 44.4 ± 1.1 46.7 ± 1.0
Flower17 49.7 ± 1.0 44.9 ± 1.5 38.8 ± 1.1 56.4 ± 0.9 49.8 ± 0.6 58.9 ± 0.4 49.7 ± 1.0 57.8 ± 0.9 60.5 ± 0.6
Caltech102 59.3 ± 0.6 58.6 ± 0.5 55.3 ± 0.5 59.7 ± 0.5 58.5 ± 0.6 59.5 ± 0.6 59.3 ± 0.6 60.4 ± 0.5 62.3 ± 0.4
Handwritten 91.1 ± 0.1 64.8 ± 1.6 64.7 ± 0.5 79.4 ± 0.2 91.8 ± 1.9 90.9 ± 0.1 91.1 ± 0.1 87.4 ± 0.0 92.0 ± 1.8
Flower102 46.0 ± 0.5 42.7 ± 0.2 - 56.7 ± 0.5 56.9 ± 0.3 54.9 ± 0.4 46.0 ± 0.5 58.6 ± 0.5 59.4 ± 0.3
Purity
Wdbc 91.0 ± 0.0 91.0 ± 0.0 91.0 ± 0.0 81.5 ± 0.0 79.4 ± 0.0 91.0 ± 0.0 91.0 ± 0.0 90.5 ± 0.0 93.0 ± 0.0
ProteinFold 37.4 ± 1.7 33.7 ± 1.1 31.2 ± 1.0 41.9 ± 1.4 43.7 ± 0.8 39.3 ± 1.5 37.4 ± 1.7 41.8 ± 1.5 44.3 ± 1.4
Flower17 51.9 ± 1.5 46.2 ± 2.0 39.2 ± 1.3 59.7 ± 1.6 51.4 ± 0.7 62.4 ± 0.7 51.9 ± 1.5 60.9 ± 1.2 63.4 ± 1.0
Caltech102 36.2 ± 1.0 34.9 ± 0.9 29.6 ± 0.8 36.8 ± 0.8 34.3 ± 0.9 36.7 ± 1.3 36.2 ± 1.0 38.0 ± 0.7 40.4 ± 0.8
Handwritten 96.0 ± 0.0 65.8 ± 2.1 65.5 ± 0.9 88.7 ± 0.1 95.4 ± 3.5 95.8 ± 0.0 96.0 ± 0.0 93.6 ± 0.0 96.1 ± 2.5
Flower102 32.3 ± 0.6 27.8 ± 0.4 - 46.3 ± 0.8 48.0 ± 0.6 44.6 ± 0.8 32.3 ± 0.6 48.6 ± 0.7 49.6 ± 0.7
Rand Index
Wdbc 67.2 ± 0.0 67.2 ± 0.0 67.2 ± 0.0 39.7 ± 0.0 34.5 ± 0.0 67.2 ± 0.0 67.2 ± 0.0 65.5 ± 0.0 73.8 ± 0.0
ProteinFold 14.4 ± 1.8 12.1 ± 0.7 7.8 ± 0.4 17.2 ± 1.5 20.1 ± 1.1 16.1 ± 1.5 14.4 ± 1.8 17.6 ± 1.9 20.3 ± 2.0
Flower17 32.2 ± 1.3 27.2 ± 1.8 20.6 ± 1.1 39.9 ± 1.3 31.6 ± 0.8 44.1 ± 0.4 32.2 ± 1.3 41.5 ± 1.5 44.8 ± 0.7
Caltech102 18.4 ± 0.9 17.3 ± 0.7 13.4 ± 0.8 18.8 ± 0.8 16.8 ± 0.9 18.8 ± 1.0 18.4 ± 0.9 19.8 ± 0.7 21.8 ± 0.7
Handwritten 91.3 ± 0.0 51.8 ± 2.3 50.4 ± 1.2 77.2 ± 0.2 91.6 ± 3.5 91.0 ± 0.1 91.3 ± 0.0 86.5 ± 0.1 91.9 ± 3.0
Flower102 15.5 ± 0.5 12.1 ± 0.4 - 25.5 ± 0.6 27.2 ± 0.6 25.5 ± 1.0 15.5 ± 0.5 28.5 ± 0.8 28.8 ± 0.9

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 43 / 51

 Parameter-free Localized SimpleMKKM Experimental Results

Clustering Evolution
Wdbc ProteinFold Flower17
Performance (%)

Performance (%)

Performance (%)
90 60
ACC Pur 40
NMI RI ACC Pur
80 ACC Pur NMI RI
30 NMI RI 50
70

60 20 40

5 10 15 20 5 10 15 20 5 10 15 20
Number of Iterations Number of Iterations Number of Iterations
Caltech102 Handwritten Flower102
60
60
Performance (%)

Performance (%)

Performance (%)
ACC Pur 96 50
50 NMI RI ACC Pur
NMI RI 40 ACC Pur
40 94
NMI RI
30
30
92 20
20
5 10 15 20 5 10 15 20 5 10 15 20
Number of Iterations Number of Iterations Number of Iterations

图 9: The evolution of the learned H by the proposed algorithm with iterations.

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 44 / 51

 Parameter-free Localized SimpleMKKM Experimental Results

Objective Value
Wdbc ProteinFold Flower17
6.6 10
Objective value

Objective value

Objective value
6.4 22
9.5
6.2
20
6
9
5.8 18

5 10 15 20 5 10 15 20 5 10 15 20
Number of Iterations Number of Iterations Number of Iterations
Caltech102 Handwritten Flower102
14.4
Objective value

Objective value

Objective value
14.2 280
75
14 260
13.8
70 240
13.6
220
13.4 65
5 10 15 20 5 10 15 20 5 10 15 20
Number of Iterations Number of Iterations Number of Iterations

图 10: The objective of the proposed parameter free localized SimpleMKKM

evolves with iterations.

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 45 / 51

 Conclusion and Future Work

Content

1 Introduction

2 SimpleMKKM: Simple Multiple Kernel K-means

3 Localized SimpleMKKM

4 Parameter-free Localized SimpleMKKM

5 Conclusion and Future Work

6 Related References

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 46 / 51

 Conclusion and Future Work

Conclusion and Future Work

Conclusion
We propose a new (simple yet effective) formulation for MKKM, and
develop an efficient algorithm to solve it. The generalization error
bound of the proposed algorithm is derived.
We improve SimpleMKKM by proposing a localized variant, and
further design a parameter-free formulation to eliminate its
hyper-parameters.
Our work provides a more effective approach to fuse multi-view data
for clustering, which could trigger novel research on MKC.

Ongoing Work Based on SimpleMKKM

Incomplete SimpleMKKM
min-max Optimization Applied into Late Fusion MVC

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 47 / 51

 Related References

Content

1 Introduction

2 SimpleMKKM: Simple Multiple Kernel K-means

3 Localized SimpleMKKM

4 Parameter-free Localized SimpleMKKM

5 Conclusion and Future Work

6 Related References

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 48 / 51

 Related References

Partial Related References

1 X. Liu, et al: SimpleMKKM: Simple Multiple Kernel K-means. IEEE
T-PAMI (Revise and Resubmit as NEW) https://arxiv.org/pdf/2005.04975
2 X. Liu, et al: Parameter-free Localized SimpleMKKM. JMLR (Under
Review)
3 X. Liu, et al: Localized Simple Multiple Kernel K-means. ICCV 2021, pp.
9293-9301
4 X. Liu: Incomplete Multiple Kernel Alignment Maximization for Clustering.
IEEE T-PAMI, DOI: 10.1109/TPAMI.2021.3116948
5 X. Liu, et al: Efficient and Effective Regularized Incomplete Multi-view
Clustering. IEEE T-PAMI, 43(8): 2634-2646 (2021)
6 X. Liu, et al: Multiple Kernel k-means with Incomplete Kernels. IEEE
T-PAMI 42(5): 1191-1204 (2020)
7 X. Liu, et al: Absent Multiple Kernel Learning Algorithms. IEEE T-PAMI
42(6): 1303-1316 (2020)
8 X. Liu, et al: Late Fusion Incomplete Multi-view Clustering. IEEE T-PAMI
41(10): 2410-2423 (2019)
刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 49 / 51
 Related References

Acknowledgement

祝恩 朱信忠 唐厂 周思航 王思齐 郭西风

王思为 刘吉元 涂文轩 梁伟轩 张培 张毅

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 50 / 51

 Related References

Q&A

Thanks for your listening!

Recruiting several post-doctors in machine learning!

Homepage: https://xinwangliu.github.io/

Email: [email protected]

刘新旺 (NUDT) SimpleMKKM SDAI2021 前沿技术讲习班 51 / 51