Efficient algorithms for estimating

multi-view mixture models

Daniel Hsu

Microsoft Research, New England

Outline

Multi-view mixture models

Multi-view method-of-moments

Some applications and open questions

Concluding remarks
 Part 1. Multi-view mixture models

Multi-view mixture models

Unsupervised learning and mixture models
Multi-view mixture models
Complexity barriers

Multi-view method-of-moments

Some applications and open questions

Concluding remarks
Unsupervised learning

I Many modern applications of machine learning:

I high-dimensional data from many diverse sources,
I but mostly unlabeled.
Unsupervised learning

I Many modern applications of machine learning:

I high-dimensional data from many diverse sources,
I but mostly unlabeled.

I Unsupervised learning: extract useful info from this data.

I Disentangle sub-populations in data source.
I Discover useful representations for downstream stages of
learning pipeline (e.g., supervised learning).
Mixture models

Simple latent variable model: mixture model

h h ∈ [k ] := {1, 2, . . . , k } (hidden);
~x ∈ Rd (observed);

~x Pr[ h = j ] = wj ; ~x h ∼ Ph ;

so ~x has a mixture distribution

P(~x ) = w1 P1 (~x ) + w2 P2 (~x ) + · · · + wk Pk (~x ).

Mixture models

Simple latent variable model: mixture model

h h ∈ [k ] := {1, 2, . . . , k } (hidden);
~x ∈ Rd (observed);

~x Pr[ h = j ] = wj ; ~x h ∼ Ph ;

so ~x has a mixture distribution

P(~x ) = w1 P1 (~x ) + w2 P2 (~x ) + · · · + wk Pk (~x ).

Typical use: learn about constituent sub-populations

(e.g., clusters) in data source.
Multi-view mixture models

Can we take advantage of diverse sources of information?

Multi-view mixture models

Can we take advantage of diverse sources of information?

h h ∈ [k ],

~x1 ~x2 ··· ~x` ~x1 ∈ Rd1 , ~x2 ∈ Rd2 , . . . , ~x` ∈ Rd` .

k = # components, ` = # views (e.g., audio, video, text).

View 1: ~x1 ∈ Rd1 View 2: ~x2 ∈ Rd2 View 3: ~x3 ∈ Rd3

Multi-view mixture models

Can we take advantage of diverse sources of information?

h h ∈ [k ],

~x1 ~x2 ··· ~x` ~x1 ∈ Rd1 , ~x2 ∈ Rd2 , . . . , ~x` ∈ Rd` .

k = # components, ` = # views (e.g., audio, video, text).

View 1: ~x1 ∈ Rd1 View 2: ~x2 ∈ Rd2 View 3: ~x3 ∈ Rd3

Multi-view mixture models

Multi-view assumption:
Views are conditionally independent given the component.

View 1: ~x1 ∈ Rd1 View 2: ~x2 ∈ Rd2 View 3: ~x3 ∈ Rd3

Larger k (# components): more sub-populations to disentangle.

Larger ` (# views): more non-redundant sources of information.
Semi-parametric estimation task

“Parameters” of component distributions:

Mixing weights wj := Pr[h = j], j ∈ [k ];

~ v ,j := E[~xv |h = j] ∈ Rdv ,
Conditional means µ j ∈ [k ], v ∈ [`].

Goal: Estimate mixing weights and conditional means from

independent copies of (~x1 , ~x2 , . . . , ~x` ).
Semi-parametric estimation task

“Parameters” of component distributions:

Mixing weights wj := Pr[h = j], j ∈ [k ];

~ v ,j := E[~xv |h = j] ∈ Rdv ,
Conditional means µ j ∈ [k ], v ∈ [`].

Goal: Estimate mixing weights and conditional means from

independent copies of (~x1 , ~x2 , . . . , ~x` ).

Questions:
1. How do we estimate {wj } and {~
µv ,j } without observing h?
2. How many views ` are sufficient to learn with poly(k )
computational / sample complexity?
Some barriers to efficient estimation

Challenge: many difficult parametric estimation tasks reduce to

this estimation problem.
Some barriers to efficient estimation

Challenge: many difficult parametric estimation tasks reduce to

this estimation problem.

Cryptographic barrier: discrete HMM pa-

rameter estimation as hard as learning parity
functions with noise (Mossel-Roch, ’06).
Some barriers to efficient estimation

Challenge: many difficult parametric estimation tasks reduce to

this estimation problem.

Cryptographic barrier: discrete HMM pa-

rameter estimation as hard as learning parity
functions with noise (Mossel-Roch, ’06).

Statistical barrier: Gaussian mixtures in R1

can require exp(Ω(k )) samples to estimate
parameters, even if components are well-
separated (Moitra-Valiant, ’10).
Some barriers to efficient estimation

Challenge: many difficult parametric estimation tasks reduce to

this estimation problem.

Cryptographic barrier: discrete HMM pa-

rameter estimation as hard as learning parity
functions with noise (Mossel-Roch, ’06).

Statistical barrier: Gaussian mixtures in R1

can require exp(Ω(k )) samples to estimate
parameters, even if components are well-
separated (Moitra-Valiant, ’10).

In practice: resort to local search (e.g., EM), often subject to

slow convergence and inaccurate local optima.
Making progress: Gaussian mixture model
Gaussian mixture model: problem becomes easier if assume
some large minimum separation between component means
(Dasgupta, ’99):
k~
µi − µ
~jk
sep := min .
i6=j max{σi , σj }
Making progress: Gaussian mixture model
Gaussian mixture model: problem becomes easier if assume
some large minimum separation between component means
(Dasgupta, ’99):
k~
µi − µ
~jk
sep := min .
i6=j max{σi , σj }

I sep = Ω(d c ): interpoint distance-based methods / EM

(Dasgupta, ’99; Dasgupta-Schulman, ’00; Arora-Kannan, ’00)
I sep = Ω(k c ): first use PCA to k dimensions
(Vempala-Wang, ’02; Kannan-Salmasian-Vempala, ’05;
Achlioptas-McSherry, ’05)
I Also works for mixtures of log-concave distributions.
Making progress: Gaussian mixture model
Gaussian mixture model: problem becomes easier if assume
some large minimum separation between component means
(Dasgupta, ’99):
k~
µi − µ
~jk
sep := min .
i6=j max{σi , σj }

I sep = Ω(d c ): interpoint distance-based methods / EM

(Dasgupta, ’99; Dasgupta-Schulman, ’00; Arora-Kannan, ’00)
I sep = Ω(k c ): first use PCA to k dimensions
(Vempala-Wang, ’02; Kannan-Salmasian-Vempala, ’05;
Achlioptas-McSherry, ’05)
I Also works for mixtures of log-concave distributions.
I No minimum separation requirement: method-of-moments
but exp(Ω(k )) running time / sample size
(Kalai-Moitra-Valiant, ’10; Belkin-Sinha, ’10; Moitra-Valiant, ’10)
Making progress: discrete hidden Markov models
Hardness reductions create HMMs with degenerate output and
next-state distributions.

0.6

1 2 3 4 5 6 7 8
≈

1 2 3 4 5 6 7 8 +0.4
Pr[~xt = ·|ht = 1]
1 2 3 4 5 6 7 8
0.6 Pr[~xt = ·|ht = 2] + 0.4 Pr[~xt = ·|ht = 3]
Making progress: discrete hidden Markov models
Hardness reductions create HMMs with degenerate output and
next-state distributions.

0.6

1 2 3 4 5 6 7 8
≈

1 2 3 4 5 6 7 8 +0.4
Pr[~xt = ·|ht = 1]
1 2 3 4 5 6 7 8
0.6 Pr[~xt = ·|ht = 2] + 0.4 Pr[~xt = ·|ht = 3]

These instances are avoided by assuming parameter matrices

are full-rank (Mossel-Roch, ’06; Hsu-Kakade-Zhang, ’09)
What we do

This work: given ≥ 3 views, mild non-degeneracy conditions

imply efficient algorithms for estimation.
What we do

This work: given ≥ 3 views, mild non-degeneracy conditions

imply efficient algorithms for estimation.
I Non-degeneracy condition for multi-view mixture model:
Conditional means {~
µv ,1 , µ ~ v ,k } are linearly
~ v ,2 , . . . , µ
independent for each view v ∈ [`], and w ~ > ~0.
Requires high-dimensional observations (dv ≥ k )!
What we do

This work: given ≥ 3 views, mild non-degeneracy conditions

imply efficient algorithms for estimation.
I Non-degeneracy condition for multi-view mixture model:
Conditional means {~
µv ,1 , µ ~ v ,k } are linearly
~ v ,2 , . . . , µ
independent for each view v ∈ [`], and w ~ > ~0.
Requires high-dimensional observations (dv ≥ k )!

I New efficient learning guarantees for parametric models

(e.g., mixtures of Gaussians, general HMMs)
What we do

This work: given ≥ 3 views, mild non-degeneracy conditions

imply efficient algorithms for estimation.
I Non-degeneracy condition for multi-view mixture model:
Conditional means {~
µv ,1 , µ ~ v ,k } are linearly
~ v ,2 , . . . , µ
independent for each view v ∈ [`], and w ~ > ~0.
Requires high-dimensional observations (dv ≥ k )!

I New efficient learning guarantees for parametric models

(e.g., mixtures of Gaussians, general HMMs)
I General tensor decomposition framework applicable to
a wide variety of estimation problems.
 Part 2. Multi-view method-of-moments

Multi-view mixture models

Multi-view method-of-moments
Overview
Structure of moments
Uniqueness of decomposition
Computing the decomposition
Asymmetric views

Some applications and open questions

Concluding remarks
The plan

I First, assume views are (conditionally) exchangeable,

and derive basic algorithm.
The plan

I First, assume views are (conditionally) exchangeable,

and derive basic algorithm.

I Then, provide reduction from general multi-view setting to

exchangeable case.

−→
Simpler case: exchangeable views

(Conditionally) exchangeable views: assume the views have

the same conditional means, i.e.,

E[ ~xv |h = j ] ≡ µ
~j, j ∈ [k ], v ∈ [`].
Simpler case: exchangeable views

(Conditionally) exchangeable views: assume the views have

the same conditional means, i.e.,

E[ ~xv |h = j ] ≡ µ
~j, j ∈ [k ], v ∈ [`].

Motivating setting: bag-of-words model,

~x1 , ~x2 , . . . , ~x` ≡ ` exchangeable words in a document.

One-hot encoding:
~xv = ~ei ⇔ v -th word in document is i-th word in vocab
(where ~ei ∈ {0, 1}d has 1 in i-th position, 0 elsewhere).

µj )i = E[(~xv )i |h = j] = Pr[~xv = ~ei |h = j],

(~ i ∈ [d], j ∈ [k ].
Key ideas

1. Method-of-moments: conditional means are revealed by

appropriate low-rank decompositions of moment matrices
and tensors.
2. Third-order tensor decomposition is uniquely
determined by directions of (locally) maximum skew.
3. The required local optimization can be efficiently
performed in poly time.
Algebraic structure in moments
Recall: E[ ~xv |h = j ] = µ
~j.
Algebraic structure in moments
Recall: E[ ~xv |h = j ] = µ
~j.
By conditional independence and exchangeability of
~x1 , ~x2 , . . . , ~x` given h,

Pairs := E[~x1 ⊗ ~x2 ]

 
= E E[~x1 |h] ⊗ E[~x2 |h] = E[~
µh ⊗ µ
~ h]
k
X
= wi µ ~ i ∈ Rd×d .
~i ⊗ µ
i=1
Algebraic structure in moments
Recall: E[ ~xv |h = j ] = µ
~j.
By conditional independence and exchangeability of
~x1 , ~x2 , . . . , ~x` given h,

Pairs := E[~x1 ⊗ ~x2 ]

 
= E E[~x1 |h] ⊗ E[~x2 |h] = E[~
µh ⊗ µ
~ h]
k
X
= wi µ ~ i ∈ Rd×d .
~i ⊗ µ
i=1

Triples := E[~x1 ⊗ ~x2 ⊗ ~x3 ]

k
X
= ~i ⊗ µ
wi µ ~ i ∈ Rd×d×d ,
~i ⊗ µ etc.
i=1

(If only we could extract these “low-rank” decompositions . . . )

2nd moment: subspace spanned by conditional means
2nd moment: subspace spanned by conditional means

µi } linearly independent)
Non-degeneracy assumption ({~
2nd moment: subspace spanned by conditional means

Non-degeneracy assumption ({~ µi } linearly independent)

Pk
~i ⊗ µ
=⇒ Pairs = i=1 wi µ ~ i symmetric psd and rank k
2nd moment: subspace spanned by conditional means

Non-degeneracy assumption ({~ µi } linearly independent)

Pk
~i ⊗ µ
=⇒ Pairs = i=1 wi µ ~ i symmetric psd and rank k
=⇒ Pairs equips k -dim subspace span{~ µ1 , µ ~ k } with
~ 2, . . . , µ
inner product
Pairs(~x , ~y ) := ~x > Pairs~y .
2nd moment: subspace spanned by conditional means

Non-degeneracy assumption ({~ µi } linearly independent)

Pk
~i ⊗ µ
=⇒ Pairs = i=1 wi µ ~ i symmetric psd and rank k
=⇒ Pairs equips k -dim subspace span{~ µ1 , µ ~ k } with
~ 2, . . . , µ
inner product
Pairs(~x , ~y ) := ~x > Pairs~y .

However, {~
µi } not generally determined by just Pairs
µi } are not necessarily orthogonal).
(e.g., {~
2nd moment: subspace spanned by conditional means

Non-degeneracy assumption ({~ µi } linearly independent)

Pk
~i ⊗ µ
=⇒ Pairs = i=1 wi µ ~ i symmetric psd and rank k
=⇒ Pairs equips k -dim subspace span{~ µ1 , µ ~ k } with
~ 2, . . . , µ
inner product
Pairs(~x , ~y ) := ~x > Pairs~y .

However, {~
µi } not generally determined by just Pairs
µi } are not necessarily orthogonal).
(e.g., {~
Must look at higher-order moments?
3rd moment: (cross) skew maximizers

Claim: Up to third-moment (i.e., 3 views) suffices.

View Triples : Rd × Rd × Rd → R as trilinear form.
3rd moment: (cross) skew maximizers

Claim: Up to third-moment (i.e., 3 views) suffices.

View Triples : Rd × Rd × Rd → R as trilinear form.

Theorem
Each isolated local maximizer ~η ∗ of

max Triples(~η , ~η , ~η ) s.t. Pairs(~η , ~η ) ≤ 1

~ ∈Rd
η

satisfies, for some i ∈ [k ],

√ 1
Pairs ~η ∗ = wi µ
~i, Triples(~η ∗ , ~η ∗ , ~η ∗ ) = √ .
wi
3rd moment: (cross) skew maximizers

Claim: Up to third-moment (i.e., 3 views) suffices.

View Triples : Rd × Rd × Rd → R as trilinear form.

Theorem
Each isolated local maximizer ~η ∗ of

max Triples(~η , ~η , ~η ) s.t. Pairs(~η , ~η ) ≤ 1

~ ∈Rd
η

satisfies, for some i ∈ [k ],

√ 1
Pairs ~η ∗ = wi µ
~i, Triples(~η ∗ , ~η ∗ , ~η ∗ ) = √ .
wi

Also: these maximizers can be found efficiently and robustly.

Variational analysis

max Triples(~η , ~η , ~η ) s.t. Pairs(~η , ~η ) ≤ 1

~ ∈Rd
η
Variational analysis

max Triples(~η , ~η , ~η ) s.t. Pairs(~η , ~η ) ≤ 1

~ ∈Rd
η

Pk Pk
(Substitute Pairs = i=1 ~i ⊗ µ
wi µ ~ i and Triples = i=1 ~i ⊗ µ
wi µ ~i ⊗ µ
~ i .)
Variational analysis
k
X k
X
max ~ i )3 s.t.
wi (~η > µ ~ i )2 ≤ 1
wi (~η > µ
~ ∈Rd
η
i=1 i=1
Pk Pk
(Substitute Pairs = i=1 ~i ⊗ µ
wi µ ~ i and Triples = i=1 ~i ⊗ µ
wi µ ~i ⊗ µ
~ i .)
Variational analysis
k
X k
X
max ~ i )3 s.t.
wi (~η > µ ~ i )2 ≤ 1
wi (~η > µ
~ ∈Rd
η
i=1 i=1
√
(Let θi := ~ i ) for i ∈ [k ].)
wi (~η > µ
Variational analysis
k k
X 1 √ X √
max ~ i )3 s.t.
√ ( wi ~η > µ ~ i )2 ≤ 1
( wi ~η > µ
~ ∈Rd
η wi
i=1 i=1
√
(Let θi := ~ i ) for i ∈ [k ].)
wi (~η > µ
Variational analysis
k k
X 1 X
max √ θi3 s.t. θi2 ≤ 1
~ k
θ∈R wi
i=1 i=1
√
(Let θi := ~ i ) for i ∈ [k ].)
wi (~η > µ
Variational analysis
k k
X 1 X
max √ θi3 s.t. θi2 ≤ 1
~ k
θ∈R wi
i=1 i=1
√
(Let θi := ~ i ) for i ∈ [k ].)
wi (~η > µ

Isolated local maximizers θ~∗ (found via gradient ascent) are

~e1 = (1, 0, 0, . . . ), ~e2 = (0, 1, 0, . . . ), etc.

which means that each ~η ∗ satisfies, for some i ∈ [k ],

(
1 j =i
wj ~η ∗> µ
p

~j =
0 j 6= i.
Variational analysis
k k
X 1 X
max √ θi3 s.t. θi2 ≤ 1
~ k
θ∈R wi
i=1 i=1
√
(Let θi := ~ i ) for i ∈ [k ].)
wi (~η > µ

Isolated local maximizers θ~∗ (found via gradient ascent) are

~e1 = (1, 0, 0, . . . ), ~e2 = (0, 1, 0, . . . ), etc.

which means that each ~η ∗ satisfies, for some i ∈ [k ],

(
1 j =i
wj ~η ∗> µ
p

~j =
0 j 6= i.
Therefore k
X
√
Pairs ~η ∗ = ~ j ~η ∗> µ
wj µ ~ j = wi µ ~i.
j=1
Extracting all isolated local maximizers

1. Start with T := Triples.

Extracting all isolated local maximizers

1. Start with T := Triples.

2. Find isolated local maximizer of

T (~η , ~η , ~η ) s.t. Pairs(~η , ~η ) ≤ 1

via gradient ascent from random ~η ∈ range(Pairs).

Say maximum is λ∗ and maximizer is ~η ∗ .
Extracting all isolated local maximizers

1. Start with T := Triples.

2. Find isolated local maximizer of

T (~η , ~η , ~η ) s.t. Pairs(~η , ~η ) ≤ 1

via gradient ascent from random ~η ∈ range(Pairs).

Say maximum is λ∗ and maximizer is ~η ∗ .
3. Deflation: replace T with T − λ∗ ~η ∗ ⊗ ~η ∗ ⊗ ~η ∗ .
Goto step 2.
Extracting all isolated local maximizers

1. Start with T := Triples.

2. Find isolated local maximizer of

T (~η , ~η , ~η ) s.t. Pairs(~η , ~η ) ≤ 1

via gradient ascent from random ~η ∈ range(Pairs).

Say maximum is λ∗ and maximizer is ~η ∗ .
3. Deflation: replace T with T − λ∗ ~η ∗ ⊗ ~η ∗ ⊗ ~η ∗ .
Goto step 2.

A variant of this runs in polynomial time (w.h.p.), and is

robust to perturbations to Pairs and Triples.
General case: asymmetric views

Each view v has different set of conditional means

{~
µv ,1 , µ ~ v ,k } ⊂ Rdv .
~ v ,2 , . . . , µ
General case: asymmetric views

Each view v has different set of conditional means

{~
µv ,1 , µ ~ v ,k } ⊂ Rdv .
~ v ,2 , . . . , µ

Reduction: transform ~x1 and ~x2 to “look like” ~x3 via linear
transformations.

−→
Asymmetric cross moments

Define asymmetric cross moment:

Pairsu,v := E[~xu ⊗ ~xv ].

Asymmetric cross moments

Define asymmetric cross moment:

Pairsu,v := E[~xu ⊗ ~xv ].

Transforming view v to view 3:

Cv →3 := E[~x3 ⊗ ~xu ] E[~xv ⊗ ~xu ]† ∈ Rd3 ×dv

where † denotes Moore-Penrose pseudoinverse.

Asymmetric cross moments

Define asymmetric cross moment:

Pairsu,v := E[~xu ⊗ ~xv ].

Transforming view v to view 3:

Cv →3 := E[~x3 ⊗ ~xu ] E[~xv ⊗ ~xu ]† ∈ Rd3 ×dv

where † denotes Moore-Penrose pseudoinverse.

Simple exercise to show

E[Cv →3~xv |h = j] = µ
~ 3,j

so Cv →3~xv behaves like ~x3 (as far as our algorithm can tell).
Part 3. Some applications and open questions

Multi-view mixture models

Multi-view method-of-moments

Some applications and open questions

Mixtures of Gaussians
Hidden Markov models and other models
Topic models
Open questions

Concluding remarks
Mixtures of axis-aligned Gaussians

Mixture of axis-aligned Gaussian in Rn , with component means

µ
~ 1, µ ~ k ∈ Rn ; no minimum separation requirement.
~ 2, . . . , µ
h

x1 x2 ··· xn
Mixtures of axis-aligned Gaussians

Mixture of axis-aligned Gaussian in Rn , with component means

µ
~ 1, µ ~ k ∈ Rn ; no minimum separation requirement.
~ 2, . . . , µ
h

x1 x2 ··· xn

Assumptions:
I non-degeneracy: component means span k dim subspace.
I weak incoherence condition: component means not
perfectly aligned with coordinate axes — similar to
spreading condition of (Chaudhuri-Rao, ’08).
Mixtures of axis-aligned Gaussians

Mixture of axis-aligned Gaussian in Rn , with component means

µ
~ 1, µ ~ k ∈ Rn ; no minimum separation requirement.
~ 2, . . . , µ
h

x1 x2 ··· xn

Assumptions:
I non-degeneracy: component means span k dim subspace.
I weak incoherence condition: component means not
perfectly aligned with coordinate axes — similar to
spreading condition of (Chaudhuri-Rao, ’08).
Then, randomly partitioning coordinates into ` ≥ 3 views
guarantees (w.h.p.) that non-degeneracy holds in all ` views.
Hidden Markov models and others

h1 h2 h3

~x1 ~x2 ~x3

Hidden Markov models and others

h1 h2 h3 h
−→
~x1 ~x2 ~x3 ~x1 ~x2 ~x3
Hidden Markov models and others

h1 h2 h3 h
−→
~x1 ~x2 ~x3 ~x1 ~x2 ~x3

Other models:
1. Mixtures of Gaussians (Hsu-Kakade, ITCS’13)
2. HMMs (Anandkumar-Hsu-Kakade, COLT’12)
3. Latent Dirichlet Allocation
(Anandkumar-Foster-Hsu-Kakade-Liu, NIPS’12)
4. Latent parse trees (Hsu-Kakade-Liang, NIPS’12)
5. Independent Component Analysis
(Arora-Ge-Moitra-Sachdeva, NIPS’12; Hsu-Kakade, ITCS’13)
Bag-of-words clustering model
µj )i = Pr[ see word i in document | document topic is j ].
(~
I Corpus: New York Times (from UCI), 300000 articles.
I Vocabulary size: d = 102660 words.
I Chose k = 50.
I For each topic j, show top 10 words i.
Bag-of-words clustering model
µj )i = Pr[ see word i in document | document topic is j ].
(~
I Corpus: New York Times (from UCI), 300000 articles.
I Vocabulary size: d = 102660 words.
I Chose k = 50.
I For each topic j, show top 10 words i.

sales run school drug player

economic inning student patient tiger_wood
consumer hit teacher million won
major game program company shot
home season official doctor play
indicator home public companies round
weekly right children percent win
order games high cost tournament
claim dodger education program tour
scheduled left district health right
Bag-of-words clustering model

palestinian tax cup point yard

israel cut minutes game game
israeli percent oil team play
yasser_arafat bush water shot season
peace billion add play team
israeli plan tablespoon laker touchdown
israelis bill food season quarterback
leader taxes teaspoon half coach
official million pepper lead defense
attack congress sugar games quarter
Bag-of-words clustering model

percent al_gore car book taliban

stock campaign race children attack
market president driver ages afghanistan
fund george_bush team author official
investor bush won read military
companies clinton win newspaper u_s
analyst vice racing web united_states
money presidential track writer terrorist
investment million season written war
economy democratic lap sales bin
Bag-of-words clustering model

com court show film music

www case network movie song
site law season director group
web lawyer nbc play part
sites federal cb character new_york
information government program actor company
online decision television show million
mail trial series movies band
internet microsoft night million show
telegram right new_york part album

etc.
Some open questions

What if k > dv ? (relevant to overcomplete dictionary learning)

Some open questions

What if k > dv ? (relevant to overcomplete dictionary learning)

I Apply some non-linear transformations ~xv 7→ fv (~xv )?
Some open questions

What if k > dv ? (relevant to overcomplete dictionary learning)

I Apply some non-linear transformations ~xv 7→ fv (~xv )?
I Combine views, e.g., via tensor product

x̃1,2 := ~x1 ⊗ ~x2 , x̃3,4 := ~x3 ⊗ ~x4 , x̃5,6 := ~x5 ⊗ ~x6 , etc. ?
Some open questions

What if k > dv ? (relevant to overcomplete dictionary learning)

I Apply some non-linear transformations ~xv 7→ fv (~xv )?
I Combine views, e.g., via tensor product

x̃1,2 := ~x1 ⊗ ~x2 , x̃3,4 := ~x3 ⊗ ~x4 , x̃5,6 := ~x5 ⊗ ~x6 , etc. ?

Can we relax the multi-view assumption?

Some open questions

What if k > dv ? (relevant to overcomplete dictionary learning)

I Apply some non-linear transformations ~xv 7→ fv (~xv )?
I Combine views, e.g., via tensor product

x̃1,2 := ~x1 ⊗ ~x2 , x̃3,4 := ~x3 ⊗ ~x4 , x̃5,6 := ~x5 ⊗ ~x6 , etc. ?

Can we relax the multi-view assumption?

I Allow for richer hidden state?
(e.g., independent component analysis)
Some open questions

What if k > dv ? (relevant to overcomplete dictionary learning)

I Apply some non-linear transformations ~xv 7→ fv (~xv )?
I Combine views, e.g., via tensor product

x̃1,2 := ~x1 ⊗ ~x2 , x̃3,4 := ~x3 ⊗ ~x4 , x̃5,6 := ~x5 ⊗ ~x6 , etc. ?

Can we relax the multi-view assumption?

I Allow for richer hidden state?
(e.g., independent component analysis)
I “Gaussianization” via random projection?
 Part 4. Concluding remarks

Multi-view mixture models

Multi-view method-of-moments

Some applications and open questions

Concluding remarks
Concluding remarks

Take-home messages:
Concluding remarks

Take-home messages:
I Power of multiple views: Can take advantage of diverse /
non-redundant sources of information in unsupervised
learning.
Concluding remarks

Take-home messages:
I Power of multiple views: Can take advantage of diverse /
non-redundant sources of information in unsupervised
learning.
I Overcoming complexity barriers: Some provably hard
estimation problems become easy after ruling out
“degenerate” cases.
Concluding remarks

Take-home messages:
I Power of multiple views: Can take advantage of diverse /
non-redundant sources of information in unsupervised
learning.
I Overcoming complexity barriers: Some provably hard
estimation problems become easy after ruling out
“degenerate” cases.
I “Blessing of dimensionality” for estimators based on
method-of-moments.
 Thanks!
(Co-authors: Anima Anandkumar, Dean Foster, Rong Ge, Sham
Kakade, Yi-Kai Liu, Matus Telgarsky)

http://arxiv.org/abs/1210.7559