Label propagation on graphs

Leonid E. Zhukov

School of Data Analysis and Artificial Intelligence

Department of Computer Science
National Research University Higher School of Economics

Structural Analysis and Visualization of Networks

Leonid E. Zhukov (HSE) Lecture 17 19.05.2015 1 / 26

Lecture outline

1 Label propagation problem

2 Collective classification
Iterative classification

3 Semi-supervised learning
Random walk based methods
Graph regularization

Leonid E. Zhukov (HSE) Lecture 17 19.05.2015 2 / 26

Label propagation

Label propagation - labeling of all nodes in a graph structure

Subset of nodes is labeled: categorical/numeric/binary values
Extend labeling to all nodes on the graph
Classification in networked data, network classification, structured
inference

Leonid E. Zhukov (HSE) Lecture 17 19.05.2015 3 / 26

Label propagation problem

Structure can help only if labels/values of linked nodes are correlated

Social networks show assortative mixing - bias in favor of connections
between network nodes with similar characteristics:
– homophily: similar characteristics → connections
– influence: connections → similar characteristics
Can apply to constructed (induced) similarity networks

Leonid E. Zhukov (HSE) Lecture 17 19.05.2015 4 / 26

Network classification

Supervised learning approach

Given graph nodes V = Vl ∪ Vu :
– nodes Vl given labels Yl
– nodes Vu do not have lables
Need to find Yu
Labels can be binary, multi-class, real values
Features (attributes) can be computed for every node φi :
– local node features (if available)
– link features available (labels from neighbors, attributes from
neighbours, node degrees, connectivity patterns)
Feature (design) matrix Φ = (Φl , Φu )

Leonid E. Zhukov (HSE) Lecture 17 19.05.2015 5 / 26

Network learning components

Local classifier. This is a local learned model, predicts node label

based on node attributes. No network information
Relational classier. Takes into account labels and attributes of node
neighbors. Uses neighborhood network information
Collective classifier. Estimates unknown values together applying
relational classifier iteratively. Strongly depends on network structure

Leonid E. Zhukov (HSE) Lecture 17 19.05.2015 6 / 26

Collective classification

Algorithm: Iterative classification method

Input: Graph G (V , E ), labels Yl
Output: labels Ŷ
Compute Φ(0)
(0)
Train classifier on (Φl , Yl )
(0)
Predict Yu
repeat
(t)
Compute Φu
Train classifier on (Φ(t) , Y (t) )
(t+1) (t)
Predict Yu from Φu
(t)
until Yu converges;
Ŷ ← Y (t)

Leonid E. Zhukov (HSE) Lecture 17 19.05.2015 7 / 26

Iterative classification

Leonid E. Zhukov (HSE) Lecture 17 19.05.2015 8 / 26

Relational classifiers

Weighted-vote relational neighbor classifier:

1 X
P(yi = c|Ni ) = Aij P(yj = c|Nj )
Z
i∈Ni

Network only Bayes classifier:

P(Ni |c)P(c)
P(yi = c|Ni ) =
P(Ni )

where
1 Y
P(Ni |c) = P(yj = ŷj |yi = c)
Z
j∈Ni

Leonid E. Zhukov (HSE) Lecture 17 19.05.2015 9 / 26

Semi-supervised learning

Graph-based semi-supervised learning

Given partially labeled dataset
Data: X = Xl ∪ Xu
– small set of labeled data (Xl , Yl )
– large set of unlabeled data Xu
Similarity graph over data points G (V , E ), where every vertex vi
corresponds to a data point xi
Transductive learning: learn a function that predicts labels Yu for the
unlabeled input Xu

Leonid E. Zhukov (HSE) Lecture 17 19.05.2015 10 / 26

Random walk methods
Consider random walk with absorbing states - labeled nodes Vl
Probability ŷi [c] for node vi ∈ Vu to have label c,
X
ŷi [c] = pij∞ yj [c]
j∈Vl

where yi [c] - probability distribution over labels,

pij = P(i → j) - one stept probaility transition matrix
If output requires single label per node, assign the most probable
In matrix form
Ŷ = P ∞ Y
where Y = (Yl , 0), Ŷ = (Yl , Ŷu )

Leonid E. Zhukov (HSE) Lecture 17 19.05.2015 11 / 26

Random walk methods

Random walk matrix: P = D −1 A

Random walk with absorbing states
   
Pll Plu I 0
P= =
Pul Puu Pul Puu

At the t → ∞ limit:
   
t I 0 I 0
lim P = P∞ ∞ =
t→∞ n )P
( n=0 Puu ul Puu (I − Puu )−1 Pul 0

Leonid E. Zhukov (HSE) Lecture 17 19.05.2015 12 / 26

Random walk methods

Matrix equation
    
Ŷl I 0 Yl
=
Ŷu (I − Puu )−1 Pul 0 Yu

Solution

Ŷl = Yl
Ŷu = (I − Puu )−1 Pul Yl

(I − Puu ) is non-singular for all label connected graphs (is always

possible to reach a labeled node from any unlabeled node)

Leonid E. Zhukov (HSE) Lecture 17 19.05.2015 13 / 26

Label propagation

Algorithm: Label propagation, Zhu et. al 2002

Input: Graph G (V , E ), labels Yl
Output: labels Ŷ
P
Compute Dii = j Aij
Compute P = D −1 A
Initialize Y (0) = (Yl , 0), t=0
repeat
Y (t+1) ← P · Y (t)
(t+1) (t)
Yl ← Yl
until Y (t) converges;
Ŷ ← Y (t)

Leonid E. Zhukov (HSE) Lecture 17 19.05.2015 14 / 26

Label propagation

Iteration
Y (t) = P Y (t−1)
(t) (t−1)
Yu = Pul Yl + Puu Yu
t
(t)
X
n−1 t
Yu = Puu Pul Yl + Puu Yu
n=1

Convergence
Ŷ = lim Y (t) = (I − Puu )−1 Pul Yl
t→∞

Leonid E. Zhukov (HSE) Lecture 17 19.05.2015 15 / 26

Label spreading

Algorithm: Label spreading, Zhou et. al 2004

Input: Graph G (V , E ), labels Yl
Output: labels Ŷ
P
Compute Dii = j Aij ,
Compute S = D −1/2 AD −1/2
Initialize Y (0) = (Yl , 0), t=0
repeat
Y (t+1) ← αSY (t) + (1 − α)Y (0)
t ←t +1
until Y (t) converges;

Leonid E. Zhukov (HSE) Lecture 17 19.05.2015 16 / 26

Label spreading

Iterations
Y (t) = αSY (t−1) + (1 − α)Y (0)
At the t → ∞ limit:
 
lim Y (t) = lim αSY (t−1) + (1 − α)Y (0)
t→∞ t→∞

Ŷ = αS Ŷ + (1 − α)Y (0)
Solution
Ŷ = (1 − α)(I − αS)−1 Y (0)

Leonid E. Zhukov (HSE) Lecture 17 19.05.2015 17 / 26

Regularization on graphs

Find labeling Ŷ = (Ŷl , Ŷu ) that

Consistent with initial labeling:
X
(ŷi − yi )2 = ||Ŷl − Yl ||2
i∈Vl

Consistent with graph structure (regression function smoothness):

1 X
Aij (ŷi − ŷj )2 = Ŷ T (D − A)Ŷ = Ŷ T LŶ
2
i,j∈V

Stable (additional regularization):

X
 ŷi2 = ||Ŷ ||2
i∈V

Leonid E. Zhukov (HSE) Lecture 17 19.05.2015 18 / 26

Regularization on graphs

Minimization with respect to Ŷ , arg minŶ Q(Ŷ )

Label propagation [Zhu, 2002]:

1 X
Q(Ŷ ) = Aij (ŷi − ŷj )2 = Ŷ T LŶ , with fixed Ŷl = Yl
2
i,j∈V

Label spread [Zhou, 2003]:

!2
1X ŷ ŷj
√i − √
X
Q(Ŷ ) = Aij +µ (ŷi − yi )2
2 di dj
ij∈V i∈V

Q(Ŷ ) = Ŷ T LŶ + µ||Ŷ − Y ||2

L = I − S = I − D −1/2 AD −1/2

Leonid E. Zhukov (HSE) Lecture 17 19.05.2015 19 / 26

Regularization on graphs

Laplacian regularization [Belkin, 2003]

1X X
Q(Ŷ ) = Aij (ŷi − ŷj )2 + µ (ŷi − yi )2
2
ij∈V i∈Vl

Q(Ŷ ) = Ŷ T LŶ + µ||Ŷl − Yl ||2

Use eigenvectors (e1 ..ep ) from smallest eigenvalues of L = D − A:

Lej = λj ej

Construct classifier (regression function) on eigenvectors

X p
X
Err (a) = (yi − aj eji )2
i∈Vl j=1
Pp
Predict value (classify) ŷi = j=1 aj eji , class ci = sign(ŷi )
Leonid E. Zhukov (HSE) Lecture 17 19.05.2015 20 / 26
Laplacian regularization

Algorithm: Laplacian regularization, Belkin and Niyogy, 2003

Input: Graph G (V , E ), labels Yl
Output: labels Ŷ
P
Compute Dii = j Aij
Compute L = D − A
Compute p eigenvectors e1 ..ep with smallest eigenvalues of L, Le = λe
Minimize over a1 ...ap
arg mina1 ,...ap li=1 (yi − pj=1 aj eji )2 , a = (E T E )−1 E T Yl
P P
Pp
Label vi by the sign( j=1 aj eji )

Leonid E. Zhukov (HSE) Lecture 17 19.05.2015 21 / 26

Label propagation example

Leonid E. Zhukov (HSE) Lecture 17 19.05.2015 22 / 26

Label propagation example

Leonid E. Zhukov (HSE) Lecture 17 19.05.2015 23 / 26

Label propagation example

Leonid E. Zhukov (HSE) Lecture 17 19.05.2015 24 / 26

Label propagation example

Leonid E. Zhukov (HSE) Lecture 17 19.05.2015 25 / 26

References

S. A. Macskassy, F. Provost, Classification in Networked Data: A

Toolkit and a Univariate Case Study. Journal of Machine Learning
Research 8, 935-983, 2007
Bengio Yoshua, Delalleau Olivier, Roux Nicolas Le. Label Propagation
and Quadratic Criterion. Chapter in Semi-Supervised Learning, Eds.
O. Chapelle, B. Scholkopf, and A. Zien, MIT Press 2006
Smriti Bhagat, Graham Cormode, S. Muthukrishnan. Node
classification in social networks. Chapter in Social Network Data
Analytics, Eds. C. Aggrawal, 2011, pp 115-148
D. Zhou, O. Bousquet, T. Lal, J. Weston, and B. Scholkopf. Learning
with local and global consistency. In NIPS, volume 16, 2004.
X. Zhu, Z. Ghahramani, and J. Lafferty. Semi-supervised learning
using Gaussian fields and harmonic functions. In ICML, 2003.
M. Belkin, P. Niyogi, V. Sindhwani. Manifold regularization: A
geometric framework for learning from labeled and unlabeled
examples. J. Mach. Learn. Res., 7, 2399-2434, 2006
Leonid E. Zhukov (HSE) Lecture 17 19.05.2015 26 / 26