Recent Advances on Graph Analytics

and Its Applications in Healthcare

KDD 2020 Tutorial
August 23, morning
Fei Wang, Peng Cui, Jian Pei, Yangqiu Song, Chengxi Zang,
http://www.calvinzang.com/kdd2020_tutorial_medical_graph_analytics.html
KDD’20 Tutorial on Recent Advances on Graph Analytics and
Its Applications in Healthcare

Network Embedding and Graph Neural Networks

Peng Cui
Tsinghua University
 3

Healthcare and Graph

Many healthcare problems can be modeled as graph problems.

Drug retargeting Adverse drug reaction

http://www.cytoscape.org/

https://www.future-science.com/doi/10.4155/fmc.13.202
 4

Networks are not learning-friendly

Pipeline for network analysis
G = ( V, E )
Network
Data
Inapplicability of
ML methods
Feature
Extraction
Links Topology
Pattern
Discovery

Network
Applications
 5

Learning from networks

Network
GCN
Embedding
 6

Network Embedding
G = ( V, E ) G=(V)
Vector Space

generate

embed

• Easy to parallel
• Can apply classical ML methods
 7

The goal of network embedding

Goal Support network inference in vector space

Reflect network Maintain network

structure properties

A C

Transitivity
Transform network nodes into vectors that are fit for
off-the-shelf machine learning models.
 8

Graph Neural Networks

Design a learning mechanism on graph.

 Basic idea: recursive definition of states

 A simple example: PageRank

F. Scarselli, et al. The graph neural network model. IEEE TNN, 2009.
 9

Graph Convolutional Networks (GCN)

 Main idea: pass messages between pairs of nodes & agglomerate

 Stacking multiple layers like standard CNNs:

 State-of-the-art results on node classification

T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. ICLR, 2017.
 10

Network Embedding and GCN

Input Model Output
Feature

Network Task results

Graph
Embedding

Topology to Vector Embedding

Feature GCN Task results

Fusion of Topology and Features

Unsupervised v.s. (Semi-)Supervised

 11

Learning from networks

Network
GCN
Embedding
 12

The intrinsic problems NE is solving

Reducing representation dimensionality while preserving necessary

topological structures and properties.
Nodes & Links Non-transitivity

Node Neighborhood Asymmetric Transitivity

Pair-wise Proximity Uncertainty

Community Dynamic

Hyper Edges Heterogeneity

Global Structure Interpretability

Preserving Arbitrary-Order Proximity
• Shifting across different orders/weights:

Embedding1
Shifting

Embedding2
Eigen-decomposition
Λ 𝑋𝑋
Embedding3

Embedding4
Efficient!
……
• Preserving arbitrary-order proximity
• Low marginal cost
• Accurate and efficient

Z. Zhang,
Z. Zhang, et
et al.
al.Arbitrary-Order
Arbitrary-Order Proximity
Proximity Preserved
Preserved Network
Network Embedding.
Embedding. KDD,
KDD, 2018.
2018.
Preserving Arbitrary-Order Proximity
• High-order proximity: a polynomial function of the adjacency matrix
𝑆𝑆 = 𝑓𝑓 𝐴𝐴 = 𝑤𝑤1 𝐴𝐴1 + 𝑤𝑤2 𝐴𝐴2 + ⋯ + 𝑤𝑤𝑞𝑞 𝐴𝐴𝑞𝑞
• 𝑞𝑞: order; 𝑤𝑤1 …𝑤𝑤𝑞𝑞 : weights, assuming to be non-negative
• 𝐴𝐴: could be replaced by other variations (such as the Laplacian matrix)

• Objective function: matrix factorization

∗ ∗ 𝑇𝑇 2
min 𝑆𝑆 − 𝑈𝑈 𝑉𝑉 𝐹𝐹
𝑈𝑈 ∗ ,𝑉𝑉 ∗
• 𝑈𝑈 ∗ , 𝑉𝑉 ∗ ∈ ℝ𝑁𝑁×𝑑𝑑 : left/right embedding vectors
• d: dimensionality of the space

• Optimal solution: Singular Value Decomposition (SVD)

• 𝑈𝑈, Σ, 𝑉𝑉 : top-d SVD results
𝑈𝑈 ∗ = 𝑈𝑈 Σ, 𝑉𝑉 ∗ = 𝑉𝑉 Σ

Z. Zhang, et al. Arbitrary-Order Proximity Preserved Network Embedding. KDD, 2018.

Preserving Arbitrary-Order Proximity
• Eigen-decomposition reweighting

Efficient!
Eigen-decomposition
𝐴𝐴 Λ 𝑋𝑋

Time Consuming! Polynomial ℱ · Efficient! Polynomial ℱ ·

𝑆𝑆 Eigen-decomposition ℱ Λ
𝑋𝑋
Time Consuming!

Z. Zhang, et al. Arbitrary-Order Proximity Preserved Network Embedding. KDD, 2018.

Experimental Results
• Link Prediction

+100% +200%

Z. Zhang, et al. Arbitrary-Order Proximity Preserved Network Embedding. KDD, 2018.

 17

Hyper-network embedding
𝒆𝒆𝟏𝟏
𝒆𝒆𝟐𝟐

𝑨𝑨𝟐𝟐 𝐴𝐴2

𝑼𝑼𝟐𝟐 𝐿𝐿2 U2
𝑳𝑳𝟐𝟐

𝑳𝑳𝟏𝟏 𝐿𝐿1
𝑨𝑨𝟏𝟏 𝐴𝐴1

𝑼𝑼𝟏𝟏 𝑈𝑈1
𝒆𝒆𝟑𝟑
𝒆𝒆𝟒𝟒

Networks Hyper-Networks
• A hyper-network is a network in which an edge can include any number
of nodes
 18

Hyper-edges are often indecomposable

reaction

Person

paper venue

Drug
Author 1 Author 3

Author 2

Adverse Drug Network Bibliographic Network

Ke Tu, et al. Structural Deep Embedding for Hyper-Networks. AAAI, 2018.

 19

Structural Deep Network for Hyper-network

Unsupervised Heterogeneous Component
Node Type a Node Type b Node Type c
𝐴𝐴𝑎𝑎i , 𝐴𝐴𝑗𝑗𝑏𝑏 , 𝐴𝐴𝑐𝑐𝑘𝑘 … … …
second-order preserving
First Layer autoencoder autoencoder autoencoder

𝑋𝑋i𝑎𝑎 , 𝑋𝑋𝑗𝑗𝑏𝑏 , 𝑋𝑋𝑘𝑘𝑐𝑐 … … …

Second Layer Non-linear mapping

…
𝐿𝐿𝑖𝑖𝑖𝑖𝑖𝑖

.
…

Third Layer tuple-wise

first order preserving
similarity function
𝑆𝑆𝑖𝑖𝑖𝑖𝑖𝑖 +1 -1

Supervised Binary Component

Ke Tu, et al. Structural Deep Embedding for Hyper-Networks. AAAI, 2018.

 20

Experiment: link prediction

~14%

The overall performance Performance on networks of

different sparsity

Ke Tu, et al. Structural Deep Embedding for Hyper-Networks. AAAI, 2018.

 21

Learning from networks

Network
GCN
Embedding
 22

The intrinsic problem GCN is solving

Fusing topology and features in the way of smoothing features with

the assistance of topology.

N d d

N N N

X =
Robust GCN
Adversarial attacks
small perturbations in graph structures and node attributes
great challenges for applying GCNs to node classification
Robust GCN

Gaussian Based hidden representations: Attention mechanism: Sampling process:

Variance terms absorb the Remedy the propagation Explicitly considers mathematical
effects of adversarial attacks of adversarial attacks relevance between means and
variances
Dingyuan Zhu, Ziwei Zhang, Peng Cui, Wenwu Zhu. Robust Graph Convolutional Networks Against Adversarial Attacks. KDD, 2019.
Robust GCN
 Node Classification on Clean Datasets

 Against Non-targeted Adversarial Attacks

Dingyuan Zhu, Ziwei Zhang, Peng Cui, Wenwu Zhu. Robust Graph Convolutional Networks Against Adversarial Attacks. KDD, 2019.
Disentangled GCN
• A real-world graph is typically formed due to many latent factors.

 Existing GNNs/GCNs:
Social Circles  A holistic approach, that takes in
the whole neighborhood to produce
a single node representation.

 We suggest:
 To disentangle the latent factors.
(By segmenting the heterogeneous parts, and learning
multiple factor-specific representations for a node.)

 Robustness (e.g., not overreact to an irrelevant

factor) & Interpretability.
Disentangled GCN
• We present DisenGCN, the disentangled graph convolutional network.
• DisenConv, a disentangled multichannel convolutional layer (figure below).
• Each channel convolutes features related with a single latent factor.

Jianxin Ma, Peng Cui, Kun Kuang, Xin Wang, Wenwu Zhu. Disentangled Graph Convolutional Networks. ICML, 2019.
Disentangled GCN

Jianxin Ma, Peng Cui, Kun Kuang, Xin Wang, Wenwu Zhu. Disentangled Graph Convolutional Networks. ICML, 2019.
Some interesting questions for GCN…
 30

What if the problem is topology-driven?

 Since GCN is filtering features, it is inevitably feature-driven
 Structure only provides auxiliary information (e.g. for filtering/smoothing)
 When feature plays the key role, GNN performs good …
 How about the contrary?
 Synthesis data: stochastic block model + random features
Method Results
Random 10.0
GCN 18.3±1.1
DeepWalk 99.0±0.1
 31

Does GCN fuse feature and topology optimally?

Fusion Capability of GCNs

 Ideal Solution: extract the most correlated information for task

Case 1 Case 2

Random topology Correlated Features Correlated Topology Random Features

MLP(100%) > GCN(75.2%) DeepWalk(100%) > GCN(87%)

Xiao Wang, Meiqi Zhu, Deyu Bo, Peng Cui, Chuan Shi, Jian Pei. AM-GCN: Adaptive Multi-channel Graph Convolutional Networks.
ACM SIGKDD, 2020.
 32

Rethinking: Is GCN truly a Deep Learning method?

 Recall GNN formulation:

𝐻𝐻 𝑘𝑘+1
= 𝜎𝜎 𝑆𝑆𝐻𝐻 𝑘𝑘
𝑊𝑊 𝑘𝑘 � −1/2 𝐴𝐴̃ 𝐷𝐷
, 𝑆𝑆 = 𝐷𝐷 � −1/2
 How about removing the non-linear component:
𝐻𝐻 𝑘𝑘+1 = 𝑆𝑆𝐻𝐻 𝑘𝑘 𝑊𝑊 𝑘𝑘

 Stacking multiple layers and add softmax classification:

𝑌𝑌� = 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠 𝐻𝐻 𝐾𝐾
= 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠 𝑆𝑆𝑆𝑆 … 𝑆𝑆𝐻𝐻 0 𝑊𝑊 0 𝑊𝑊 1 … 𝑊𝑊 𝐾𝐾−1

= 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠 𝑆𝑆 𝐾𝐾 𝐻𝐻 0 𝑊𝑊
High-order proximity
Wu, Felix, et al. Simplifying graph convolutional networks. ICML, 2019.
 33

Rethinking: Is GCN truly a Deep Learning method?

 This simplified GNN (SGC) shows remarkable results:

Node classification Text Classification

Wu, Felix, et al. Simplifying graph convolutional networks. ICML, 2019.

 34

Network Embedding v.s. GCN

There is no better one, but there is more proper one.
Network GCN Feature-based
Embedding Learning

Node
Topology
Features
 35

Summaries and Conclusions

 Unsupervised v.s. (Semi-)Supervised

 Topology-driven v.s. Feature-driven
 For different healthcare tasks, there is no best one, but there is more
proper one.
 36

A Survey on Network Embedding

Peng Cui, Xiao Wang, Jian Pei, Wenwu Zhu. A Survey on Network Embedding. IEEE TKDE,
2019.
 37

Deep Learning on Graphs: A Survey

Ziwei Zhang, Peng Cui, Wenwu Zhu. Deep Learning on Graphs: A Survey. Arxiv, 2019.
 38

Thanks!

Peng Cui
[email protected]
http://pengcui.thumedialab.com