Hypergraph Neural Networks

Yifan Feng† , Haoxuan You‡ , Zizhao Zhang‡ , Rongrong Ji† , Yue Gao‡∗
†
Fujian Key Laboratory of Sensing and Computing for Smart City,
School of Information Science and Engineering, Xiamen University, 361005, China
‡
BNRist, KLISS, School of Software, Tsinghua University, 100084, China.
{evanfeng97, haoxuanyou}@gmail.com, [email protected], {zz-z14,gaoyue}@tsinghua.edu.cn
arXiv:1809.09401v2 [cs.LG] 14 Dec 2018

Abstract Tweets/Microblogs
In this paper, we present a hypergraph neural networks
(HGNN) framework for data representation learning, which Visual
can encode high-order data correlation in a hypergraph struc- connections
ture. Confronting the challenges of learning representation
for complex data in real practice, we propose to incorpo-
rate such data structure in a hypergraph, which is more flexi-
ble on data modeling, especially when dealing with complex
Text
data. In this method, a hyperedge convolution operation is
connections
designed to handle the data correlation during representation
learning. In this way, traditional hypergraph learning proce-
dure can be conducted using hyperedge convolution opera-
tions efficiently. HGNN is able to learn the hidden layer rep- Social
resentation considering the high-order data structure, which connections
is a general framework considering the complex data correla-
tions. We have conducted experiments on citation network
classification and visual object recognition tasks and com-
Figure 1: Examples of complex connections on social me-
pared HGNN with graph convolutional networks and other
traditional methods. Experimental results demonstrate that dia data. Each color point represents a tweet or microblog,
the proposed HGNN method outperforms recent state-of-the- and there could be visual connections, text connections and
art methods. We can also reveal from the results that the pro- social connections among them.
posed HGNN is superior when dealing with multi-modal data
compared with existing methods.
Figure 1 provides examples of complex connections on so-
Introduction cial media data. On one hand, the data correlation can be
Graph-based convolutional neural networks (Kipf and more complex than pairwise relationship, which is difficult
Welling 2017), (Defferrard, Bresson, and Vandergheynst to be modeled by a graph structure. On the other hand, the
2016) have attracted much attention in recent years. Dif- data representation tends to be multi-modal, such as the vi-
ferent from traditional convolutional neural networks, graph sual connections, text connections and social connections in
convolution is able to encode the graph structure of different this example. Under such circumstances, traditional graph
input data using a neural network model and it can be used structure has the limitation to formulate the data correlation,
in the semi-supervised learning procedure. Graph convolu- which limits the application of graph convolutional neural
tional neural networks have shown superiority on represen- networks. Under such circumstance, it is important and ur-
tation learning compared with traditional neural networks gent to further investigate better and more general data struc-
due to its ability of using data graph structure. ture model to learn representation.
In traditional graph convolutional neural network meth- To tackle this challenging issue, in this paper, we propose
ods, the pairwise connections among data are employed. It a hypergraph neural networks (HGNN) framework, which
is noted that the data structure in real practice could be be- uses the hypergraph structure for data modeling. Compared
yond pairwise connections and even far more complicated. with simple graph, on which the degree for all edges is
Confronting the scenarios with multi-modal data, the situa- mandatory 2, a hypergraph can encode high-order data cor-
tion for data correlation modelling could be more complex. relation (beyond pairwise connections) using its degree-free
∗
Corresponding author. This work was finished when Yifan hyperedges, as shown in Figure 2. In Figure 2, the graph is
Feng visited Tsinghua University. represented using the adjacency matrix, in which each edge
Copyright c 2019, Association for the Advancement of Artificial connects just two vertices. On the contrary, a hypergraph
Intelligence (www.aaai.org). All rights reserved. is easy to be expanded for multi-modal and heterogeneous
 Graph: Hypergraph:
H1: e1 e2 e3
n8 n1 0 1 0
n8 n2
n1 e2 n7 1 0 0

Data type 1
n1 n7
n3 0 0 1

n4 1 0 0
n2 n6 n2 n6 H: e1 e2 e3 em-1 em
e1 n5 0 0 1
n1 0 1 0 1 0
e3 n6 0 1 0
n2 1 0 0 1 0
n3 n5 n3 n5 n7 0 1 1
n3 0 0 1 0 1
n4 n4 n8 1 0 0
Concat n4 1 0 0 0 1
Hyperedge group 1 n5 0 0 1 1 0
n6
W: n1 n2 n3 n4 n5 n6 n7 n8
0 1 0 1 0

HN: em-1 em n7 0 1 1 0 1
n1 0 0 0 0 1 1 0 0 n8 n1 1 0 n8 1 0 0 0 1

Data type N
n2 0 0 0 0 0 1 0 0 n1 n7 n2 1 0
n3 0 0 0 0 0 1 0 1 em-1 (H1) (HN)
n3 0 1
n4 0 0 0 0 0 0 1 0 n2 n6 0 1
n4
n5 1 0 0 0 0 0 0 0
n5 1 0
n6 1 1 1 0 0 0 0 0
em n6
n3 n5 1 0
n7 0 0 0 1 0 0 0 0
n7
n4 0 1
n8 0 0 1 0 0 0 0 0
n8 0 1
Hyperedge group N

Figure 2: The comparison between graph and hypergraph.

data representation using its flexible hyperedges. For exam- 2. We have conducted extensive experiments on citation net-
ple, a hypergraph can jointly employ multi-modal data for work classification and visual object classification tasks.
hypergraph generation by combining the adjacency matrix, Comparisons with state-of-the-art methods demonstrate
as illustrated in Figure 2. Therefore, hypergraph has been the effectiveness of the proposed HGNN framework. Ex-
employed in many computer vision tasks such as classifi- periments also indicate the better performance of the pro-
cation and retrieval tasks (Gao et al. 2012). However, tra- posed method when dealing with multi-modal data.
ditional hypergraph learning methods (Zhou, Huang, and
Schölkopf 2007) suffer from their high computation com-
plexity and storage cost, which limits the wide application
Related Work
of hypergraph learning methods. In this section, we briefly review existing works of hyper-
In this paper, we propose a hypergraph neural networks graph learning and neural networks on graph.
framework (HGNN) for data representation learning. In this
method, the complex data correlation is formulated in a hy- Hypergraph learning
pergraph structure, and we design a hyperedge convolution In many computer vision tasks, the hypergraph structure has
operation to better exploit the high-order data correlation been employed to model high-order correlation among data.
for representation learning. More specifically, HGNN is a Hypergraph learning is first introduced in (Zhou, Huang, and
general framework which can incorporate with multi-modal Schölkopf 2007), as a propagation process on hypergraph
data and complicated data correlations. Traditional graph structure. The transductive inference on hypergraph aims to
convolutional neural networks can be regarded as a special minimize the label difference among vertices with stronger
case of HGNN. To evaluate the performance of the pro- connections on hypergraph. In (Huang, Liu, and Metaxas
posed HGNN framework, we have conducted experiments 2009), hypergraph learning is further employed in video ob-
on citation network classification and visual object recog- ject segmentation. (Huang et al. 2010) used the hypergraph
nition tasks. The experimental results on four datasets and structure to model image relationship and conducted trans-
comparisons with graph convolutional network (GCN) and ductive inference process for image ranking. To further im-
other traditional methods have shown better performance prove the hypergraph structure, research attention has been
of HGNN. These results indicate that the proposed HGNN attracted for leaning the weights of hyperedges, which have
method is more effective on learning data representation us- great influence on modeling the correlation of data. In (Gao
ing high-order and complex correlations. et al. 2013), a l2 regularize on the weights is introduced to
The main contributions of this paper are two-fold: learn optimal hyperedge weights. In (Hwang et al. 2008),
1. We propose a hypergraph neural networks framework, the correlation among hyperedges is further explored by a
i.e., HGNN, for representation learning using hypergraph assumption that highly correlated hyperedges should have
structure. HGNN is able to formulate complex and high- similar weights. Regarding the multi-modal data, in (Gao et
order data correlation through its hypergraph structure al. 2012), multi-hypergraph structure is introduced to assign
and can be also efficient using hyperedge convolution weights for different sub-hypergraphs, which corresponds to
operations. It is effective on dealing with multi-modal different modalities.
data/features. Moreover, GCN (Kipf and Welling 2017)
can be regarded as a special case of HGNN, for which the Neural networks on graph
edges in simple graph can be regarded as 2-order hyper- Since many irregular data that do not own a grid-like struc-
edges which connect just two vertices. ture can only be represented in the form of graph, extending
 Node Feature
Multimodal Data Hyperedge Generation Hypergraph Neural Networks
Training Data Testing Data

Hyperedge
Data Type 1

group 1

Hypergraph

Hypergraph
Conv

Conv
Hyperedge
group m
Data Type m

HConv 1 HConv n

Labels

Node Features

Figure 3: The proposed HGNN framework.

neural networks to graph structure has attracted great atten- matrix of edge weights. The hypergraph G can be denoted
tion from researchers. In (Gori, Monfardini, and Scarselli by a |V| × |E| incidence matrix H, with entries defined as
2005) and (Scarselli et al. 2009), the neural network on 
graph is first introduced to apply recurrent neural networks 1, if v ∈ e
h(v, e) = (1)
to deal with graphs. For generalizing convolution network to 0, if v 6∈ e,
graph, the methods are divided into spectral and non-spectral
approaches. PFor a vertex v ∈ V, its degree is defined as d(v) =
For spectral approaches, the convolution operation is for- P e). For an edge e ∈ E, its degree is defined
e∈E ω(e)h(v,
mulated in spectral domain of graph. (Bruna et al. 2014) in- as δ(e) = v∈V h(v, e). Further, De and Dv denote the di-
troduces the first graph CNN, which uses the graph Lapla- agonal matrices of the edge degrees and the vertex degrees,
cian eigenbasis as an analogy of the Fourier transform. respectively.
In (Henaff, Bruna, and LeCun 2015), the spectral filters Here let us consider the node(vertex) classification prob-
can be parameterized with smooth coefficients to make lem on hypergraph, where the node labels should be smooth
them spatial-localized. In (Defferrard, Bresson, and Van- on the hypergraph structure. The task can be formulated as
dergheynst 2016), a Chebyshev expansion of the graph a regularization framework as introduced by (Zhou, Huang,
Laplacian is further used to approximate the spectral filters. and Schölkopf 2007):
Then, in (Kipf and Welling 2017), the chebyshev polynomi- arg min {Remp (f ) + Ω(f )}, (2)
als are simplified into 1-order polynomials to form an effi- f
cient layer-wise propagation model.
For spatial approaches, the convolution operation is de- where Ω(f ) is a regularize on hypergraph, Remp (f ) denotes
fined in groups of spatial close nodes. In (Atwood and the supervised empirical loss, f (·) is a classification func-
Towsley 2016), the powers of a transition matrix is em- tion. The regularize Ω(f ) is defined as:
ployed to define the neighborhood of nodes. (Monti et al. 1X X w(e)h(u, e)h(v, e)
2017) uses the local path operators in the form of Gaussian Ω(f ) =
mixture models to generalize convolution in spatial domain. 2 δ(e)
e∈E {u,v}∈V
In (Velickovic et al. 2018), the attention mechanisms is in- (3)
 f (u) f (v) 2
troduced into the graph to build attention-based architecture p −p ,
to perform the node classification task on graph. d(u) d(v)
−1/2 −1/2
We let θ = Dv HWD−1 >
e H Dv and ∆ = I − Θ.
Hypergraph Neural Networks Then, the normalized Ω(f ) can be written as
In this section, we introduce our proposed hypergraph neu-
ral networks (HGNN). We first briefly introduce hypergraph Ω(f ) = f > ∆, (4)
learning, and then the spectral convolution on hypergraph
is provided. Following, we analyze the relations between where ∆ is positive semi-definite, and usually called the hy-
HGNN and existing methods. In the last part of the section, pergraph Laplacian.
some implementation details will be given.
Spectral convolution on hypergraph
Hypergraph learning statement Given a hypergraph G = (V, E, ∆) with n vertices,
We first review the hypergraph analysis theory. Different since the hypergraph Laplacian ∆ is a n × n positive
from simple graph, a hyperedge in a hypergraph connects semi-definite matrix, the eigen decomposition ∆ = ΦΛΦ>
two or more vertices. A hypergraph is defined as G = can be employed to get the orthonormal eigen vectors
(V, E, W), which includes a vertex set V, a hyperedge set E. Φ = diag(φ1 , . . . , φn ) and a diagonal matrix Λ =
Each hyperedge is assigned with a weight by W, a diagonal diag(λ1 , . . . , λn ) containing corresponding non-negative
 n8
<latexit sha1_base64="zka5wiLNeuIzdnNzOY6ZgfZJctY=">AAAB63icbVBNS8NAEJ3Ur1q/qh69LBbBU0lEsMeiF48V7Ae0oWy2m3bp7ibsToQS+he8eFDEq3/Im//GpM1BWx8MPN6bYWZeEEth0XW/ndLG5tb2Tnm3srd/cHhUPT7p2CgxjLdZJCPTC6jlUmjeRoGS92LDqQok7wbTu9zvPnFjRaQfcRZzX9GxFqFgFHNJDxuVYbXm1t0FyDrxClKDAq1h9WswiliiuEYmqbV9z43RT6lBwSSfVwaJ5TFlUzrm/Yxqqrj108Wtc3KRKSMSRiYrjWSh/p5IqbJ2poKsU1Gc2FUvF//z+gmGDT8VOk6Qa7ZcFCaSYETyx8lIGM5QzjJCmRHZrYRNqKEMs3jyELzVl9dJ56ruuXXv4brWvC3iKMMZnMMleHADTbiHFrSBwQSe4RXeHOW8OO/Ox7K15BQzp/AHzucPPCWNsQ==</latexit>
n7 n2
<latexit sha1_base64="bjIC+35MlRj3rmKoxI1XWIuyzrA=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKUI9FLx4r2g9oQ9lsN+3SzSbsToQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSNnGqGW+xWMa6G1DDpVC8hQIl7yaa0yiQvBNMbud+54lrI2L1iNOE+xEdKREKRtFKD2pQG5QrbtVdgKwTLycVyNEclL/6w5ilEVfIJDWm57kJ+hnVKJjks1I/NTyhbEJHvGepohE3frY4dUYurDIkYaxtKSQL9fdERiNjplFgOyOKY7PqzcX/vF6K4bWfCZWkyBVbLgpTSTAm87/JUGjOUE4toUwLeythY6opQ5tOyYbgrb68Ttq1qudWvfurSuMmj6MIZ3AOl+BBHRpwB01oAYMRPMMrvDnSeXHenY9la8HJZ07hD5zPH/3pjZc=</latexit>
n8
<latexit sha1_base64="zka5wiLNeuIzdnNzOY6ZgfZJctY=">AAAB63icbVBNS8NAEJ3Ur1q/qh69LBbBU0lEsMeiF48V7Ae0oWy2m3bp7ibsToQS+he8eFDEq3/Im//GpM1BWx8MPN6bYWZeEEth0XW/ndLG5tb2Tnm3srd/cHhUPT7p2CgxjLdZJCPTC6jlUmjeRoGS92LDqQok7wbTu9zvPnFjRaQfcRZzX9GxFqFgFHNJDxuVYbXm1t0FyDrxClKDAq1h9WswiliiuEYmqbV9z43RT6lBwSSfVwaJ5TFlUzrm/Yxqqrj108Wtc3KRKSMSRiYrjWSh/p5IqbJ2poKsU1Gc2FUvF//z+gmGDT8VOk6Qa7ZcFCaSYETyx8lIGM5QzjJCmRHZrYRNqKEMs3jyELzVl9dJ56ruuXXv4brWvC3iKMMZnMMleHADTbiHFrSBwQSe4RXeHOW8OO/Ox7K15BQzp/AHzucPPCWNsQ==</latexit>
n7
<latexit sha1_base64="DpErsjR/8ayk1yFvuxTfBiYrwsg=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEaI9FLx4r2g9oQ9lsN+3SzSbsToQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSNnGqGW+xWMa6G1DDpVC8hQIl7yaa0yiQvBNMbud+54lrI2L1iN
sha1_base64="DpErsjR/8ayk1yFvuxTfBiYrwsg=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEaI9FLx4r2g9oQ9lsN+3SzSbsToQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSNnGqGW+xWMa6G1DDpVC8hQIl7yaa0yiQvBNMbud+54lrI2L1iNOE+xEdKREKRtFKD2pQG5QrbtVdgKwTLycVyNEclL/6w5ilEVfIJDWm57kJ+hnVKJjks1I/NTyhbEJHvGepohE3frY4dUYurDIkYaxtKSQL9fdERiNjplFgOyOKY7PqzcX/vF6KYd3PhEpS5IotF4WpJBiT+d9kKDRnKKeWUKaFvZWwMdWUoU2nZEPwVl9eJ+2rqudWvfvrSuMmj6MIZ3AOl+BBDRpwB01oAYMRPMMrvDnSeXHenY9la8HJZ07hD5zPHwWMjZw=</latexit>

e2 e1
n1
<latexit sha1_base64="DpErsjR/8ayk1yFvuxTfBiYrwsg=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEaI9FLx4r2g9oQ9lsN+3SzSbsToQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSNnGqGW+xWMa6G1DDpVC8hQIl7yaa0yiQvBNMbud+54lrI2L1iNOE+xEdKREKRtFKD2pQG5QrbtVdgKwTLycVyNEclL/6w5ilEVfIJDWm57kJ+hnVKJjks1I/NTyhbEJHvGepohE3frY4dUYurDIkYaxtKSQL9fdERiNjplFgOyOKY7PqzcX/vF6KYd3PhEpS5IotF4WpJBiT+d9kKDRnKKeWUKaFvZWwMdWUoU2nZEPwVl9eJ+2rqudWvfvrSuMmj6MIZ3AOl+BBDRpwB01oAYMRPMMrvDnSeXHenY9la8HJZ07hD5zPHwWMjZw=</latexit>

e2 e1 n1
n6
<latexit sha1_base64="TbOVm4a1bj7EWv2q2BG16ZUEIEA=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPRi8eK9gPaUDbbSbt0swm7G6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTAXXxvO+ndLa+sbmVnm7srO7t3/gHh61dJIphk2WiER1QqpRcIlNw43ATqqQxqHAdji+nfntJ1SaJ/LRTFIMYjqUPOKMGis9yL7fd6tezZuDrBK/IFUo0Oi7X71BwrIYpWGCat31vdQEOVWGM4HTSi/TmFI2pkPsWippjDrI56dOyZlVBiRKlC1pyFz9PZHTWOtJHNrOmJqRXvZm4n9eNzPRdZBzmWYGJVssijJBTEJmf5MBV8iMmFhCmeL2VsJGVFFmbDoVG4K//PIqaV3UfK/m319W6zdFHGU4gVM4Bx+uoA530IAmMBjCM7zCmyOcF+fd+Vi0lpxi5hj+wPn8AfxljZY=</latexit>

<latexit sha1_base64="mfO9ARkQyRLj4TbGagbpzJgRzY0=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKUI9FLx4r2g9oQ9lsJ+3SzSbsboQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKHSPJaPZpqgH9GR5CFn1FjpAQe1QbniVt0FyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP66UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ+1a1XOr3v1VpXGTx1GEMziHS/CgDg24gya0gMEInuEV3hzhvDjvzseyteDkM6fwB87nD/AzjY4=</latexit>

e3 e2 n7 n2 n2
n2
<latexit sha1_base64="tbNUust/TB+TBhNg9a465x4qcNI=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqseiF48V7Qe0oWy2k3bpZhN2N0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3n1BpHstHM0nQj+hQ8pAzaqz0IPu1frniVt05yCrxclKBHI1++as3iFkaoTRMUK27npsYP6PKcCZwWuqlGhPKxnSIXUsljVD72fzUKTmzyoCEsbIlDZmrvycyGmk9iQLbGVEz0sveTPzP66YmvPYzLpPUoGSLRWEqiInJ7G8y4AqZERNLKFPc3krYiCrKjE2nZEPwll9eJa2LqudWvfvLSv0mj6MIJ3AK5+DBFdThDhrQBAZDeIZXeHOE8+K8Ox+L1oKTzxzDHzifPwQIjZs=</latexit>

<latexit sha1_base64="bjIC+35MlRj3rmKoxI1XWIuyzrA=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKUI9FLx4r2g9oQ9lsN+3SzSbsToQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSNnGqGW+xWMa6G1DDpVC8hQIl7yaa0yiQvBNMbud+54lrI2L1iNOE+xEdKREKRtFKD2pQG5QrbtVdgKwTLycVyNEclL/6w5ilEVfIJDWm57kJ+hnVKJjks1I/NTyhbEJHvGepohE3frY4dUYurDIkYaxtKSQL9fdERiNjplFgOyOKY7PqzcX/vF6K4bWfCZWkyBVbLgpTSTAm87/JUGjOUE4toUwLeythY6opQ5tOyYbgrb68Ttq1qudWvfurSuMmj6MIZ3AOl+BBHRpwB01oAYMRPMMrvDnSeXHenY9la8HJZ07hD5zPH/3pjZc=</latexit>

e1 e1
<latexit sha1_base64="6zA7gEs4vLWyBlUv9CAvgcAHwFg=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPRi8eK9gPaUDbbSbt0swm7G6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTAXXxvO+ndLa+sbmVnm7srO7t3/gHh61dJIphk2WiER1QqpRcIlNw43ATqqQxqHAdji+nfntJ1SaJ/LRTFIMYjqUPOKMGis9YN/vu1Wv5s1BVolfkCoUaPTdr94gYVmM0jBBte76XmqCnCrDmcBppZdpTCkb0yF2LZU0Rh3k81On5MwqAxIlypY0ZK7+nshprPUkDm1nTM1IL3sz8T+vm5noOsi5TDODki0WRZkgJiGzv8mAK2RGTCyhTHF7K2EjqigzNp2KDcFffnmVtC5qvlfz7y+r9ZsijjKcwCmcgw9XUIc7aEATGAzhGV7hzRHOi/PufCxaS04xcwx/4Hz+AO6vjY0=</latexit>
<latexit sha1_base64="Ixs6byDxI13PHmv3i/ziV98fHIw=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0m0oMeiF48V7Qe0oWy2k3bpZhN2N0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3n1BpHstHM0nQj+hQ8pAzaqz0gP3LfrniVt05yCrxclKBHI1++as3iFkaoTRMUK27npsYP6PKcCZwWuqlGhPKxnSIXUsljVD72fzUKTmzyoCEsbIlDZmrvycyGmk9iQLbGVEz0sveTPzP66YmvPYzLpPUoGSLRWEqiInJ7G8y4AqZERNLKFPc3krYiCrKjE2nZEPwll9eJa2LqudWvftapX6Tx1GEEziFc/DgCupwBw1oAoMhPMMrvDnCeXHenY9Fa8HJZ47hD5zPH/G3jY8=</latexit>

e3 n3
e 3 n5
<latexit sha1_base64="6zA7gEs4vLWyBlUv9CAvgcAHwFg=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPRi8eK9gPaUDbbSbt0swm7G6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTAXXxvO+ndLa+sbmVnm7srO7t3/gHh61dJIphk2WiER1QqpRcIlNw43ATqqQxqHAdji+nfntJ1SaJ/LRTFIMYjqUPOKMGis9YN/vu1Wv5s1BVolfkCoUaPTdr94gYVmM0jBBte76XmqCnCrDmcBppZdpTCkb0yF2LZU0Rh3k81On5MwqAxIlypY0ZK7+nshprPUkDm1nTM1IL3sz8T+vm5noOsi5TDODki0WRZkgJiGzv8mAK2RGTCyhTHF7K2EjqigzNp2KDcFffnmVtC5qvlfz7y+r9ZsijjKcwCmcgw9XUIc7aEATGAzhGV7hzRHOi/PufCxaS04xcwx/4Hz+AO6vjY0=</latexit>

em nn
n3 n5
<latexit sha1_base64="Ixs6byDxI13PHmv3i/ziV98fHIw=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0m0oMeiF48V7Qe0oWy2k3bpZhN2N0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3n1BpHstHM0nQj+hQ8pAzaqz0gP3LfrniVt05yCrxclKBHI1++as3iFkaoTRMUK27npsYP6PKcCZwWuqlGhPKxnSIXUsljVD72fzUKTmzyoCEsbIlDZmrvycyGmk9iQLbGVEz0sveTPzP66YmvPYzLpPUoGSLRWEqiInJ7G8y4AqZERNLKFPc3krYiCrKjE2nZEPwll9eJa2LqudWvftapX6Tx1GEEziFc/DgCupwBw1oAoMhPMMrvDnCeXHenY9Fa8HJZ47hD5zPH/G3jY8=</latexit>

<latexit sha1_base64="D3jROYEN/QbXF/wsb0gVOC1ZS38=">AAAB6nicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzQOSJcxOepMhM7PLzKwQQj7BiwdFvPpF3vwbJ8keNLGgoajqprsrSgU31ve/vcLa+sbmVnG7tLO7t39QPjxqmiTTDBssEYluR9Sg4AoblluB7VQjlZHAVjS6nfmtJ9SGJ+rRjlMMJR0oHnNGrZMesCd75Ypf9ecgqyTISQVy1Hvlr24/YZlEZZmgxnQCP7XhhGrLmcBpqZsZTCkb0QF2HFVUogkn81On5MwpfRIn2pWyZK7+nphQacxYRq5TUjs0y95M/M/rZDa+DidcpZlFxRaL4kwQm5DZ36TPNTIrxo5Qprm7lbAh1ZRZl07JhRAsv7xKmhfVwK8G95eV2k0eRxFO4BTOIYArqMEd1KEBDAbwDK/w5gnvxXv3PhatBS+fOYY/8D5/AEmujck=</latexit>

<latexit sha1_base64="p9y3cs0G0q7hp9YjBH6OpaJsVlg=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0nEoseiF48V7Qe0oWy2k3bpZhN2N0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3n1BpHstHM0nQj+hQ8pAzaqz0IPu1frniVt05yCrxclKBHI1++as3iFkaoTRMUK27npsYP6PKcCZwWuqlGhPKxnSIXUsljVD72fzUKTmzyoCEsbIlDZmrvycyGmk9iQLbGVEz0sveTPzP66YmvPYzLpPUoGSLRWEqiInJ7G8y4AqZERNLKFPc3krYiCrKjE2nZEPwll9eJa2LqudWvfvLSv0mj6MIJ3AK5+DBFdThDhrQBAZDeIZXeHOE8+K8Ox+L1oKTzxzDHzifPwKEjZo=</latexit>

n4 n3
<latexit sha1_base64="viX2VKqEK4KOxQmuXZwEBoDvRBE=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPRi8eK9gPaUDbbSbt0swm7G6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTAXXxvO+ndLa+sbmVnm7srO7t3/gHh61dJIphk2WiER1QqpRcIlNw43ATqqQxqHAdji+nfntJ1SaJ/LRTFIMYjqUPOKMGis9yL7su1Wv5s1BVolfkCoUaPTdr94gYVmM0jBBte76XmqCnCrDmcBppZdpTCkb0yF2LZU0Rh3k81On5MwqAxIlypY0ZK7+nshprPUkDm1nTM1IL3sz8T+vm5noOsi5TDODki0WRZkgJiGzv8mAK2RGTCyhTHF7K2EjqigzNp2KDcFffnmVtC5qvlfz7y+r9ZsijjKcwCmcgw9XUIc7aEATGAzhGV7hzRHOi/PufCxaS04xcwx/4Hz+AFjojdM=</latexit>

<latexit sha1_base64="yXoDxELpx3VhIJk9qVncqGfvwS0=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0m0oMeiF48V7Qe0oWy2k3bpZhN2N0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3n1BpHstHM0nQj+hQ8pAzaqz0IPuX/XLFrbpzkFXi5aQCORr98ldvELM0QmmYoFp3PTcxfkaV4UzgtNRLNSaUjekQu5ZKGqH2s/mpU3JmlQEJY2VLGjJXf09kNNJ6EgW2M6JmpJe9mfif101NeO1nXCapQckWi8JUEBOT2d9kwBUyIyaWUKa4vZWwEVWUGZtOyYbgLb+8SloXVc+teve1Sv0mj6MIJ3AK5+DBFdThDhrQBAZDeIZXeHOE8+K8Ox+L1oKTzxzDHzifP/9tjZg=</latexit>

n4 <latexit sha1_base64="qj1TO30AZnbRJEF2eK9SuYWxaeA=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkUI9FLx4r2g9oQ9lsN+3SzSbsToQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSNnGqGW+xWMa6G1DDpVC8hQIl7yaa0yiQvBNMbud+54lrI2L1iNOE+xEdKREKRtFKD2pQG5QrbtVdgKwTLycVyNEclL/6w5ilEVfIJDWm57kJ+hnVKJjks1I/NTyhbEJHvGepohE3frY4dUYurDIkYaxtKSQL9fdERiNjplFgOyOKY7PqzcX/vF6K4bWfCZWkyBVbLgpTSTAm87/JUGjOUE4toUwLeythY6opQ5tOyYbgrb68TtpXVc+teve1SuMmj6MIZ3AOl+BBHRpwB01oAYMRPMMrvDnSeXHenY9la8HJZ07hD5zPHwEAjZk=</latexit>
<latexit sha1_base64="qj1TO30AZnbRJEF2eK9SuYWxaeA=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkUI9FLx4r2g9oQ9lsN+3SzSbsToQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSNnGqGW+xWMa6G1DDpVC8hQIl7yaa0yiQvBNMbud+54lrI2L1iNOE+xEdKREKRtFKD2pQG5QrbtVdgKwTLycVyNEclL/6w5ilEVfIJDWm57kJ+hnVKJjks1I/NTyhbEJHvGepohE3frY4dUYurDIkYaxtKSQL9fdERiNjplFgOyOKY7PqzcX/vF6K4bWfCZWkyBVbLgpTSTAm87/JUGjOUE4toUwLeythY6opQ5tOyYbgrb68TtpXVc+teve1SuMmj6MIZ3AOl+BBHRpwB01oAYMRPMMrvDnSeXHenY9la8HJZ07hD5zPHwEAjZk=</latexit>
<latexit sha1_base64="yXoDxELpx3VhIJk9qVncqGfvwS0=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0m0oMeiF48V7Qe0oWy2k3bpZhN2N0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3n1BpHstHM0nQj+hQ8pAzaqz0IPuX/XLFrbpzkFXi5aQCORr98ldvELM0QmmYoFp3PTcxfkaV4UzgtNRLNSaUjekQu5ZKGqH2s/mpU3JmlQEJY2VLGjJXf09kNNJ6EgW2M6JmpJe9mfif101NeO1nXCapQckWi8JUEBOT2d9kwBUyIyaWUKa4vZWwEVWUGZtOyYbgLb+8SloXVc+teve1Sv0mj6MIJ3AK5+DBFdThDhrQBAZDeIZXeHOE8+K8Ox+L1oKTzxzDHzifP/9tjZg=</latexit>
<latexit sha1_base64="p9y3cs0G0q7hp9YjBH6OpaJsVlg=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0nEoseiF48V7Qe0oWy2k3bpZhN2N0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3n1BpHstHM0nQj+hQ8pAzaqz0IPu1frniVt05yCrxclKBHI1++as3iFkaoTRMUK27npsYP6PKcCZwWuqlGhPKxnSIXUsljVD72fzUKTmzyoCEsbIlDZmrvycyGmk9iQLbGVEz0sveTPzP66YmvPYzLpPUoGSLRWEqiInJ7G8y4AqZERNLKFPc3krYiCrKjE2nZEPwll9eJa2LqudWvfvLSv0mj6MIJ3AK5+DBFdThDhrQBAZDeIZXeHOE8+K8Ox+L1oKTzxzDHzifPwKEjZo=</latexit>

e3
Node Feature Hyperedge Feature Node Feature
N ⇥ C1

E ⇥ C2

N ⇥ C2
N ⇥ C2
Node Feature Edge Feature Node Feature
N: Node Number
Transform Gathering Aggregating
E: Hyperedge Number
<latexit sha1_base64="iL6zRC2XmWpyDMX0fR/mBzocyR4=">AAAB83icbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GOxF09SwX5AE8pmu2mXbjZhdyKU0L/hxYMiXv0z3vw3btsctPpg4PHeDDPzwlQKg6775ZTW1jc2t8rblZ3dvf2D6uFRxySZZrzNEpnoXkgNl0LxNgqUvJdqTuNQ8m44ac797iPXRiTqAacpD2I6UiISjKKV/Dvio4i5Ic2BN6jW3Lq7APlLvILUoEBrUP30hwnLYq6QSWpM33NTDHKqUTDJZxU/MzylbEJHvG+ponZRkC9unpEzqwxJlGhbCslC/TmR09iYaRzazpji2Kx6c/E/r59hdB3kQqUZcsWWi6JMEkzIPAAyFJozlFNLKNPC3krYmGrK0MZUsSF4qy//JZ2LuufWvfvLWuOmiKMMJ3AK5+DBFTTgFlrQBgYpPMELvDqZ8+y8Oe/L1pJTzBzDLzgf37rlkNE=</latexit>

<latexit sha1_base64="XsY80slebKt2ayXYT46tJsGAa2U=">AAAB83icbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoMdiETxWsB/QhLLZbtqlm03YnQil9G948aCIV/+MN/+N2zYHbX0w8Hhvhpl5YSqFQdf9dgobm1vbO8Xd0t7+weFR+fikbZJMM95iiUx0N6SGS6F4CwVK3k01p3EoeSccN+Z+54lrIxL1iJOUBzEdKhEJRtFK/h3xUcTckEa/1i9X3Kq7AFknXk4qkKPZL3/5g4RlMVfIJDWm57kpBlOqUTDJZyU/MzylbEyHvGeponZRMF3cPCMXVhmQKNG2FJKF+ntiSmNjJnFoO2OKI7PqzcX/vF6G0U0wFSrNkCu2XBRlkmBC5gGQgdCcoZxYQpkW9lbCRlRThjamkg3BW315nbRrVc+teg9XlfptHkcRzuAcLsGDa6jDPTShBQxSeIZXeHMy58V5dz6WrQUnnzmFP3A+fwCuYpDJ</latexit>

<latexit sha1_base64="3r+HEMsLu6mYRJ4k0vZFNb1c9Xo=">AAAB83icbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoMdiL56kgv2AJpTNdtMu3WzC7kQopX/DiwdFvPpnvPlv3LY5aOuDgcd7M8zMC1MpDLrut1PY2Nza3inulvb2Dw6PyscnbZNkmvEWS2SiuyE1XArFWyhQ8m6qOY1DyTvhuDH3O09cG5GoR5ykPIjpUIlIMIpW8u+JjyLmhjT6tX654lbdBcg68XJSgRzNfvnLHyQsi7lCJqkxPc9NMZhSjYJJPiv5meEpZWM65D1LFbWLguni5hm5sMqARIm2pZAs1N8TUxobM4lD2xlTHJlVby7+5/UyjG6CqVBphlyx5aIokwQTMg+ADITmDOXEEsq0sLcSNqKaMrQxlWwI3urL66Rdq3pu1Xu4qtRv8ziKcAbncAkeXEMd7qAJLWCQwjO8wpuTOS/Ou/OxbC04+cwp/IHz+QO8aZDS</latexit>
<latexit sha1_base64="3r+HEMsLu6mYRJ4k0vZFNb1c9Xo=">AAAB83icbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoMdiL56kgv2AJpTNdtMu3WzC7kQopX/DiwdFvPpnvPlv3LY5aOuDgcd7M8zMC1MpDLrut1PY2Nza3inulvb2Dw6PyscnbZNkmvEWS2SiuyE1XArFWyhQ8m6qOY1DyTvhuDH3O09cG5GoR5ykPIjpUIlIMIpW8u+JjyLmhjT6tX654lbdBcg68XJSgRzNfvnLHyQsi7lCJqkxPc9NMZhSjYJJPiv5meEpZWM65D1LFbWLguni5hm5sMqARIm2pZAs1N8TUxobM4lD2xlTHJlVby7+5/UyjG6CqVBphlyx5aIokwQTMg+ADITmDOXEEsq0sLcSNqKaMrQxlWwI3urL66Rdq3pu1Xu4qtRv8ziKcAbncAkeXEMd7qAJLWCQwjO8wpuTOS/Ou/OxbC04+cwp/IHz+QO8aZDS</latexit>
Node Feature Node Feature Hyperedge Feature Node Feature

F gure 4 The us ra on of he hyperedge convo u on ayer

e genva ues Then he Four er ransform for a s gna x = Then he convo u on opera on can be s mp fied o he
(x1 xn ) n hypergraph s defined as x̂ = Φ> x where fo ow ng express on
he e gen vec ors are regarded as he Four er bases and he
1 −1/2
e genva ues are n erpre ed as frequenc es The spec ra con- g?x≈ θD H(W + I)D−1 > −1/2
e H Dv x
vo u on of s gna x and fi er g can be deno ed as 2 v (9)
g ? x = Φ((Φ g) (Φ x)) = Φg(Λ)Φ x (5) > > > ≈ θD−1/2
v HWD−1 > −1/2
e H Dv x
where deno es he e emen -w se Hadamard produc and where (W + I) can be regarded as he we gh of he hyper-
g(Λ) = d ag(g(λ1 ) g(λn )) s a func on of he Four er edges W s n a zed as an den y ma r x wh ch means
coeffic en s However he compu a on cos n forward and equa we gh s for a hyperedges
nverse Four er ransform s O(n2 ) To so ve he prob- When we have a hypergraph s gna X ∈ Rn×C1 w h n
em we can fo ow (Defferrard Bresson and Vandergheyns nodes and C1 d mens ona fea ures our hyperedge convo u-
2016) o parame r ze g(Λ) w h K order po ynom a s Fur- on can be formu a ed by
hermore we use he runca ed Chebyshev expans on as
one such po ynom a Chebyshv po ynom a s Tk (x) s recur- Y = D−1/2
v HWD−1 > −1/2
e H Dv XΘ (10)
s ve y compu ed by Tk (x) = 2xTk−1 (x) − Tk−2 (x) w h C1 ×C2
where W = d ag(w1 wn ) Θ ∈ R s he param-
T0 (x) = 1 and T1 (x) = x Thus he g(Λ) can be parame-
e er o be earned dur ng he ra n ng process The fi er Θ
r ed as
K s app ed over he nodes n hypergraph o ex rac fea ures
Af er convo u on we can ob a n Y ∈ Rn×C2 wh ch can be
X
g?x≈ ˜
θk Tk (∆)x (6)
k=0
used for c ass fica on
˜ s he Chebyshev po ynom a of order k w h
where Tk (∆) Hypergraph neural networks analysis
˜ = 2 ∆ − I In Equa on 6 he ex-
sca ed Lap ac an ∆ λmax F gure 3 us ra es he de a s of he hypergraph neura ne -
pans ve compu a on of Lap ac an E gen vec ors s exc uded
works Mu -moda y da ase s are d v ded n o ra n ng da a
and on y ma r x powers add ons and mu p ca ons are
and es ng da a and each da a con a ns severa nodes w h
nc uded wh ch br ngs fur her mprovemen n compu a on
fea ures Then mu p e hyperedge s ruc ure groups are con-
comp ex y We can fur her e K = 1 o m he order
s ruc ed from he comp ex corre a on of he mu -moda y
of convo u on opera on due o ha he Lap ac an n hyper-
da ase s We conca ena e he hyperedge groups o genera e
graph can a ready we represen he h gh-order corre a on
he hypergraph ad acen ma r x H The hypergraph ad acen
be ween nodes I s a so sugges ed n (K pf and We ng
ma r x H and he node fea ure are fed n o he HGNN o ge
2017) ha λmax ≈ 2 because of he sca e adap ab y of
he node ou pu abe s As n roduced n he above sec on
neura ne works Then he convo u on opera on can be fur-
we can bu d a hyperedge convo u ona ayer f (X W Θ)
her s mp fied o
n he fo ow ng formu a on
g ? x ≈ θ0 x − θ1 D−1/2 HWD−1 > −1/2
e H Dv x (7)
X( +1) = σ(D−1/2
v HWD−1 > −1/2 ( ) ( )
e H Dv X Θ ) (11)
where θ0 and θ1 s parame ers of fi ers over a nodes We
fur her use a s ng e parame er θ o avo d he overfi ng prob- where X(1) ∈ RN ×C s he s gna of hypergraph a ayer
em wh ch s defined as X(0) = X and σ deno es he non near ac va on func on
θ1 = − 21 θ

The HGNN mode s based on he spec ra convo u on on
−1/2 > −1/2 (8)
θ0 = 12 θDv HD−1 e H Dv
he hypergraph Here we fur her nves ga e HGNN n he
property of exploiting high-order correlation among data. As edge groups and then various hyperedges are fused together
is shown in Figure 4, the HGNN layer can perform node- to model the complex relationship on data.
edge-node transform, which can better refine the features
using the hypergraph structure. More specifically, at first, Experiments
the initial node feature X(1) is processed by learnable fil- In this section, we evaluate our proposed hypergraph neu-
ter matrix Θ(1) to extract C2 -dimensional feature. Then, the ral networks on two task: citation network classification and
node feature is gathered according to the hyperedge to form visual object recognition. We also compare the proposed
the hyperedge feature RE×C2 , which is implemented by the method with graph convolutional networks and other state-
multiplication of H> ∈ RE×N . Finally the output node fea- of-the-art methods.
ture is obtained by aggregating their related hyperedge fea-
ture, which is achieved by multiplying matrix H. Denote Dataset Cora Pumbed
that Dv and De play a role of normalization in Equation 11.
Thus, the HGNN layer can efficiently extract the high-order Nodes 2708 19717
correlation on hypergraph by the node-edge-node transform. Edges 5429 44338
Feature 1433 500
Training node 140 60
Relations to existing methods When the hyperedges only Validation node 500 500
connect two vertices, the hypergraph is simplified into a sim- Testing node 1000 1000
ple graph and the Laplacian ∆ is also coincident with the Classes 7 3
Laplacian of simple graph up to a factor of 12 . Compared
with the existing graph convolution methods, our HGNN can Table 1: Summary of the citation classification datasets.
naturally model high-order relationship among data, which
is effectively exploited and encoded in forming feature ex-
traction. Compared with the traditional hypergraph method, Citation network classification
our model is highly efficient in computation without the in-
verse operation of Laplacian ∆. It should also be noted that Datasets In this experiment, the task is to classify citation
our HGNN has great expansibility toward multi-modal fea- data. Here, two widely used citation network datasets, i.e.,
ture with the flexibility of hyperedge generation. Cora and Pubmed (Sen et al. 2008) are employed. The ex-
perimental setup follows the settings in (Yang, Cohen, and
Implementation Salakhutdinov 2016). In both of those two datasets, the fea-
ture for each data is the bag-of-words representation of doc-
Hypergraph construction In our visual object classifica- uments. The data connection, i.e., the graph structure, in-
tion task, the features of N visual object data can be repre- dicates the citations among those data. To generate the hy-
>
sented as X = [x1 , . . . , xn ] . We build the hypergraph ac- pergraph structure for HGNN, each time one vertex in the
cording to the distance between two features. More specif- graph is selected as the centroid and its connected vertices
ically, Euclidean distance is used to calculate d(xi , xj ). In are used to generate one hyperedge including the centroid
the construction, each vertex represents one visual object, itself. Through this we can obtain the same size incidence
and each hyperedge is formed by connecting one vertex and matrix compared with the original graph. It is noted that as
its K nearest neighbors, which brings N hyperedges that there are no more information for data relationship, the gen-
links K + 1 vertices. And thus, we get the incidence ma- erated hypergraph constructure is quite similar to the graph.
trix H ∈ RN ×N with N × (K + 1) entries equaling to 1 The Cora dataset contains 2708 data and 5% are used as la-
while others equaling to 0. In the citation network classifi- beled data for training. The Pubmed dataset contains 19717
cation, where the data are organized in graph structure, each data, and only 0.3% are used for training. The detailed de-
hyperedge is built by linking one vertex and their neighbors scription for the two datasets listed in Table 1.
according to the adjacency relation on graph. So we also get
N hyperedges and H ∈ RN×N . Experimental settings In this experiment, a two-layer
HGNN is applied. The feature dimension of the hidden layer
Model for node classification In the problem of node is set as 16 and the dropout (Srivastava et al. 2014) is em-
classification, we build the HGNN model as in Figure 3. ployed to avoid overfitting with drop rate p = 0.5. We
The dataset is divided into training data and test data. Then choose the ReLU as the nonlinear activation function. Dur-
hypergraph is constructed as the section above, which gen- ing the training process, we use Adam optimizer (Kingma
erates the incidence matrix H and corresponding De . We and Ba 2014) to minimize our cross-entropy loss function
build a two-layer HGNN model to employ the powerful ca- with a learning rate of 0.001. We have also compared the
pacity of HGNN layer. And the softmax function is used to proposed HGNN with recent methods in these experiments.
generate predicted labels. During training, the cross-entropy
loss for the training data is back-propagated to update the pa- Results and discussion The results of the experimental re-
rameters Θ and in testing, the labels of test data is predicted sults and comparisons on the citation network dataset are
for evaluating the performance. When there are multi-modal shown in Table 2. For our HGNN model, we report the aver-
information incorporate them by the construction of hyper- age classification accuracy of 100 runs on Core and Pumbed,
which is 81.6% and 80.1%. As shown in the results, the pro- the affinity matrix A is generated to represent the relation-
posed HGNN model can achieve the best or comparable per- ship among different vertices, and Aij can be calculated by:
formance compared with the state-of-the-art methods. Com- 2Dij 2
pared with GCN, the proposed HGNN method can achieve a )Aij = exp(− (12)
slight improvement on the Cora dataset and 1.1% improve- ∆
ment on the Pubmed dataset. We note that the generated hy- where Dij indicates the Euclidean distance between node
pergraph structure is quite similar to the graph structure as i and node j. ∆ is the average pairwise distance between
there is neither extra nor more complex information in these nodes. For the GCN experiment with two features con-
data. Therefore, the gain obtained by HGNN is not very sig- structed simple graphs, we simply average the two modal-
nificant. ity adjacency matrices to get the fused graph structure for
comparison.
Method Cora Pubmed
Dataset ModelNet40 NTU
DeepWalk (Perozzi, Al-Rfou, 67.2% 65.3%
and Skiena 2014) Objects 12311 2012
ICA (Lu and Getoor 2003) 75.1% 73.9% MVCNN Feature 4096 4096
Planetoid (Yang, Cohen, and 75.7% 77.2% GVCNN Feature 2048 2048
Salakhutdinov 2016) Training node 9843 1639
Chebyshev (Defferrard, Bres- 81.2% 74.4% Testing node 2468 373
son, and Vandergheynst 2016) Classes 40 67
GCN (Kipf and Welling 2017) 81.5% 79.0%
Table 3: The detailed information of the ModelNet40 and
HGNN 81.6% 80.1% the NTU datasets.

Table 2: Classification results on the Cora and Pubmed

datasets. Neighborhood GVCNN-based MVCNN-based
Centeroid hyperedges hyperedges
ei e1 e2 e3 em-1em e1 e2 e3 em-1em
n3 n1 1 n1 n1
n2
n8
1
n2 n2
n3
Visual object classification n9 n5 n4
0
n3 Concat n3
H=
0
n2 n4 n5 1 n4 n4
n1
n6 0 n5 n5
Datasets and experimental settings In this experiment, n6 n7
n7
n8
0
1
n6
n7
n6
n7
n9
the task is to classify visual objects. Two public benchmarks 0
n8 n8

are employed here, including the Princeton ModelNet40 Hyperedge Generation Incidence Matrix Generation
dataset (Wu et al. 2015) and the National Taiwan University
(NTU) 3D model dataset (Chen et al. 2003), as shown in Ta-
ble 3. The ModelNet40 dataset consists of 12,311 objects Figure 5: An example of hyperedge generation in the vi-
from 40 popular categories, and the same training/testing sual object classification task. Left: For each node we ag-
split is applied as introduced in (Wu et al. 2015), where gregate its N neighbor nodes by Euclidean distance to gen-
9,843 objects are used for training and 2,468 objects are erate a hyperedge. Right: To generate the multi-modality hy-
used for testing. The NTU dataset is composed of 2,012 pergraph adjacent matrix we concatenate adjacent matrix of
3D shapes from 67 categories, including car, chair, chess, two modality.
chip, clock, cup, door, frame, pen, plant leaf and so on. In
the NTU dataset, 80% data are used for training and the Hypergraph structure construction on visual datasets
other 20% data are used for testing. In this experiment, each In experiments on ModelNet40 and NTU datasets, two hy-
3D object is represented by the extracted features. Here, pergraph construction methods are employed. The first one
two recent state-of-the-art shape representation methods are is based on single modality feature and the other one is based
employed, including Multi-view Convolutional Neural Net- on multi-modality feature. In the first case, only one feature
work (MVCNN) (Su et al. 2015) and Group-View Convolu- is used. Each time one object in the dataset is selected as the
tional Neural Network (GVCNN) (Feng et al. 2018). These centroid, and its 10 nearest neighbors in the selected feature
two methods are selected due to that they have shown sat- space are used to generate one hyperedge including the cen-
isfactory performance on 3D object representation. We fol- troid itself, as shown in Figure 5. Then, a hypergraph G with
low the experimental settings of MVCNN and GVCNN to N hyperedges can be constructed. In the second case, mul-
generate multiple views of each 3D object. Here, 12 virtual tiple features are used to generate a hypergraph G modeling
cameras are employed to capture views with a interval angle complex multi-modality correlation. Here, for the ith modal-
of 30 degree, and then both the MVCNN and the GVCNN ity data, a hypergraph adjacent matrix Hi is constructed ac-
features are extracted accordingly. cordingly. After all the hypergraphs from different features
To compare with GCN method, it is noted that there is no have been generated, these adjacent matrices Hi can be con-
available graph structure in the ModelNet40 dataset and the catenated to build the multi-modality hypergraph adjacent
NTU dataset. Therefore, we construct a probability graph matrix H. In this way, the hypergraphs using single modal
based on the distance of nodes. Given the features of data, feature and multi-modal features can be constructed.
 Features for Structure
Feature GVCNN MVCNN GVCNN+MVCNN
GCN HGNN GCN HGNN GCN HGNN
GVCNN (Feng et al. 2018) 91.8% 92.6% 91.5% 91.8% 92.8% 96.6%
MVCNN (Su et al. 2015) 92.5% 92.9% 86.7% 91.0% 92.3% 96.6%
GVCNN+MVCNN - - - - 94.4% 96.7%

Table 4: Comparison between GCN and HGNN on the ModelNet40 dataset.

Features for Structure

Feature GVCNN MVCNN GVCNN+MVCNN
GCN HGNN GCN HGNN GCN HGNN
GVCNN ((Feng et al. 2018)) 78.8% 82.5% 78.8% 79.1% 75.9% 84.2%
MVCNN ((Su et al. 2015)) 74.0% 77.2% 71.3% 75.6% 73.2% 83.6%
GVCNN+MVCNN − − − − 76.1% 84.2%

Table 5: Comparison between GCN and HGNN on the NTU dataset.

Classification For example, HGNN achieves gains of 8.3%, 10.4% and

Method 8.1% compared with GCN when GVCNN, MVCNN and
Accuracy
GVCNN+MVCNN are used as the object features on the
PointNet (Qi et al. 2017a) 89.2% NTU dataset, respectively.
PointNet++ (Qi et al. 2017b) 90.7%
PointCNN (Li et al. 2018) 91.8% The better performance can be dedicated to the employed
SO-Net (Li, Chen, and Lee 2018) 93.4% hypergraph structure. The hypergraph structure is able to
HGNN 96.7% convey complex and high-order correlations among data,
which can better represent the underneath data relation-
Table 6: Experimental comparison among recent classifica- ship compared with graph structure or the methods without
tion methods on ModelNet40 dataset. graph structure. Moreover, when multi-modal data/features
are available, HGNN has the advantage of combining such
multi-modal information in the same structure by its flexible
Results and discussions Experiments and comparisons hyperedges. Compared with traditional hypergraph learning
on the visual object recognition task are shown in Table 4 methods, which may suffer from the high computational
and Table 5, respectively. For the ModelNet40 dataset, we complexity and storage cost, the proposed HGNN frame-
have compared the proposed method using two features with work is much more efficient through the hyperedge convo-
recent state-of-the-are methods in Table 6. As shown in the lution operation.
results, we can have the following observations:
Conclusion
1. The proposed HGNN method outperforms the state-of-
In this paper, we propose a framework of hypergraph neural
the-art object recognition methods in the ModelNet40
networks (HGNN). In this method, HGNN generalizes the
dataset. More specifically, compared with PointCNN and
convolution operation to the hypergraph learning process.
SO-Net, the proposed HGNN method can achieve gains
The convolution on spectral domain is conducted with hy-
of 4.8% and 3.2%, respectively. These results demonstrate
pergraph Laplacian and further approximated by truncated
the superior performance of the proposed HGNN method
chebyshev polynomials. HGNN is a more general frame-
on visual object recognition.
work which is able to handle the complex and high-order
2. Compared with GCN, the proposed method achieves bet- correlations through the hypergraph structure for representa-
ter performance in all experiments. As shown in Ta- tion learning compared with traditional graph. We have con-
ble 4 and Table 5, when only one feature is used for ducted experiments on citation network classification and
graph/hypergraph structure generation, HGNN can ob- visual object recognition tasks to evaluate the performance
tain slightly improvement. For example, when GVCNN of the proposed HGNN method. Experimental results and
is used as the object feature and MVCNN is used for comparisons with the state-of-the-art methods demonstrate
graph/hypergraph structure generation, HGNN achieves better performance of the proposed HGNN model. HGNN
gains of 0.3% and 2.0% compared with GCN on the is able to take complex data correlation into representation
ModelNet40 and the NTU datasets, respectively. When learning and thus lead to potential wide applications in many
more features, i.e., both GVCNN and MVCNN, are tasks, such as visual recognition, retrieval and data classifi-
used for graph/hypergraph structure generation, HGNN cation.
achieves much better performance compared with GCN.
 Acknowledgements Metaxas, D. 2009. ] Video Object Segmentation by Hy-
This work was supported by National Key R&D Pro- pergraph Cut. In Proc. CVPR, 1738–1745. IEEE.
gram of China (Grant No. 2017YFC0113000, and [Hwang et al. 2008] Hwang, T.; Tian, Z.; Kuangy, R.; and
No.2016YFB1001503), and National Natural Science Kocher, J.-P. 2008. Learning on Weighted Hypergraphs
Funds of China (U1701262, U1705262, 61671267, to Integrate Protein Interactions and Gene Expressions for
61772443, 61572410), National Science and Technology Cancer Outcome Prediction. In Proc. ICDM, 293–302.
Major Project (No. 2016ZX01038101), MIIT IT funds IEEE.
(Research and application of TCN key technologies) of [Kingma and Ba 2014] Kingma, D. P., and Ba, J. 2014.
China, and The National Key Technology R and D Program Adam: A Method for Stochastic Optimization. In Proc.
(No. 2015BAG14B01-02), Post Doctoral Innovative Talent ICLR.
Support Program under Grant BX201600094, China Post-
[Kipf and Welling 2017] Kipf, T. N., and Welling, M. 2017.
Doctoral Science Foundation under Grant 2017M612134,
Semi-Supervised Classification with Graph Convolutional
Scientific Research Project of National Language Commit-
Networks. In Proc. ICLR.
tee of China (Grant No. YB135-49), and Nature Science
Foundation of Fujian Province, China (No. 2017J01125 and [Li et al. 2018] Li, Y.; Bu, R.; Sun, M.; and Chen, B. 2018.
No. 2018J01106). PointCNN. In NIPS.
[Li, Chen, and Lee 2018] Li, J.; Chen, B. M.; and Lee, G. H.
References 2018. SO-Net: Self-Organizing Network for Point Cloud
Analysis. In Proc. CVPR, 9397–9406.
[Atwood and Towsley 2016] Atwood, J., and Towsley, D.
2016. Diffusion-Convolutional Neural Networks. In NIPS, [Lu and Getoor 2003] Lu, Q., and Getoor, L. 2003. Link-
1993–2001. based Classification. In Proc. ICML, 496–503.
[Bruna et al. 2014] Bruna, J.; Zaremba, W.; Szlam, A.; and [Monti et al. 2017] Monti, F.; Boscaini, D.; Masci, J.;
LeCun, Y. 2014. Spectral Networks and Locally Connected Rodola, E.; Svoboda, J.; and Bronstein, M. M. 2017. Geo-
Networks on Graphs. In Proc. ICLR. metric Deep Learning on Graphs and Manifolds Using Mix-
ture Model CNNs. In Proc. CVPR, volume 1, 3.
[Chen et al. 2003] Chen, D.-Y.; Tian, X.-P.; Shen, Y.-T.; and
Ouhyoung, M. 2003. On Visual Similarity Based 3D Model [Perozzi, Al-Rfou, and Skiena 2014] Perozzi, B.; Al-Rfou,
Retrieval. In Computer Graphics Forum, volume 22, 223– R.; and Skiena, S. 2014. Deepwalk: Online Learning of
232. Wiley Online Library. Social Representations. In Proc. SIGKDD, 701–710. ACM.
[Defferrard, Bresson, and Vandergheynst 2016] Defferrard, [Qi et al. 2017a] Qi, C. R.; Su, H.; Mo, K.; and Guibas, L. J.
M.; Bresson, X.; and Vandergheynst, P. 2016. Convolutional 2017a. PointNet: Deep Learning on Point Sets for 3D Clas-
Neural Networks on Graphs with Gast Localized Spectral sification and Segmentation. Proc. CVPR 1(2):4.
Filtering. In NIPS, 3844–3852. [Qi et al. 2017b] Qi, C. R.; Yi, L.; Su, H.; and Guibas, L. J.
[Feng et al. 2018] Feng, Y.; Zhang, Z.; Zhao, X.; Ji, R.; and 2017b. PointNet++: Deep Hierarchical Feature Learning on
Gao, Y. 2018. Gvcnn: Group-View Convolutional Neural Point Sets in a Metric Space. In NIPS, 5105–5114.
Networks for 3D Shape Recognition. In Proc. CVPR, 264– [Scarselli et al. 2009] Scarselli, F.; Gori, M.; Tsoi, A. C.; Ha-
272. genbuchner, M.; and Monfardini, G. 2009. The Graph Neu-
[Gao et al. 2012] Gao, Y.; Wang, M.; Tao, D.; Ji, R.; and Dai, ral Network Model. IEEE Transactions on Neural Networks
Q. 2012. 3-D Object Retrieval and Recognition with Hyper- 20(1):61–80.
graph Analysis. IEEE Transactions on Image Processing [Sen et al. 2008] Sen, P.; Namata, G.; Bilgic, M.; Getoor, L.;
21(9):4290–4303. Galligher, B.; and Eliassi-Rad, T. 2008. Collective Classifi-
[Gao et al. 2013] Gao, Y.; Wang, M.; Zha, Z.-J.; Shen, J.; Li, cation in Network Data. AI magazine 29(3):93.
X.; and Wu, X. 2013. Visual-Textual Joint Relevance Learn- [Srivastava et al. 2014] Srivastava, N.; Hinton, G.;
ing for Tag-based Social Image Search. IEEE Transactions Krizhevsky, A.; Sutskever, I.; and Salakhutdinov, R.
on Image Processing 22(1):363–376. 2014. Dropout: A Simple Way to Prevent Neural Net-
[Gori, Monfardini, and Scarselli 2005] Gori, M.; Monfar- works from Overfitting. The Journal of Machine Learning
dini, G.; and Scarselli, F. 2005. A New Model for Learning Research 15(1):1929–1958.
in Graph Domains. In Proc. IJCNN, volume 2, 729–734. [Su et al. 2015] Su, H.; Maji, S.; Kalogerakis, E.; and
IEEE. Learned-Miller, E. 2015. Multi-View Convolutional Neural
[Henaff, Bruna, and LeCun 2015] Henaff, M.; Bruna, J.; and Networks for 3D Shape Recognition. In Proc. ICCV, 945–
LeCun, Y. 2015. Deep Convolutional Networks on Graph- 953.
Structured Data. arXiv preprint arXiv:1506.05163. [Velickovic et al. 2018] Velickovic, P.; Cucurull, G.;
[Huang et al. 2010] Huang, Y.; Liu, Q.; Zhang, S.; and Casanova, A.; Romero, A.; Lio, P.; and Bengio, Y. 2018.
Metaxas, D. N. 2010. Image Retrieval via Probabilistic Hy- Graph Attention networks. In Proc. ICLR, volume 1.
pergraph Ranking. In Proc. CVPR, 3376–3383. IEEE. [Wu et al. 2015] Wu, Z.; Song, S.; Khosla, A.; Yu, F.; Zhang,
[Huang, Liu, and Metaxas 2009] Huang, Y.; Liu, Q.; and L.; Tang, X.; and Xiao, J. 2015. 3D ShapeNets: A Deep Rep-
 resentation for Volumetric Shapes. In Proc. CVPR, 1912–
1920.
[Yang, Cohen, and Salakhutdinov 2016] Yang, Z.; Cohen,
W. W.; and Salakhutdinov, R. 2016. Revisiting Semi-
Supervised Learning with Graph Embeddings. Proc. ICML.
[Zhou, Huang, and Schölkopf 2007] Zhou, D.; Huang, J.;
and Schölkopf, B. 2007. Learning with Hypergraphs: Clus-
tering, Classification, and Embedding. In NIPS, 1601–1608.