IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 34, NO.

4, APRIL 2023 1705

A Spatiotemporal Deep Learning Approach for

Unsupervised Anomaly Detection in Cloud Systems
Zilong He , Pengfei Chen , Xiaoyun Li , Yongfeng Wang , Guangba Yu, Cailin Chen,
Xinrui Li, and Zibin Zheng , Senior Member, IEEE

Abstract— Anomaly detection is a critical task for maintaining

the performance of a cloud system. Using data-driven methods
to address this issue is the mainstream in recent years. However,
due to the lack of labeled data for training in practice, it is
necessary to enable an anomaly detection model trained on
contaminated data in an unsupervised way. Besides, with the
increasing complexity of cloud systems, effectively organizing
data collected from a wide range of components of a system and
modeling spatiotemporal dependence among them become a chal-
lenge. In this article, we propose TopoMAD, a stochastic seq2seq
model which can robustly model spatial and temporal dependence
among contaminated data. We include system topological infor-
mation to organize metrics from different components and apply
Fig. 1. Example of multivariate time series with topological information.
sliding windows over metrics collected continuously to capture
the temporal dependence. We extract spatial features with the
help of graph neural networks and temporal features with long its state continuously. Metrics, such as CPU usage, num-
short-term memory networks. Moreover, we develop our model
based on variational auto-encoder, enabling it to work well ber of I/O requests per second, network throughput, and so
robustly even when trained on contaminated data. Our approach on, can provide significant insights into the running state
is validated on the run-time performance data collected from two of a cloud system, which helps the anomaly detection and
representative cloud systems, namely, a big data batch processing troubleshooting.
system and a microservice-based transaction processing system. To reduce the cost of system accidents with the use of
The experimental results show that TopoMAD outperforms some
metrics we collect, anomaly detection on these metrics is
state-of-the-art methods on these two data sets.
first performed. Since metrics collected are time series data,
Index Terms— Cloud computing, neural networks, unsuper- an efficient and accurate algorithm for anomaly detection on
vised anomaly detection, variational auto-encoder (VAE). time series is needed. So far, with the trend of developing
artificial intelligence for IT operations [1], there have been
I. I NTRODUCTION plenty of works conducted on anomaly detection for time

W ITH the rapid development of cloud technology,

the complexity and scale of cloud systems are continu-
ally increasing, which gives rise to frequent system accidents
series with the use of AI technology. Some of them perform
anomaly detection on univariate time series [2]–[6]. While
with the increasing complexity of cloud systems, more and
and the downgraded performance. To ensure the performance more metrics can be collected and utilized to model a system
and reliability of a cloud system, operators need to monitor state [7]. As a result, methods only designed for univariate
time series inevitably reveal their weakness. There also exist
Manuscript received 4 October 2019; revised 12 April 2020 and 26 July
2020; accepted 22 September 2020. Date of publication 16 October 2020; some methods which take multivariate time series as input
date of current version 5 April 2023. This work was supported in part [8]–[12], but they only concatenate different time series
by the National Key Research and Development Program of China under directly without effective organization. In a complex cloud
Grant 2019YFB1804002, in part by the Research Development Plan of
Key Areas in Guangdong Province under Grant 2020B010165002, in part
system, more information about the collected metrics can
by the Natural Science Foundation of China under Grant U1811462 and be obtained and utilized to help organize these metrics. For
Grant 61802448, in part by the Basic and Applied Basic Research of example, the topology (see Fig. 1) originated from the system
Guangzhou under Grant 202002030328, and in part by the Natural Sci- can help gain a graph-based representation of the system
ence Foundation of Guangdong Province under Grant 2019A1515012229.
(Corresponding author: Pengfei Chen.) state, which we also denote as a topological multivariate time
The authors are with the School of Data Science and Computer, Sun Yat-sen series below. Especially, in a microservice system, it becomes
University, Guangzhou 510006, China (e-mail: [email protected]; more critical to leverage the topological information. However,
[email protected]; [email protected]; wangyf226@
mail2.sysu.edu.cn; [email protected]; [email protected]; the problem of how to effectively construct and further perform
[email protected]; [email protected]). anomaly detection for such graph-based representations still
This article has supplementary downloadable material available at remains unsettled.
https://ieeexplore.ieee.org, provided by the authors.
Color versions of one or more of the figures in this article are available
A deep neural network is a powerful tool for model-
online at https://ieeexplore.ieee.org. ing dependence in data with complex structure. Therefore,
Digital Object Identifier 10.1109/TNNLS.2020.3027736 its application for anomaly detection has attracted many
2162-237X © 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.

Authorized licensed use limited to: Zhejiang University. Downloaded on January 27,2025 at 01:13:45 UTC from IEEE Xplore. Restrictions apply.
1706 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 34, NO. 4, APRIL 2023

researchers’ attention in recent years. Recent research [3], them into traditional approaches and deep learning-based
[12]–[14] discovers that a variational auto-encoder (VAE) [15] approaches.
shows superiority for the task of anomaly detection. There- Traditional approaches [20] include statistical approaches,
fore, we develop an anomaly detector based on the design such as Gaussian-based model, classification-based
of a VAE. Beyond its advantages declared in the previous approaches, such as one-class support vector machines
literature, we also demonstrate its effectiveness for training (OC-SVMs) [21], nearest neighbor-based approaches, such as
on contaminated data (i.e., data including normal data and local outlier factor (LOF) [22], and so on. Usually, a traditional
abnormal data) without reliance on labels. anomaly detector is associated with an assumption about the
In this article, we propose a topology-aware multivariate normal data, and its effectiveness will degrade when the data
time series anomaly detector (TopoMAD), which combines cannot fit in the corresponding assumption.
graph neural networks [16], [17], long short-term memory Compared with traditional approaches, deep learning-based
(LSTM) [18], [19], and VAE [15] to perform unsupervised models are better at modeling complex dependence in
anomaly detection for a cloud system. We evaluate our model data, and thus, gain a lot of attention. For example, deep
using metrics collected from an environment running big auto-encoder [23] can be used to perform dimension reduc-
data batching systems, such as Hadoop and Spark, with fault tion by utilizing its multiple nonlinear transformations. For
injections and metrics collected from a microservice-based anomaly detection, the reconstruction error of an observation
application, where faults are injected occasionally. is used as its anomaly score. For anomaly detection on
The contributions of TopoMAD are summarized as follows. time series, Donut [3] leverages VAE [15] to model the
1) TopoMAD introduces an unsupervised anomaly detec- reconstruction probability [13] of univariate time series and
tion approach, which considers the topological informa- perform anomaly detection based on this measure. LSTM-AD
tion originated from a cloud system. We combine this [8] learns a prediction model using stacked LSTM networks,
topological information with metrics collected from a and the prediction error is used as the measure of an anomaly.
cloud system to construct a graph-based representation LSTM-ED [9] proposes an LSTM-based seq2seq auto-encoder
for anomaly detection. that learns to reconstruct normal multivariate time series and
2) TopoMAD glues graph neural networks and LSTM uses the reconstruction error to detect anomalies. Deep struc-
as the basic structure of VAE to perform anomaly tured energy-based models (DSEBMs) [10] connects an EBM
detection in a topological time series. We make use of with a regularized auto-encoder to model the data distribution,
state-of-the-art graph neural networks, such as graph and the energy score or the reconstruction error is used to
convolution network (GCN) [16] and graph attention perform anomaly detection. For sequential data, a recurrent
network (GAT) [17] to extract information from the formulation of EBMs can be employed. Though some of these
metrics organized in a predefined topology, together with approaches have tackled the problem of modeling complex
LSTM to extract information from sliding windows over temporal dependence for time series, we still need an approach
time. The spatiotemporal information extracted by graph to effectively model the spatiotemporal dependence for data
neural networks together with LSTM, can help improve which are collected from a specific topology continuously.
the anomaly detection performance in cloud systems.
3) TopoMAD makes use of VAE [15], a stochastic model, B. Anomaly Detection in Cloud Systems
to perform anomaly detection for a cloud system in a
In Section II-A, we have given a brief introduction about
fully unsupervised way. Instead of making an assump-
those general unsupervised anomaly detectors. Most of them
tion of data we use, we train our model on all data we
have been used in cloud systems [3], [24], [25]. In this section,
collect, no matter it is normal or abnormal. Moreover,
we shall focus on some other aspects of anomaly detection in
we propose an unsupervised threshold selection method
cloud systems.
and examine this method based on our model.
1) Anomaly Types: According to some work [26]–[31],
4) We open-sourced two data sets,1 including metrics col-
anomalies in cloud systems can be categorized into two types,
lected from an environment running big data batching
including external impairments and internal application faults.
systems and metrics collected from a running microser-
External impairments refer to unexpected overloads, such as
vice system. These data sets can be used to reproduce
a memory hog, induced by another co-located application or
the results of this article or to motivate further research.
infrastructure failures, such as disk failure, network disconnec-
This article is organized as follows. The related work
tion, OS crash, and so on. Internal application faults denote
is introduced in Section II. In Section III, we present the
anomalies caused by the misconfiguration or software bugs of
preliminaries of each main technique in TopoMAD. Section IV
the corresponding running application. Different causes can
outlines and specifies our approach. In Section V, we evalu-
result in different symptoms. Multiple work [26], [29], [30]
ate our approach using data collected from two application
has been conducted on the analysis of failure characteristics
scenarios. Section VI concludes this article.
in cloud systems. For example, Zhou et al. [29] investigate
II. R ELATED W ORK application faults in microservice systems with different root
A. Unsupervised Anomaly Detectors causes and the debugging practice for them.
Enormous methods of unsupervised anomaly detection have 2) Anomaly Indicators: There are multiple types of indica-
been developed in the past years. In this section, we categorize tors which can reveal the state of a cloud system and further
help detect and analyze anomalies. Examples of these types
1 https://github.com/QAZASDEDC/TopoMAD include metrics [3], [5], [32], [33], logs [34]–[37], and system

Authorized licensed use limited to: Zhejiang University. Downloaded on January 27,2025 at 01:13:45 UTC from IEEE Xplore. Restrictions apply.
HE et al.: SPATIOTEMPORAL DEEP LEARNING APPROACH FOR UNSUPERVISED ANOMALY DETECTION IN CLOUD SYSTEMS 1707

calls [38], [39]. The anomaly detection process varies accord- unified GCN to perform human action recognition. Inspired
ing to different types of indicators. In this article, we focus by these models, we develop TopoMAD to include and
on anomaly detection in cloud systems with the use of system extract topological information with the help of graph neural
metrics. System metrics, such as resource utilization and I/O networks to perform unsupervised anomaly detection in cloud
response time, are widely used as anomaly indicators in cloud systems.
systems and have proved their effectiveness in plenty of
works [5], [28], [32], and [40]. Also, there exist works which III. P RELIMINARIES
succeed in finding new effective anomaly indicators in some A. Cloud System Topologies
specific scenarios, such as queue lengths of microservices used To illustrate what is the topological information originated
in [40]. from the system, we first define nodes and edges in cloud
3) Temporal Model: Many indicators presented in system topology and expound their characteristics.
Section II-B2 can be aggregated as time-series data by 1) Nodes: We have mentioned in Section II-B that each
applying a sliding window, which helps reveal the system’s system component is treated as a node in the cloud system
local state in time. Some work [41] calculates statistical topology. The definition of a system component can vary
features (such as mean, standard deviation, and gradient) depending on the different granularity of division. Examples of
of data in the sliding window as temporal features. There a course-grained division might be the role-based (e.g., master
also exists complex but more powerful temporal modeling or slave) division in Hadoop and the service-based division
techniques, such as time series decomposition [32], spectral in a microservice system, while examples of a fine-grained
residual [6], and some deep learning-based methods [3], division might be the node-based division in Hadoop and
[8]–[10] mentioned in Section II-A. the pod-based division in a Kubernetes driven microservice
4) Spatial Model: Copious literature on system performance system. In a fine-grained topology, several nodes (e.g., pods)
diagnosis involves constructing a graph-based representation that can be grouped into one node (e.g., services) in a
for a system. Some of them [11], [42]–[44] analyze metrics course-grained topology can share the same set of edges with
collected from a system in a pairwise way and construct an similar behaviors.
invariant graph which takes each metric as a node. On the 2) Edges: Multiple relationships of system components can
other hand, some work [27], [45], [46] treat each of the be chosen to be an edge. For example, components that
system components as a node with multiple attributes and locate in the same physical machine can share an edge,
employ edges to represent the connectivity between system components whose load are balanced can share an edge,
components. In this article, we prefer the latter way because components that have the same update setting can share an
analyzing metrics pairwise is time consuming and infeasible edge, and so on. Most of these relationships are dynamic.
for anomaly detection in real time when more and more In this article, we define there is an edge between two system
metrics are collected from cloud systems. components if they interact with each other. According to this
5) Threshold Selection: To apply an unsupervised anomaly definition, we can construct the topology in advance based
detector to practice, a corresponding threshold is a necessary upon some tracing-based analysis [53], [54] or network traffic
and influential hyperparameter. Threshold selection is integral correlation-based analysis [27], [55]–[57]. These methods are
to anomaly detection in cloud systems. Some work [11], [12], important complements to our approach.
[47] has provided some practices for threshold selection. For
instance, peaks-over-threshold (POT) [12] utilizes the extreme
value theory [2] to perform automatic threshold selection. B. Problem Statement
As for dynamic threshold calculation, nonparametric-dynamic- We list the notations used in this article with their descrip-
thresholding (NDT) [47] selects a threshold from the set tions in Table I. Anomaly detection for a topological mul-
 = μ(es ) + zσ (es ) with μ(es ) and σ (es ) as the mean and tivariate time series is to determine whether an observation
standard deviation of anomaly scores over a sliding window X t is an anomaly or not. We divide this objective into two
and z chosen from a range between two and ten to maximize steps. First, we calculate an anomaly score S(X t |X t−W :t−1 , E)
an equation predefined by us. for X t considering its recent history X t−W :t−1 and its topol-
ogy E. Then, we obtain an anomaly result by comparing
C. Graph Neural Networks for Spatiotemporal Modeling S(X t |X t−W :t−1 , E) with a threshold τ , which is selected in
an unsupervised manner. If the score is higher than τ , an alert
Learning patterns from spatiotemporal graphs is will be triggered.
increasingly important in many applications. Many current
approaches [48] apply graph neural networks together with
RNN and CNN to simultaneously consider spatial and C. Basics of Graph Neural Networks
temporal relations. For instance, diffusion convolutional Graph neural networks are modern deep learning techniques
recurrent neural network (RNN) [49] replaces the matrix designed for graph data. They are helpful when we need
multiplications in GRU [50] with a diffusion convolution to model the spatial dependence in metrics collected from
operation to accomplish the task of traffic forecasting. different components with connectivity and extract information
CNN-GCN [51] overcomes this problem through a complete from the topology. In this article, we apply two representative
convolutional structure which interleaves 1-D-CNN with graph neural networks, namely, GCN [16] and GAT [17],
GCN. ST-GCN [52] extends a temporal flow as graph edges as the basic layer in an LSTM cell to capture the spatial
and then extracts both temporal and spatial features using a dependence in the topology. In the following, we will introduce

Authorized licensed use limited to: Zhejiang University. Downloaded on January 27,2025 at 01:13:45 UTC from IEEE Xplore. Restrictions apply.
1708 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 34, NO. 4, APRIL 2023

TABLE I update the states of all nodes with LSTM. Peng et al. [61]
N OTATIONS AND D ESCRIPTIONS partition a document graph into two directed acyclic graphs
and constructs the LSTMs accordingly. In the constructed
LSTM, different precedents of a unit are calculated via full
parametrization or edge-type embedding and then summed to
obtain the output, states, and gates. Jain et al. [62] parame-
terize a spatiotemporal graph with a factor graph [63] and
then represent each factor with an RNN. Compared with these
methods, we directly replace the linear layers in LSTM with
state-of-the-art graph neural layers to perform spatiotemporal
learning. A more detailed description is given in Section IV-D.
the layerwise propagation rules of these two techniques in
detail. E. Basics of Variational Auto-Encoder
GCN [16] proposes a graph convolution layer which accepts VAE [15] is a deep probabilistic graphical model. It is a
an N × F matrix X (l) containing features of each node and the powerful tool for modeling the relationship between observed
adjacency matrix A of the topology as inputs. It propagates variables x and their corresponding latent variables z with
forward with the following rule: reduced dimension. VAE assumes that x can be generated
X (l+1) = D̃ − 2 Ã D̃ − 2 X (l) W(l) .
1 1
(1) through a process which first samples z from some prior
distribution pθ (z) and then samples x from pθ (x|z), which is
Here, Ã = A + I denotes the adjacency matrix with inserted  derived from a neural network with parameter θ . Since pθ (z|x)
self-loops. D̃ is its diagonal degree matrix with D̃ii = j Ãi j is intractable, VAE introduces a recognition model qφ (z|x) to

and W(l) ∈ R F×F is a layer-specific trainable weight matrix. approximate it. Parameters φ and θ can be trained through
An activation function can also be applied to the result X l+1 maximizing the variational lower bound L(θ, φ; x) [(4)] on the
to introduce nonlinearity. marginal likelihood with stochastic gradient variational Bayes
GAT [17] is similar to GCN. It utilizes attention mech- estimator [15]
anisms [58], [59] to fuse the neighboring nodes and then  
learn hidden representations for each node in a graph. After L(θ, φ; x) = −DKL qφ (z|x) pθ (z) + Eqφ (z|x) (log( pθ (x|z))).
inputting X (l) = {X 1(l) , X 2(l) , . . . , X (l)
N }, new hidden representa- (4)
tions can be computed as follows:
A typical choice for pθ (x|z) or qφ (z|x) is a multivariate
1  k
K
Gaussian with a diagonal covariance structure, whose mean
X i(l+1) = α Wk X (l) (2) and the covariance matrix is derived by neural networks. And
K k=1 j ∈N i, j j
i in general, a standard normal distribution N (0, I ) is chosen
where the attention coefficients αi, j are computed as for the prior of z.
    For the purpose of anomaly detection given a specific
exp LeakyReLU a  Wxi  Wx j input x, reconstruction probability Eqφ (z|x) (log( pθ (x|z))) [13]
αi, j =     . (3)
k∈Ni exp LeakyReLU a   [Wxi  Wxk ] is adopted and calculated using Monte Carlo integration [64]
  as follows:
Here, W ∈ R F ×F and a ∈ R2F are trainable parameters.
1    (l) 
L
K denotes the number of independent attention mechanisms
executed and Ni denotes the first-order neighbors of node Eqφ (x|z) (log( pθ (x|z))) = log pθ x|z (5)
L l=1
i (including node i ). LeakyReLU(·) is kind of an activation
function. Nonlinearity can also be introduced here by applying where L denotes the number of samples and z (l) ,
an activation function to the output. l = 1, 2, . . . , L are sampled from qφ (z|x).

D. Basics of Long Short-Term Memory Networks IV. M ETHODOLOGY

Recurrent neural networks (RNNs) are discrete-time In this section, we will introduce our motivation and
state–space models which can recursively process an input then elaborate on the overall structure and other details of
sequence and model its temporal dependence. Since a simple TopoMAD.
RNN has limited ability to learn the long-term dependence
in a sequence, LSTM was invented to address this problem.
Using gating mechanisms, LSTM can decide whether to forget A. Motivation
or update certain information in the transferred cell state ct−1 There are two key ideas in TopoMAD, namely, the inclusion
considering the input state x t and transferred hidden state h t−1 , of topological information to represent a system state and
and then calculates new cell state ct and new hidden state h t . the unsupervised setting at the stage of model training and
There have been some methods which apply RNN, includ- threshold selection.
ing LSTM on graph-structured data. For example, Liang Nowadays, most popular cloud systems can be recognized
et al. [60] utilize confidence-driven search to specify the as distributed systems with multiple interconnected com-
node updating sequence for a graph and then it sequentially ponents. The components mentioned here might be virtual

Authorized licensed use limited to: Zhejiang University. Downloaded on January 27,2025 at 01:13:45 UTC from IEEE Xplore. Restrictions apply.
HE et al.: SPATIOTEMPORAL DEEP LEARNING APPROACH FOR UNSUPERVISED ANOMALY DETECTION IN CLOUD SYSTEMS 1709

machines, microservices, and so on. When we focus on a cer-

tain component in these systems, we can find the behavior of a
component can be analyzed through the following two factors.
One factor is the task the component is responsible for and
currently performing. Metrics of the component can provide
insights into this factor. Considering a single-selected metric
can provide metric-level information while considering mul-
tiple metrics of the component can provide component-level
information. The other factor is the interacting relationship
of the components. Considering this can provide topological
information because behaviors of a component can be induced
by requests or return values of other components. The inte-
gration of topological information and component-level infor- Fig. 2. Overall structure of TopoMAD.
mation of each component forms system-level information,
which can provide a more comprehensive insight into the of anomalies in the testing phase. By contrast, variational
whole running system. auto-encoder learns the distribution of the latent variables
Previous research based on deep learning usually neglects instead of the latent variables themselves. The first RHS
the topological information. They either focus on anomaly term −DKL (qφ (z|x) pθ (z)) in the training objective [i.e., (4)]
detection on each particular component or even each particular makes input data “compete for” a suitable latent space by
metric of the whole system individually, or totally ignore the driving qφ (z|x) close to a standard normal distribution. As a
intrinsic communicating behaviors within a system and treat result, anomalies, which occur infrequently, will tend to lose
the whole system as a cumbersome component. We argue that the competition and result in a low reconstruction probability
paying attention to detecting anomalies at the system level with even when they have appeared in the training data.
topological information, rather than only at the component Threshold selection is an essential phase of anomaly detec-
level or metric level is worth considering. First, the decision of tion in practice. However, quite a few methods, such as [3]
whether a component is in an abnormal state needs to consider and [9] neglect this phase and only enumerate all thresholds
its connected components sometimes. Second, training and to gain the best F1 score and use it as the performance metric
maintaining an individual model for each component will of an anomaly detector. We argue that the best F1 score is
become more and more labor intensive as the cloud system not a good measure of the performance of an unsupervised
enlarges its complexity. Third, roughly treating the whole anomaly detection model, because the performance achieved
system as a cumbersome component loses insight into its inner denoted by the best F1 score is impractical and unachievable
topology and might, thus, raise the difficulty of modeling its without the support of labels. Therefore, we advise using an
normal behavior. unsupervised way to select a threshold before evaluating model
Including the topological information to represent a system performance.
state and utilizing graph neural networks to learn spatial
dependence of metrics with the help of this topological infor- B. Overall Structure
mation can bring three benefits. First, feature extractors of Fig. 2 displays the overall structure of TopoMAD. Data
graph neural networks are shared amongst the same kind of collected is processed to gain a tensor X recording metrics
metrics from different components, which helps catch similar collected from each node and an array E which describes
pattern amongst the same metric type with unified feature the system topology by recording its edges. In the stage of
learning. Second, the role of a particular component can be model training, the model is trained with historical data in
defined by its connections with other components through an offline batch processing way. After the model is trained
a graph neural operation, which provides convenience for properly, we select a threshold according to the distribution
performing end-to-end learning on patterns and behavior of all of anomaly scores of the training data. Then, in the stage
components from a system. Third, the topological information of online anomaly detection, the anomaly score of a new
can guide models to concentrate attention on interactions of observation is calculated using this properly trained model.
components with a direct connection in reality, which helps If the anomaly score of observation is higher than the threshold
prevent overfitting of our model. we select, an alert will be triggered.
TopoMAD follows an unsupervised setting, which means
that labels of data keep unavailable throughout the stage of
model training and threshold selection. As a result, we can- C. Data Integration and Preprocessing
not deliberately select normal data to be the training data Fig. 3 shows an example of an observation from a Hadoop
set. In this case, a stochastic model, such as VAE, has cluster with one master node and four slave nodes. An example
superiority over a deterministic model, such as auto-encoder. of the corresponding input data of Fig. 3 is visualized in Fig. 4.
Auto-encoder can be thought as a model which learns deter- During data preprocessing, we transform different metrics
ministic mappings from input data to their corresponding collected from different nodes through data standardization
latent variables; therefore, when training data includes anom- and then apply sliding windows of length W over these
alies, auto-encoder can also learn to reconstruct them well, observations. As for the edge set array E, we do not consider
which increases the difficulty of detecting the same kind the construction of a dynamic topology in this article because

Authorized licensed use limited to: Zhejiang University. Downloaded on January 27,2025 at 01:13:45 UTC from IEEE Xplore. Restrictions apply.
1710 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 34, NO. 4, APRIL 2023

Fig. 3. Graph-based representation of metrics collected from a Hadoop

cluster.

Fig. 5. Inner structure of GraphLSTM.

Fig. 4. Example of the input data in Fig. 3.

a real-time topology can change with high frequency owing to

load balance or other reasons. It is hard to assign a real-time
topology to each time step of the collected metrics.
Fig. 6. Overall graphical model of TopoMAD at the stage of inference.

D. Building Block: GraphLSTM

Before presenting the network architecture of our model
proposed in TopoMAD, we first introduce the basic build-
ing block to concurrently model the spatial and temporal
dependence in the input data. We name this building block
GraphLSTM for its nature as a combination of graph neural
networks and LSTM. Specifically, we replace fully connected
layers in LSTM with graph neural layers, such as GCN [16]
and GAT [17] to build a GraphLSTM cell. Fig. 5 shows the
inner structure of GraphLSTM.
All the states and gates in GraphLSTM are graph-based rep-
resentations containing features of each node and a topology Fig. 7. Overall graphical model of TopoMAD at the stage of training.
denoted by an input edge set array E. When a new input X t
from a time series X arrives at time t, GraphLSTM updates The overall network is a stochastic seq2seq
the cell state cti and calculates the new hidden state h it of a auto-encoder [65]. Let t0 = t − W + 1 and
certain node i in a topology considering the input X tNi and past X t0 :t = {X t0 , X t0 +1 , . . . , X t−1 , X t } denote an input sequence
Ni
states ct−1 , hN i
t−1 of itself and its neighbors. The computations
and E denote an input edge set array. The encoder computes
conducted by GraphLSTM can be formulated as (6) in the the distribution parameters of q(z t |X t0 :t ) after iteratively
following, where “∗G ” denotes the graph neural operator and feeding every time step of X t0 :t into it. GraphLSTM is
“∗” denotes the Hadamard product: used as part of the encoder to capture the spatiotemporal
     dependence among nodes and long time. After consuming the
ft = sigmoid W f ∗G h t−1 , x t , E + b f entire input sequence, we get a summarized fixed-dimension
    
i t = sigmoid Wi ∗G h t−1 , x t , E + bi tensor h t which is further passed to a couple of multilayer
     perceptrons to estimate the parameters μ(X t0 :t ) and σ (X t0 :t )
gt = tanh Wg ∗G h t−1 , x t , E + bg
of q(z t |X t0 :t ) = N (μ(X t0 :t ), σ (X t0 :t )). Multilayer perceptrons
ct = f t ∗ ct−1 + i t ∗ gt
     here are learned individually for each node. Likewise, similar
ot = sigmoid Wo ∗G h t−1 , x t , E + bo architecture is adopted as the decoder to reconstruct the
h t = ot ∗ tanh(ct ). (6) sequence in reverse order, and the states of the decoder
are initialized using the final states of the encoder. During
training, the input for the decoder per time step comes
E. Network Architecture from the original input sequence with probability λ or the
In this section, we will introduce the detailed neural network reconstructed sequence with probability 1 − λ (see Fig. 7),
architecture of the model proposed in TopoMAD. Fig. 6 shows namely, a scheduled sampling process [66]. In our model,
a visualization of our model at the stage of inference. λ is initialized to be 1 and will reduce by half at the end

Authorized licensed use limited to: Zhejiang University. Downloaded on January 27,2025 at 01:13:45 UTC from IEEE Xplore. Restrictions apply.
HE et al.: SPATIOTEMPORAL DEEP LEARNING APPROACH FOR UNSUPERVISED ANOMALY DETECTION IN CLOUD SYSTEMS 1711

of an epoch if the validation loss does not decrease. While attention. Therefore, we also calculate an anomaly score from
during inference, the input for the decoder per time step a component perspective as follows:
always comes from its preceding reconstruction (see Fig. 6).
1     i (l)
D−1 L
Specifically, at time step t − 1, we use the mean value, μ(z t ), St = − max log pθ X t |z t+d , X t+1:t+d
of p(X t |z t ) = N (μ(z t ), σ (z t )) as the input for the decoder. 0≤i<N L ∗ D
d=0 l=1
(10)
F. Offline Model Training where N denotes the number of components in the system and
Let φ denote the encoder network parameters and θ denote X ti denotes the metrics of component i at time t.
the decoder network parameters. Similar to VAE, our model For online anomaly detection, whether to trigger an alert
is trained by optimizing the variational lower bound on the at time t is decided based upon tempSt , which can be imme-
marginal likelihood. Given a topological time series X t0 :t , diately calculated at time t and D − 1 consecutive updated
where t0 = t − W + 1 denotes the beginning of the sliding anomaly scores preceding to it. When one of these anomaly
window, the loss function is derived as follows: scores is higher than the threshold, an anomaly is detected.
      Algorithm 2 in Appendix A of the supplementary mater-
L θ, φ; X t0 :t = −DKL qφ z t |X t0 :t  pθ (z t ) ial shows the pseudocode of our online anomaly detection
   
+ Eqφ (zt |X t0 :t ) log pθ X t0 :t |z t algorithm.
   
= −DKL qφ z t |X t0 :t N (0, I )
H. Threshold Selection
1   
L t
+ log pθ X j |z t(l) , X j +1:t . (7) An anomaly score can reveal how anomalous an observation
L l=1 j =t is, but in practice, a threshold is still needed to trigger an alert
0
and instruct operators to take actions.
Here, L denotes the sampling number to estimate the expec- We propose a threshold selection method based on an
tation. In this article, we set L = 1 since it has been reported assumption that anomaly scores of normal data locate in an
in [15] that one sample is sufficient as long as the minibatch area with a high density, while anomaly scores of abnormal
size is large enough. The pseudocode for training the model data locate in an area with a low density. The distance of
GraphLSTM-VAE in TopoMAD is shown in Algorithm 1 in these two areas is relatively long. Specifically, we define the
Appendix A of the supplementary material. distance of two anomaly scores sets S<τ and S>τ separated by
a threshold τ as follows:
G. Computing Anomaly Scores min(S>τ ) − max(S<τ )
d(S<τ , S>τ ) = (11)
min(S>τ ) + max(S<τ ) − 2 ∗ min(S<τ )
As shown in Section III-E, the reconstruction probability
Eqφ (zt |X t0 :t ) (log( pθ (X t |z t ))) can be immediately calculated at where max(S) denotes the maximal element in S, and min(S)
time t and used to indicate whether X t is anomalous. We fol- denotes the minimal element.
low it and use its additive inverse as the anomaly score of an Based upon this assumption, we select a threshold which
observation X t to make a higher anomaly score imply a more maximizes the distance [11)] between the two sets cut from
anomalous observation. The temporary anomaly score at time the training data set by this threshold from a range provided
t is formulated as follows: by an operator. The pseudocode of the threshold selection
algorithm is demonstrated in Algorithm 3 in Appendix A of
tempSt = −Eqφ (zt |X t0 :t ) (log( pθ (X t |z t ))). (8) the supplementary material.
If operators want the selected threshold to adapt to the
It has been reported by [3] that anomalous observations
dynamic changes in the system, they can use the automatic
usually occur continuously. In practice, it is acceptable to
threshold selection method on newly collected data and update
trigger an alert within a short delay. We allow the anomaly
the selected threshold routinely to maintain its effectiveness.
score of an observation X t to be adjusted according to some
of its succeeding observations. As a stochastic sequence-to- V. E XPERIMENT VALIDATION
sequence model, the model in TopoMAD not only learns
In this section, we conduct experiments to validate our
to reconstruct X t but also learns to reconstruct W − 1
model and answer the following questions.
observations preceding to X t , which provides convenience for
us to update the anomaly score of an observation X t . The 1) Can TopoMAD outperform other approaches in anomaly
final anomaly score St for an observation X t is formulated as detection for topological multivariate time series col-
follows: lected from a cloud system?
2) How does each component of TopoMAD affect the
1   
D−1 L
(l) performance?
St = − log pθ X t |z t+d , X t+1:t+d (9) 3) How effective is the topology?
L ∗ D d=0 l=1
4) How can we interpret the results of TopoMAD?
where L denotes the sampling number and D denotes the 5) How is the robustness of TopoMAD?
number of times we calculate or update an anomaly score 6) How is the efficiency of TopoMAD?
(i.e., the tolerance of detection delay). 7) How effective is our unsupervised threshold selection
In some cases, a relative low reconstruction probability of method in TopoMAD? Can it recommend a relatively
a particular component in a system is enough to draw our better threshold?

Authorized licensed use limited to: Zhejiang University. Downloaded on January 27,2025 at 01:13:45 UTC from IEEE Xplore. Restrictions apply.
1712 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 34, NO. 4, APRIL 2023

TABLE II
F EATURES OF O UR T WO D ATA S ETS

A. Data Sets
To demonstrate the effectiveness of our model, we conduct
experiments using two data sets, of which one is collected from
an environment running a big data batch processing system
Fig. 8. Pod-level topology of the Hipster-Shop where pods belonging to the
(MBD), and the other is from a microservice-based transaction same service are plotted with the same color.
processing system (MMS). The features of these two data sets
are shown in Table II. collected from each pod and recorded in the data set MMS.
We obtain the data set MBD from a cluster containing Then anomalies are labeled according to the fault injection
five nodes with one master and four slaves. During the log. The pod-level topology, which can be used to construct
experiments, we continuously generate multiple random work- a graph-based representation for metrics in this data set,
loads with random parameters using HiBench [67], a big is demonstrated in Fig. 8.
data benchmark published by Intel. At the same time, ran-
dom faults, including external impairments and application B. Performance Metrics
faults, are injected irregularly with random parameters. The All the anomaly detectors we evaluate in this article can
injected external impairments include system resource hog return an anomaly score for each observation, and therefore,
(high CPU/memory/disk-IO load), network failure (delay or we consider using the average precision (AP) of the abnormal
packets loss), and the application faults are simulated through class as the performance metric to compare them. AP is con-
injecting some delays or exceptions into Hadoop distributed sidered more suitable than Area Under the receiver operating
file system (HDFS) (causing symptoms similar to HDFS-448 characteristic Curve (AUC) when dealing with a highly skewed
[68] and HDFS-8160 [69]). We monitor and collect 26 metrics data set [74], which means the normal data dominates the
per node, including CPU idle, CPU I/O wait, CPU softirq, whole data set. It can be calculated as follows:
CPU system, CPU user, disk I/O wait per second, disk I/O TP TP
in progress per second, disk used percentage, disk read speed, P = , R=
T P + FP T P + FN 
disk write speed, kernel entropy, load1, load5, load15, memory  A PC
active, memory available percentage, memory cached, memory AP = (Ri − Ri−1 )Pi , m A P = (12)
dirty, memory free, memory used percentage, network bytes i
N(Classes)
received rate, network bytes sent rate, network TCP time wait, where T P, F P, and F N refer to true positive, false positive,
number of processes blocked, number of running processes, and false negative, respectively, and Pi and Ri denote the
and the total number of processes. These metrics are selected precision and recall at the i th threshold. The mean of A P
for two reasons. First, they are representative system metrics from all classes is known as mAP.
used in a series of previous works [12], [27], [32], [43], [46], F1 score, the harmonic mean of precision and recall, is also
and we try to cover many types, ranging from CPU, memory, widely used as a performance metric for anomaly detection
disk, and network to processes, as we can to gain a thorough when the observations have been classified into normal or
insight into the running state of the system. In addition, abnormal with a threshold. It can be calculated as follows:
these metrics are easy to acquire using tools, such as Perf, P·R
Telegraf [70], Amazon CloudWatch [71], and so on. These F1 = 2 · . (13)
metrics are collected for three days and constitute the data set P+R
MBD with anomalies labeled on the basis of the injection log. We use the F1 score to compare the effectiveness of different
We can refer to Fig. 3 for the topology input of this data set. threshold selection methods.
There is no doubt that other performance metrics can be fed
into our algorithm. C. Experiment Setup
With respect to the data set MMS, we adopt We compare our model in TopoMAD with seven baseline
Hipster-Shop [72], a web-based e-commerce microservice anomaly detectors, including a Gaussian-based anomaly detec-
benchmark where users can browse hipster goods, add them tor, OC-SVM [21], LOF [22], a simple auto-encoder [23],
to the cart and purchase them. This benchmark is deployed LSTM-AD [8], LSTM-ED [9], and RNN-EBM [10], which
in a Kubernetes [73] cluster with 12 VMs. A load generator have been introduced in Section II-A. Since the anomaly detec-
is included in Hipster-Shop to mimic visits to the website. tion for metrics with the help of topological information is
Moreover, we inject anomalies with random parameters, such kind of an unprecedented attempt, for other anomaly detectors,
as CPU/disk-IO hog, network delay, and container hang to we conduct experiments in the following two ways. The first
some pods during the experiments. Metrics, including CPU way is to concatenate all metrics from different components
usage, memory usage, network receive rate, network transmit into one huge vector and use this vector as the input of the
rate, pod latency, pod workload, and pod success rate, are evaluated anomaly detector. The second way is to perform

Authorized licensed use limited to: Zhejiang University. Downloaded on January 27,2025 at 01:13:45 UTC from IEEE Xplore. Restrictions apply.
HE et al.: SPATIOTEMPORAL DEEP LEARNING APPROACH FOR UNSUPERVISED ANOMALY DETECTION IN CLOUD SYSTEMS 1713

Fig. 9. AP of TopoMAD and other baseline anomaly detectors on MBD. Fig. 10. AP of TopoMAD and other baseline anomaly detectors on MMS.

anomaly detection on each component one by one and use

the average anomaly score of all components as the anomaly
score of the whole cloud system.
We employ Pytorch and Pytorch Geometric [75] to imple-
ment the model in TopoMAD and its variants. Then, we train
them and other baseline anomaly detectors on a server with
four NVIDIA Tesla P100-SXM2 GPUs.
Hyperparameters are chosen based on Bayesian optimiza-
tion [76], [77]. Since a lightweight model is usually preferred Fig. 11. mAP of TopoMAD and other baseline anomaly detectors on MBD.
when performing anomaly detection in cloud systems, we will
not consider using a wide and deep network architecture in
our experiments of deep learning-based methods. For both
data sets, we set the time window size as 10, which spans
5 min in MBD and 10 min in MMS. We set the hidden
dim per component as 3, the number of recurrent layers per
GraphLSTM cell as 2, the learning rate 10−4 , the batch size is
32, and the sampling number for VAE as 20 at the inference
stage. If the choice of graph neural layer in GraphLSTM is
GAT, the number of heads is set to 8, and the dropout rate is Fig. 12. mAP of TopoMAD and other baseline anomaly detectors on MMS.
set to 0.4. The tolerance of delay is set to 5 min. We calculate
For instance, LOF behaves well on the data set MMS while
anomaly scores from a component perspective [(10)] for
it shows a poor performance on the data set MBD. This is
both data sets. During experiments, the input for traditional
because we routinely scale the load generator when collecting
anomaly detectors is each individual observation, while the
the data of MMS, and the workload is relatively stable under
input for deep leaning-based anomaly detectors in each sliding
the same level of workload. As a result, normal observations
window applied over the observations. In particular, for a
of MMS tend to locate in a dense neighborhood area. As for
simple auto-encoder which is not customized for time series
the data set MBD, there are multiple workloads with random
data, we flatten the time series as its input, just like [3] does.
parameters running in the cluster, leading to more complex
Detecting anomalies in cloud systems should work online.
workload features. Therefore, when used in data set MBD,
Moreover, the performance data are collected continuously.
LOF does not behave and in the data set MMS.
Therefore, we take the former two days of data for training
The overall results of the evaluated algorithms seem not
with the rest one day for testing. Experiments on each data
very high in our experiments. Two reasons account for this
set using each anomaly detector are repeated five times, and
phenomenon. First, we inject faults with different kinds and
the average results are recorded for comparison.
severities to the system. Some faults result in a very subtle
impact on our system. In Fig. 13, we show the performance
D. Overall Performance of TopoMAD using MMS with more severe faults (e.g., higher
In this section, we will answer the question “Can Topo- network delays). When TopoMAD is evaluated in data with
MAD outperform other approaches in anomaly detection for severer faults, the result will be better. Second, there exist
topological multivariate time series collected from a cloud some fluctuations which are not caused by the injected faults
system?” Figs. 9–12 show comparisons (including AP and in the collected metrics. These fluctuations do not last for a
mAP) between our models in TopoMAD with other baseline long time and might be induced by dynamic changes in the
anomaly detectors trained in two ways mentioned earlier. It can running environment. They are common in reality. However,
be seen that the model in TopoMAD outperforms or matches these fluctuations are usually detected as anomalies with an
all baseline anomaly detectors. In the following, we will unsupervised anomaly detector because they indeed occur
discuss some baseline anomaly detectors individually in detail. less frequently and deviate far from normal data. In practice,
In our experiments, the performance of traditional anomaly if operators treat this kind of fluctuations as false alerts, they
detectors varies a lot from each other. This is mainly because can set some rules to filter the alerts (e.g., only a corresponding
the performance of a traditional anomaly detector highly alert lasts long enough will it be sent to the operators) or train a
depends on the corresponding assumption of data distribution. supervised anomaly detector to discriminate these fluctuations
When the corresponding assumption of a traditional anomaly from real anomalies. In the following, we shall focus on the
detector is violated, its performance will seriously downgrade. comparison among different anomaly detectors.

Authorized licensed use limited to: Zhejiang University. Downloaded on January 27,2025 at 01:13:45 UTC from IEEE Xplore. Restrictions apply.
1714 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 34, NO. 4, APRIL 2023

Fig. 13. Performance of TopoMAD on MMS under different levels of sever-

ities. A higher level of severity results in anomalies with larger deviations.

Deep learning-based models are good at modeling complex Fig. 14. ROC curves on both data sets, MBD and MMS. (a) MBD. (b) MMS.
dependencies embedded in data, but this advantage can turn
into a disadvantage when these deep anomaly detectors are
trained on contaminated training data. With this problem
out of consideration when performing unsupervised anomaly
detection, attempts to improve model fitting capacity are very
likely to backfire. For MBD, LSTM-ED performs the best
among all baseline anomaly detectors. For MMS, if we utilize
the concatenate policy, RNN-EBM achieves the best result
among all baseline deep anomaly detectors, and if we apply Fig. 15. AP of model variants on two data sets.
the one-by-one policy, a simple auto-encoder is the best in all
Finally, the ROC curves of anomaly detectors with top three
baseline deep anomaly detectors.
performances on each data set are displayed in Fig. 14 to show
LSTM-AD achieves the worst result in all deep
the performance tradeoff of different models more distinctly.
learning-based models evaluated. As a prediction-based
From the ROC curves on MBD, we can see that the false
model, LSTM-AD heavily relies on the fact that the time
positive rate of TopoMAD is lower compared with the other
series should be predictable; therefore, it is not surprising
two baseline models under a relatively low threshold. As for
that LSTM-AD performs the worst when our data sets do
MMS, the performance of TopoMAD is similar to LOF and
not satisfy this setting. By contrast, the assumption that
steadily better than RNN-EBM under different thresholds.
observations are reconstructable is weaker and more reliable.
Other deep learning-based models we evaluate, including
auto-encoder, LSTM-ED, RNN-EBM, and TopoMAD are E. Effects of Major Components
based upon this more reliable assumption, and thus, perform In this section, we will answer the question “How does
better than LSTM-AD in our two data sets. each component of TopoMAD affect the performance?”
LSTM-ED is a reconstruction-based model, and our model The results are displayed in Fig. 15. We include the mAP
in TopoMAD can be seen as its extension with consideration of LSTM-ED here because our model is kind of its exten-
of topological information and contaminated training data. sion. We select the best performance metric gained by
In addition to the lack of these considerations, the sub- LSTM-ED among two policies mentioned above to display
optimal performance of LSTM-ED is also caused by the here. “GCN ONLY” and “GAT ONLY” denote model variants
discrepancy between input distribution at training and testing of TopoMAD gained by removing its variational component
phases because the decoder in LSTM-ED uses ground truth which degrades our model into a deterministic model. “LIN-
observations as input at the training stage while reconstructed EAR + VAE” denotes a model variant gained by replac-
observations at the inference stage. To show the comparison ing the GraphLSTM in TopoMAD with a simple LSTM,
between LSTM-ED and TopoMAD more clearly, we provide which loses sight of the topological information. “GCN +
a case study in Appendix B of the supplementary material. VAE” and “GAT + VAE” denote model variants with two
As shown in Section V-C, we evaluate the baseline anomaly different choices of graph neural layers when implementing
detectors using two training policies, namely, a concatenate GraphLSTM in TopoMAD.
policy and a one-by-one policy. For MBD, nearly all base- 1) Effect of the Inclusion of Graph Neural Networks: The
line models achieve better performance using the concatenate objective of including graph neural networks in TopoMAD is
policy (note that deep anomaly detectors with the concatenate to extract features from neighbors of each component. We can
policy have higher space and time complexity than TopoMAD see in Fig. 15 that “GCN ONLY” and “GAT ONLY” are
and those with the one-by-one policy and we will discuss superior to LSTM-ED, and “GCN + VAE” and “GAT + VAE”
this more specifically in Section V-I). Nevertheless, no matter outperform or match “LINEAR + VAE” in both data sets.
which policy they utilize, TopoMAD remains considerable That is, no matter in a deterministic model or in a stochastic
superiority over them. In Section IV-A, we claim three benefits model, utilizing graph neural networks to explicitly model the
of TopoMAD, namely, the unified feature learning, the con- spatial dependence among components in a cloud system is
venience for end-to-end learning and the ability to prevent beneficial.
overfitting. These three benefits are based upon the comparison 2) Effect of VAE: It can be seen in Fig. 15 that “LINEAR +
between our model and other methods trained using the two VAE,” “GCN + VAE,” and “GAT + VAE” all outperform
policies declared earlier. their corresponding deterministic variants. The superiority of

Authorized licensed use limited to: Zhejiang University. Downloaded on January 27,2025 at 01:13:45 UTC from IEEE Xplore. Restrictions apply.
HE et al.: SPATIOTEMPORAL DEEP LEARNING APPROACH FOR UNSUPERVISED ANOMALY DETECTION IN CLOUD SYSTEMS 1715

TABLE III
D IFFERENT T OPOLOGY I NPUTS ’ AP ON T WO D ATA S ETS

VAE as an anomaly detector over a simple auto-encoder has

been demonstrated in [13], which can be concluded as the
benefits of representing latent variables and reconstructions as
stochastic variables and using probability measures as anomaly
scores. In addition, we have also illustrated VAE’s superiority Fig. 16. Three-dimensional hidden representations of each node calculated
when model training is performed on all data (also denoted as by TopoMAD for MBD.
contaminated data) in Section IV-A. Therefore, we recommend
VAE instead of a deterministic auto-encoder when performing TABLE IV
unsupervised anomaly detection. AP OF T OPO MAD U NDER D IFFERENT A NOMALY R ATIOS

F. Impact of the Topology Input

In this section, we will answer the question “How effective
is the topology?” We validate the rationality of our definition
of topology input by comparing our method with the perfor-
mance gained by inputting a random edge set array.
The results in Table III demonstrate that the topology input
also plays an important role in the superior performance of
TopoMAD. However, a misconfiguration of the topology input
does not harm so much. This is because we maintain individual
parameters for different nodes outside the GraphLSTM (as
described in Section IV-E, multilayer perceptrons outside the
GraphLSTM are learned individually for each node), allowing Fig. 17. Convergence of TopoMAD on both data sets. (a) MBD. (b) MMS.
TopoMAD to focus on the learning of each node one by one
after processing the topological information. With this design, data share similar hidden representations with normal data,
the output of our model will not be highly dependent on the decoder cannot learn to reconstruct them well, resulting in
the topology input and yet can gain benefit from it. Thus, their higher anomaly scores. Su et al. [12] also draw a similar
our model can also work even when it is confused by a conclusion about how VAE-based anomaly detectors work.
misconfiguration of the topology input. The second reason
is that actually components in a system have all kinds of
H. Robustness of TopoMAD
connections with each other and our idea is only to construct a
simplification of all these connections. From this point of view, In this section, we will answer the question “How is the
we cannot say an edge we define between the two components robustness of TopoMAD?”
is wrong. What we can say is that it does not represent a direct First, we evaluate our model under different degrees of
relation. Yet a better way to construct an effective graph-based data contamination to figure out the relationship between the
representation for a cloud system deserves further research. performance of TopoMAD and the data anomaly ratio. The
training data sets with different anomaly ratios are generated
through sampling normal data in the original data set until it
G. Interpretability of TopoMAD reaches a corresponding anomaly ratio. Table IV displays the
In this section, we will answer the question “How can we performance of TopoMAD under the anomaly ratios of 10%,
interpret the results of TopoMAD?” We will take advantage 15%, and 20%.
of the visualization of hidden representations calculated by From Table IV, we can see that the performance of Topo-
TopoMAD to explain why it tends to give an anomalous MAD will not degrade too much with the increase of the
observation a higher anomaly score. In Fig. 16, we show anomaly ratio. This is because the hidden representation
the 3-D hidden representations of each node calculated by calculated by VAE for an observation is a stochastic variable.
TopoMAD with the anomalous observations in red and the Even when the anomaly ratio is relatively high, it is still far
normal observations in blue. lower than the normal data ratio. As a result, it is difficult
From Fig. 16, we can find that there is an overlap between for the decoder to learn to reconstruct the anomaly well when
the hidden representations of anomalous observations and in each epoch, and the calculated latent is sampled from a
normal observations. The hidden representations are used normal distribution and can be quite different from the latent
as the input of the decoder to reconstruct the observations. calculated for this observation in previous epoches.
Because normal data occur more frequently, TopoMAD can In addition, we present the convergence of TopoMAD under
capture the normal pattern in the training data and reconstruct different anomaly ratios in Fig. 17. It can be seen that Topo-
normal data very well. Meanwhile, because most anomalous MAD can still converge stably at high anomaly proportions.

Authorized licensed use limited to: Zhejiang University. Downloaded on January 27,2025 at 01:13:45 UTC from IEEE Xplore. Restrictions apply.
1716 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 34, NO. 4, APRIL 2023

TABLE V TABLE VII

T IME C OMPLEXITY OF D EEP L EARNING R ECONSTRUCTION -BASED E XECUTION T IME OF T OPO MAD W ITH R EGARD TO T OPOLOGIES
A NOMALY D ETECTION M ETHODS OF D IFFERENT S CALES

TABLE VI
S PACE C OMPLEXITY OF D EEP L EARNING R ECONSTRUCTION -BASED
A NOMALY D ETECTION M ETHODS

Fig. 18. Distribution of anomaly scores on MMS.

Last but not least, we discuss the robustness of TopoMAD

with regard to initializations. Under the condition of multiple
random initializations, the performance of TopoMAD varies.
The worst AP TopoMAD achieves on MBD is 0.381, and that
on MMS is 0.399. The worst performance of TopoMAD can
still remain superiority over all baseline models in MBD and
baseline models except for LOF in MMS.
Fig. 19. F1 scores of different threshold selection methods on both data sets.

J. Effect of Our Threshold Selection Method

I. Efficiency of TopoMAD
In this section, we will answer the question “How effective
In this section, we will answer the question “How is the is our unsupervised threshold selection method in Topo-
efficiency of TopoMAD?” MAD? Can it recommend a relatively better threshold?”
Usually, there are a huge amount of data we can collect We compare our unsupervised threshold selection method with
from a large-scale cloud system, and the computational cost two unsupervised threshold selection methods POT [12] and
of an anomaly detector will inevitably increase with the NDT [47] and a supervised one which enumerates thresholds
enlargement of the cloud system. In the following, we will to achieve the best F1 score in the training data set with the
first give a theoretical analysis about the time complexity help of labels. The objective of our method is to recommend
and the space complexity of TopoMAD and other deep a relatively good threshold right after the training stage, and
learning reconstruction-based anomaly detection methods. In therefore, we assume that the testing data set remains unseen
Tables V and VI, we present the complexity of a single layer until a threshold is selected. For the supervised method here,
in fully connected feed-forward neural networks-based models a threshold which can achieve the best F1 score in the training
(FC), RNNs-based models, and TopoMAD when dealing with data set is selected and applied to the testing data set. For
data with the window size W , the node number |V |, the edge POT, we evaluate hyperparameters used in [12] and display
number |E|, the number of features per node F, and the hidden the best result achieved. For NDT, we select the historical
dim per node F  . Note that |E| |V |2 usually holds true in a window size from 15 min, 30 min, 1 h, and 2 h according to
cloud system topology. From Tables V and VI, we can see that the performance they achieve.
the time complexity and the space complexity of TopoMAD Operators can narrow the search range of thresholds by
are only higher than those of the RNNs-based models trained observing the distribution of anomaly scores on the training
in a one-by-one way. Besides, there exists some work [78] data set. To give an example, we show anomaly scores of
which can help improve efficiency when training large graph the training data of MMS in Fig. 18, where the x-axis denotes
neural networks. anomaly scores of observations, and the y-axis denotes random
In addition to a theoretical analysis, we also conduct exper- numbers which help flatten out the displayed anomaly scores
iments to evaluate the execution time of TopoMAD under so that we can recognize their density. The corresponding
topologies of varying complexities. We generate synthetic data threshold can be selected from a roughly narrowed range
sets using the data set MMS by doubling the number of pods where the density of anomaly scores begins to decrease (such
from each service. The metrics of newly generated pods are as 5∼50 here).
copied from the original data set. The window size and the The F1 scores obtained are displayed in Fig 19. We can
hidden dim are fixed to 1 during the experiments. Table VII see that the performance of our method is even slightly higher
shows the execution time of TopoMAD for testing the entire than the supervised one (the best threshold on the training data
testing data set. The growth rate of the execution time is set does not always guarantee to be the best on the testing
acceptable for anomaly detection in cloud systems. data set). This demonstrates that an unsupervised threshold

Authorized licensed use limited to: Zhejiang University. Downloaded on January 27,2025 at 01:13:45 UTC from IEEE Xplore. Restrictions apply.
HE et al.: SPATIOTEMPORAL DEEP LEARNING APPROACH FOR UNSUPERVISED ANOMALY DETECTION IN CLOUD SYSTEMS 1717

selection method can also gain a relatively good threshold. 3) Research in combining some other state-of-the-art deep
Considering the lack of operational data with labels, in reality, anomaly detectors to gain a better performance.
we recommend selecting a threshold in an unsupervised way.
R EFERENCES
K. Discussions [1] Y. Dang, Q. Lin, and P. Huang, “AIOps: Real-world challenges and
research innovations,” in Proc. IEEE/ACM 41st Int. Conf. Softw. Eng.,
1) Assumptions of Unsupervised Anomaly Detectors: Due Companion (ICSE-Companion), May 2019, pp. 4–5.
to the lack of labeled data for training, the effectiveness [2] A. Siffer, P.-A. Fouque, A. Termier, and C. Largouet, “Anomaly detec-
of an unsupervised anomaly detector relies heavily on its tion in streams with extreme value theory,” in Proc. 23rd ACM SIGKDD
assumption. As introduced in Section II-A, assumptions of Int. Conf. Knowl. Discovery Data Mining, Aug. 2017, pp. 1067–1075.
[3] H. Xu et al., “Unsupervised anomaly detection via variational auto-
traditional anomaly detectors include stochastic model-based encoder for seasonal KPIs in Web applications,” in Proc. World Wide
assumptions, density-based assumptions, classification-based Web Conf. World Wide Web WWW, 2018, pp. 187–196.
assumptions, and so on. As for deep learning-based anomaly [4] Z. Li, W. Chen, and D. Pei, “Robust and unsupervised KPI anomaly
detection based on conditional variational autoencoder,” in Proc. IEEE
detectors examined in this article, putting aside their superi- 37th Int. Perform. Comput. Commun. Conf. (IPCCC), Nov. 2018,
ority in modeling dependence in data with complex structure, pp. 1–9.
their assumptions are actually quite simple. What they assume [5] W. Chen et al., “Unsupervised anomaly detection for intricate KPIs via
adversarial training of VAE,” in Proc. IEEE INFOCOM Conf. Comput.
is that normal data are easier to generate than abnormal Commun., Apr. 2019, pp. 1891–1899.
data. This generation process can be further subdivided into [6] H. Ren et al., “Time-series anomaly detection service at microsoft,” in
reconstruction based or prediction based. The quality of an Proc. 25th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining,
assumption is determined by the data rather than the assump- Jul. 2019, pp. 3009–3017.
[7] Z. Zheng, J. Zhu, and M. R. Lyu, “Service-generated big data and big
tion itself; therefore, it is no wonder a simple traditional data-as-a-service: An overview,” in Proc. IEEE Int. Congr. Big Data,
model can achieve same or even superior performance com- Jun. 2013, pp. 403–410.
pared with some state-of-the-art deep learning-based models, [8] P. Malhotra, L. Vig, G. Shroff, and P. Agarwal, “Long short term
memory networks for anomaly detection in time series,” in Proc.
as visualized in Figs. 9 and 10. Integration of the capacity Eur. Symp. Artif. Neural Netw., Comput. Intell. Mach. Learn., 2015,
of modeling dependence in data with complex structure in pp. 89–94.
deep learning-based models and a diversity of assumptions in [9] P. Malhotra, A. Ramakrishnan, G. Anand, L. Vig, P. Agarwal, and
G. Shroff, “LSTM-based encoder-decoder for multi-sensor anomaly
traditional models is worth of a try, which is denoted as a deep detection,” in Proc. Anomaly Detection Workshop 33rd Int. Conf. Mach.
hybrid model in [79]. Learn., 2016, pp. 1–5.
2) Anomaly Localization: Once an anomaly is detected, [10] S. Zhai, Y. Cheng, W. Lu, and Z. Zhang, “Deep structured energy based
system operators usually need to locate the real root causes models for anomaly detection,” in Proc. 33rd Int. Conf. Int. Conf. Mach.
Learn., 2016, pp. 1100–1109.
of this anomaly. To locate an anomaly, operators can take the [11] C. Zhang et al., “A deep neural network for unsupervised anomaly
anomaly scores of all components generated by TopoMAD as detection and diagnosis in multivariate time series data,” in Proc. AAAI
the input of some anomaly localization algorithms, such as Conf. Artif. Intell., 2019, pp. 1409–1416.
[12] Y. Su, Y. Zhao, C. Niu, R. Liu, W. Sun, and D. Pei, “Robust anomaly
CauseInfer [27] and microscope [80]. For example, when an detection for multivariate time series through stochastic recurrent neural
anomaly in a microservice system is detected by TopoMAD, network,” in Proc. 25th ACM SIGKDD Int. Conf. Knowl. Discovery Data
operators can locate the anomaly using the cause inference Mining, Jul. 2019, pp. 2828–2837.
[13] J. An and S. Cho, “Variational autoencoder based anomaly detec-
algorithm proposed in microscope [80], which traverses across tion using reconstruction probability,” Special Lect. IE, vol. 2, no. 1,
the service topology to find the root cause. pp. 1–18, Dec. 2015.
[14] L. Li, J. Yan, H. Wang, and Y. Jin, “Anomaly detection of time series
with smoothness-inducing sequential variational auto-encoder,” IEEE
VI. C ONCLUSION Trans. Neural Netw. Learn. Syst., early access, Apr. 13, 2020, doi: 10.
Since anomaly detection is essential to operate a cloud 1109/TNNLS.2020.2980749.
[15] D. P. Kingma and M. Welling, “Auto-encoding variational Bayes,” in
system, this article proposes a topology-aware multivariate Proc. Int. Conf. Learn. Represent., 2014, pp. 1–14.
time series anomaly detector, namely, TopoMAD for unsu- [16] T. N. Kipf and M. Welling, “Semi-supervised classification with graph
pervised anomaly detection. To achieve that, we propose a convolutional networks,” in Proc. Int. Conf. Learn. Represent., 2017,
pp. 1–13.
novel architecture of a neural network by integrating the [17] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and
technologies of graph neural networks, LSTM and VAE. Our Y. Bengio, “Graph attention networks,” in Proc. Int. Conf. Learn.
approach can robustly and effectively model the complex Represent., 2018, pp. 1–12.
[18] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural
spatiotemporal dependence in contaminated data. The time Comput., vol. 9, no. 8, pp. 1735–1780, 1997.
complexity of TopoMAD is quantitatively analyzed, which [19] F. A. Gers, J. Schmidhuber, and F. Cummins, “Learning to forget:
shows that TopoMAD is efficient to analyze a large-scale Continual prediction with LSTM,” Neural Comput., vol. 12, no. 10,
cloud system. Moreover, a threshold selection algorithm is pp. 2451–2471, Oct. 2000.
[20] V. Chandola, A. Banerjee, and V. Kumar, “Anomaly detection: A survey,”
also proposed to help optimize TopoMAD. The experimental ACM Comput. Surv., vol. 41, no. 3, p. 15, 2009.
results conducted on two real-world data sets demonstrate our [21] B. Schölkopf, R. C. Williamson, A. J. Smola, J. Shawe-Taylor, and
model’s superior performance over other baseline anomaly J. C. Platt, “Support vector method for novelty detection,” in Proc. Adv.
Neural Inf. Process. Syst., 2000, pp. 582–588.
detectors. [22] M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander, “LOF: Identifying
In future works, we will focus on the following. density-based local outliers,” in Proc. Int. Conf. Manage. Data (SIG-
1) Research in online learning techniques for TopoMAD. MOD), vol. 29, 2000, pp. 93–104.
[23] S. Hawkins, H. He, G. Williams, and R. Baxter, “Outlier detection
2) The choice of metrics which are better at exposing the using replicator neural networks,” in Proc. Int. Conf. Data Warehousing
failure of a cloud system. Knowl. Discovery. Berlin, Germany: Springer, 2002, pp. 170–180.

Authorized licensed use limited to: Zhejiang University. Downloaded on January 27,2025 at 01:13:45 UTC from IEEE Xplore. Restrictions apply.
1718 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 34, NO. 4, APRIL 2023

[24] S. Fu, J. Liu, and H. Pannu, “A hybrid anomaly detection frame- [46] M. Zasadziński, M. Solé, A. Brandon, V. Muntés-Mulero, and
work in cloud computing using one-class and two-class support vector D. Carrera, “‘Next stop’ noops’: Enabling cross-system diagnostics
machines,” in Proc. Int. Conf. Adv. Data Mining Appl. Berlin, Germany: through graph-based composition of logs and metrics,” in Proc. IEEE
Springer, 2012, pp. 726–738. Int. Conf. Cluster Comput. (CLUSTER). Sep. 2018, pp. 212–222.
[25] T. Huang et al., “An LOF-based adaptive anomaly detection scheme for [47] K. Hundman, V. Constantinou, C. Laporte, I. Colwell, and T. Soder-
cloud computing,” in Proc. IEEE 37th Annu. Comput. Softw. Appl. Conf. strom, “Detecting spacecraft anomalies using LSTMs and nonparametric
Workshops, Jul. 2013, pp. 206–211. dynamic thresholding,” in Proc. 24th ACM SIGKDD Int. Conf. Knowl.
[26] K. V. Vishwanath and N. Nagappan, “Characterizing cloud computing Discovery Data Mining, Jul. 2018, pp. 387–395.
hardware reliability,” in Proc. 1st ACM Symp. Cloud Comput. SoCC, [48] Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and P. S. Yu, “A com-
2010, pp. 193–204. prehensive survey on graph neural networks,” 2019, arXiv:1901.00596.
[27] P. Chen, Y. Qi, P. Zheng, and D. Hou, “CauseInfer: Automatic and [Online]. Available: http://arxiv.org/abs/1901.00596
distributed performance diagnosis with hierarchical causality graph in [49] Y. Li, R. Yu, C. Shahabi, and Y. Liu, “Diffusion convolutional recurrent
large distributed systems,” in Proc. IEEE INFOCOM Conf. Comput. neural network: Data-driven traffic forecasting,” in Proc. Int. Conf.
Commun., Apr. 2014, pp. 1887–1895. Learn. Represent., 2018, pp. 1–14.
[28] Q. Lin et al., “Predicting node failure in cloud service systems,” in Proc. [50] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Empirical eval-
26th ACM Joint Meeting Eur. Softw. Eng. Conf. Symp. Found. Softw. Eng. uation of gated recurrent neural networks on sequence modeling,”
ESEC/FSE, 2018, pp. 480–490. 2014, arXiv:1412.3555. [Online]. Available: http://arxiv.org/abs/1412.
[29] X. Zhou et al., “Fault analysis and debugging of microservice sys- 3555
tems: Industrial survey, benchmark system, and empirical study,” IEEE [51] B. Yu, H. Yin, and Z. Zhu, “Spatio-temporal graph convolutional
Trans. Softw. Eng., early access, Dec. 18, 2018, doi: 10.1109/TSE.2018. networks: A deep learning framework for traffic forecasting,” in Proc.
2887384. 27th Int. Joint Conf. Artif. Intell., Jul. 2018, pp. 3634–3640.
[30] X. Zhou et al., “Latent error prediction and fault localization for [52] S. Yan, Y. Xiong, and D. Lin, “Spatial temporal graph convolutional
microservice applications by learning from system trace logs,” in Proc. networks for skeleton-based action recognition,” in Proc. 32nd AAAI
27th ACM Joint Meeting Eur. Softw. Eng. Conf. Symp. Found. Softw. Conf. Artif. Intell., 2018, pp. 7444–7452.
Eng., Aug. 2019, pp. 683–694. [53] P. Barham, A. Donnelly, R. Isaacs, and R. Mortier, “Using magpie for
[31] H. Liu, S. Lu, M. Musuvathi, and S. Nath, “What bugs cause production request extraction and workload modelling,” in Proc. OSDI, vol. 4, 2004,
cloud incidents,” in Proc. Workshop Hot Topics Operating Syst., 2019, p. 18.
pp. 155–162. [54] R. Fonseca, G. Porter, R. H. Katz, and S. Shenker, “X-trace: A pervasive
[32] O. Vallis, J. Hochenbaum, and A. Kejariwal, “A novel technique for network tracing framework,” in Proc. 4th USENIX Symp. Netw. Syst.
long-term anomaly detection in the cloud,” in Proc. 6th USENIX Design Implement. (NSDI), 2007, pp. 271–284.
Workshop Hot Topics Cloud Comput. (HotCloud), 2014, p. 15. [55] X. Chen, M. Zhang, Z. M. Mao, and P. Bahl, “Automating network
[33] C. Wang, V. Talwar, K. Schwan, and P. Ranganathan, “Online detection application dependency discovery: Experiences, limitations, and new
of utility cloud anomalies using metric distributions,” in Proc. IEEE solutions,” in Proc. OSDI, vol. 8, 2008, pp. 117–130.
Netw. Oper. Manage. Symp. NOMS, Apr. 2010, pp. 96–103. [56] P. Barham et al., “Constellation: Automated discovery of service and
[34] M. Du, F. Li, G. Zheng, and V. Srikumar, “DeepLog: Anomaly detection host dependencies in networked systems,” Microsoft Res., Redmond,
and diagnosis from system logs through deep learning,” in Proc. ACM WA, USA, Tech. Rep. MSR-TR-2008-67, 2008, pp. 1–14.
SIGSAC Conf. Comput. Commun. Secur., Oct. 2017, pp. 1285–1298. [57] J. Hwang, G. Liu, S. Zeng, F. Y. Wu, and T. Wood, “Topology discovery
[35] W. Meng et al., “LogAnomaly: Unsupervised detection of sequential and service classification for distributed-aware clouds,” in Proc. IEEE
and quantitative anomalies in unstructured logs,” in Proc. 28th Int. Int. Conf. Cloud Eng., Mar. 2014, pp. 385–390.
Joint Conf. Artif. Intell., Aug. 2019, pp. 4739–4745, doi: 10.24963/ijcai. [58] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation
2019/658. by jointly learning to align and translate,” in Proc. Int. Conf. Learn.
[36] X. Zhang et al., “Robust log-based anomaly detection on unstable log Represent., 2015, pp. 1–15.
data,” in Proc. 27th ACM Joint Meeting Eur. Softw. Eng. Conf. Symp. [59] A. Vaswani et al., “Attention is all you need,” in Proc. Adv. Neural Inf.
Found. Softw. Eng. ESEC/FSE, 2019, pp. 807–817. Process. Syst., 2017, pp. 5998–6008.
[37] S. He, J. Zhu, P. He, and M. R. Lyu, “Experience report: System log [60] X. Liang, X. Shen, J. Feng, L. Lin, and S. Yan, “Semantic object parsing
analysis for anomaly detection,” in Proc. IEEE 27th Int. Symp. Softw. with graph lstm,” in Proc. Eur. Conf. Comput. Vis. Berlin, Germany:
Rel. Eng. (ISSRE), Oct. 2016, pp. 207–218. Springer, 2016, pp. 125–143.
[38] N. Khadke, M. P. Kasick, S. P. Kavulya, J. Tan, and P. Narasimhan, [61] N. Peng, H. Poon, C. Quirk, K. Toutanova, and W.-T. Yih, “Cross-
“Transparent system call based performance debugging for cloud com- sentence N-ary relation extraction with graph LSTMs,” Trans. Assoc.
puting,” Presented at the Workshop Manag. Syst. Automatically Dynam- Comput. Linguistics, vol. 5, pp. 101–115, Dec. 2017.
ically, 2012. [62] A. Jain, A. R. Zamir, S. Savarese, and A. Saxena, “Structural-RNN:
[39] W. Sha, Y. Zhu, M. Chen, and T. Huang, “Statistical learning for Deep learning on spatio-temporal graphs,” in Proc. IEEE Conf. Comput.
anomaly detection in cloud server systems: A multi-order Markov chain Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 5308–5317.
framework,” IEEE Trans. Cloud Comput., vol. 6, no. 2, pp. 401–413, [63] F. R. Kschischang, B. J. Frey, and H.-A. Loeliger, “Factor graphs and
Jun. 2018. the sum-product algorithm,” IEEE Trans. Inf. Theory, vol. 47, no. 2,
[40] Y. Gan et al., “Seer: Leveraging big data to navigate the complexity pp. 498–519, Feb. 2001.
of performance debugging in cloud microservices,” in Proc. 24th Int. [64] C. Robert and G. Casella, Monte Carlo Statistical Methods. Berlin,
Conf. Architectural Support Program. Lang. Operating Syst., 2019, Germany: Springer, 2013.
pp. 19–33. [65] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning
[41] M. Ma, W. Lin, D. Pan, and P. Wang, “MS-rank: Multi-metric and self- with neural networks,” in Proc. Adv. Neural Inf. Process. Syst., 2014,
adaptive root cause diagnosis for microservice applications,” in Proc. pp. 3104–3112.
IEEE Int. Conf. Web Services (ICWS), Jul. 2019, pp. 60–67. [66] S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer, “Scheduled sampling
[42] A. B. Sharma, H. Chen, M. Ding, K. Yoshihira, and G. Jiang, “Fault for sequence prediction with recurrent neural networks,” in Proc. Int.
detection and localization in distributed systems using invariant rela- Conf. Neural Inf. Process. Syst., 2015, pp. 1171–1179.
tionships,” in Proc. 43rd Annu. IEEE/IFIP Int. Conf. Dependable Syst. [67] S. Huang, J. Huang, J. Dai, T. Xie, and B. Huang, “The HiBench bench-
Netw. (DSN), Jun. 2013, pp. 1–8. mark suite: Characterization of the MapReduce-based data analysis,” in
[43] P. Chen, Y. Qi, X. Li, and L. Su, “An ensemble MIC-based approach for Proc. IEEE 26th Int. Conf. Data Eng. Workshops (ICDEW ), Mar. 2010,
performance diagnosis in big data platform,” in Proc. IEEE Int. Conf. pp. 41–51.
Big Data, Oct. 2013, pp. 78–85. [68] (2020). HDFS-448. Accessed: 2020. [Online]. Available: https://issues.
[44] W. Cheng, K. Zhang, H. Chen, G. Jiang, Z. Chen, and W. Wang, apache.org/jira/browse/HDFS-448
“Ranking causal anomalies via temporal and dynamical analysis on [69] (2020). HDFS-8160. Accessed: 2020. [Online]. Available: https://issues.
vanishing correlations,” in Proc. 22Nd ACM SIGKDD Int. Conf. Knowl. apache.org/jira/browse/HDFS-8160
Discovery Data Mining, 2016, pp. 805–814. [70] (2020). Telegraf. Accessed: 2020. [Online]. Available: https://www.
[45] Z. Zheng, T. Chao Zhou, M. R. Lyu, and I. King, “Component ranking influxdata.com/time-series-platform/telegraf/
for fault-tolerant cloud applications,” IEEE Trans. Services Comput., [71] (2020). Cloudwatch. Accessed: 2020. [Online]. Available: https://aws.
vol. 5, no. 4, pp. 540–550, 4th Quart., 2012. amazon.com/cn/cloudwatch/

Authorized licensed use limited to: Zhejiang University. Downloaded on January 27,2025 at 01:13:45 UTC from IEEE Xplore. Restrictions apply.
HE et al.: SPATIOTEMPORAL DEEP LEARNING APPROACH FOR UNSUPERVISED ANOMALY DETECTION IN CLOUD SYSTEMS 1719

[72] (2019). Hipster Shop: Cloud-Native Microservices Demo Appli- Yongfeng Wang received the B.E. degree from the
cation. Accessed: 2019. [Online]. Available: https://github.com/ School of Data and Computer Science, Sun Yat-sen
GoogleCloudPlatform/microservices-demo University, Guangzhou, China.
[73] (2019). Kubernetes. Accessed: 2019. [Online]. Available: https:// His current research areas include persistent mem-
kubernetes.io/ ory, storage system, and cloud computing.
[74] J. Davis and M. Goadrich, “The relationship between precision-recall
and ROC curves,” in Proc. 23rd Int. Conf. Mach. Learn. ICML, 2006,
pp. 233–240, doi: 10.1145/1143844.1143874.
[75] M. Fey and J. Eric Lenssen, “Fast graph representation learning
with PyTorch geometric,” 2019, arXiv:1903.02428. [Online]. Available:
http://arxiv.org/abs/1903.02428
[76] P. I. Frazier, “A tutorial on Bayesian optimization,” 2018,
arXiv:1807.02811. [Online]. Available: http://arxiv.org/abs/1807.02811
[77] J. Snoek, H. Larochelle, and R. P. Adams, “Practical Bayesian optimiza-
tion of machine learning algorithms,” in Proc. Adv. Neural Inf. Process. Guangba Yu is currently pursuing the Ph.D. degree
Syst., 2012, pp. 2951–2959. with the School of Data and Computer Science, Sun
[78] W.-L. Chiang, X. Liu, S. Si, Y. Li, S. Bengio, and C.-J. Hsieh, Yat-sen University, Guangzhou, China.
“Cluster-GCN: An efficient algorithm for training deep and large graph His current research areas include distributed sys-
convolutional networks,” in Proc. 25th ACM SIGKDD Int. Conf. Knowl. tem, cloud computing, and AI-driven operations.
Discovery Data Mining, Jul. 2019, pp. 257–266.
[79] R. Chalapathy and S. Chawla, “Deep learning for anomaly detection:
A survey,” 2019, arXiv:1901.03407. [Online]. Available: http://arxiv.org/
abs/1901.03407
[80] J. Lin, P. Chen, and Z. Zheng, “Microscope: Pinpoint performance issues
with causal graphs in micro-service environments,” in Proc. Int. Conf.
Service-Oriented Comput. Berlin, Germany: Springer, 2018, pp. 3–20.

Cailin Chen is currently a Post-Graduate Researcher

Zilong He received the B.E. degree from Sun
with the School of Data and Computer Science, Sun
Yat-sen University, Guangzhou, China, in 2019.
Yat-sen University, Guangzhou, China. Her current
He is currently a Post-Graduate Researcher with
research area includes AI-driven operations.
the School of Data and Computer Science, Sun
Yat-sen University. His current research areas
include anomaly detection algorithms and AI-driven
operations.

Pengfei Chen received the Ph.D. degree from the

Department of Computer Science, Xi’an Jiaotong Xinrui Li received the B.E. degree from the School
University, Xi’an, China, in 2016. of Data and Computer Science, Sun Yat-sen Univer-
He visited the IBM Thomas J. Watson Research sity, Guangzhou, China.
Center, Ossining, NY, USA, in 2017. He is cur-
rently an Associate Professor with the School of
Data and Computer Science, Sun Yat-sen Univer-
sity, Guangzhou, where he is also a Ph.D. Advisor.
He has strong skill in cloud computing. He current
research interests include distributed systems, artifi-
cial intelligence for IT operations, cloud computing,
microservice, and blockchain.

Xiaoyun Li is currently pursuing the Ph.D. degree Zibin Zheng (Senior Member, IEEE) received
with the School of Data and Computer Science, Sun the Ph.D. degree from The Chinese University of
Yat-sen University, Guangzhou, China. Hong Kong, Hong Kong, in 2011.
Her current research areas include log analysis and He is currently a Professor with the School of
AI-driven operations. Data and Computer Science, Sun Yat-sen University,
Guangzhou, China. His research interests include
blockchain, services computing, software engineer-
ing, and financial big data.

Authorized licensed use limited to: Zhejiang University. Downloaded on January 27,2025 at 01:13:45 UTC from IEEE Xplore. Restrictions apply.