Computational Visual Media

https://doi.org/10.1007/s41095-020-0191-7

Review Article

A survey of visual analytics techniques for machine learning

Jun Yuan1 , Changjian Chen1 , Weikai Yang1 , Mengchen Liu2 , Jiazhi Xia3 , and Shixia Liu1 ( )


c The Author(s) 2020.

Abstract Visual analytics for machine learning has and machine learning techniques to facilitate the
recently evolved as one of the most exciting areas in the analysis and understanding of the major components
field of visualization. To better identify which research in the learning process, with an aim to improve
topics are promising and to learn how to apply relevant performance. For example, visual analytics research for
techniques in visual analytics, we systematically review explaining the inner workings of deep convolutional
259 papers published in the last ten years together neural networks has increased the transparency of
with representative works before 2010. We build a deep learning models and has received ongoing and
taxonomy, which includes three first-level categories: increasing attention recently [1–4].
techniques before model building, techniques during The rapid development of visual analytics
modeling building, and techniques after model building.
techniques for machine learning yields an emerging
Each category is further characterized by representative
need for a comprehensive review of this area to
analysis tasks, and each task is exemplified by a
support the understanding of how visualization
set of recent influential works. We also discuss and
techniques are designed and applied to machine
highlight research challenges and promising potential
future research opportunities useful for visual analytics learning pipelines. There have been several initial
researchers. efforts to summarize the advances in this field from
different viewpoints. For example, Liu et al. [5]
Keywords visual analytics; machine learning; data summarized visualization techniques for text analysis.
quality; feature selection; model under-
Lu et al. [6] surveyed visual analytics techniques for
standing; content analysis
predictive models. Recently, Liu et al. [1] presented
a paper on the analysis of machine learning models
1 Introduction from the visual analytics viewpoint. Sacha et al. [7]
analyzed a set of example systems and proposed
The recent success of artificial intelligence applications
an ontology for visual analytics assisted machine
depends on the performance and capabilities of
learning. However, existing surveys either focus on
machine learning models [1]. In the past ten years,
a specific area of machine learning (e.g., text mining
a variety of visual analytics methods have been
[5], predictive models [6], model understanding [1])
proposed to make machine learning more explainable,
or aim to sketch an ontology [7] based on a set of
trustworthy, and reliable. These research efforts fully
example techniques only.
combine the advantages of interactive visualization
In this paper, we aim to provide a comprehensive
survey of visual analytics techniques for machine
1 BNRist, Tsinghua University, Beijing 100086, China. learning, which focuses on every phase of the
E-mail: J. Yuan, [email protected]; C. machine learning pipeline. We focus on works in
Chen, [email protected]; W. Yang, ywk19@
the visualization community. Nevertheless, the AI
mails.tsinghua.edu.cn; S. Liu, [email protected] ( ).
community has also made solid contributions to the
2 Microsoft, Redmond 98052, USA, E-mail: mengcliu@
microsoft.com. study of visually explaining feature detectors in deep
3 Central South University, Changsha 410083, China, learning models. For example, Selvaraju et al. [8]
E-mail: [email protected]. tried to identify the part of an image to which its
Manuscript received: 2020-07-12; accepted: 2020-08-04 classification result is sensitive, by computing class

1
2 J. Yuan, C. Chen, W. Yang, et al.

activation maps. Readers can refer to the surveys of machine learning techniques over the past ten years,
Zhang and Zhu [9] and Hohman et al. [3] for more this field has been attracting ever more research
details. We have collected 259 papers from related top- attention.
tier venues in the past ten years through a systematical 2.2 Taxonomy
procedure. Based on the machine learning pipeline,
In this section, we comprehensively analyze the
we divide this literature as relevant to three stages:
collected visual analytics works to systematically
before, during, and after model building. We analyze
understand the major research trends. These works
the functions of visual analytics techniques in the three
are categorized based on a typical machine learning
stages and abstract typical tasks, including improving
pipeline [11] used to solve real-world problems. As
data quality and feature quality before model building,
shown in Fig. 1, such a pipeline contains three stages:
model understanding, diagnosis, and steering during
(1) data pre-processing before model building, (2)
model building, and data understanding after model
machine learning model building, and (3) deployment
building. Each task is illustrated by a set of carefully
after the model is built. Accordingly, visual analytics
selected examples. We highlight six prominent
techniques for machine learning can be mapped into
research directions and open problems in the field of
these three stages: techniques before model building,
visual analytics for machine learning. We hope that
techniques during model building, and techniques
this survey promotes discussion of machine learning
after model building.
related visual analytics techniques and acts as a
starting point for practitioners and researchers wishing 2.2.1 Techniques before model building
to develop visual analytics tools for machine learning. The major goal of visual analytics techniques before
model building is to help model developers better
2 Survey landscape prepare the data for model building. The quality
of the data is mainly determined by the data itself
2.1 Paper selection and the features used. Accordingly, there are two
In this paper, we focus on visual analytics techniques research directions, visual analytics for data quality
that help to develop explainable, trustworthy, improvement and feature engineering.
and reliable machine learning applications. To Data quality can be improved in various ways,
comprehensively survey visual analytics techniques such as completing missing data attributes and
for machine learning, we performed an exhaustive correcting wrong data labels. Previously, these tasks
manual review of relevant top-tier venues in the past were mainly conducted manually or by automatic
ten years (2010–2020): these were InfoVis, VAST, methods, such as learning-from-crowds algorithms
Vis (later SciVis), EuroVis, PacificVis, IEEE TVCG, [12] which aim to estimate ground-truth labels from
CGF, and CG&A. The manual review was conducted noisy crowd-sourced labels. To reduce experts’ efforts
by three Ph.D. candidates with more than two years or further improve the results of automatic methods,
of research experience in visual analytics. We followed some works employ visual analytics techniques to
the manual review process used in a text visualization interactively improve the data quality. Table 1 shows
survey [5]. Specifically, we first considered the titles that in recent years, this topic has gained increasing
of papers from these venues to identify candidate research attention.
papers. Next, we reviewed the abstracts of the Feature engineering is used to select the best
candidate papers to further determine whether they features to train the model. For example, in computer
concerned visual analytics techniques for machine vision, we could use HOG (Histogram of Oriented
learning. If the title and abstract did not provide Gradient) features instead of using raw image pixels.
clear information, the full text was gone through to In visual analytics, interactive feature selection
make a final decision. In addition to the exhaustive provides an interactive and iterative feature selection
manual review of the above venues, we also searched process. In recent years, in the deep learning era,
for the representative related works that appeared feature selection and construction are mostly
earlier or in other venues, such as the Profiler [10]. conducted via neural networks. Echoing this trend,
After this process, 259 papers were selected. Table 1 there is reducing research attention in recent years
presents detailed statistics. Due to the increase in (2016–2020) in this direction (see Table 1).
A survey of visual analytics techniques for machine learning 3

Table 1 Categories of visual analytics techniques for machine learning and representative works in each category (number of papers given in
brackets)
Technique category Papers Trend
[14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24],
Improving data quality (31) [25],[26], [27], [10], [28], [29], [30], [31], [32], [33], [34],
Before model building
[35],[36], [37], [38], [39], [40], [41], [42], [43]
Improving feature quality (6) [44], [45], [46], [47], [48], [49]
[50], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60], [61],
Model understanding (30) [62], [63], [64], [65], [66], [67], [68], [69], [70], [71], [72], [73],
[74], [75], [76], [77], [78], [79]

Model diagnosis (19) [80], [81], [82], [83], [84], [85], [86], [87], [88], [89], [90], [91],
During model building [92], [93], [94], [95], [96], [97], [98]
[99], [100], [101], [102], [13], [103], [104], [105], [106], [107],
[108], [109], [110], [111], [112], [113], [114], [115], [116],
Model steering (29)
[117], [118], [119], [120], [121], [122], [123], [124], [125],
[126]
[127], [128], [129], [130], [131], [132], [133], [134], [135],
[136], [137], [138], [139], [140], [141], [142], [143],
Understanding static data
[144], [145], [146], [147], [148], [149], [150], [151], [152],
analysis results (43)
[153], [154], [155], [156], [157], [158], [159], [160], [161],
[162],[163], [164], [165], [166], [167], [168], [169]
[170], [171], [172], [173], [174], [175], [176], [177], [178],
[179], [180], [181], [182], [183], [184], [185], [186], [187],
[188], [189], [190], [191], [192], [193], [194], [195], [196],
After model building
[197], [198], [199], [200], [201], [202], [203], [204], [205],
[206], [207], [208], [209], [210], [211], [212], [213], [214],
Understanding dynamic data [215], [216], [217], [218], [219], [220], [221], [222], [223],
analysis results (101) [224], [225], [226], [227], [228], [229], [230], [231], [232],
[233], [234], [235], [236], [237], [238], [239], [240], [241],
[242], [243], [244], [245], [246], [247], [248], [249], [250],
[251], [252], [253], [254], [255], [256], [257], [258], [259],
[260], [261], [262], [263], [264], [265], [266], [267], [268],
[269], [270]

Fig. 1 An overview of visual analytics research for machine learning.

2.2.2 Techniques during model building visual analytics methods to facilitate model building
Model building is a central stage in building a is also a growing research direction in visualization
successful machine learning application. Developing (see Table 1). In this survey, we categorize current
4 J. Yuan, C. Chen, W. Yang, et al.

methods by their analysis goal: model understanding, 3.1 Improving data quality
diagnosis, or steering. Model understanding methods Data includes instances and their labels [273]. From
aim to visually explain the working mechanisms of a this perspective, existing efforts for improving data
model, such as how changes in parameters influence quality either concern instance-level improvement, or
the model and why the model gives a certain output label-level improvement.
for a specific input. Model diagnosis methods target 3.1.1 Instance-level improvement
diagnosing errors in model training via interactive
At the instance level, many visual analytics methods
exploration of the training process. Model steering focus on detecting and correcting anomalies in data,
methods are mainly aimed at interactively improving such as missing values and duplication. For example,
model performance. For example, to refine a topic Kandel et al. [10] proposed Profiler to aid the
model, Utopian [13] enables users to interactively discovery and assessment of anomalies in tabular data.
merge or split topics, and automatically modify other Anomaly detection methods are applied to detect data
topics accordingly. anomalies, which are classified into different types
2.2.3 Techniques after model building subsequently. Then, linked summary visualizations
After a machine learning model has been built and are automatically recommended to facilitate the
deployed, it is crucial to help users (e.g., domain discovery of potential causes and consequences of
these anomalies. VIVID [14] was developed to handle
experts) understand the model output in an intuitive
missing values in longitudinal cohort study data.
way, to promote trust in the model output. To this
Through multiple coordinated visualizations, experts
end, there are many visual analytics methods to
can identify the root causes of missing values (e.g., a
explore model output, for a variety of applications.
particular group who do not participate in follow-up
Unlike methods for model understanding during
examinations), and replace missing data using an
model building, these methods usually target model
appropriate imputation model. Anomaly removal is
users rather than model developers. Thus, the
often an iterative process which must be repeated.
internal workings of a model are not illustrated,
Illustrating provenance in this iterative process allows
but the focus is on the intuitive presentation and users to be aware of changes in data quality and
exploration of model output. As these methods are to build trust in the processed data. Thus, Bors
often data-driven or application-driven, in this survey, et al. [20] proposed DQProv Explorer to support
we categorize these methods by the type of data being the analysis of data provenance, using a provenance
analyzed, particularly as static data or temporal data. graph to support the navigation of data states and a
quality flow to present changes in data quality over
3 Techniques before model building time. Recently, another type of data anomaly, out-of-
distribution (OoD) samples, has received extensive
Two major tasks required before building a model attention [274, 275]. OoD samples are test samples
are data processing and feature engineering. They that are not well covered by training data, which is a
are critical, as practical experience indicates that low- major source of model performance degradation. To
quality data and features degrade the performance tackle this issue, Chen et al. [21] proposed OoDAnalyzer
of machine learning models [271, 272]. Data quality to detect and analyze OoD samples. An ensemble OoD
issues include missing values, outliers, and noise in detection method, combining both high- and low-level
instances and their labels. Feature quality issues features, was proposed to improve detection accuracy.
include irrelevant features, redundancy between A grid visualization of the detection result (see Fig. 2) is
features, etc. While manually addressing these utilized to explore OoD samples in context and explain
issues is time-consuming, automatic methods may the underlying reasons for their presence. In order
suffer from poor performance. Thus, various visual to generate grid layouts at interactive rates during
analytics techniques have been developed to reduce the exploration, a kNN-based grid layout algorithm
experts’ effort as well as to simultaneously improve motivated by Hall’s theorem was developed.
the performance of automatic methods of producing When considering time-series data, several
high-quality data and features [303]. challenges arise as time has distinct characteristics
A survey of visual analytics techniques for machine learning 5

Fig. 2 OoDAnalyzer, an interactive method to detect out-of-distribution samples and explain them in context. Reproduced with permission
from Ref. [21], 
c IEEE 2020.

that induce specific quality issues that require To tackle uncertainties in data quality improve-
analysis in a temporal context. To tackle this issue, ment, Bernard et al. [17] developed a visual
Arbesser et al. [15] proposed a visual analytics analytics tool to exhibit the changes in the data
system, Visplause, to visually assess time-series data and uncertainties caused by different preprocessing
quality. Anomaly detection results, e.g., frequencies methods. This tool enables experts to become aware
of anomalies and their temporal distributions, are of the effects of these methods and to choose suitable
shown in a tabular layout. In order to address ones, to reduce task-irrelevant parts while preserving
the scalability problem, data are aggregated in a task-relevant parts of the data.
hierarchy based on meta-information, which enables As data have the risk of exposing sensitive
analysis of a group of anomalies (e.g., abnormal time information, several recent studies have focused on
series of the same type) simultaneously. Besides preserving data privacy during the data quality
automatically detected anomalies, KYE [23] also improvement process. For tabular data, Wang et
supports the identification of additional anomalies al. [41] developed a Privacy Exposure Risk Tree to
overlooked by automatic methods. Time-series data display privacy exposure risks in the data and a
are presented in a heatmap view, where abnormal Utility Preservation Degree Matrix to exhibit how
patterns (e.g., regions with unusually high values) the utility changes as privacy-preserving operations
indicate potential anomalies. Click stream data are applied. To preserve privacy in network datasets,
are a widely studied kind of time-series data in Wang et al. [40] presented a visual analytics system,
the field of visual analytics. To better analyze GraphProtector. To preserve important structures
and refine click stream data, Segmentifier [22] was of networks, node priorities are first specified
proposed to provide an iterative exploration process based on their importance. Important nodes are
for segmentation and analysis. Users can explore assigned low priorities, reducing the possibility of
segments in three coordinated views at different modifying these nodes. Based on node priorities
granularities and refine them by filtering, partitioning, and utility metrics, users can apply and compare a
and transformation. Every refinement step results set of privacy-preserving operations and choose the
in new segments, which can be further analyzed and most suitable one according to their knowledge and
refined. experience.
6 J. Yuan, C. Chen, W. Yang, et al.

3.1.2 Label-level improvement [31] was proposed to improve crowdsourced labels by

According to whether the data have noisy labels, validating uncertain instance labels and unreliable
existing works can be classified as either methods workers. Three coordinated visualizations, a con-
for improving the quality of noisy labels, or allowing fusion (see Fig. 3(a)), an instance (see Fig. 3(b)), and
interactive labeling. a worker visualization (see Fig. 3(c)), were developed
Crowdsourcing provides a cost-effective way to to facilitate the identification and validation of
collect labels. However, annotations provided by uncertain instance labels and unreliable workers.
crowd workers are usually noisy [271, 276]. Many Based on expert validation, further instances and
methods have been proposed to remove noise in workers are recommended for validation by an
labels. Willett et al. [42] developed a crowd- iterative and progressive verification procedure.
assisted clustering method to remove redundant Although the aforementioned methods can
explanations provided by crowd workers. Explanations effectively improve crowdsourced labels, crowd
are clustered into groups, and the most representative information is not available in many real-world
ones are preserved. Park et al. [35] proposed datasets. For example, the ImageNet dataset [277]
C2 A that visualizes crowdsourced annotations and only contains the labels cleaned by automatic noise
worker behavior to help doctors identify malignant removal methods. To tackle these datasets, Xiang
tumors in clinical videos. Using C2 A, doctors can et al. [43] developed DataDebugger to interactively
discard most tumor-free video segments and focus improve data quality by utilizing user-selected trusted
on the ones that most likely to contain tumors. To items. Hierarchical visualization combined with
analyze the accuracy of crowdsourcing workers, Park an incremental projection method and an outlier
et al. [34] developed CMed that visualizes clinical biased sampling method facilitates the exploration
image annotations by crowdsourcing, and workers’ and identification of trusted items. Based on these
behavior. By clustering workers according to their identified trusted items, a data correction algorithm
annotation accuracy and analyzing their logged events, propagates labels from trusted items to the whole
experts are able to find good workers and observe the dataset. Paiva et al. [33] assumed that instances
effects of workers’ behavior patterns. LabelInspect misclassified by a trained classifier were likely to

Fig. 3 LabelInspect, an interactive method to verify uncertain instance labels and unreliable workers. Reproduced with permission from
Ref. [31], 
c IEEE 2019.
A survey of visual analytics techniques for machine learning 7

be mislabeled instances. Based on this assumption, tweets [37], and argumentation mining [38]. For
they employed a Neighbor Joining Tree enhanced by example, to annotate text fragments in argumentation
multidimensional projections to help users explore mining tasks, Sperrle et al. [38] developed a language
misclassified instances and correct mislabeled ones. model for fragment recommendation. A layered
After correction, the classifier is refined using the visual abstraction is utilized to support five relevant
corrected labels, and a new round of correction starts. analysis tasks required by text fragment annotation.
Bäuerle et al. [16] developed three classifier-guided In addition to developing systems for interactive
measures to detect data errors. Data errors are then labeling, some empirical experiments were conducted
presented in a matrix and a scatter plot, allowing to demonstrate their effectiveness. For example,
experts to reason about and resolve errors. Bernard et al. [18] conducted experiments to show the
All the above methods start with a set of labeled superiority of user-centered visual interactive labeling
data with noise. However, many datasets do not over model-centered active learning. A quantitative
contain such a label set. To tackle this issue, many analysis [19] was also performed to evaluate user
visual analytics methods have been proposed for strategies for selecting samples in the labeling process.
interactive labeling. Reducing labeling effort is a major Results show that in early phases, data-based (e.g.,
goal of interactive labeling. To this end, Moehrmann clusters and dense areas) user strategies work well.
et al. [32] used an SOM-based visualization to place However, in later phases, model-based (e.g., class
similar images together, allowing users to label multiple separation) user strategies perform better.
similar images of the same class in one go. This
strategy is also used by Khayat et al. [28] to identify 3.2 Improving feature quality
social spambot groups with similar anomalous behavior, A typical method to improve feature quality is
Kurzhals et al. [29] to label mobile eye-tracking selecting useful features that contribute most to the
data, and Halter et al. [24] to annotate and analyze prediction, i.e., feature selection [278]. A common
primary color strategies used in films. Apart from feature selection strategy is to select a subset of
placing similar items together, other strategies, like features that minimizes the redundancy between
filtering, have also been applied to find items of interest them and maximizes the relevance between them
for labeling. Filtering and sorting are utilized in and targets (e.g., classes of instances) [46]. Along
MediaTable [36] to find similar video segments. A table this line, several methods have been developed to
visualization is utilized to present video segments and interactively analyze the redundancy and relevance
their attributes. Users can filter out irrelevant segments of features. For example, Seo and Shneiderman
and sort on attributes to order relevant segments, [48] proposed a rank-by-feature framework, which
allowing users to label several segments of the same ranks features by relevance. They visualized ranking
class simultaneously. Stein et al. [39] provided a rule- results with tables and matrices. Ingram et al. [44]
based filtering engine to find patterns of interest in proposed a visual analytics system, DimStiller, which
soccer match videos. Experts can interactively specify allows users to explore features and their relationships
rules through a natural language GUI. and interactively remove irrelevant and redundant
Recently, to enhance the effectiveness of interactive features. May et al. [46] proposed SmartStripes
labeling, various visual analytics methods have to select different feature subsets for different data
combined visualization techniques with machine subsets. A matrix-based layout is utilized to
learning techniques, such as active learning. The exhibit the relevance and redundancy of features.
concept of “intra-active labeling” was first introduced Mühlbacher and Piringer [47] developed a partition-
by Höferlin et al. [26]; it enhances active learning based visualization for the analysis of the relevance
with human knowledge. Users are not only able to of features or feature pairs. The features or feature
query instances and label them via active learning, pairs are partitioned into subdivisions, which allows
but also to understand and steer machine learning users to explore the relevance of features (or feature
models interactively. This concept is also used in pairs) at different levels of detail. Parallel coordinates
text document retrieval [25], sequential data retrieval visualization was utilized by Tam et al. [49] to identify
[30], trajectory classification [27], identifying relevant features that could discriminate between different
8 J. Yuan, C. Chen, W. Yang, et al.

clusters. Krause et al. [45] ranked features across outputs; these were bird occurrence predictions in
different feature selection algorithms, cross-validation their application. The tool also reveals how these
folds, and classification models. Users are able to parameters are related to each other in the prediction
interactively select the features and models that lead model. Zhang et al. [266] proposed a visual analytics
to the best performance. method to visualize how variables affect statistical
Besides selecting existing features, constructing indicators in a logistic regression model.
new features is also useful in model building. For 4.1.2 Understanding model behaviours
example, FeatureInsight [279] was proposed to Another aspect is how the model works to produce
construct new features for text classification. By the desired outputs. There are three main types
visually examining classifier errors and summarizing of methods used to explain model behaviours,
the root causes of these errors, users are able to namely network-centric, instance-centric, and hybrid
create new features that can correctly discriminate methods. Network-centric methods aim to explore
misclassified documents. To improve the generalization the model structure and interpret how different parts
capability of new features, visual summaries are used of the model (e.g., neurons or layers in convolutional
to analyze a set of errors instead of individual errors. neural networks) cooperate with each other to
produce the final outputs. Earlier works employ
4 Techniques during model building directed graph layouts to visualize the structure of
Machine learning models are usually regarded as black neural networks [280], but visual clutter becomes a
boxes because of their lack of interpretability, which serious problem as the model structure becoming
hinders their practical use in risky scenarios such as increasingly complex. To tackle this problem,
self-driving cars and financial investment. Current Liu et al. [62] developed CNNVis to visualize
visual analytics techniques in model building explore deep convolutional neural networks (see Fig. 4). It
how to reveal the underlying working mechanisms leverages clustering techniques to group neurons with
of machine learning models and then help model similar roles as well as their connections in order to
developers to build well-formed models. First address visual clutter caused by their huge quantity.
of all, model developers require a comprehensive This tool helps experts understand the roles of the
understanding of models in order to release them neurons and their learned features, and moreover,
from a time-consuming trial-and-error process. When how low-level features are aggregated into high-level
the training process fails or the model does not ones through the network. Later, Wongsuphasawat
provide satisfactory performance, model developers et al. [77] designed a graph visualization for
need to diagnose the issues occurring in the training exploring the machine learning model architecture in
process. Finally, there is a need to assist in model Tensorflow [281]. They conducted a series of graph
steering as much time is spent in improving model transformations to compute a legible interactive
performance during the model building process. graph layout from a given low-level dataflow graph
Echoing these needs, researchers have developed to display the high-level structure of the model.
many visual analytics methods to enhance model Instance-centric methods aim to provide
understanding, diagnosis, and steering [1, 2]. instance-level analysis and exploration, as well
as understanding the relationships between instances.
4.1 Model understanding
Rauber et al. [69] visualized the representations
Works related to model understanding belong to two learned from each layer in the neural network
classes: those understanding the effects of parameters, by projecting them onto 2D scatterplots. Users
and those understanding model behaviour. can identify clusters and confusion areas in the
4.1.1 Understanding the effects of parameters representation projections and, therefore, understand
One aspect of model understanding is to inspect the representation space learned by the network.
how the model outputs change with changes in Furthermore, they can study how the representation
model parameters. For example, Ferreira et al. [54] space evolves during training so as to understand the
developed BirdVis to explore the relationships network’s learning behaviour. Some visual analytics
between different parameter configurations and model techniques for understanding recurrent neural
A survey of visual analytics techniques for machine learning 9

Fig. 4 CNNVis, a network-centric visual analytics technique to understand deep convolutional neural networks with millions of neurons and
connections. Reproduced with permission from Ref. [62], 
c IEEE 2017.

networks (RNNs) also adopt such an instance-centric model structure with a computational graph and the
design. LSTMVis [73] developed by Strobelt et al. activation relationships between instances, subsets,
utilizes parallel coordinates to present the hidden and classes using a projected view.
states, to support the analysis of changes in the hidden In recent years, there have been some efforts to
states over texts. RNNVis [65] developed by Ming et use a surrogate explainable model to explain model
al. clusters the hidden state units (each hidden state behaviours. The major benefit of such methods
unit is a dimension of the hidden state vector in an is that they do not require users to investigate
RNN) as memory chips and words as word clouds. the model itself. Thus, they are more useful for
Their relationships are modeled as a bipartite graph, those with no or limited machine learning knowledge.
which supports sentence-level explanations in RNNs. Treating the classifier as a black box, Ming et
Hybrid methods combine the above two methods al. [66] first extracted rule-based knowledge from
and leverage both of their strengths. In particular, the input and output of the classifier. These rules are
instance-level analysis can be enhanced with the then visualized using RuleMatrix, which supports
context of the network architecture. Such contexts interactive exploration of the extracted rules by
benefit the understanding of the network’s working practitioners, improving the interpretability of the
mechanism. For instance, Hohman et al. [56] model. Wang et al. [75] developed DeepVID to
proposed Summit, to reveal important neurons and generate a visual interpretation for image classifiers.
critical neuron associations contributing to the model Given an image of interest, a deep generative model
prediction. It integrates an embedding view to was first used to generate samples near it. These
summarize the activations between classes and an generated samples were used to train a simpler and
attribute graph view to reveal influential connections more interpretable model, such as a linear regression
between neurons. Kahng et al. [59] proposed ActiVis classifier, which helps explain how the original model
for large-scale deep neural networks. It visualizes the makes the decision.
10 J. Yuan, C. Chen, W. Yang, et al.

4.2 Model diagnosis being reliably applied to real-world applications

Visual analytical techniques for model diagnosis may [84, 91]. Cao et al. [84] proposed AEVis to analyze
either analyze the training results, or analyze the how adversarial examples fool neural networks. The
training dynamics. system (see Fig. 5) takes both normal and adversarial
examples as input and extracts their datapaths for
4.2.1 Analyzing training results
model prediction. It then employs a river-based
Tools have been developed for diagnosing classifiers metaphor to show the diverging and merging patterns
based on their performance [81, 82, 86, 93]. For of the extracted datapaths, which reveal where the
example, Squares [93] used boxes to represent adversarial samples mislead the model. Ma et al. [91]
samples and group them according to their prediction designed a series of visual representations from
classes. Using different textures to encode true/false overview to detail to reveal how data poisoning will
positives/negatives, this tool allows fast and accurate make a model misclassify a specific sample. By
estimation of performance metrics at multiple levels of comparing the distributions of the poisoned and
detail. Recently, the issue of model fairness has drawn normal training data, experts can deduce the reason
growing attention [80, 83, 97]. For example, Ahn et for the misclassification of the attacked sample.
al. [80] proposed a framework named FairSight and
implemented a visual analytics system to support the 4.3 Analyzing training dynamics
analysis of fairness in ranking problems. They divided Recent efforts have also been concentrated on
the machine learning pipeline into three phases (data, analyzing the training dynamics. These techniques
model, and outcome) and then measured the bias are intended for debugging the training process of
both at individual and group levels using different machine learning models. For example, DGMTracker
measures. Based on these measures, developers [89] assists experts to discover reasons for the
can iteratively identify those features that cause failed training process of deep generative models. It
discrimination and remove them from the model. utilizes a blue-noise polyline sampling algorithm
Researchers are also interested in exploring potential to simultaneously keep the outliers and the major
vulnerabilities in models that prevent them from distribution of the training dynamics in order to help

Fig. 5 AEVis, a visual analytics system for analyzing adversarial samples. It shows diverging and merging patterns in the extracted datapaths
with a river-based visualization, and critical feature maps with a layer-level visualization. Reproduced with permission from Ref. [84], 
c IEEE 2020.
A survey of visual analytics techniques for machine learning 11

experts detect the potential root cause of a failure. Users can directly refine the target model with
It also employs a credit assignment algorithm to visual analytics techniques. A typical example is
disclose the interactions between neurons to facilitate ProtoSteer [116], a visual analytics system that
the diagnosis of failure propagation. Attention has also enables editing prototypes to refine a prototype
been given to the diagnosis of the training process of sequence network named ProSeNet [282]. ProtoSteer
deep reinforcement learning. Wang et al. [96] proposed uses four coordinated views to present the information
DQNViz for the understanding and diagnosis of deep about the learned prototypes in ProSeNet. Users
Q-networks for a Breakout game. At the overview can refine these prototypes by adding, deleting, and
level, DQNViz presents changes in the overall statistics revising specific prototypes. The model is then
during the training process with line charts and retrained with these user-specific prototypes for
stacked area charts. Then at the detail level, it uses performance gain. In addition, van der Elzen and van
segment clustering and a pattern mining algorithm Wijk [122] proposed BaobabView to support experts
to help experts identify common as well as suspicious to construct decision trees iteratively using domain
patterns in the event-sequences of the agents in Q- knowledge. Experts can refine the decision tree with
networks. As another example, He et al. [87] proposed direct operations, including growing, pruning, and
DynamicsExplorer to diagnose an LSTM trained to optimizing the internal nodes, and can evaluate the
control a ball-in-maze game. To support quick identi- refined one with various visual representations.
fication of where training failures arise, it visualizes Besides direct model updates, users can also correct
ball trajectories with a trajectory variability plot, as flaws in the results or provide extra knowledge,
well as their clusters using a parallel coordinates plot. allowing the model to be updated implicitly to
4.4 Model steering produce improved results based on human feedback.
Several works have focused on incorporating user
There are two major strategies for model steering: knowledge into topic models to improve their results
refining the model with human knowledge, and [13, 105, 106, 109, 124, 125]. For instance, Yang et
selecting the best model from a model ensemble. al. [125] presented ReVision that allows users to
4.4.1 Model refinement with human knowledge steer hierarchical clustering results by leveraging an
Several visual analytics techniques have been evolutionary Bayesian rose tree clustering algorithm
developed to place users into the loop of the model with constraints. As shown in Fig. 6, the constraints
refinement process, through flexible interaction. and the clustering results are displayed with an

Fig. 6 ReVision, a visual analytics system integrating a constrained hierarchical clustering algorithm with an uncertainty-aware, tree-based
visualization to help users interactively refine hierarchical topic modeling results. Reproduced with permission from Ref. [125], 
c IEEE 2020.
12 J. Yuan, C. Chen, W. Yang, et al.

uncertainty-aware tree-based visualization to guide treats all model output as a large collection and
the steering of the clustering results. Users can refine analyzes the static structure. For dynamic data, in
the constraint hierarchy by dragging. Documents are addition to understanding the analysis results at each
then re-clustered based on the modified constraints. time point, the system focuses on illustrating the
Other human-in-the-loop models have also stimulated evolution of data over time, which is learned by the
the development of visual analytic systems to support analysis model.
such kinds of model refinement. For instance, 5.1 Understanding static data analysis results
Liu et al. [112] proposed MutualRanker using an
We summarize the research on understanding static
uncertainty-based mutual reinforcement graph model
data analysis according to the type of data. Most
to retrieve important blogs, users, and hashtags from
research focuses on textual data analysis, while fewer
microblog data. It shows ranking results, uncertainty,
works study the understanding of other types of data
and its propagation with the help of a composite
analysis.
visualization; users can examine the most uncertain
items in the graph and adjust their ranking scores. 5.1.1 Textual data analysis
The model is incrementally updated by propagating The most widely studied topic is visual text analytics,
adjustments throughout the graph. which tightly integrates interactive visualization
techniques with text mining techniques (e.g.,
4.4.2 Model selection from an ensemble
document clustering, topic models, and word
Another strategy for model steering is to select the best embedding) to help users better understand a large
model from a model ensemble, which is usually found amount of textual data [5].
in clustering [102, 118, 121] and regression models [99, Some early works employed simple visualizations
103, 113, 119]. Clustrophile 2 [102] is a visual analytics to directly convey the results of classical text
system for visual clustering analysis, which guides user mining techniques, such as text summarization,
selection of appropriate input features and clustering categorization, and clustering. For example, Görg
parameters through recommendations based on user- et al. [143] developed a multi-view visualization
selected results. BEAMES [103] was designed for consisting of a list view, a cluster view, a word
multimodel steering and selection in regression tasks. cloud, a grid view, and a document view, to visually
It creates a collection of regression models by varying illustrate analysis results of document summarization,
algorithms and their corresponding hyperparameters, document clustering, sentiment analysis, entity
with further optimization by interactive weighting of identification, and recommendation. By combining
data instances and interactive feature selection and interactive visualization with text mining techniques,
weighting. Users can inspect them and then select a smooth and informative exploration environment
an optimal model according to different aspects of is provided to users.
performance, such as their residual scores and mean Most later research has focused on combining well-
squared errors. designed interactive visualization with state-of-the-
art text mining techniques, such as topic models and
5 Techniques after model building deep learning models, to provide deeper insights into
textual data. To provide an overview of the relevant
Existing visual analytics efforts after model building topics discussed in multiple sources, Liu et al. [159]
aim to help users understand and gain insights first utilized a correlated topic model to extract topic
from model outputs, such as high-dimensional data graphs from multiple text sources. A graph matching
analysis results [5, 283]. As these methods are often algorithm is then developed to match the topic graphs
data-driven, we categorize the corresponding methods from different sources, and a hierarchical clustering
according to the type of data analyzed. The temporal method is employed to generate hierarchies of topic
property of data is critical in visual design. Thus, we graphs. Both the matched topic graph and hierarchies
classify methods as those understanding static data are fed into a hybrid visualization which consists of
analysis results, and those understanding dynamic a radial icicle plot and a density-based node-link
data analysis results. A visual analytics system for diagram (see Fig. 7(a)), to support exploration and
understanding static data analysis results usually analysis of common and distinctive topics discussed
A survey of visual analytics techniques for machine learning 13

in multiple sources. Dou et al. [136] introduced analyzed flow fields through an LDA model by
DemographicVis to analyze different demographic defining pathlines as documents and features as
groups on social media based on the content generated words, respectively. After modeling, the original
by users. An advanced topic model, latent Dirichlet pathlines and extracted topics were projected into a
allocation (LDA) [284], is employed to extract topic two-dimensional space using multidimensional scaling,
features from the corpus. Relationships between the and several previews were generated to render the
demographic information and extracted features are pathlines for important topics. Recently, a visual
explored through a parallel sets visualization [285], analytics tool, SMARTexplore [129], was developed
and different demographic groups are projected onto to help analysts find and understand interesting
the two-dimensional space based on the similarity patterns within and between dimensions, including
of their topics of interest (see Fig. 7(b)). Recently, correlations, clusters, and outliers. To this end,
some deep learning models have also been adopted it tightly couples a table-based visualization with
because of their better performance. For example, pattern matching and subspace analysis.
Berger et al. [128] proposed cite2vec to visualize
the latent themes in a document collection via 5.2 Understanding dynamic data analysis
document usage (e.g., citations). It extended a results
famous word2vec model, the skip-gram model [286],
In addition to understanding the results of static
to generate the embedding for both words and
data analysis, it is also important to investigate
documents by considering the citation information
and analyze how latent themes in data change over
and the textual content together. The words are
time. For example, a system can help politicians
projected into a two-dimensional space using t-SNE
to make timely decisions if it provides an overview
first, and the documents are projected onto the same
of major public opinions on social media and how
space, where both the document-word relationship
they change over time. Most existing works focus on
and document–document relationships are considered
understanding the analysis results of a data corpus
simultaneously.
where each data item is associated with a time
5.1.2 Other data analysis stamp. According to whether the system supports the
In addition to textual data, other types of data have analysis of streaming data, we may further classify
also been studied. For example, Hong et al. [146] existing works on visual dynamic data analysis as

Fig. 7 Examples of static text visualization. (a) TopicPanorama extracts topic graphs from multiple sources and reveals relationships between
them using graph layout. Reproduced with permission from Ref. [159],  c IEEE 2014. (b) DemographicVis measures similarity between different
users after analyzing their posting contents, and reveals their relationships using t-SNE projection. Reproduced with permission from Ref. [136],
c IEEE 2015.
14 J. Yuan, C. Chen, W. Yang, et al.

offline and online. In offline analysis, all data are their dynamic changes over time. Xu et al. [259]
available before analysis, while online analysis tackles leveraged a topic competition model to extract
streaming data that is incoming during the analysis dynamic competition between topics and the effects
process. of opinion leaders on social media. Sun et al. [238]
5.2.1 Offline analysis. extended the competition model to a “coopetition”
(cooperation and competition) model to help
Offline analysis research can be classified according
understand the more complex interactions between
to the analysis task: topic analysis, event analysis,
evolving topics. Wang et al. [246] proposed IdeaFlow,
and trajectory analysis.
a visual analytics system for learning the lead-
Understanding topic evolution in a large text
lag relationships across different social groups over
corpus over time is an important topic, attracting
time. However, these works use a flat structure
much attention. Most existing works adopt a river
to model topics, which hampers their usage in
metaphor to convey changes in the text corpus over
the era of big data for handling large-scale text
time. ThemeRiver [204] is one of the pioneering corpora. Fortunately, there are already initial efforts
works, using the river metaphor to reveal changes in in coupling hierarchical topic models with interactive
the volumes of different themes. To better understand visualization to favor the understanding of the main
the content change of a document corpus, TIARA content in a large text corpus. For example, Cui et
[220, 248] utilizes an LDA model [287] to extract al. [191] extracted a sequence of topic trees using
topics from the corpus and reveal their changes an evolutionary Bayesian rose tree algorithm [289]
over time. However, only observing volumes and and then calculated the tree cut for each tree. These
content change is not enough for complex analysis tree cuts are used to approximate the topic trees and
tasks where users want to explore relationships display them in a river metaphor, which also reveals
between different topics and their changes over time. dynamic relationships between the topics, including
Therefore, later works have focused on understanding topic birth, death, splitting, and merging.
relationships between topics (e.g., topic splitting and Event analysis targets revealing common or
merging) and their evolving patterns over time. For semantically important sequential patterns in ordered
example, Cui et al. [190] first extracted topic splitting sequences of events [149, 202, 222, 226]. To facilitate
and merging patterns from a document collection visual exploration of large scale event sequences and
using an incremental hierarchical Dirichlet process pattern discovery, several visual analytics methods
model [288]. Then a river metaphor with a set have been proposed. For example, Liu et al. [222]
of well-designed glyphs was developed to visually developed a visual analytics method for click stream
illustrate the aforementioned topic relationships and data. Maximal sequential patterns are discovered and

Fig. 8 TextFlow employs a river-based metaphor to show topic birth, death, merging, and splitting. Reproduced with permission from
Ref. [190], 
c IEEE 2011.
A survey of visual analytics techniques for machine learning 15

pruned from the click stream data. The extracted regarded as a document. Parallel coordinates were
patterns and original data are well illustrated at four used to visualize the distribution of streets over topics,
granularities: patterns, segments, sequences, and where each axis represents a topic, and each polyline
events. Guo et al. [202] developed EventThread, represents a street. The evolution of the topics
which uses a tensor-based model to transform the was visualized as topic routes that connect similar
event sequence data into an n-dimensional tensor. topics between adjacent time windows. More recently,
Latent patterns (threads) are extracted with a tensor Zhou et al. [269] treated origin-destination flows
decomposition technique, segmented into stages, and as words and trajectories as paragraphs, respectively.
then clustered. These threads are represented as Therefore, a word2Vec model was used to generate the
segmented linear stripes, and a line map metaphor is vectorized representation for each origin-destination
used to reveal the changes between different stages. flow. t-SNE was then employed to project the
Later, EventThread was extended to overcome the embedding of the flows into two-dimensional space,
limitation of the fixed length of each stage [201]. where analysts can check the distributions of the
The authors proposed an unsupervised stage analysis origin-destination flows and select some for display
algorithm to effectively identify the latent stages on the map. Besides directly analyzing the original
in event sequences. Based on this algorithm, an trajectory data, other papers try to augment the
interactive visualization tool was developed to reveal trajectories with auxiliary information to reduce the
and analyze the evolution patterns across stages. burden on visual exploration. Kruger et al. [212]
Other works focus on understanding movement data clustered destinations with DBScan and then used
(e.g., GPS records) analysis results. Andrienko et Foursquare to provide detailed information about the
al. [174] extracted movement events from trajectories destinations (e.g., shops, university, residence). Based
and then performed spatio-temporal clustering for on the enriched data, frequent patterns were extracted
aggregation. These clusters are visualized using spatio- and displayed in the visualization (see Fig. 9); icons
temporal envelopes to help analysts find potential on the time axis help understand these patterns. Chen
traffic jams in the city. Chu et al. [189] adopted et al. [186] mined trajectories from geo-tagged social
an LDA model for mining latent movement patterns media and displayed keywords extracted from the
in taxi trajectories. The movement of each taxi, text content, helping users explore the semantics of
represented by the traversed street names, was trajectories.

Fig. 9 Kruger et al. enrich trajectory data semantically. Frequent routes and destinations are visualized in the geographic view (top), while the
frequent temporal patterns are mined and displayed in the temporal view (bottom). Reproduced with permission from Ref. [212], c IEEE 2015.
16 J. Yuan, C. Chen, W. Yang, et al.

5.2.2 Online analysis learning has achieved promising results in both

Online analysis is especially necessary for streaming academia and real-world applications, there are still
data, such as text streams. As a pioneering work several long-term research challenges. Here, we
for online analysis of text streams, Thom et al. discuss and highlight major challenges and potential
[240] proposed ScatterBlog to analyze geo-located research opportunities in this area.
tweet streams. The system uses Twitter4J to get 6.1 Opportunities before model building
streaming tweets and extracts location, time, user
ID, and tokenized terms in the tweets. To efficiently 6.1.1 Improving data quality for weakly supervised
analyze a tweet stream, an incremental clustering learning
algorithm was employed to cluster similar tweets. Weakly supervised learning builds models from
Based on the clustering results, spatio-temporal data with quality issues, including inaccurate labels,
anomalies were detected and reported to users in incomplete labels, and inexact labels. Improving
real-time. To reduce user effort for filtering and data quality can boost the performance of weakly
monitoring in ScatterBlogs, Bosch et al. [177] proposed supervised learning models [290]. Most existing
ScatterBlogs2, which enhanced ScatterBlogs with methods focus on inaccurate data (e.g., noisy
machine learning techniques. In particular, an SVM- crowdsourced annotations and label errors) quality
based classifier was built for filtering tweets of interest, issues, and interactive labeling related to incomplete
and an LDA model was employed to generate a topic data (e.g., none or only a few data are labeled)
overview. To efficiently handle high-volume text quality issues. However, fewer efforts are devoted
streams, Liu et al. [219] developed TopicStream to to the better exploitation of unlabeled data related
help users analyze hierarchical topic evolution in high- to incomplete data quality issues as well as inexact
volume text streams. In TopicStream, an evolutionary data (e.g., coarse-grained labels that are not
topic tree was built from text streams, and a tree cut exact as required) quality issues. This paves the
algorithm was developed to reduce visual clutter and way for potential future research. Firstly, the
enable users to focus on topics of interest. Combining potential of visual analytics techniques to address
a river metaphor and a visual sedimentation metaphor, the incompleteness issue is not fully exploited. For
the tool effectively illustrates the overall hierarchical example, improving the quality of unlabeled data
topic evolution as well as how newly arriving textual is critical for semi-supervised learning [290, 291],
documents are gradually aggregated into the existing which is tightly combined with a small amount of
topics over time. Triggered by TopicStream, Wu et labeled data during training to infer the correct
al. [252] developed StreamExplorer, which enables the mapping from the data set to the label set. One
tracking and comparison of a social stream. In typical example is graph-based semi-supervised
particular, an entropy-based event detection method learning [291], which depends on the relationship
was developed to detect events in the social media between labeled and unlabeled data. Automatically
stream. They are further visualized in a multi-level constructed relationships (graphs) are sometimes
visualization, including a glyph-based timeline, a map poor in quality, resulting in model performance
visualization, and interactive lenses. In addition to degradation. A major cause behind these poor-quality
text streams, other types of streaming data are also graphs is that automatic graph construction methods
analyzed. For example, Lee et al. [213] employed a long usually rely on global parameters (e.g., a global k
short-term memory model for road traffic congestion value in the kNN graph construction method), which
forecasting and visualized the results with a Volume- may be locally inappropriate. As a consequence, it
Speed Rivers visualization. Propagation of congestion is necessary to utilize visualization to illustrate how
was also extracted and visualized, helping analysts labels are propagated along graph edges, to facilitate
understand causality within the detected congestion. understanding of how local graph structures affect
model performance. Based on such understanding,
experts can adaptively modify the graph to gradually
6 Research opportunities
create a higher-quality graph.
Although visual analytics research for machine Secondly, although the inexact data quality issue
A survey of visual analytics techniques for machine learning 17

is common in real-world applications [292], it has it in a comprehensible manner.

received little attention from the field of visual Moreover, redundancy exists in extracted deep
analytics. This issue refers to the situation where features [297]. Removing redundant features can
labels are inexact, e.g., coarse-grained labels, such lead to several benefits, such as reducing storage
as arise in computed tomography (CT) scans. The requirements and improving generalization [278].
labels of CT scans usually come from corresponding However, without a clear understanding of the exact
diagnosis reports that describe whether patients have meaning of features, it is hard to judge whether a
certain medical problems (e.g., a tumor). For a feature is redundant. Thus, an interesting future
CT scan with tumors, we only know that one or topic is to develop a visual analytics method to convey
more slices in the scan contain tumors. However, we feature redundancy in a comprehensible way, to allow
do not know which slices contain tumors as well as experts to explore it, and remove redundant features.
the exact tumor locations in these slices. Although
6.2 Opportunities during model building
various machine learning methods [293, 294] have
been proposed to learn from such coarse-grained 6.2.1 Online training diagnosis
labels, they may lead to poor performance [290] Existing visual analytics tools for model diagnosis
due to the lack of exact information. Fine-grained mostly work offline: the data for diagnosis is collected
validation is still required to improve data quality. after the training process is finished. They have
To this end, one potential solution is to combine shown their capability for revealing the root causes
interactive visualization with learning algorithms to of failed training processes. However, as modern
better illustrate the root cause of bad performance machine learning models become more and more
by examining the overall data distribution and complex, training processes can last for days or
the wrongly predictions, to develop an interactive even weeks. Offline diagnosis severely restricts the
verification process for providing more finely-grained ability of visual analytics to assist in training. Thus,
labels while minimizing expert effort. there is a significant need to develop visual analytics
6.1.2 Explainable feature engineering tools for online diagnosis of the training process so
Most existing work for improving feature quality that model developers can identify anomalies and
focuses on tabular or textual data from traditional promptly make corresponding adjustments to the
analysis models. The features of these data are process. This can save much time in the trial-and-
naturally interpretable, which makes the feature error model building process. The key challenge for
engineering process simple. However, features online diagnosis is to detect anomalies in the training
extracted by deep neural networks perform better process in a timely manner. While it remains a
than handcrafted ones [295, 296]. These deep features difficult task to develop algorithms for automatically
are hard to interpret due to the black box nature of and accurately detecting anomalies in real-time,
deep neural networks, which brings several challenges interactive visualization promises a way to locate
for feature engineering. potential errors in the training process. Differing
Firstly, the extracted features are obtained in a from offline diagnosis, the data of the training process
data-driven process, which may poorly represent the will be continuously fed into the online analysis tool.
original images/videos when the datasets are biased. Thus, progressive visualization techniques are needed
For example, given a dataset with only dark dogs and to produce meaningful visualization results of partial
light cats, the extracted features may emphasize color streaming data. These techniques can help experts
and ignore other discriminating concepts, like shapes monitor online model training processes and identify
of faces and ears. Without a clear understanding of possible issues rapidly.
these biased features, it is hard to correct them in 6.2.2 Interactive model refinement
a suitable way. Thus, an interesting topic for future Recent works have explored the utilization of
work is to utilize interactive visualization to disclose uncertainty to facilitate interactive model refinement
why the features are biased. The key challenge [106, 112, 124, 125]. There are many methods to
here is how to measure the information preserved or assign uncertainty scores to model outputs (e.g.,
discarded by the extracted features and to visualize based on confidence scores produced by classifiers),
18 J. Yuan, C. Chen, W. Yang, et al.

and visual hints can be used to guide users to examine such multi-modal learning models. Many machine
model outputs with high uncertainty. Models learning models have been proposed to learn joint
uncertainty will be recomputed after user refinement, representations of multi-modal data, including natural
and users can perform iteratively until they are language, visual signals, and vocal signals [298, 299].
satisfied with the results. Furthermore, additional Accordingly, an interesting future direction is how
information can also be leveraged to provide users to effectively visualize learned joint representations
with more intelligent guidance to facilitate a fast and of multi-modal data in an all-in-one manner, to
accurate model refinement process. However, the facilitate the understanding of the data and their
room for improving interactive model refinement is relationships. Various classic multi-modal tasks can be
still largely unexplored by researchers. One possible employed to enhance natural interactions in the field
direction is that since the refinement process usually of visual analytics. For example, in the vision-and-
requires several iterations, guidance in later iterations language scenario, the visual grounding task (identify
can be learned from users’ previous interactions. the corresponding image area given the description)
For example, in a clustering application, users may can be used to provide a natural interface to support
define some must-link or cannot-link constraints on natural-language-based image retrieval in a visual
some instance pairs, and such constraints can be environment.
used to instruct a model to split or merge some
6.3.2 Analyzing concept drift
clusters in the intermediate result. In addition, prior
knowledge can be used to predict where refinements In real-world applications, it is often assumed that
are needed. For example, model outputs may the mapping from input data to output values (e.g.,
conflict with certain public or domain knowledge, prediction label) is static. However, as data continues
especially for unsupervised models (e.g., nonlinear to arrive, the mapping between the input data and
matrix factorization and latent Dirichlet allocation for output values may change in unexpected ways [300].
topic modeling), which should be considered in the In such a situation, a model trained on historical
refinement process. Therefore, such a knowledge- data may no longer work properly on new data. This
based strategy focuses on revealing unreasonable usually causes noticeable performance degradation
results produced by the models, allowing users to when the application data does not match the training
refine the models by adding constraints to them. data. Such a non-stationary learning problem over
time is known as concept drift. As more and
6.3 Opportunities after model building
more machine learning applications directly consume
6.3.1 Understanding multi-modal data streaming data, it is important to detect and analyze
Existing works on content analysis have achieved concept drift and minimize the resulting performance
great success in understanding single-modal data, such degradation [301, 302]. In the field of machine
as texts, images, and videos. However, real-world learning, three main research topics, have been
applications often contain multi-modal data, which studied: drift detection, drift understanding, and
combines several different content forms, such as text, drift adaptation. Machine learning researchers have
audio, and images. For example, a physician diagnoses proposed many automatic algorithms to detect and
a patient after considering multiple kinds of data, adapt to concept drift. Although these algorithms
such as the medical record (text), laboratory reports can improve the adaptability of learning models in an
(tables), and CT scans (images). When analyzing uncertain environment, they only provide a numerical
such multi-modal data, in-depth relationships between value to measure the degree of drift at a given time.
different modals cannot be well captured by simply This makes it hard to understand why and where drift
combining knowledge learned from single-modal occurs. If the adaptation algorithms fail to improve
models. It is more promising to employ multi- the model performance, the black-box behavior of
modal machine learning techniques and leverage their the adaptation models makes it difficult to diagnose
capability to disclose insights across different forms the root cause of performance degradation. As a
of data. To this end, a more powerful visual analytics result, model developers need tools that intuitively
system is crucial for understanding the output of illustrate how data distributions have changed over
A survey of visual analytics techniques for machine learning 19

time, which samples cause drift, and how the training References
samples and models can be adjusted to overcoming
[1] Liu, S. X.; Wang, X. T.; Liu, M. C.; Zhu, J. Towards
such drift. This requirement naturally leads to a
better analysis of machine learning models: A visual
visual analytics paradigm where the expert interacts
analytics perspective. Visual Informatics Vol. 1, No.
and collaborates in concept drift detection and
1, 48–56, 2017.
adaptation algorithms by putting the human in the
[2] Choo, J.; Liu, S. X. Visual analytics for explainable
loop. The major challenges here are how to (i) visually
deep learning. IEEE Computer Graphics and
represent the evolving patterns of streaming data over
Applications Vol. 38, No. 4, 84–92, 2018.
time and effectively compare data distributions at
[3] Hohman, F.; Kahng, M.; Pienta, R.; Chau, D. H.
different points in time, and (ii) tightly integrate
Visual analytics in deep learning: An interrogative
such streaming data visualization with drift detection
survey for the next frontiers. IEEE Transactions on
and adaptation algorithms to form an interactive and
Visualization and Computer Graphics Vol. 25, No. 8,
progressive analysis environment with the human in
2674–2693, 2019.
the loop.
[4] Zeiler, M. D.; Fergus, R. Visualizing and understanding
convolutional networks. In: Computer Vision–ECCV
7 Conclusions 2014. Lecture Notes in Computer Science, Vol. 8689.
Fleet, D.; Pajdla, T.; Schiele, B.; Tuytelaars, T. Eds.
This paper has comprehensively reviewed recent Springer Cham, 818–833, 2014.
progress and developments in visual analytics
[5] Liu, S. X.; Wang, X. T.; Collins, C.; Dou, W. W.;
techniques for machine learning. These techniques Ouyang, F.; El-Assady, M.; Jiang, L.; Keim, D.
are classified into three groups by the corresponding A. Bridging text visualization and mining: A task-
analysis stage: techniques before, during, and after driven survey. IEEE Transactions on Visualization
model building. Each category is detailed by typical and Computer Graphics Vol. 25, No. 7, 2482–2504,
analysis tasks, and each task is illustrated by a set of 2019.
representative works. By comprehensively analyzing [6] Lu, Y. F.; Garcia, R.; Hansen, B.; Gleicher, M.;
existing visual analytics research for machine learning, Maciejewski, R. The state-of-the-art in predictive
we also suggest six directions for future machine- visual analytics. Computer Graphics Forum Vol. 36,
learning-related visual analytics research, including No. 3, 539–562, 2017.
improving data quality for weakly supervised learning [7] Sacha, D.; Kraus, M.; Keim, D. A.; Chen, M. VIS4ML:
and explainable feature engineering before model An ontology for visual analytics assisted machine
building, online training diagnosis and intelligent learning. IEEE Transactions on Visualization and
model refinement during model building, and multi- Computer Graphics Vol. 25, No. 1, 385–395, 2019.
modal data understanding and concept drift analysis [8] Selvaraju, R. R.; Cogswell, M.; Das, A.; Vedantam, R.;
after model building. We hope this survey has Parikh, D.; Batra, D. Grad-CAM: Visual explanations
provided an overview of visual analytics research from deep networks via gradient-based localization.
for machine learning, facilitating understanding of International Journal of Computer Vision Vol. 128,
state-of-the-art knowledge in this area, and shedding 336–359, 2020.

light on future research. [9] Zhang, Q. S.; Zhu, S. C. Visual interpretability for
deep learning: A survey. Frontiers of Information
Technology & Electronic Engineering Vol. 19, No. 1,
Acknowledgements
27–39, 2018.
This research is supported by the National Key [10] Kandel, S.; Parikh, R.; Paepcke, A.; Hellerstein, J.
R&D Program of China (Nos. 2018YFB1004300 M.; Heer, J. Profiler: Integrated statistical analysis
and 2019YFB1405703), the National Natural Science and visualization for data quality assessment. In:
Foundation of China (Nos. 61761136020, 61672307, Proceedings of the International Working Conference
61672308, and 61936002), TC190A4DA/3, and in on Advanced Visual Interfaces, 547–554, 2012.
part by Tsinghua–Kuaishou Institute of Future Media [11] Marsland, S. Machine Learning: an Algorithmic
Data. Perspective. Chapman and Hall/CRC, 2015.
20 J. Yuan, C. Chen, W. Yang, et al.

[12] Hung, N. Q. V.; Thang, D. C.; Weidlich, M.; Aberer, [23] Gschwandtner, T.; Erhart, O. Know your enemy:
K. Minimizing efforts in validating crowd answers. Identifying quality problems of time series data.
In: Proceedings of the ACM SIGMOD International In: Proceedings of the IEEE Pacific Visualization
Conference on Management of Data, 999–1014, 2015. Symposium, 205–214, 2018.
[13] Choo, J.; Lee, C.; Reddy, C. K.; Park, H. UTOPIAN: [24] Halter, G.; Ballester-Ripoll, R.; Flueckiger, B.;
User-driven topic modeling based on interactive Pajarola, R. VIAN: A visual annotation tool for film
nonnegative matrix factorization. IEEE Transactions analysis. Computer Graphics Forum Vol. 38, No. 3,
on Visualization and Computer Graphics Vol. 19, No. 119–129, 2019.
12, 1992–2001, 2013. [25] Heimerl, F.; Koch, S.; Bosch, H.; Ertl, T. Visual
[14] Alemzadeh, S.; Niemann, U.; Ittermann, T.; Völzke, classifier training for text document retrieval. IEEE
H.; Schneider, D.; Spiliopoulou, M.; Bühler, K.; Preim, Transactions on Visualization and Computer Graphics
B. Visual analysis of missing values in longitudinal Vol. 18, No. 12, 2839–2848, 2012.
cohort study data. Computer Graphics Forum Vol. 39, [26] Höferlin, B.; Netzel, R.; Höferlin, M.; Weiskopf,
No. 1, 63–75, 2020. D.; Heidemann, G. Inter-active learning of ad-hoc
[15] Arbesser, C.; Spechtenhauser, F.; Muhlbacher, T.; classifiers for video visual analytics. In: Proceedings
Piringer, H. Visplause: Visual data quality assessment of the Conference on Visual Analytics Science and
of many time series using plausibility checks. IEEE Technology, 23–32, 2012.
Transactions on Visualization and Computer Graphics [27] Soares Junior, A.; Renso, C.; Matwin, S. ANALYTiC:
Vol. 23, No. 1, 641–650, 2017. An active learning system for trajectory classification.
[16] Bäuerle, A.; Neumann, H.; Ropinski, T. Classifier- IEEE Computer Graphics and Applications Vol. 37,
guided visual correction of noisy labels for image No. 5, 28–39, 2017.
classification tasks. Computer Graphics Forum Vol. [28] Khayat, M.; Karimzadeh, M.; Zhao, J. Q.; Ebert, D. S.
39, No. 3, 195–205, 2020. VASSL: A visual analytics toolkit for social spambot
[17] Bernard, J.; Hutter, M.; Reinemuth, H.; Pfeifer, labeling. IEEE Transactions on Visualization and
H.; Bors, C.; Kohlhammer, J. Visual-interactive pre- Computer Graphics Vol. 26, No. 1, 874–883, 2020.
processing of multivariate time series data. Computer [29] Kurzhals, K.; Hlawatsch, M.; Seeger, C.; Weiskopf,
Graphics Forum Vol. 38, No. 3, 401–412, 2019. D. Visual analytics for mobile eye tracking. IEEE
[18] Bernard, J.; Hutter, M.; Zeppelzauer, M.; Fellner, D.; Transactions on Visualization and Computer Graphics
Sedlmair, M. Comparing visual-interactive labeling Vol. 23, No. 1, 301–310, 2017.
with active learning: An experimental study. IEEE [30] Lekschas, F.; Peterson, B.; Haehn, D.; Ma, E.;
Transactions on Visualization and Computer Graphics Gehlenborg, N.; Pfister, H. 2019. PEAX: interactive
Vol. 24, No. 1, 298–308, 2018. visual pattern search in sequential data using
[19] Bernard, J.; Zeppelzauer, M.; Lehmann, M.; Müller, unsupervised deep representation learning. bioRxiv
M.; Sedlmair, M. Towards user-centered active 597518, https://doi.org/10.1101/597518, 2020.
learning algorithms. Computer Graphics Forum Vol. [31] Liu, S. X.; Chen, C. J.; Lu, Y. F.; Ouyang, F.
37, No. 3, 121–132, 2018. X.; Wang, B. An interactive method to improve
[20] Bors, C.; Gschwandtner, T.; Miksch, S. Capturing crowdsourced annotations. IEEE Transactions on
and visualizing provenance from data wrangling. IEEE Visualization and Computer Graphics Vol. 25, No.
Computer Graphics and Applications Vol. 39, No. 6, 1, 235–245, 2019.
61–75, 2019. [32] Moehrmann, J.; Bernstein, S.; Schlegel, T.; Werner,
[21] Chen, C. J.; Yuan, J.; Lu, Y. F.; Liu, Y.; Su, H.; G.; Heidemann, G. Improving the usability of
Yuan, S. T.; Liu, S. X. OoDAnalyzer: Interactive hierarchical representations for interactively labeling
analysis of out-of-distribution samples. IEEE large image data sets. In: Human-Computer
Transactions on Visualization and Computer Graphics Interaction. Design and Development Approaches.
doi: 10.1109/TVCG.2020.2973258, 2020. Lecture Notes in Computer Science, Vol. 6761. Jacko,
[22] Dextras-Romagnino, K.; Munzner, T. Segmen++ J. A. Ed. Springer Berlin, 618–627, 2011.
tifier: Interactive refinement of clickstream data. [33] Paiva, J. G. S.; Schwartz, W. R.; Pedrini, H.;
Computer Graphics Forum Vol. 38, No. 3, 623–634, Minghim, R. An approach to supporting incremental
2019. visual data classification. IEEE Transactions on
A survey of visual analytics techniques for machine learning 21

Visualization and Computer Graphics Vol. 21, No. In: Proceedings of the IEEE Conference on Visual
1, 4–17, 2015. Analytics Science and Technology, 57–68, 2019.
[34] Park, J. H.; Nadeem, S.; Boorboor, S.; Marino, [44] Ingram, S.; Munzner, T.; Irvine, V.; Tory, M.;
J.; Kaufman, A. E. CMed: Crowd analytics Bergner, S.; Möller, T. DimStiller: Workflows for
for medical imaging data. IEEE Transactions on dimensional analysis and reduction. In: Proceedings
Visualization and Computer Graphics doi: 10.1109/ of the IEEE Conference on Visual Analytics Science
TVCG.2019.2953026, 2019. and Technology, 3–10, 2010.
[35] Park, J. H.; Nadeem, S.; Mirhosseini, S.; Kaufman, [45] Krause, J.; Perer, A.; Bertini, E. INFUSE: Interactive
A. C2A: Crowd consensus analytics for virtual feature selection for predictive modeling of high
colonoscopy. In: Proceedings of the IEEE Conference dimensional data. IEEE Transactions on Visualization
on Visual Analytics Science and Technology, 21–30, and Computer Graphics Vol. 20, No. 12, 1614–1623,
2016. 2014.
[36] De Rooij, O.; van Wijk, J. J.; Worring, M. MediaTable: [46] May, T.; Bannach, A.; Davey, J.; Ruppert, T.;
Interactive categorization of multimedia collections. Kohlhammer, J. Guiding feature subset selection with
IEEE Computer Graphics and Applications Vol. 30, an interactive visualization. In: Proceedings of the
No. 5, 42–51, 2010. IEEE Conference on Visual Analytics Science and
[37] Snyder, L. S.; Lin, Y. S.; Karimzadeh, M.; Goldwasser, Technology, 111–120, 2011.
D.; Ebert, D. S. Interactive learning for identifying [47] Muhlbacher, T.; Piringer, H. A partition-based
relevant tweets to support real-time situational framework for building and validating regression
awareness. IEEE Transactions on Visualization and models. IEEE Transactions on Visualization and
Computer Graphics Vol. 26, No. 1, 558–568, 2020. Computer Graphics Vol. 19, No. 12, 1962–1971, 2013.
[38] Sperrle, F.; Sevastjanova, R.; Kehlbeck, R.; El- [48] Seo, J.; Shneiderman, B. A rank-by-feature framework
Assady, M. VIANA: Visual interactive annotation for interactive exploration of multidimensional data.
of argumentation. In: Proceedings of the Conference Information Visualization Vol. 4, No. 2, 96–113, 2005.
on Visual Analytics Science and Technology, 11–22, [49] Tam, G. K. L.; Fang, H.; Aubrey, A. J.; Grant, P. W.;
2019. Rosin, P. L.; Marshall, D.; Chen, M. Visualization of
[39] Stein, M.; Janetzko, H.; Breitkreutz, T.; Seebacher, time-series data in parameter space for understanding
D.; Schreck, T.; Grossniklaus, M.; Couzin, I. D.; facial dynamics. Computer Graphics Forum Vol. 30,
Keim, D. A. Director’s cut: Analysis and annotation No. 3, 901–910, 2011.
of soccer matches. IEEE Computer Graphics and [50] Broeksema, B.; Baudel, T.; Telea, A.; Crisafulli, P.
Applications Vol. 36, No. 5, 50–60, 2016. Decision exploration lab: A visual analytics solution
[40] Wang, X. M.; Chen, W.; Chou, J. K.; Bryan, C.; for decision management. IEEE Transactions on
Guan, H. H.; Chen, W. L.; Pan, R.; Ma, K.-L. Visualization and Computer Graphics Vol. 19, No.
GraphProtector: A visual interface for employing 12, 1972–1981, 2013.
and assessing multiple privacy preserving graph [51] Cashman, D.; Patterson, G.; Mosca, A.; Watts,
algorithms. IEEE Transactions on Visualization and N.; Robinson, S.; Chang, R. RNNbow: Visualizing
Computer Graphics Vol. 25, No. 1, 193–203, 2019. learning via backpropagation gradients in RNNs.
[41] Wang, X. M.; Chou, J. K.; Chen, W.; Guan, H. IEEE Computer Graphics and Applications Vol. 38,
H.; Chen, W. L.; Lao, T. Y.; Ma, K.-L. A utility- No. 6, 39–50, 2018.
aware visual approach for anonymizing multi-attribute [52] Collaris, D.; van Wijk, J. J. ExplainExplore:
tabular data. IEEE Transactions on Visualization and Visual exploration of machine learning explanations.
Computer Graphics Vol. 24, No. 1, 351–360, 2018. In: Proceedings of the IEEE Pacific Visualization
[42] Willett, W.; Ginosar, S.; Steinitz, A.; Hartmann, B.; Symposium, 26–35, 2020.
Agrawala, M. Identifying redundancy and exposing [53] Eichner, C.; Schumann, H.; Tominski, C. Making
provenance in crowdsourced data analysis. IEEE parameter dependencies of time-series segmentation
Transactions on Visualization and Computer Graphics visually understandable. Computer Graphics Forum
Vol. 19, No. 12, 2198–2206, 2013. Vol. 39, No. 1, 607–622, 2020.
[43] Xiang, S.; Ye, X.; Xia, J.; Wu, J.; Chen, Y.; Liu, [54] Ferreira, N.; Lins, L.; Fink, D.; Kelling, S.; Wood,
S. Interactive correction of mislabeled training data. C.; Freire, J.; Silva, C. BirdVis: Visualizing and
22 J. Yuan, C. Chen, W. Yang, et al.

understanding bird populations. IEEE Transactions [65] Ming, Y.; Cao, S.; Zhang, R.; Li, Z.; Chen, Y.;
on Visualization and Computer Graphics Vol. 17, No. Song, Y.; Qu, H. Understanding hidden memories
12, 2374–2383, 2011. of recurrent neural networks. In: Proceedings of the
[55] Fröhler, B.; Möller, T.; Heinzl, C. GEMSe: IEEE Conference on Visual Analytics Science and
Visualization-guided exploration of multi-channel Technology, 13–24, 2017.
segmentation algorithms. Computer Graphics Forum [66] Ming, Y.; Qu, H. M.; Bertini, E. RuleMatrix:
Vol. 35, No. 3, 191–200, 2016. Visualizing and understanding classifiers with rules.
[56] Hohman, F.; Park, H.; Robinson, C.; Polo Chau, D. IEEE Transactions on Visualization and Computer
H. Summit: Scaling deep learning interpretability by Graphics Vol. 25, No. 1, 342–352, 2019.
visualizing activation and attribution summarizations. [67] Murugesan, S.; Malik, S.; Du, F.; Koh, E.; Lai, T. M.
IEEE Transactions on Visualization and Computer DeepCompare: Visual and interactive comparison of
Graphics Vol. 26, No. 1, 1096–1106, 2020. deep learning model performance. IEEE Computer
[57] Jaunet, T.; Vuillemot, R.; Wolf, C. DRLViz: Graphics and Applications Vol. 39, No. 5, 47–59, 2019.
Understanding decisions and memory in deep [68] Nie, S.; Healey, C.; Padia, K.; Leeman-Munk, S.;
reinforcement learning. Computer Graphics Forum Benson, J.; Caira, D.; Sethi, S.; Devarajan, R.
Vol. 39, No. 3, 49–61, 2020. Visualizing deep neural networks for text analytics.
[58] Jean, C. S.; Ware, C.; Gamble, R. Dynamic change In: Proceedings of the IEEE Pacific Visualization
arcs to explore model forecasts. Computer Graphics Symposium, 180–189, 2018.
Forum Vol. 35, No. 3, 311–320, 2016.
[69] Rauber, P. E.; Fadel, S. G.; Falcao, A. X.; Telea, A.
[59] Kahng, M.; Andrews, P. Y.; Kalro, A.; Chau, D.
C. Visualizing the hidden activity of artificial neural
H. ActiVis: Visual exploration of industry-scale
networks. IEEE Transactions on Visualization and
deep neural network models. IEEE Transactions on
Computer Graphics Vol. 23, No. 1, 101–110, 2017.
Visualization and Computer Graphics Vol. 24, No. 1,
[70] Rohlig, M.; Luboschik, M.; Kruger, F.; Kirste, T.;
88–97, 2018.
Schumann, H.; Bogl, M.; Alsallakh, B.; Miksch. S.
[60] Kahng, M.; Thorat, N.; Chau, D. H. P.; Viegas, F. B.;
Supporting activity recognition by visual analytics.
Wattenberg, M. GAN lab: Understanding complex
In: Proceedings of the IEEE Conference on Visual
deep generative models using interactive visual
Analytics Science and Technology, 41–48, 2015.
experimentation. IEEE Transactions on Visualization
[71] Scheepens, R.; Michels, S.; van de Wetering, H.; van
and Computer Graphics Vol. 25, No. 1, 310–320, 2019.
Wijk, J. J. Rationale visualization for safety and
[61] Kwon, B. C.; Anand, V.; Severson, K. A.; Ghosh,
security. Computer Graphics Forum Vol. 34, No. 3,
S.; Sun, Z. N.; Frohnert, B. I.; Lundgren, M.; Ng,
191–200, 2015.
K. DPVis: Visual analytics with hidden Markov
models for disease progression pathways. IEEE [72] Shen, Q.; Wu, Y.; Jiang, Y.; Zeng, W.; LAU, A.
Transactions on Visualization and Computer Graphics K. H.; Vianova, A.; Qu, H. Visual interpretation of
doi: 10.1109/TVCG.2020.2985689, 2020. recurrent neural network on multi-dimensional time-
[62] Liu, M. C.; Shi, J. X.; Li, Z.; Li, C. X.; Zhu, J.; Liu, series forecast. In: Proceedings of the IEEE Pacific
S. X. Towards better analysis of deep convolutional Visualization Symposium, 61–70, 2020.
neural networks. IEEE Transactions on Visualization [73] Strobelt, H.; Gehrmann, S.; Pfister, H.; Rush, A.
and Computer Graphics Vol. 23, No. 1, 91–100, 2017. M. LSTMVis: A tool for visual analysis of hidden
[63] Liu, S. S.; Li, Z. M.; Li, T.; Srikumar, V.; Pascucci, state dynamics in recurrent neural networks. IEEE
V.; Bremer, P. T. NLIZE: A perturbation-driven Transactions on Visualization and Computer Graphics
visual interrogation tool for analyzing and interpreting Vol. 24, No. 1, 667–676, 2018.
natural language inference models. IEEE Transactions [74] Wang, J. P.; Gou, L.; Yang, H.; Shen, H. W. GANViz:
on Visualization and Computer Graphics Vol. 25, No. A visual analytics approach to understand the
1, 651–660, 2019. adversarial game. IEEE Transactions on Visualization
[64] Migut, M.; van Gemert, J.; Worring, M. Interactive and Computer Graphics Vol. 24, No. 6, 1905–1917,
decision making using dissimilarity to visually 2018.
represented prototypes. In: Proceedings of the [75] Wang, J. P.; Gou, L.; Zhang, W.; Yang, H.; Shen, H.
IEEE Conference on Visual Analytics Science and W. DeepVID: Deep visual interpretation and diagnosis
Technology, 141–149, 2011. for image classifiers via knowledge distillation. IEEE
A survey of visual analytics techniques for machine learning 23

Transactions on Visualization and Computer Graphics [85] Diehl, A.; Pelorosso, L.; Delrieux, C.; Matković, K.;
Vol. 25, No. 6, 2168–2180, 2019. Ruiz, J.; Gröller, M. E.; Bruckner, S. Albero: A
[76] Wang, J.; Zhang, W.; Yang, H. SCANViz: visual analytics approach for probabilistic weather
Interpreting the symbol-concept association captured forecasting. Computer Graphics Forum Vol. 36, No.
by deep neural networks through visual analytics. 7, 135–144, 2017.
In: Proceedings of the IEEE Pacific Visualization [86] Gleicher, M.; Barve, A.; Yu, X. Y.; Heimerl, F. Boxer:
Symposium, 51–60, 2020. Interactive comparison of classifier results. Computer
[77] Wongsuphasawat, K.; Smilkov, D.; Wexler, J.; Wilson, Graphics Forum Vol. 39, No. 3, 181–193, 2020.
J.; Mane, D.; Fritz, D.; Krishnan, D.; Viegas, F. B.; [87] He, W.; Lee, T.-Y.; van Baar, J.; Wittenburg, K.;
Wattenberg, M. Visualizing dataflow graphs of deep Shen, H.-W. DynamicsExplorer: Visual analytics for
learning models in TensorFlow. IEEE Transactions robot control tasks involving dynamics and LSTM-
on Visualization and Computer Graphics Vol. 24, No. based control policies. In: Proceedings of the IEEE
1, 1–12, 2018. Pacific Visualization Symposium, 36–45, 2020.
[78] Zhang, C.; Yang, J.; Zhan, F. B.; Gong, X.; [88] Krause, J.; Dasgupta, A.; Swartz, J.;
Brender, J. D.; Langlois, P. H.; Barlowe, S.; Zhao, Aphinyanaphongs, Y.; Bertini, E. A workow
Y. A visual analytics approach to high-dimensional for visual diagnostics of binary classifiers using
logistic regression modeling and its application to instance-level explanations. In: Proceedings of the
an environmental health study. In: Proceedings of IEEE Conference on Visual Analytics Science and
the IEEE Pacific Visualization Symposium, 136–143, Technology, 162–172, 2017.
2016. [89] Liu, M. C.; Shi, J. X.; Cao, K. L.; Zhu, J.; Liu, S. X.
[79] Zhao, X.; Wu, Y. H.; Lee, D. L.; Cui, W. W. iForest: Analyzing the training processes of deep generative
Interpreting random forests via visual analytics. IEEE models. IEEE Transactions on Visualization and
Transactions on Visualization and Computer Graphics Computer Graphics Vol. 24, No. 1, 77–87, 2018.
Vol. 25, No. 1, 407–416, 2019. [90] Liu, S. X.; Xiao, J. N.; Liu, J. L.; Wang, X. T.; Wu,
[80] Ahn, Y.; Lin, Y. R. FairSight: Visual analytics for J.; Zhu, J. Visual diagnosis of tree boosting methods.
fairness in decision making. IEEE Transactions on IEEE Transactions on Visualization and Computer
Visualization and Computer Graphics Vol. 26, No. 1, Graphics Vol. 24, No. 1, 163–173, 2018.
1086–1095, 2019. [91] Ma, Y. X.; Xie, T. K.; Li, J. D.; Maciejewski,
[81] Alsallakh, B.; Hanbury, A.; Hauser, H.; Miksch, R. Explaining vulnerabilities to adversarial machine
S.; Rauber, A. Visual methods for analyzing learning through visual analytics. IEEE Transactions
probabilistic classification data. IEEE Transactions on Visualization and Computer Graphics Vol. 26, No.
on Visualization and Computer Graphics Vol. 20, No. 1, 1075–1085, 2020.
12, 1703–1712, 2014. [92] Pezzotti, N.; Hollt, T.; van Gemert, J.; Lelieveldt,
[82] Bilal, A.; Jourabloo, A.; Ye, M.; Liu, X. M.; Ren, L. B. P. F.; Eisemann, E.; Vilanova, A. DeepEyes:
2018. Do convolutional neural networks learn class Progressive visual analytics for designing deep neural
hierarchy? IEEE Transactions on Visualization and networks. IEEE Transactions on Visualization and
Computer Graphics Vol. 24, No. 1, 152–162, 2018. Computer Graphics Vol. 24, No. 1, 98–108, 2018.
[83] Cabrera, A. A.; Epperson, W.; Hohman, F.; Kahng, [93] Ren, D. H.; Amershi, S.; Lee, B.; Suh, J.; Williams,
M.; Morgenstern, J.; Chau, D. H.; FAIRVIS: Visual J. D. Squares: Supporting interactive performance
analytics for discovering intersectional bias in machine analysis for multiclass classifiers. IEEE Transactions
learning. In: Proceedings of the IEEE Conference on on Visualization and Computer Graphics Vol. 23, No.
Visual Analytics Science and Technology, 46–56, 2019. 1, 61–70, 2017.
[84] Cao, K. L.; Liu, M. C.; Su, H.; Wu, J.; Zhu, [94] Spinner, T.; Schlegel, U.; Schafer, H.; El-Assady,
J.; Liu, S. X. Analyzing the noise robustness M. explAIner: A visual analytics framework for
of deep neural networks. IEEE Transactions interactive and explainable machine learning. IEEE
on Visualization and Computer Graphics doi: Transactions on Visualization and Computer Graphics
10.1109/TVCG.2020.2969185, 2020. Vol. 26, No. 1, 1064–1074, 2020.
24 J. Yuan, C. Chen, W. Yang, et al.

[95] Strobelt, H.; Gehrmann, S.; Behrisch, M.; Perer, [105] Dou, W. W.; Yu, L.; Wang, X. Y.; Ma, Z. Q.; Ribarsky,
A.; Pfister, H.; Rush, A. M. Seq2seq-Vis: A W. HierarchicalTopics: Visually exploring large text
visual debugging tool for sequence-to-sequence models. collections using topic hierarchies. IEEE Transactions
IEEE Transactions on Visualization and Computer on Visualization and Computer Graphics Vol. 19, No.
Graphics Vol. 25, No. 1, 353–363, 2019. 12, 2002–2011, 2013.
[96] Wang, J. P.; Gou, L.; Shen, H. W.; Yang, H. DQNViz: [106] El-Assady, M.; Kehlbeck, R.; Collins, C.; Keim, D.;
A visual analytics approach to understand deep Q- Deussen, O. Semantic concept spaces: Guided topic
networks. IEEE Transactions on Visualization and model refinement using word-embedding projections.
Computer Graphics Vol. 25, No. 1, 288–298, 2019. IEEE Transactions on Visualization and Computer
[97] Wexler, J.; Pushkarna, M.; Bolukbasi, T.; Graphics Vol. 26, No. 1, 1001–1011, 2020.
Wattenberg, M.; Viegas, F.; Wilson, J. The what-if [107] El-Assady, M.; Sevastjanova, R.; Sperrle, F.; Keim,
tool: Interactive probing of machine learning models. D.; Collins, C. Progressive learning of topic modeling
IEEE Transactions on Visualization and Computer parameters: A visual analytics framework. IEEE
Graphics Vol. 26, No. 1, 56–65, 2019. Transactions on Visualization and Computer Graphics
[98] Zhang, J. W.; Wang, Y.; Molino, P.; Li, L. Z.; Vol. 24, No. 1, 382–391, 2018.
Ebert, D. S. Manifold: A model-agnostic framework [108] El-Assady, M.; Sperrle, F.; Deussen, O.; Keim,
for interpretation and diagnosis of machine learning D.; Collins, C. Visual analytics for topic model
models. IEEE Transactions on Visualization and optimization based on user-steerable speculative
Computer Graphics Vol. 25, No. 1, 364–373, 2019. execution. IEEE Transactions on Visualization and
[99] Bogl, M.; Aigner, W.; Filzmoser, P.; Lammarsch, Computer Graphics Vol. 25, No. 1, 374–384, 2019.
T.; Miksch, S.; Rind, A. Visual analytics for model [109] Kim, H.; Drake, B.; Endert, A.; Park, H.
selection in time series analysis. IEEE Transactions ArchiText: Interactive hierarchical topic modeling.
on Visualization and Computer Graphics Vol. 19, No. IEEE Transactions on Visualization and Computer
12, 2237–2246, 2013. Graphics doi: 10.1109/TVCG.2020.2981456, 2020.
[100] Cashman, D.; Perer, A.; Chang, R.; Strobelt, H. [110] Kwon, B. C.; Choi, M. J.; Kim, J. T.; Choi, E.; Kim,
Ablate, variate, and contemplate: Visual analytics for Y. B.; Kwon, S.; Sun, J.; Choo, J. RetainVis: Visual
discovering neural architectures. IEEE Transactions analytics with interpretable and interactive recurrent
on Visualization and Computer Graphics Vol. 26, No. neural networks on electronic medical records. IEEE
1, 863–873, 2020. Transactions on Visualization and Computer Graphics
[101] Cavallo, M.; Demiralp, Ç. Track xplorer: A system Vol. 25, No. 1, 299–309, 2019.
for visual analysis of sensor-based motor activity [111] Lee, H.; Kihm, J.; Choo, J.; Stasko, J.; Park,
predictions. Computer Graphics Forum Vol. 37, No. H. iVisClustering: An interactive visual document
3, 339–349, 2018. clustering via topic modeling. Computer Graphics
[102] Cavallo, M.; Demiralp, C. Clustrophile 2: Guided Forum Vol. 31, No. 3, 1155–1164, 2012.
visual clustering analysis. IEEE Transactions on [112] Liu, M. C.; Liu, S. X.; Zhu, X. Z.; Liao, Q. Y.; Wei,
Visualization and Computer Graphics Vol. 25, No. F. R.; Pan, S. M. An uncertainty-aware approach for
1, 267–276, 2019. exploratory microblog retrieval. IEEE Transactions
[103] Das, S.; Cashman, D.; Chang, R.; Endert, A. on Visualization and Computer Graphics Vol. 22, No.
BEAMES: Interactive multimodel steering, selection, 1, 250–259, 2016.
and inspection for regression tasks. IEEE Computer [113] Lowe, T.; Forster, E. C.; Albuquerque, G.; Kreiss, J.
Graphics and Applications Vol. 39, No. 5, 20–32, 2019. P.; Magnor, M. Visual analytics for development and
[104] Dingen, D.; van’t Veer, M.; Houthuizen, P.; Mestrom, evaluation of order selection criteria for autoregressive
E. H. J.; Korsten, E. H. H. M.; Bouwman, processes. IEEE Transactions on Visualization and
A. R. A.; van Wijk. J. J. RegressionExplorer: Computer Graphics Vol. 22, No. 1, 151–159, 2016.
Interactive exploration of logistic regression models [114] MacInnes, J.; Santosa, S.; Wright, W. Visual
with subgroup analysis. IEEE Transactions on classification: Expert knowledge guides machine
Visualization and Computer Graphics Vol. 25, No. learning. IEEE Computer Graphics and Applications
1, 246–255, 2019. Vol. 30, No. 1, 8–14, 2010.
A survey of visual analytics techniques for machine learning 25

[115] Migut, M.; Worring, M. Visual exploration [126] Zhao, K. Y.; Ward, M. O.; Rundensteiner, E. A.;
of classification models for risk assessment. In: Higgins, H. N. LoVis: Local pattern visualization for
Proceedings of the IEEE Conference on Visual model refinement. Computer Graphics Forum Vol. 33,
Analytics Science and Technology, 11–18, 2010. No. 3, 331–340, 2014.
[116] Ming, Y.; Xu, P. P.; Cheng, F. R.; Qu, H. M.; Ren, [127] Alexander, E.; Kohlmann, J.; Valenza, R.; Witmore,
L. ProtoSteer: Steering deep sequence model with M.; Gleicher, M. Serendip: Topic model-driven visual
prototypes. IEEE Transactions on Visualization and exploration of text corpora. In: Proceedings of the
Computer Graphics Vol. 26, No. 1, 238–248, 2020. IEEE Conference on Visual Analytics Science and
[117] Muhlbacher, T.; Linhardt, L.; Moller, T.; Piringer, Technology, 173–182, 2014.
H. TreePOD: Sensitivity-aware selection of Pareto- [128] Berger, M.; McDonough, K.; Seversky, L. M. Cite2vec:
optimal decision trees. IEEE Transactions on Citation-driven document exploration via word
Visualization and Computer Graphics Vol. 24, No. embeddings. IEEE Transactions on Visualization
1, 174–183, 2018. and Computer Graphics Vol. 23, No. 1, 691–700,
[118] Packer, E.; Bak, P.; Nikkila, M.; Polishchuk, V.; Ship, 2017.
H. J. Visual analytics for spatial clustering: Using [129] Blumenschein, M.; Behrisch, M.; Schmid, S.; Butscher,
a heuristic approach for guided exploration. IEEE S.; Wahl, D. R.; Villinger, K.; Renner, B.; Reiterer,
Transactions on Visualization and Computer Graphics H.; Keim, D. A. SMARTexplore: Simplifying high-
Vol. 19, No. 12, 2179–2188, 2013. dimensional data analysis through a table-based
[119] Piringer, H.; Berger, W.; Krasser, J. HyperMoVal: visual analytics approach. In: Proceedings of the
Interactive visual validation of regression models for IEEE Conference on Visual Analytics Science and
real-time simulation. Computer Graphics Forum Vol. Technology, 36–47, 2018.
29, No. 3, 983–992, 2010. [130] Bradel, L.; North, C.; House, L. Multi-model semantic
[120] Sacha, D.; Kraus, M.; Bernard, J.; Behrisch, interaction for text analytics. In: Proceedings of the
M.; Schreck, T.; Asano, Y.; Keim, D. A. IEEE Conference on Visual Analytics Science and
SOMFlow: Guided exploratory cluster analysis with Technology, 163–172, 2014.
self-organizing maps and analytic provenance. IEEE [131] Broeksema, B.; Telea, A. C.; Baudel, T. Visual
Transactions on Visualization and Computer Graphics analysis of multi-dimensional categorical data sets.
Vol. 24, No. 1, 120–130, 2018. Computer Graphics Forum Vol. 32, No. 8, 158–169,
[121] Schultz, T.; Kindlmann, G. L. Open-box spectral 2013.
clustering: Applications to medical image analysis. [132] Cao, N.; Sun, J. M.; Lin, Y. R.; Gotz, D.; Liu, S. X.;
IEEE Transactions on Visualization and Computer Qu, H. M. FacetAtlas: Multifaceted visualization for
Graphics Vol. 19, No. 12, 2100–2108, 2013. rich text corpora. IEEE Transactions on Visualization
[122] Van den Elzen, S.; van Wijk, J. J. BaobabView: and Computer Graphics Vol. 16, No. 6, 1172–1181,
Interactive construction and analysis of decision trees. 2010.
In: Proceedings of the IEEE Conference on Visual [133] Chandrasegaran, S.; Badam, S. K.; Kisselburgh, L.;
Analytics Science and Technology, 151–160, 2011. Ramani, K.; Elmqvist, N. Integrating visual analytics
[123] Vrotsou, K.; Nordman, A. Exploratory visual support for grounded theory practice in qualitative
sequence mining based on pattern-growth. IEEE text analysis. Computer Graphics Forum Vol. 36, No.
Transactions on Visualization and Computer Graphics 3, 201–212, 2017.
Vol. 25, No. 8, 2597–2610, 2019. [134] Chen, S. M.; Andrienko, N.; Andrienko, G.; Adilova,
[124] Wang, X. T.; Liu, S. X.; Liu, J. L.; Chen, J. F.; L.; Barlet, J.; Kindermann, J.; Nguyen, P. H.;
Zhu, J.; Guo, B. N. TopicPanorama: A full picture of Thonnard, O.; Turkay, C. LDA ensembles for
relevant topics. IEEE Transactions on Visualization interactive exploration and categorization of behaviors.
and Computer Graphics Vol. 22, No. 12, 2508–2521, IEEE Transactions on Visualization and Computer
2016. Graphics Vol. 26, No. 9, 2775–2792, 2020.
[125] Yang, W. K.; Wang, X. T.; Lu, J.; Dou, W. W.; Liu, [135] Correll, M.; Witmore, M.; Gleicher, M. Exploring
S. X. Interactive steering of hierarchical clustering. collections of tagged text for literary scholarship.
IEEE Transactions on Visualization and Computer Computer Graphics Forum Vol. 30, No. 3, 731–740,
Graphics doi: 10.1109/TVCG.2020.2995100, 2020. 2011.
26 J. Yuan, C. Chen, W. Yang, et al.

[136] Dou, W.; Cho, I.; ElTayeby, O.; Choo, J.; Wang, [146] Hong, F.; Lai, C.; Guo, H.; Shen, E.; Yuan, X.; Li.
X.; Ribarsky, W.; DemographicVis: Analyzing S. FLDA: Latent Dirichlet allocation based unsteady
demographic information based on user generated flow analysis. IEEE Transactions on Visualization
content. In: Proceedings of the IEEE Conference and Computer Graphics Vol. 20, No.12, 2545–2554,
on Visual Analytics Science and Technology, 57–64, 2014.
2015. [147] Hoque, E.; Carenini, G. ConVis: A visual text analytic
[137] El-Assady, M.; Gold, V.; Acevedo, C.; Collins, system for exploring blog conversations. Computer
C.; Keim, D. ConToVi: Multi-party conversation Graphics Forum Vol. 33, No. 3, 221–230, 2014.
exploration using topic-space views. Computer [148] Hu, M. D.; Wongsuphasawat, K.; Stasko, J.
Visualizing social media content with SentenTree.
Graphics Forum Vol. 35, No. 3, 431–440, 2016.
IEEE Transactions on Visualization and Computer
[138] El-Assady, M.; Sevastjanova, R.; Keim, D.; Collins,
Graphics Vol. 23, No. 1, 621–630, 2017.
C. ThreadReconstructor: Modeling reply-chains to
[149] Jänicke, H.; Borgo, R.; Mason, J. S. D.; Chen,
untangle conversational text through visual analytics.
M. SoundRiver: Semantically-rich sound illustration.
Computer Graphics Forum Vol. 37, No. 3, 351–365,
Computer Graphics Forum Vol. 29, No. 2, 357–366,
2018.
2010.
[139] Filipov, V.; Arleo, A.; Federico, P.; Miksch, S. CV3:
[150] Jänicke, S.; Wrisley, D. J. Interactive visual alignment
Visual exploration, assessment, and comparison of of medieval text versions. In: Proceedings of the
CVs. Computer Graphics Forum Vol. 38, No. 3, 107– IEEE Conference on Visual Analytics Science and
118, 2019. Technology, 127–138, 2017.
[140] Fried, D.; Kobourov, S. G. Maps of computer science. [151] Jankowska, M.; Kefiselj, V.; Milios, E. Relative
In: Proceedings of the IEEE Pacific Visualization N-gram signatures: Document visualization at the
Symposium, 113–120, 2014. level of character n-grams. In: Proceedings of the
[141] Fulda, J.; Brehmer, M.; Munzner, T. IEEE Conference on Visual Analytics Science and
TimeLineCurator: Interactive authoring of visual Technology, 103–112, 2012.
timelines from unstructured text. IEEE Transactions [152] Ji, X. N.; Shen, H. W.; Ritter, A.; Machiraju, R.;
on Visualization and Computer Graphics Vol. 22, No. Yen, P. Y. Visual exploration of neural document
1, 300–309, 2016. embedding in information retrieval: Semantics and
[142] Glueck, M.; Naeini, M. P.; Doshi-Velez, F.; Chevalier, feature selection. IEEE Transactions on Visualization
F.; Khan, A.; Wigdor, D.; Brudno, M. PhenoLines: and Computer Graphics Vol. 25, No. 6, 2181–2192,
Phenotype comparison visualizations for disease 2019.
subtyping via topic models. IEEE Transactions on [153] Kakar, T.; Qin, X.; Rundensteiner, E. A.; Harrison, L.;
Visualization and Computer Graphics Vol. 24, No. 1, Sahoo, S. K.; De, S. DIVA: Exploration and validation
of hypothesized drug-drug interactions. Computer
371–381, 2018.
Graphics Forum Vol. 38, No. 3, 95–106, 2019.
[143] Gorg, C.; Liu, Z. C.; Kihm, J.; Choo, J.; Park, H.;
[154] Kim, H.; Choi, D.; Drake, B.; Endert, A.; Park,
Stasko, J. Combining computational analyses and
H. TopicSifter: Interactive search space reduction
interactive visualization for document exploration
through targeted topic modeling. In: Proceedings of
and sensemaking in jigsaw. IEEE Transactions on
the IEEE Conference on Visual Analytics Science and
Visualization and Computer Graphics Vol. 19, No. 10,
Technology, 35–45, 2019.
1646–1663, 2013.
[155] Kim, M.; Kang, K.; Park, D.; Choo, J.; Elmqvist,
[144] Guo, H.; Laidlaw, D. H. Topic-based exploration and N. TopicLens: Efficient multi-level visual topic
embedded visualizations for research idea generation. exploration of large-scale document collections. IEEE
IEEE Transactions on Visualization and Computer Transactions on Visualization and Computer Graphics
Graphics Vol. 26, No. 3, 1592–1607, 2020. Vol. 23, No. 1, 151–160, 2017.
[145] Heimerl, F.; John, M.; Han, Q.; Koch, S.; Ertl. T. [156] Kochtchi, A.; von Landesberger, T.; Biemann, C.
DocuCompass: Effective exploration of document Networks of names: Visual exploration and semi-
landscapes. In: Proceedings of the IEEE Conference automatic tagging of social networks from newspaper
on Visual Analytics Science and Technology, 11–20, articles. Computer Graphics Forum Vol. 33, No. 3,
2016. 211–220, 2014.
A survey of visual analytics techniques for machine learning 27

[157] Li, M. Z.; Choudhury, F.; Bao, Z. F.; Samet, [167] Xie, X.; Cai, X. W.; Zhou, J. P.; Cao, N.; Wu, Y.
H.; Sellis, T. ConcaveCubes: Supporting cluster- C. A semantic-based method for visualizing large
based geographical visualization in large data scale. image collections. IEEE Transactions on Visualization
Computer Graphics Forum Vol. 37, No. 3, 217–228, and Computer Graphics Vol. 25, No. 7, 2362–2377,
2018. 2019.
[158] Liu, S.; Wang, B.; Thiagarajan, J. J.; Bremer, [168] Zhang, L.; Huang, H. Hierarchical narrative collage
P. T.; Pascucci, V. Visual exploration of high- for digital photo album. Computer Graphics Forum
dimensional data through subspace analysis and Vol. 31, No. 7, 2173–2181, 2012.
dynamic projections. Computer Graphics Forum Vol. [169] Zhao, J.; Chevalier, F.; Collins, C.; Balakrishnan,
34, No. 3, 271–280, 2015. R. Facilitating discourse analysis with interactive
[159] Liu, S.; Wang, X.; Chen, J.; Zhu, J.; Guo, B. visualization. IEEE Transactions on Visualization
TopicPanorama: A full picture of relevant topics. and Computer Graphics Vol. 18, No. 12, 2639–2648,
In: Proceedings of the IEEE Conference on Visual 2012.
Analytics Science and Technology, 183–192, 2014. [170] Alsakran, J.; Chen, Y.; Luo, D. N.; Zhao, Y.; Yang,
[160] Liu, X.; Xu, A.; Gou, L.; Liu, H.; Akkiraju, R.; J.; Dou, W. W.; Liu, S. Real-time visualization of
Shen, H. W. SocialBrands: Visual analysis of public streaming text with a force-based dynamic system.
perceptions of brands on social media. In: Proceedings IEEE Computer Graphics and Applications Vol. 32,
of the IEEE Conference on Visual Analytics Science No. 1, 34–45, 2012.
and Technology, 71–80, 2016. [171] Alsakran, J.; Chen, Y.; Zhao, Y.; Yang, J.; Luo, D.
[161] Oelke, D.; Strobelt, H.; Rohrdantz, C.; Gurevych, I.; STREAMIT: Dynamic visualization and interactive
Deussen, O. Comparative exploration of document exploration of text streams. In: Proceedings of the
collections: A visual analytics approach. Computer IEEE Pacific Visualization Symposium, 131–138,
Graphics Forum Vol. 33, No. 3, 201–210, 2014. 2011.
[162] Park, D.; Kim, S.; Lee, J.; Choo, J.; Diakopoulos, N.; [172] Andrienko, G.; Andrienko, N.; Anzer, G.; Bauer,
Elmqvist, N. ConceptVector: text visual analytics via P.; Budziak, G.; Fuchs, G.; Hecker, D.; Weber,
interactive lexicon building using word embedding. H.; Wrobel, S. Constructing spaces and times for
IEEE Transactions on Visualization and Computer tactical analysis in football. IEEE Transactions
Graphics Vol. 24, No. 1, 361–370, 2018. on Visualization and Computer Graphics doi:
[163] Paulovich, F. V.; Toledo, F. M. B.; Telles, G. P.; 10.1109/TVCG.2019.2952129, 2019.
Minghim, R.; Nonato, L. G. Semantic wordification [173] Andrienko, G.; Andrienko, N.; Bremm, S.; Schreck, T.;
of document collections. Computer Graphics Forum von Landesberger, T.; Bak, P.; Keim, D. Space-in-time
Vol. 31, No. 3pt3, 1145–1153, 2012. and time-in-space self-organizing maps for exploring
[164] Shen, Q. M.; Zeng, W.; Ye, Y.; Arisona, S. M.; spatiotemporal patterns. Computer Graphics Forum
Schubiger, S.; Burkhard, R.; Qu, H. StreetVizor: Vol. 29, No. 3, 913–922, 2010.
Visual exploration of human-scale urban forms based [174] Andrienko, G.; Andrienko, N.; Hurter, C.; Rinzivillo,
on street views. IEEE Transactions on Visualization S.; Wrobel, S. Scalable analysis of movement data
and Computer Graphics Vol. 24, No. 1, 1004–1013, for extracting and exploring significant places. IEEE
2018. Transactions on Visualization and Computer Graphics
[165] Von Landesberger, T.; Basgier, D.; Becker, M. Vol. 19, No. 7, 1078–1094, 2013.
Comparative local quality assessment of 3D medical [175] Blascheck, T.; Beck, F.; Baltes, S.; Ertl, T.; Weiskopf,
image segmentations with focus on statistical shape D. Visual analysis and coding of data-rich user
model-based algorithms. IEEE Transactions on behavior. In: Proceedings of the IEEE Conference
Visualization and Computer Graphics Vol. 22, No. on Visual Analytics Science and Technology, 141–150,
12, 2537–2549, 2016. 2016.
[166] Wall, E.; Das, S.; Chawla, R.; Kalidindi, B.; Brown, [176] Bögl, M.; Filzmoser, P.; Gschwandtner, T.;
E. T.; Endert, A. Podium: Ranking data using Lammarsch, T.; Leite, R. A.; Miksch, S.; Rind, A.
mixed-initiative visual analytics. IEEE Transactions Cycle plot revisited: Multivariate outlier detection
on Visualization and Computer Graphics Vol. 24, No. using a distance-based abstraction. Computer
1, 288–297, 2018. Graphics Forum Vol. 36, No. 3, 227–238, 2017.
28 J. Yuan, C. Chen, W. Yang, et al.

[177] Bosch, H.; Thom, D.; Heimerl, F.; Puttmann, Interactive visual discovering of movement patterns
E.; Koch, S.; Kruger, R.; Worner, M.; Ertl, T. from sparsely sampled geo-tagged social media data.
ScatterBlogs2: real-time monitoring of microblog IEEE Transactions on Visualization and Computer
messages through user-guided filtering. IEEE Graphics Vol. 22, No. 1, 270–279, 2016.
Transactions on Visualization and Computer Graphics [187] Chen, Y.; Chen, Q.; Zhao, M.; Boyer, S.;
Vol. 19, No. 12, 2022–2031, 2013. Veeramachaneni, K.; Qu, H. DropoutSeer: Visualizing
[178] Buchmüller, J.; Janetzko, H.; Andrienko, G.; learning patterns in massive open online courses for
Andrienko, N.; Fuchs, G.; Keim, D. A. Visual dropout reasoning and prediction. In: Proceedings of
analytics for exploring local impact of air traffic. the IEEE Conference on Visual Analytics Science and
Computer Graphics Forum Vol. 34, No. 3, 181–190, Technology, 111–120, 2016.
2015. [188] Chen, Y. Z.; Xu, P. P.; Ren, L. Sequence synopsis:
[179] Cao, N.; Lin, C. G.; Zhu, Q. H.; Lin, Y. R.; Teng, Optimize visual summary of temporal event data.
X.; Wen, X. D. Voila: Visual anomaly detection and IEEE Transactions on Visualization and Computer
monitoring with streaming spatiotemporal data. IEEE Graphics Vol. 24, No. 1, 45–55, 2018.
Transactions on Visualization and Computer Graphics [189] Chu, D.; Sheets, D. A.; Zhao, Y.; Wu, Y.; Yang,
Vol. 24, No. 1, 23–33, 2018. J.; Zheng, M.; Chen, G. Visualizing hidden themes
[180] Cao, N.; Lin, Y. R.; Sun, X. H.; Lazer, D.; Liu, S. of taxi movement with semantic transformation.
X.; Qu, H. M. Whisper: Tracing the spatiotemporal In: Proceedings of the IEEE Pacific Visualization
process of information diffusion in real time. IEEE Symposium, 137–144, 2014.
Transactions on Visualization and Computer Graphics [190] Cui, W. W.; Liu, S. X.; Tan, L.; Shi, C. L.;
Vol. 18, No. 12, 2649–2658, 2012. Song, Y. Q.; Gao, Z. K.; Qu, H. M.; Tong, X.
[181] Cao, N.; Shi, C. L.; Lin, S.; Lu, J.; Lin, Y. R.; Lin, TextFlow: Towards better understanding of evolving
C. Y. TargetVue: Visual analysis of anomalous user topics in text. IEEE Transactions on Visualization
behaviors in online communication systems. IEEE and Computer Graphics Vol. 17, No. 12, 2412–2421,
Transactions on Visualization and Computer Graphics 2011.
Vol. 22, No. 1, 280–289, 2016. [191] Cui, W. W.; Liu, S. X.; Wu, Z. F.; Wei, H. How
[182] Chae, J.; Thom, D.; Bosch, H.; Jang, Y.; Maciejewski, hierarchical topics evolve in large text corpora. IEEE
R.; Ebert, D. S.; Ertl, T. Spatiotemporal social Transactions on Visualization and Computer Graphics
media analytics for abnormal event detection and Vol. 20, No. 12, 2281–2290, 2014.
examination using seasonal-trend decomposition. In: [192] Di Lorenzo, G.; Sbodio, M.; Calabrese, F.; Berlingerio,
Proceedings of the IEEE Conference on Visual M.; Pinelli, F.; Nair, R. AllAboard: Visual exploration
Analytics Science and Technology, 143–152, 2012. of cellphone mobility data to optimise public transport.
[183] Chen, Q.; Yue, X. W.; Plantaz, X.; Chen, Y. Z.; Shi, IEEE Transactions on Visualization and Computer
C. L.; Pong, T. C.; Qu, H. ViSeq: Visual analytics Graphics Vol. 22, No. 2, 1036–1050, 2016.
of learning sequence in massive open online courses. [193] Dou, W.; Wang, X.; Chang, R.; Ribarsky, W.
IEEE Transactions on Visualization and Computer ParallelTopics: A probabilistic approach to exploring
Graphics Vol. 26, No. 3, 1622–1636, 2020. document collections. In: Proceedings of the
[184] Chen, S.; Chen, S.; Lin, L.; Yuan, X.; Liang, J.; IEEE Conference on Visual Analytics Science and
Zhang, X. E-map: A visual analytics approach for Technology, 231–240, 2011.
exploring significant event evolutions in social media. [194] Dou, W.; Wang, X.; Skau, D.; Ribarsky, W.; Zhou,
In: Proceedings of the IEEE Conference on Visual M. X. Leadline: Interactive visual analysis of text
Analytics Science and Technology, 36–47, 2017. data through event identification and exploration.
[185] Chen, S.; Chen, S.; Wang, Z.; Liang, J.; Yuan, In: Proceedings of the IEEE Conference on Visual
X.; Cao, N.; Wu, Y. D-Map: Visual analysis of Analytics Science and Technology, 93–102, 2012.
egocentric information difiusion patterns in social [195] Du, F.; Plaisant, C.; Spring, N.; Shneiderman, B.
media. In: Proceedings of the IEEE Conference on EventAction: Visual analytics for temporal event
Visual Analytics Science and Technology, 41–50, 2016. sequence recommendation. In: Proceedings of the
[186] Chen, S. M.; Yuan, X. R.; Wang, Z. H.; Guo, IEEE Conference on Visual Analytics Science and
C.; Liang, J.; Wang, Z. C.; Zhang, X.; Zhang, J. Technology, 61–70, 2016.
A survey of visual analytics techniques for machine learning 29

[196] El-Assady, M.; Sevastjanova, R.; Gipp, B.; Keim, [206] Itoh, M.; Toyoda, M.; Zhu, C. Z.; Satoh, S.;
D.; Collins, C. NEREx: Named-entity relationship Kitsuregawa, M. Image flows visualization for inter-
exploration in multi-party conversations. Computer media comparison. In: Proceedings of the IEEE
Graphics Forum Vol. 36, No. 3, 213–225, 2017. Pacific Visualization Symposium, 129–136, 2014.
[197] Fan, M. M.; Wu, K.; Zhao, J.; Li, Y.; Wei, W.; Truong, [207] Itoh, M.; Yoshinaga, N.; Toyoda, M.; Kitsuregawa,
K. N. VisTA: Integrating machine intelligence with M. Analysis and visualization of temporal changes
visualization to support the investigation of think- in bloggers’ activities and interests. In: Proceedings
aloud sessions. IEEE Transactions on Visualization of the IEEE Pacific Visualization Symposium, 57–64,
and Computer Graphics Vol. 26, No. 1, 343–352, 2020. 2012.
[198] Ferreira, N.; Poco, J.; Vo, H. T.; Freire, J.; Silva, C. [208] Kamaleswaran, R.; Collins, C.; James, A.; McGregor,
T. Visual exploration of big spatio-temporal urban C. PhysioEx: Visual analysis of physiological event
data: A study of New York City taxi trips. IEEE streams. Computer Graphics Forum Vol. 35, No. 3,
Transactions on Visualization and Computer Graphics 331–340, 2016.
Vol. 19, No. 12, 2149–2158, 2013. [209] Karduni, A.; Cho, I.; Wessel, G.; Ribarsky, W.; Sauda,
[199] Gobbo, B.; Balsamo, D.; Mauri, M.; Bajardi, P.; E.; Dou, W. W. Urban space explorer: A visual
Panisson, A.; Ciuccarelli, P. Topic Tomographies analytics system for urban planning. IEEE Computer
(TopTom): A visual approach to distill information Graphics and Applications Vol. 37, No. 5, 50–60, 2017.
from media streams. Computer Graphics Forum Vol. [210] Krueger, R.; Han, Q.; Ivanov, N.; Mahtal, S.; Thom,
38, No. 3, 609–621, 2019. D.; Pfister, H.; Ertl, T. Bird’s-eye-large-scale visual
[200] Gotz, D.; Stavropoulos, H. DecisionFlow: Visual analytics of city dynamics using social location data.
analytics for high-dimensional temporal event Computer Graphics Forum Vol. 38, No. 3, 595–607,
sequence data. IEEE Transactions on Visualization 2019.
and Computer Graphics Vol. 20, No. 12, 1783–1792, [211] Krueger, R.; Thom, D.; Ertl, T. Visual analysis
2014. of movement behavior using web data for context
[201] Guo, S. N.; Jin, Z. C.; Gotz, D.; Du, F.; Zha, enrichment. In: Proceedings of the IEEE Pacific
H. Y.; Cao, N. Visual progression analysis of event Visualization Symposium, 193–200, 2014.
sequence data. IEEE Transactions on Visualization [212] Krueger, R.; Thom, D.; Ertl, T. Semantic
and Computer Graphics Vol. 25, No. 1, 417–426, 2019. enrichment of movement behavior with foursquare—
[202] Guo, S. N.; Xu, K.; Zhao, R. W.; Gotz, D.; Zha, A visual analytics approach. IEEE Transactions on
H. Y.; Cao, N. EventThread: Visual summarization Visualization and Computer Graphics Vol. 21, No. 8,
and stage analysis of event sequence data. IEEE 903–915, 2015.
Transactions on Visualization and Computer Graphics [213] Lee, C.; Kim, Y.; Jin, S.; Kim, D.; Maciejewski,
Vol. 24, No. 1, 56–65, 2018. R.; Ebert, D.; Ko, S. A visual analytics system for
[203] Gutenko, I.; Dmitriev, K.; Kaufman, A. E.; Barish, M. exploring, monitoring, and forecasting road traffic
A. AnaFe: Visual analytics of image-derived temporal congestion. IEEE Transactions on Visualization and
features: Focusing on the spleen. IEEE Transactions Computer Graphics Vol. 26, No. 11, 3133–3146, 2020.
on Visualization and Computer Graphics Vol. 23, No. [214] Leite, R. A.; Gschwandtner, T.; Miksch, S.; Kriglstein,
1, 171–180, 2017. S.; Pohl, M.; Gstrein, E.; Kuntner, J. EVA:
[204] Havre, S.; Hetzler, E.; Whitney, P.; Nowell, Visual analytics to identify fraudulent events. IEEE
L. ThemeRiver: Visualizing thematic changes in Transactions on Visualization and Computer Graphics
large document collections. IEEE Transactions on Vol. 24, No. 1, 330–339, 2018.
Visualization and Computer Graphics Vol. 8, No. 1, [215] Li, J.; Chen, S. M.; Chen, W.; Andrienko,
9–20, 2002. G.; Andrienko, N. Semantics-space-time cube. A
[205] Heimerl, F.; Han, Q.; Koch, S.; Ertl, T. conceptual framework for systematic analysis of
CiteRivers: Visual analytics of citation patterns. texts in space and time. IEEE Transactions on
IEEE Transactions on Visualization and Computer Visualization and Computer Graphics, Vol. 26, No.
Graphics Vol. 22, No. 1, 190–199, 2016. 4, 1789–1806, 2019.
30 J. Yuan, C. Chen, W. Yang, et al.

[216] Li, Q.; Wu, Z. M.; Yi, L. L.; Kristanto, S. N.; Qu, [226] Luo, D. N.; Yang, J.; Krstajic, M.; Ribarsky,
H. M.; Ma, X. J. WeSeer: Visual analysis for better W.; Keim, D. A. EventRiver: Visually exploring
information cascade prediction of WeChat articles. text collections with temporal references. IEEE
IEEE Transactions on Visualization and Computer Transactions on Visualization and Computer Graphics
Graphics Vol. 26, No. 2, 1399–1412, 2020. Vol. 18, No. 1, 93–105, 2012.
[217] Li, Z. Y.; Zhang, C. H.; Jia, S. C.; Zhang, J. W. Galex: [227] Maciejewski, R.; Hafen, R.; Rudolph, S.; Larew, S.
Exploring the evolution and intersection of disciplines. G.; Mitchell, M. A.; Cleveland, W. S.; Ebert, D. S.
IEEE Transactions on Visualization and Computer Forecasting hotspots: A predictive analytics approach.
Graphics Vol. 26, No. 1, 1182–1192, 2019. IEEE Transactions on Visualization and Computer
[218] Liu, H.; Jin, S. C.; Yan, Y. Y.; Tao, Y. B.; Lin, H. Graphics Vol. 17, No. 4, 440–453, 2011.
Visual analytics of taxi trajectory data via topical sub- [228] Malik, A.; Maciejewski, R.; Towers, S.; McCullough,
trajectories. Visual Informatics Vol. 3, No. 3, 140–149, S.; Ebert, D. S. Proactive spatiotemporal resource
2019. allocation and predictive visual analytics for
[219] Liu, S. X.; Yin, J. L.; Wang, X. T.; Cui, W. W.; Cao, community policing and law enforcement. IEEE
K. L.; Pei, J. Online visual analytics of text streams. Transactions on Visualization and Computer Graphics
IEEE Transactions on Visualization and Computer Vol. 20, No. 12, 1863–1872, 2014.
Graphics Vol. 22, No. 11, 2451–2466, 2016. [229] Miranda, F.; Doraiswamy, H.; Lage, M.; Zhao, K.;
[220] Liu, S.; Zhou, M. X.; Pan, S.; Song, Y.; Qian, W.; Cai, Goncalves, B.; Wilson, L.; Hsieh, M.; Silva, C. T.
W.; Lian, X. TIARA: Interactive, topic-based visual Urban pulse: Capturing the rhythm of cities. IEEE
text summarization and analysis. ACM Transactions Transactions on Visualization and Computer Graphics
on Intelligent Systems and Technology Vol. 3, No.2, Vol. 23, No. 1, 791–800, 2017.
Article No. 25, 2012. [230] Purwantiningsih, O.; Sallaberry, A.; Andary, S.;
[221] Liu, Z. C.; Kerr, B.; Dontcheva, M.; Grover, J.; Seilles, A.; Azfie, J. Visual analysis of body movement
Hoffman, M.; Wilson, A. CoreFlow: Extracting and in serious games for healthcare. In: Proceedings of
visualizing branching patterns from event sequences. the IEEE Pacific Visualization Symposium, 229–233,
Computer Graphics Forum Vol. 36, No. 3, 527–538, 2016.
2017. [231] Riehmann, P.; Kiesel, D.; Kohlhaas, M.; Froehlich,
[222] Liu, Z.; Wang, Y.; Dontcheva, M.; Hofiman, M.; B. Visualizing a thinker’s life. IEEE Transactions on
Walker, S.; Wilson, A. Patterns and sequences: Visualization and Computer Graphics Vol. 25, No. 4,
Interactive exploration of clickstreams to understand 1803–1816, 2019.
common visitor paths. IEEE Transactions on [232] Sacha, D.; Al-Masoudi, F.; Stein, M.; Schreck, T.;
Visualization and Computer Graphics Vol. 23, No.1, Keim, D. A.; Andrienko, G.; Janetzko, H. Dynamic
321–330, 2017. visual abstraction of soccer movement. Computer
[223] Lu, Y. F.; Steptoe, M.; Burke, S.; Wang, H.; Graphics Forum Vol. 36, No. 3, 305–315, 2017.
Tsai, J. Y.; Davulcu, H.; Montgomery, D.; Corman, [233] Sarikaya, A.; Correli, M.; Dinis, J. M.; O’Connor, D.
S. R.; Maciejewski, R. Exploring evolving media H.; Gleicher, M. Visualizing co-occurrence of events
discourse through event cueing. IEEE Transactions in populations of viral genome sequences. Computer
on Visualization and Computer Graphics Vol. 22, No. Graphics Forum Vol. 35, No. 3, 151–160, 2016.
1, 220–229, 2016. [234] Shi, C. L.; Wu, Y. C.; Liu, S. X.; Zhou, H.; Qu,
[224] Lu, Y. F.; Wang, F.; Maciejewski, R. Business H. M. LoyalTracker: Visualizing loyalty dynamics in
intelligence from social media: A study from the search engines. IEEE Transactions on Visualization
VAST box office challenge. IEEE Computer Graphics and Computer Graphics Vol. 20, No. 12, 1733–1742,
and Applications Vol. 34, No. 5, 58–69, 2014. 2014.
[225] Lu, Y. F.; Wang, H.; Landis, S.; Maciejewski, R. A [235] Steiger, M.; Bernard, J.; Mittelstädt, S.; Lücke-
visual analytics framework for identifying topic drivers Tieke, H.; Keim, D.; May, T.; Kohlhammer, J.
in media events. IEEE Transactions on Visualization Visual analysis of time-series similarities for anomaly
and Computer Graphics Vol. 24, No. 9, 2501–2515, detection in sensor networks. Computer Graphics
2018. Forum Vol. 33, No. 3, 401–410, 2014.
A survey of visual analytics techniques for machine learning 31

[236] Stopar, L.; Skraba, P.; Grobelnik, M.; Mladenic, D. [246] Wang, X.; Liu, S.; Chen, Y.; Peng, T.-Q.; Su, J.;
StreamStory: Exploring multivariate time series on Yang, J.; Guo, B. How ideas flow across multiple
multiple scales. IEEE Transactions on Visualization social groups. In: Proceedings of the IEEE Conference
and Computer Graphics Vol. 25, No. 4, 1788–1802, on Visual Analytics Science and Technology, 51–60,
2019. 2016.
[237] Sultanum, N.; Singh, D.; Brudno, M.; Chevalier, F. [247] Wang, Y.; Haleem, H.; Shi, C. L.; Wu, Y. H.; Zhao, X.;
Doccurate: A curation-based approach for clinical text Fu, S. W.; Qu, H. Towards easy comparison of local
visualization. IEEE Transactions on Visualization businesses using online reviews. Computer Graphics
and Computer Graphics Vol. 25, No. 1, 142–151, Forum Vol. 37, No. 3, 63–74, 2018.
2019. [248] Wei, F. R.; Liu, S. X.; Song, Y. Q.; Pan, S. M.; Zhou,
[238] Sun, G. D.; Wu, Y. C.; Liu, S. X.; Peng, T. Q.; Zhu, M. X.; Qian, W. H.; Shi, L.; Tan, L.; Zhang, Q.
J. J. H.; Liang, R. H. EvoRiver: Visual analysis of TIARA: A visual exploratory text analytic system. In:
topic coopetition on social media. IEEE Transactions Proceedings of the 16th ACM SIGKDD International
on Visualization and Computer Graphics Vol. 20, No. Conference on Knowledge Discovery and Data Mining,
12, 1753–1762, 2014. 153–162, 2010.
[239] Sung, C. Y.; Huang, X. Y.; Shen, Y. C.; Cherng, F. [249] Wei, J.; Shen, Z.; Sundaresan, N.; Ma, K.-L.
Y.; Lin, W. C.; Wang, H. C. Exploring online learners’ Visual cluster exploration of web clickstream data.
interactive dynamics by visually analyzing their time- In: Proceedings of the IEEE Conference on Visual
anchored comments. Computer Graphics Forum Vol. Analytics Science and Technology, 3–12, 2012.
36, No. 7, 145–155, 2017. [250] Wu, A. Y.; Qu, H. M. Multimodal analysis of
[240] Thom, D.; Bosch, H.; Koch, S.; Wörner, M.; video collections: Visual exploration of presentation
Ertl, T. Spatiotemporal anomaly detection through techniques in TED talks. IEEE Transactions on
visual analysis of geolocated Twitter messages. Visualization and Computer Graphics Vol. 26, No.
In: Proceedings of the IEEE Pacific Visualization 7, 2429–2442, 2020.
Symposium, 41–48, 2012. [251] Wu, W.; Zheng, Y.; Cao, N.; Zeng, H.; Ni, B.; Qu, H.;
[241] Thom, D.; Kruger, R.; Ertl, T. Can twitter save lives? Ni, L. M. MobiSeg: Interactive region segmentation
A broad-scale study on visual social media analytics using heterogeneous mobility data. In: Proceedings of
for public safety. IEEE Transactions on Visualization the IEEE Pacific Visualization Symposium, 91–100,
and Computer Graphics Vol. 22, No. 7, 1816–1829, 2017.
2016. [252] Wu, Y. C.; Chen, Z. T.; Sun, G. D.; Xie, X.; Cao, N.;
[242] Tkachev, G.; Frey, S.; Ertl, T. Local prediction Liu, S. X.; Cui, W. StreamExplorer: A multi-stage
models for spatiotemporal volume visualization. IEEE system for visually exploring events in social streams.
Transactions on Visualization and Computer Graphics IEEE Transactions on Visualization and Computer
doi: 10.1109/TVCG.2019.2961893, 2019. Graphics Vol. 24, No. 10, 2758–2772, 2018.
[243] Vehlow, C.; Beck, F.; Auwärter, P.; Weiskopf, D. [253] Wu, Y. C.; Liu, S. X.; Yan, K.; Liu, M. C.; Wu, F.
Visualizing the evolution of communities in dynamic Z. OpinionFlow: Visual analysis of opinion diffusion
graphs. Computer Graphics Forum Vol. 34, No. 1, on social media. IEEE Transactions on Visualization
277–288, 2015. and Computer Graphics Vol. 20, No. 12, 1763–1772,
[244] Von Landesberger, T.; Brodkorb, F.; Roskosch, 2014.
P.; Andrienko, N.; Andrienko, G.; Kerren, A. [254] Wu, Y. H.; Pitipornvivat, N.; Zhao, J.; Yang, S. X.;
MobilityGraphs: Visual analysis of mass mobility Huang, G. W.; Qu, H. M. egoSlider: Visual analysis
dynamics via spatio-temporal graphs and clustering. of egocentric network evolution. IEEE Transactions
IEEE Transactions on Visualization and Computer on Visualization and Computer Graphics Vol. 22, No.
Graphics Vol. 22, No. 1, 11–20, 2016. 1, 260–269, 2016.
[245] Wang, X.; Dou, W.; Ma, Z.; Villalobos, J.; Chen, Y.; [255] Xie, C.; Chen, W.; Huang, X. X.; Hu, Y. Q.; Barlowe,
Kraft, T.; Ribarsky, W. I-SI: Scalable architecture for S.; Yang, J. VAET: A visual analytics approach
analyzing latent topical-level information from social for E-transactions time-series. IEEE Transactions on
media data. Computer Graphics Forum Vol. 31, No. Visualization and Computer Graphics Vol. 20, No. 12,
3, 1275–1284, 2012. 1743–1752, 2014.
32 J. Yuan, C. Chen, W. Yang, et al.

[256] Xu, J.; Tao, Y.; Lin, H.; Zhu, R.; Yan, Y. Exploring [266] Zhang, J. W.; E, Y. L.; Ma, J.; Zhao, Y. H.; Xu, B.
controversy via sentiment divergences of aspects H.; Sun, L. T.; Chen, J.; Yuan, X. Visual analysis of
in reviews. In: Proceedings of the IEEE Pacific public utility service problems in a metropolis. IEEE
Visualization Symposium, 240–249, 2017. Transactions on Visualization and Computer Graphics
[257] Xu, J.; Tao, Y. B.; Yan, Y. Y.; Lin, H. Exploring Vol. 20, No. 12, 1843–1852, 2014.
evolution of dynamic networks via diachronic node [267] Zhao, J.; Cao, N.; Wen, Z.; Song, Y. L.; Lin,
embeddings. IEEE Transactions on Visualization and Y. R.; Collins, C. #FluxFlow: Visual analysis of
Computer Graphics Vol. 26, No. 7, 2387–2402, 2020. anomalous information spreading on social media.
IEEE Transactions on Visualization and Computer
[258] Xu, P. P.; Mei, H. H.; Ren, L.; Chen, W. ViDX:
Graphics Vol. 20, No. 12, 1773–1782, 2014.
Visual diagnostics of assembly line performance in
[268] Zhao, Y.; Luo, X. B.; Lin, X. R.; Wang, H. R.;
smart factories. IEEE Transactions on Visualization
Kui, X. Y.; Zhou, F. F.; Wang, J.; Chen, Y.; Chen,
and Computer Graphics Vol. 23, No. 1, 291–300, 2017.
W. Visual analytics for electromagnetic situation
[259] Xu, P. P.; Wu, Y. C.; Wei, E. X.; Peng, T. Q.; Liu, awareness in radio monitoring and management. IEEE
S. X.; Zhu, J. J.; Qu. H. Visual analysis of topic Transactions on Visualization and Computer Graphics
competition on social media. IEEE Transactions on Vol. 26, No. 1, 590–600, 2020.
Visualization and Computer Graphics Vol. 19, No. 12, [269] Zhou, Z. G.; Meng, L. H.; Tang, C.; Zhao, Y.; Guo, Z.
2012–2021, 2013. Y.; Hu, M. X.; Chen, W. Visual abstraction of large
[260] Yu, L.; Wu, W.; Li, X.; Li, G.; Ng, W. S.; Ng, S.-K.; scale geospatial origin-destination movement data.
Huang, Z.; Arunan, A.; Watt, H. M. iVizTRANS: IEEE Transactions on Visualization and Computer
Interactive visual learning for home and work place Graphics Vol. 25, No. 1, 43–53, 2019.
detection from massive public transportation data. [270] Zhou, Z. G.; Ye, Z. F.; Liu, Y. N.; Liu, F.; Tao, Y.
In: Proceedings of the IEEE Conference on Visual B.; Su, W. H. Visual analytics for spatial clusters
Analytics Science and Technology, 49–56, 2015. of air-quality data. IEEE Computer Graphics and
[261] Garcia Zanabria, G.; Alvarenga Silveira, J.; Poco, Applications Vol. 37, No. 5, 98–105, 2017.
J.; Paiva, A.; Batista Nery, M.; Silva, C. T.; de [271] Tian, T.; Zhu, J. Max-margin majority voting for
Abreu, S. F. A.; Nonato, L. G. CrimAnalyzer: learning from crowds. In: Proceedings of the Advances
Understanding crime patterns in São Paulo. IEEE in Neural Information Processing Systems, 1621–1629,
Transactions on Visualization and Computer Graphics 2015.
doi: 10.1109/TVCG.2019.2947515, 2019. [272] Ng, A. Machine learning and AI via brain
[262] Zeng, H. P.; Shu, X. H.; Wang, Y. B.; Wang, Y.; Zhang, simulations. 2013. Available at https://ai.stanford.edu/
∼ang/slides/DeepLearning-Mar2013.pptx.
L. G.; Pong, T. C.; Qu, H. EmotionCues: Emotion-
oriented visual summarization of classroom videos. [273] Nilsson, N. J. Introduction to Machine Learning: An
Early Draft of a Proposed Textbook. 2005. Available
IEEE Transactions on Visualization and Computer
at https://ai.stanford.edu/∼nilsson/MLBOOK.pdf.
Graphics doi: 10.1109/TVCG.2019.2963659, 2020.
[274] Lakshminarayanan, B.; Pritzel, A.; Blundell, C.
[263] Zeng, H. P.; Wang, X. B.; Wu, A. Y.; Wang, Y.;
Simple and scalable predictive uncertainty estimation
Li, Q.; Endert, A.; Qu, H. EmoCo: Visual analysis
using deep ensembles. In: Proceedings of the Advances
of emotion coherence in presentation videos. IEEE
in Neural Information Processing Systems, 6402–6413,
Transactions on Visualization and Computer Graphics
2017.
Vol. 26, No. 1, 927–937, 2019.
[275] Lee, K.; Lee, H.; Lee, K.; Shin, J. Training confidence-
[264] Zeng, W.; Fu, C. W.; Müller Arisona, S.; Erath, calibrated classifiers for detecting ut-of-distribution
A.; Qu, H. Visualizing waypoints-constrained origin- samples. arXiv preprint arXiv:1711.09325, 2018.
destination patterns for massive transportation data. [276] Liu, M. C.; Jiang, L.; Liu, J. L.; Wang, X. T.;
Computer Graphics Forum Vol. 35, No. 8, 95–107, Zhu, J.; Liu, S. X. Improving learning-from-crowds
2016. through expert validation. In: Proceedings of the
[265] Zhang, J. W.; Ahlbrand, B.; Malik, A.; Chae, J.; 26th International Joint Conference on Artificial
Min, Z. Y.; Ko, S.; Ebert, D. S. A visual analytics Intelligence, 2329–2336, 2017.
framework for microblog data analysis at multiple [277] Russakovsky, O.; Deng, J.; Su, H.; Krause, J.;
scales of aggregation. Computer Graphics Forum Vol. Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla,
35, No. 3, 441–450, 2016. A.; Bernstein, M.; Berg, A. C.; Fei-Fei, L. ImageNet
A survey of visual analytics techniques for machine learning 33

large scale visual recognition challenge. International [289] Wang, X. T.; Liu, S. X.; Song, Y. Q.; Guo, B. N.
Journal of Computer Vision Vol. 115, No. 3, 211–252, Mining evolutionary multi-branch trees from text
2015. streams. In: Proceedings of the 19th ACM SIGKDD
[278] Chandrashekar, G.; Sahin, F. A survey on International Conference on Knowledge Discovery and
feature selection methods. Computers & Electrical Data Mining, 722–730, 2013.
Engineering Vol. 40, No. 1, 16–28, 2014. [290] Li, Y. F.; Guo, L. Z.; Zhou, Z. H. Towards
[279] Brooks, M.; Amershi, S.; Lee, B.; Drucker, S. safe weakly supervised learning. IEEE Transactions
M.; Kapoor, A.; Simard, P. FeatureInsight: Visual on Pattern Analysis and Machine Intelligence doi:
support for error-driven feature ideation in text 10.1109/TPAMI.2019.2922396, 2019.
classification. In: Proceedings of the IEEE Conference [291] Li, Y. F.; Wang, S. B.; Zhou, Z. H. Graph
on Visual Analytics Science and Technology, 105–112, quality judgement: A large margin expedition. In:
2015. Proceedings of the International Joint Conference on
[280] Tzeng, F.-Y.; Ma, K.-L. Opening the black box— Artificial Intelligence, 1725–1731, 2016.
Data driven visualization of neural networks. In: [292] Zhou, Z. H. A brief introduction to weakly supervised
Proceedings of the IEEE Conference on Visualization, learning. National Science Review Vol. 5, No. 1, 44–53,
383–390, 2005. 2018.
[281] Abadi, M.; Agarwal, A.; Barham, P.; Brevdo, E.; [293] Foulds, J.; Frank, E. A review of multi-instance
Chen, Z.; Citro, C.; Corrado, G. S.; Davis, A.; Dean, learning assumptions. The Knowledge Engineering
J.; Devin, M. et al. TensorFlow: Large-scale machine Review Vol. 25, No. 1, 1–25, 2010.
learning on heterogeneous distributed systems, arXiv [294] Zhou, Z. H. Multi-instance learning from supervised
preprint arXiv:1603.04467, 2015. view. Journal of Computer Science and Technology
[282] Ming, Y.; Xu, P. P.; Qu, H. M.; Ren, L. Interpretable Vol. 21, No. 5, 800–809, 2006.
and steerable sequence learning via prototypes. In: [295] Donahue, J.; Jia, Y.; Vinyals, O.; Hofiman, J.;
Proceedings of the ACM SIGKDD International Zhang, N.; Tzeng, E.; Darrell, T. DeCAF: A deep
Conference on Knowledge Discovery & Data Mining, convolutional activation feature for generic visual
903–913, 2019. recognition. In: Proceedings of the International
[283] Liu, S. X.; Cui, W. W.; Wu, Y. C.; Liu, M. C. A Conference on Machine Learning, 647–655, 2014.
survey on information visualization: Recent advances [296] Wang, Q. W.; Yuan, J.; Chen, S. X.; Su, H.;
and challenges. The Visual Computer Vol. 30, No. 12, Qu, H. M.; Liu, S. X. Visual genealogy of deep
1373–1393, 2014. neural networks. IEEE Transactions on Visualization
[284] Ma, Z.; Dou, W.; Wang, X.; Akella, S. Tag- and Computer Graphics Vol. 26, No. 11, 3340–3352,
latent Dirichlet allocation: Understanding hashtags 2020.
and their relationships. In: Proceedings of the [297] Ayinde, B. O.; Zurada, J. M. Building eficient
IEEE/WIC/ACM International Joint Conferences on ConvNets using redundant feature pruning. arXiv
Web Intelligence and Intelligent Agent Technologies, preprint arXiv:1802.07653, 2018.
260–267, 2013. [298] Baltrusaitis, T.; Ahuja, C.; Morency, L. P. Multimodal
[285] Kosara, R.; Bendix, F.; Hauser, H. Parallel machine learning: A survey and taxonomy. IEEE
sets: Interactive exploration and visual analysis of Transactions on Pattern Analysis and Machine
categorical data. IEEE Transactions on Visualization Intelligence Vol. 41, No. 2, 423–443, 2019.
and Computer Graphics Vol. 12, No. 4, 558–568, 2006. [299] Lu, J.; Batra, D.; Parikh, D.; Lee, S. ViLBERT:
[286] Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G. S.; Pretraining task-agnostic visiolinguistic represen-
Dean, J. Distributed representations of words and tations for vision-and-language tasks. In: Proceedings
phrases and their compositionality. In: Proceedings of the Advances in Neural Information Processing
of the Advances in Neural Information Processing Systems, 13–23, 2019.
Systems, 3111–3119, 2013. [300] Lu, J.; Liu, A. J.; Dong, F.; Gu, F.; Gama, J.; Zhang,
[287] Blei, D. M.; Ng, A. Y.; Jordan, M. I. Latent Dirichlet G. Q. Learning under concept drift: A review. IEEE
allocation. Journal of Machine Learning Research Vol. Transactions on Knowledge and Data Engineering Vol.
3, 993–1022, 2003. 31, No. 12, 2346–2363, 2018.
[288] Teh, Y. W.; Jordan, M. I.; Beal, M. J.; Blei, D. [301] Yang, W.; Li, Z.; Liu, M.; Lu, Y.; Cao, K.;
M. Hierarchical dirichlet processes. Journal of the Maciejewski, R.; Liu, S. Diagnosing concept drift
American Statistical Association Vol. 101, No. 476, with visual analytics. arXiv preprint arXiv:2007.14372,
1566–1581, 2006. 2020.
34 J. Yuan, C. Chen, W. Yang, et al.

[302] Wang, X.; Chen, W.; Xia, J.; Chen, Z.; Xu, D.; Wu, Jiazhi Xia is an associate professor in
X.; Xu, M.; Schreck, T. Conceptexplorer: Visual the School of Computer Science and
analysis of concept drifts in multi-source time-series Engineering at Central South University.
He received his Ph.D. degree in computer
data. arXiv preprint arXiv:2007.15272, 2020.
science from Nanyang Technological
[303] Liu, S.; Andrienko, G.; Wu, Y.; Cao, N.; Jiang, L.; Shi, University, Singapore in 2011 and
C.; Wang, Y.-S.; Hong, S. Steering data quality with obtained his M.S. and B.S. degrees in
visual analytics: The complexity challenge. Visual computer science and technology from
Informatics Vol. 2, No. 4, 191–197, 2018. Zhejiang University in 2008 and 2005, respectively. His
research interests include data visualization, visual analytics,
and computer graphics.
Jun Yuan is currently a Ph.D. student
at Tsinghua University. His research
interests are in explainable artificial
Shixia Liu is an associate professor
intelligence. He received his B.S. degree
at Tsinghua University. Her research
from Tsinghua University.
interests include visual text analytics,
visual social analytics, interactive
machine learning, and text mining. She
has worked as a research staff member
at IBM China Research Lab and a
Changjian Chen is now a Ph.D. lead researcher at Microsoft Research
student at Tsinghua University. His Asia. She received her B.S. and M.S. degree from Harbin
research interests are in interactive Institute of Technology, and her Ph.D. degree from Tsinghua
machine learning. He received his B.S. University. She is an Associate Editor-in-Chief of IEEE
degree from the University of Science and Trans. Vis. Comput. Graph.
Technology of China.
Open Access This article is licensed under a Creative
Commons Attribution 4.0 International License, which
permits use, sharing, adaptation, distribution and reproduc-
Weikai Yang is a graduate student tion in any medium or format, as long as you give appropriate
at Tsinghua University. His research credit to the original author(s) and the source, provide a link
interest is in visual text analytics. He to the Creative Commons licence, and indicate if changes
received his B.S. degree from Tsinghua were made.
University.
The images or other third party material in this article are
included in the article’s Creative Commons licence, unless
indicated otherwise in a credit line to the material. If material
is not included in the article’s Creative Commons licence and
Mengchen Liu is a senior researcher your intended use is not permitted by statutory regulation or
at Microsoft. His research interests exceeds the permitted use, you will need to obtain permission
include explainable AI and computer directly from the copyright holder.
vision. He received his B.S. degree To view a copy of this licence, visit http://
in electronics engineering and his creativecommons.org/licenses/by/4.0/.
Ph.D. degree in computer science from Other papers from this open access journal are available
Tsinghua University. He has served as free of charge from http://www.springer.com/journal/41095.
a PC member and reviewer for various To submit a manuscript, please go to https://www.
conferences and journals. editorialmanager.com/cvmj.