0% found this document useful (0 votes)

33 views

GraphDecoder Recovering Diverse Network Graphs From Visualization Images Via Attention-Aware Learning

graph decoder

Uploaded by

sasasa

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

33 views

GraphDecoder Recovering Diverse Network Graphs From Visualization Images Via Attention-Aware Learning

graph decoder

Uploaded by

sasasa

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 17

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics.

This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3225554
IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 1

GraphDecoder: Recovering Diverse Network

Graphs from Visualization Images via
Attention-Aware Learning
Sicheng Song, Chenhui Li, Dong Li, Juntong Chen, and Changbo Wang

Abstract—DNGs are diverse network graphs with texts and different styles of nodes and edges, including mind maps, modeling
graphs, and flowcharts. They are high-level visualizations that are easy for humans to understand but difficult for machines. Inspired
by the process of human perception of graphs, we propose a method called GraphDecoder to extract data from raster images. Given
a raster image, we extract the content based on a neural network. We built a semantic segmentation network based on U-Net. We
increase the attention mechanism module, simplify the network model, and design a specific loss function to improve the model’s
ability to extract graph data. After this semantic segmentation network, we can extract the data of all nodes and edges. We then
combine these data to obtain the topological relationship of the entire DNG. We also provide an interactive interface for users to
redesign the DNGs. We verify the effectiveness of our method by evaluations and user studies on datasets collected on the internet
and generated datasets.

Index Terms—Information visualization, Chart mining, Semantic segmentation, Network graph, Attention mechanism

1 I NTRODUCTION image processing [9] and machine learning [10]. Chart

extraction focuses on extracting the original data from
D NG S (diverse network graphs) are common visual-
izations in various applications. They have differ-
ent styles of nodes and edges than the general definition
the raster image of the chart. At present, data extraction
approaches for basic charts such as bar [11, 12] and
pie charts [13, 14] are mature. There are also some
of node-link graphs [1]. Their nodes can be hollow, may
methods [15, 16] to support line charts, but they are not
contain text, and exhibit various shapes. Their edges
robust enough against multiple lines or dotted lines. It
can be curves and polylines instead of straight lines
is difficult to detect lines in the chart. OGR [17] first pro-
only. DNGs contain node-link graphs, flowcharts [2], E-
posed using morphological methods for the extraction
R diagrams [3], and mind maps [4]. Fig. 1 shows exam-
of network graphs. However, morphological methods
ples of DNGs. People often try to modify these DNGs
need to constantly adjust the threshold of binarization.
because of poor design or outdated data. However, most
In addition, such methods can only extract solid circular
visualizations are stored in the form of raster images and
nodes and cannot solve graphs with text. OGER [18]
released on various media [5]. Moreover, in the visual-
improved the edge recognition module of OGR. Vivid-
ization pipeline, designers often draw sketches on paper
Graph [1] explored extracting data from network graphs
before the visualization is formally drawn [1]. If users
with deep neural networks, but this method has many
can convert sketches into visualizations, productivity
limitations. The graph extracted in the report was a
can be increased. In these scenarios, the core problem
special kind of DNG: a solid circular node-link diagram.
is to extract the original data of graph visualizations.
The method only worked for graphs with straight edges
Obtaining original data is a complicated task that has
and circular nodes but no text. DNGs (diverse network
become an important study of visualization [6].
graphs) are more complicated than the network graphs
There are already some existing methods to solve
defined in OGER and VividGraph. There are many types
the problem of obtaining original data. Some previous
of DNGs [5, 19], including E-R diagrams of database
studies [7, 8] used steganography to write original data
design, flowcharts, mind maps, etc.
into images, but this method hides the data in the image:
We present a novel method called GraphDecoder
the data cannot reproduce the original data from the ex-
to extract data from DNGs. First, we build a deep
isting chart image without a steganography model. The
neural network based on U-Net [20] and the attention
most common method is chart data extraction, including
mechanism. We design the backbone and loss function
according to the characteristics of the chart and add a
• The authors are with the School of Computer Science and Technology,
East China Normal University, Shanghai 200062, China. E-mail: {scsong,
module with an attention mechanism to the decoder,
dongli, jtchen}@stu.ecnu.edu.cn, {chli, cbwang}@cs.ecnu.edu.cn. which improves the robustness of the network. Second,
This work was supported by the NSFC under Grant 62072183. we learn to imitate the process of human perception
(Corresponding authors: Chenhui Li and Changbo Wang.)
of a DNG and analyze the extracted data to obtain

© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: ASTAR. Downloaded on February 08,2023 at 04:50:38 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3225554
IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 2

A Input
Start
Start

Judge if the user`s logical

Output
B Input Output Output
address is aligned

Judge if the user's logical

address is aligned

Size
Size

Viraddr is a Viraddr is a Viraddr is a Viraddr is a

multiple of 2 multiple of 4 multiple of 2 multiple of 4

Return error
Return error Return error and end
and end and end

Return error
and end Assert that tlb
Assert that tlb and pagetable
and pagetable

End

Calculate the page number

End where the virtual address
is located
Calculate the page number
where the virtual address is
located

Different
conversion
Different
methods
conversion
methods

Find TLB Table Find TLB Table

Judge

PNG/JPG PNG/SVG/JSON PNG/SVG/JSON

Return error
Judge
and end
Judge Judge

Return error Return error

and end and end Return error
and end

Get the corresponding Return error Return error Get the corresponding
page table entries and end and end page table entries

Physical address is greater

than the actual physical
address
Readonly
or not

The writing
logo is set
Physical address is
greater than the actual
physical address
Readonly
or not

The writing
logo is set
C Input Output Output
Benefits
Overview

Memorize

Returns
Overview Projects
BusErrorException Simple

Projects
Set the table entry Memorize Benefits Planning Goals
in use flag
Returns Set the table
BusErrorException entry in use flag
Planning Goals

Simple Strategies
Strategies
Calculate the
Physical address
Calculate the MIND MAPPING MIND MAPPING
Physical address
Teamwork
Teamwork Ideas

Productivity Sharing
Assert that the physical
address plus size will not
Assert that the physical Sharing Productivity Creativity Innovation
exceed the memory size
address plus size will not Colleagues
exceed the memory size

Returns
Ideas
NoException Colleagues Thoughts

PNG/JPG PNG/SVG/JSON
End Creativity Innovation

PNG/JPG PNG/SVG/JSON PNG/SVG/JSON

Returns End
NoException Thoughts

Fig. 1. The graph decoder can extract DNG data from raster images and automatically retarget them. Our method can
be applied to many DNGs, including flowcharts (A), hierarchical diagrams (B), model graphs, hand-drawn sketches,
and mind maps (C).

network relational data. Our method is robust to the and a pipeline to extract DNGs.
edges of polylines and curves and supports three types (3) We put forward multiple evaluations to demon-
of nodes: rectangles, diamonds, and ellipses. We also strate that our method is effective and robust.
provide users with an interactive system. After the
user uploads a raster image, the system extracts its
underlying data. Users can redesign and modify data on
2 R ELATED W ORK
the system interface. The system can be applied in many Our work is mainly related to three aspects: chart data
scenarios, such as mind maps, flowcharts, E-R diagrams, extraction, chart redesign, and attention mechanism.
and hierarchical structure diagrams.
We perform ablation experiments on our semantic 2.1 Chart Data Extraction
segmentation model; the results show that our neural
Most visualization charts are represented as static im-
network greatly improves the ability to extract edges
ages. The purpose of chart data extraction is to extract
from graphs. We also have applied structural and per-
the original data from the bitmap. Different from some
ceptual similarity evaluations on the real corpus. After
figure extraction works [24] which focus on locating the
redrawing the extracted results with AntV [21], we
position of the image in the paper and figure classifica-
compare them with the input images. We use NetSim-
tion, chart data extraction focuses on extracting original
ile [22] to evaluate the structural similarity and the
data from figures. Chart data extraction helps designers
visual saliency map [23] to evaluate the perceptual
redesign and modify charts. There are many studies
similarity. We established an additional corpus with
on chart data extraction for simple charts such as bar
different resolutions and scales of DNGs to test the
charts, line charts, and pie charts. For bar graphs, most
robustness of our method for different scales of network
methods [9, 25, 26, 27] use the analysis of connected
structure and pixels. Our experimental results and user
components to extract bars in the charts. These meth-
study show that our method offers great application
ods are only effective for solid single-color bars. For
potential in the redesign and modification of DNGs. We
pie charts, the existing methods [9, 13, 25, 28, 29] are
share differences between deep learning and heuristics.
based on the assumption of a solid pie chart without a
We also illustrate the experience of visualization image
three-dimensional effect. These methods mainly detect
segmentation compared with natural images. Finally,
each slice by analyzing connected components [13] or
we discuss the limitations of this paper and future
deep learning methods [28, 29]. Other methods [9, 25]
work. Some of the limitations introduced by data-driven
use curve fitting to locate each pie slice. Few methods
models can be explored in correcting the chart norm.
support line charts because of the difficulty of extracting
Our contributions include three aspects:
lines. Some methods [30, 31] can only extract line charts
(1) We defined the problem of data extraction from with a single line. Some recent studies [6, 28, 29] used
DNGs. We verified the practical importance of this deep neural networks for object detection to detect
study in various application domains. bars, showing better performance than traditional im-
(2) We designed a state-of-the-art deep learning model age processing. The latest methods use deep neural

networks to learn marked patches in the chart [15] output value of these methods is one-dimensional (e.g.,
or detect key points in the chart [16], but they are the length of one line, the area of one circle, the angle
still not robust enough for multiple lines and lines of between two sides [47], or the number of nodes and
different thicknesses. Mao et al. [32] proposed a method edges [51]). However, our task needs to output the
of extracting data from meteorological facsimile charts. topological relationship of N 2 , where N represents the
Poco et al. [33] focused on heatmaps with legends. Wu number of nodes. There are also many visual attributes
et al. [34] introduced a method to handle contour maps. (e.g., color map, node shape, node position). After pre-
There are also some chart extraction frameworks and liminary experiments, we find that the semantic seg-
tools that are semiautomatic, such as ChartSense [14], mentation method is more promising. Neural networks
DataThief [35], iVoLVER [36], Plot Digitizer [37], Da- for semantic segmentation mainly include FCN [52], U-
gra [38] and Engauge Digitizer [39]. These tools use Net [20], and SegNet [53]. These models have achieved
human-supplied data to improve extraction accuracy, good results in the task of segmentation of natural
including chart types, key point positions, and line scenes. However, for chart images, the segmentation
positions. There are also some extraction works for task requires higher accuracy because tiny pixel errors
specific charts, such as ChemGrapher [40], a framework may result in large differences. Wang et al. [54] proposed
for extracting compounds. Some works [41, 42] are the application of the attention mechanism in computer
focused on science textbooks. The difference between vision to make the model learn the areas that are impor-
science textbooks and DNGs is that their objects are tant to reduce errors. SENet [55] was the first to propose
the targets of natural images, not visualization charts. the channel attention mechanism of the SE module from
Chen et al. proposed a model [43] to extract the timeline. the channelwise level, which can adaptively adjust the
They proposed the GrabCut [44] method to improve the characteristic response value of each channel. Oktay
segmentation results. For the chart types above, DNG et al. [56] designed a module with attention, which
has more attributes. In addition to the structural at- prevented target discontinuity in medical images, such
tribute (topological relationship), DNGs also have other as those of the pancreas, and achieved better results.
visual attributes (e.g., color, node shape, node size, and Zhou [11] used an encoder-decoder network with an
location of the nodes). attention mechanism to complete the task of extracting
bar graphs. In this paper, we experimentally find that
The type of visualization most relevant to DNGs is the
the attention mechanism applied to medical image seg-
network graph. OGR [17] used morphological methods
mentation can successfully improve the segmentation of
to extract network graphs. However, its network graph
DNG images. We also discuss the difference between
definition has many limitations, such as solid circle
natural images and visualization images in Sec. 8.
nodes and charts without text. These methods require
manual adjustment of the binarization threshold and
weak anti-noise ability. Users need to adjust the thresh- 2.2 Chart Redesign
olds of binarization and morphological operations, so There are many charts in the real world that do not
the robustness ability of the method is weak. OGER [18] conform to the principles of visualization design, and
improved the edge detection ability of OGR and can chart extraction enhances these visualizations. [33] We
recognize dashed lines. However, it still has the same mainly discuss three principles related to the design
limitations as OGR. Some studies [45, 46] focused on of DNGs. First, color plays an important role in chart
graphs that are close to our definition of DNGs, but their design. Wang et al. [57] applied a knowledge-based
input was a set of stroke vectors. They included more model for learning color rules to visualize images. The
prior knowledge than static bitmaps. VividGraph [1] W3C standard [58] also specifies color rules for images
used U-Net to segment network graph images. How- on web pages. The main purpose of some chart mining
ever, due to the limitation of their training data and work [33] is to recolor the chart. Second, different repre-
untuned CNN (convolutional neural network) architec- sentations of DNGs will also bring different perceptions
tures, VividGraph only works for graphs with circular to users. The entity relationship diagram [3] has been
nodes and straight edges but no text. Compared with widely approved in the field of database design, using
their work, we design our model and semantic parsing three types of nodes and solid lines to represent data
module for wider application and higher accuracy, as relationships. E-R diagrams help database designers
shown in Sec. 7. build databases faster. Buzan et al. [4] first proposed
Optimizing CNNs for feature extraction is an impor- the mind map, which helps users efficiently express
tant step in using deep learning methods. Haehn et their divergent thinking. Flow charts [2] can clearly
al. [47] proposed the possibility of using CNNs to extract demonstrate algorithms in computer programming. Our
data from charts. They evaluated the performance of system provides different retargeting schemes for differ-
four models, including MLP, LeNet [48], VGG [49], ent kinds of DNGs. Third, obtaining original data allows
and Xception [50]. Giovannangeli et al. [51] further further visualization past the raster image. Interactive
corroborated this view. However, their methods are not DNGs can ensure better user experience. There are
suitable for high-dimensional data such as DNGs. The some other types of chart mining work [59, 60, 61] that

[([240,340,197,26,0],
'Analytics &Decisions',0.97),
CTPN CRNN ([768,340,214,26,0],
'Theoretical&Empirical’,1.00),
([238,486,203,22,0],
'Data Transformations',0.98),
Text Data Extraction ([500,487,117,26,0],
'VIS2022',0.97),
{"nodes": [{
"id": 0,
([719,488,117,25,0],
"x": 247,
'Applications’,1.00),
"y": 402,
([200,576,276,26,0],
"color": "#ffffff",
'Representations&lnteraction',0.98),
"label": "Analytics &Decisions",
([760,577,197,28,0],
'Systems&Rendering’,1.00)] Semantic "fontsize": 19,
"type": "rect",
Text Data Parsing "w": 244,
"h": 63}, …… ],
"links": [{
Input "id": 0,
Neural Network "source": 2,
"target": 0,
"color": "#d9d5d0"}, ……],
"backcolor": "#ffffff"}
Output
Semantic Map

Attention Map
Graphical Data Extraction

Color Layout Data

Redesign Output Redesign and Modification Interactive System

Fig. 2. The main modules and processes of GraphDecoder.

converted raster charts into vector diagrams to make We constructed a semantic segmentation network
the charts interactive, such as intelligent question and with an attention mechanism. This network can
answer, adding table notes, and adding auxiliary lines. accurately locate pixels at which the nodes and
In this work, we provide methods and pipelines for edges are located and classify various types of
extracting data from DNGs and corresponding inter- nodes. By adding attention modules and improving
faces and applications for automatic redesign. However, the objective function, the network is robust to
we do not contribute the design principles of DNGs, and continuous curves and polylines.
we assume that the target design is prior knowledge. • Semantic Parsing Module. By analyzing the con-
nected components of the data obtained in the
3 OVERVIEW previous two modules, we obtain the approximate
original data.
Our goal is to extract the original data from DNGs. The
traditional methods [1, 18] are only suitable for certain
simple graphs. DNGs have more data attributes, com- 4 M ETHODS
plex data types, difficult edge recognition, and matching
problems between text and graphics. To solve these 4.1 Dataset
problems, we use the OCR system to preprocess raster Graph data extraction requires a large number of train-
images, use a semantic segmentation network with an ing datasets, but most works usually use small-scale
attention module to locate and classify each pixel, and datasets and keep the data private [6]. This is because
finally use the semantic parsing module to recover the most datasets require manual labeling [62], which re-
DNG. quires high-quality control. The more advanced the
We propose a framework named GraphDecoder, visual coding, the more difficult the annotation. To the
which automatically extracts original data from DNG best of our knowledge, there is no existing DNG dataset
images. Fig. 2 shows the pipeline of our framework. The for data extraction. Therefore, we built our dataset using
framework includes three components: the most popular visualization tools: D3 Library [63]
• Text Detection Module. To improve the performance in JavaScript and Matplotlib [64] and Skimage [65] in
of semantic segmentation, we first extract the text Python. The height and width of each image are defined
data in the chart. Through the OCR system, we as H and W , where H, W ∈ [320, 800]. To improve the
obtain the context and position of the text. We robustness of the model, we also enhanced our dataset,
remove the text area in the image and fill it with including cropping, stretching, adding noise, etc. Finally,
color blocks. Then, we can obtain the DNG image we selected 12, 000 images as the training set, 4, 000
without text. images as the validation set, and 4, 000 images as the test
• Segmentation Neural Network. The segmentation neu- set. Python and JS each draw one-third of the images,
ral network is the core module of our framework. and the remaining are data-augmented images.

We set the number of nodes at [0, 30], the thickness

Conv, ReLU Maxpool UpConv Attention Gate
of edges at [1, 10], and the number of edges at [0, 50].
The colors of the elements in the graphs are all random
between (0, 0, 0) and (255, 255, 255). The height and
weight of nodes are random between 5 and 100. We
assume that the types of nodes are rectangles, ellipses
and diamonds, and the types of edges are straight lines,
polylines and curves. All random attributes are evenly
distributed, ensuring that the training set covers most (a) Segmentation Neural Network
styles of DNGs. Some samples of our training dataset
are shown in the appendix.
𝐹𝐹1 × 𝐻𝐻1 × 𝑊𝑊1 𝐹𝐹3 × 𝐻𝐻1 × 𝑊𝑊1
1 × 1 𝐶𝐶𝐶𝐶𝐶 α
𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅 1 × 1 𝐶𝐶𝐶𝐶𝐶 𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆 𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅
1 × 1 𝐶𝐶𝐶𝐶𝐶 𝐻𝐻1 × 𝑊𝑊1 𝐻𝐻2 × 𝑊𝑊2
4.2 Text Detection Module 𝐹𝐹2 × 𝐻𝐻2 × 𝑊𝑊2

In the text detection module, we use CTPN [66] to locate

(b) Attention Gate
our textual area. CTPN combines a CNN with an LSTM Fig. 3. Our segmentation neural network structure is
network to effectively detect horizontally distributed based on the U-Net network, with the addition of the
text in complex scenes. CTPN uses a vertical anchor attention mechanism module and the modification of
regression mechanism to improve the performance of the loss function and the backbone network of feature
detecting small-scale text proposals. Different from the extraction.
text detection in natural images, most of the texts in
the chart originate in digital form. CTPN considers the
characteristics of this type of text compared with Faster-
RCNN [67]. The width of the vertical anchor is fixed at
16 pixels. The height varies from 11 pixels to 273 pixels
(divided by 0.7 each time) for a total of 10 anchors.
After obtaining the text area, we use CRNN [68]
to recover the text content. Inspired by Bosechen [69], (a) VividGraph [1] (b) Ours
we rotate each image by three angles, including 90◦ , Fig. 4. Examples of segmentation results for different
180◦ and 270◦ . Then, we perform text extraction to models. The challenges of model segmentation brought
obtain the most accurate results. In the end, we obtain a by DNG are shown in the box.
list of texts: {tx , ty , tw , th , ta , T ext, Conf idence}, where
tx , ty , tw , th , ta and T ext are the center coordinates,
width, height, angle and content of the text. Conf idence incorrectly identifies the polyline area as a rectangular
is the confidence of the text. We consider the text to be hollow node, as shown in the black box. Second, the
effective only when the confidence is greater than 0.95. borders of hollow nodes are misidentified as edges, as
By this method, we can suppress the background noise shown in the green box. Third, the edges are difficult to
in the chart. recognize. The broken pixel results are shown as a blue
We also preprocess the input image based on the text box. Inspired by pancreas segmentation [56] in medical
data, including removing the pixels in the text area and images, we improve the model so that it can more accu-
filling it with pixel values around the text. This method rately segment the above situation, as shown in Fig. 4(b).
reduces the unnecessary noise generated by the text in The pancreas has the characteristics of changeable shape
the graphics data in the semantic segmentation network. and area, and the phenomena of oversegmentation and
We perform the expansion processing of kernel = (2, 2) undersegmentation often occur [56]. The pancreas is
on the image, which solves the problem of edge pixel an organ that changes in shape and size with body
breakage when the edge width is 1. We input the pre- health. Oversegmentation refers to the segmentation of
processed image into the segmentation neural network organs that do not belong to the pancreas into the pan-
for the next step of extracting graphics data. creatic area. Undersegmentation represents insufficient
segmentation of the pancreas area. This is similar to the
situation in which DNG nodes and edges have variable
4.3 Segmentation Neural Network shapes and sizes, and the segmentation results are often
Compared with the network graphs defined in past discontinuous.
work [1], DNG introduces hollow nodes with more First, we found that noisy pixels causing targets to
shapes and nonstraight edges. This imposes greater be discontinuous were misidentified as the background
challenges to the segmentation model. We show some class. Most of the visualization images include more
examples of segmentation results from past models background pixels than object pixels, and we expect
trained on our dataset in Fig. 4(a). First, when mul- the model to focus more on visual codes rather than
tiple polylines enclose a rectangular area, the model the background. We add an attention gate [56] to the

network, a module that learns the target structures. As

shown in Fig. 3, F1 × H1 × W1 is the feature map output
by the deeper convolutional layer. F2 × H2 × W2 is the
lower-level feature map. F , H, and W represent features,
height, and width, respectively. After the operation (a) Attention Map (b) Ground Truth
shown in Fig. 3(b), we obtain the attention coefficient
α, α ∈ [0, 1]. We multiply it with the lower-level feature
map and output it to the upper decoder. α is defined as
follows:
(c) VividGraph (d) Ours
αil = σ2 ψ T σ1 WxT xli + WgT gi + bg + bψ

(1) Fig. 5. Due to the attention map and hybrid loss function,
where σ1 is the ReLU function, σ2 is the sigmoid func- our segmentation results are closer to the ground truth
tion, gi is the deeper-level feature map, xi is the lower- than are those of VividGraph. In the green box, the nodes
level feature map, WxT , WgT , ψ T are the convolution in our semantic map are continuous, while the nodes in
operations, and bg , bψ is the bias of the convolution. (c) are discontinuous. In the blue box, the pixels on the
Through the attention map shown in Fig. 5(a), we can edges of our semantic map are also more continuous,
determine that it depicts the border of the DNG, which which is conducive to semantic parsing.
makes the model focus more on the visual codes in
visualizations. The color opacity of the attention map
where γ is the Laplace smoothing factor, which prevents
represents the α value, which means that attention
the denominator from being zero when both yi and yi′
increases the weight of the target area and suppresses
are zero. At the same time, the risk of overfitting can
background noise.
be reduced. We set the value to 1. This loss function
During training, we found that the segmentation of
can also help us solve the problem of sample imbalance,
edge pixels is still difficult, whereas the results of other
thereby improving the performance of our network.
categories are sufficiently accurate. The reason for this is
Chen et al. [43] proposed that most models of nat-
that edge pixels in the training set are fewer than other
ural images are designed to use low-resolution, strong
pixels (node type, background type). We use a hybrid
semantics to improve detection, while the accuracy of
training loss to solve class imbalance:
segmentation suffers. We found that one way to improve
Lhybrid = Lwce + LDL (2) segmentation accuracy (especially crispy edges) under
the same resolution model is to reduce the loss of infor-
where Lwce is the weighted cross entropy with log- mation during the convolution process, since visualiza-
its [70], and LDL is the Dice loss [71]. tion images contain fewer features than natural images.
Given the ground truth of each pixel i category yi and They do not require as many convolutional layers for
the prediction of network yi′ , Lwce is defined as follows: feature extraction as natural images. We simplified the
X backbone from a four-layer encoder-decoder structure
Lwce = − (ωyi log (yi′ ) + (1 − yi ) log (1 − yi′ )) (3)
to three layers. To eliminate the impact of reducing
i
the number of encoder-decoder layers on large targets
where ω is the weight of each category. yi ∈ {0, 1, 2, 3, 4} such as nodes, we learn the structure of VGG19 [49] to
represents the background, rectangle, ellipse, diamond thicken the convolutional layers of the last two back-
and edge. Because most pixels in the graph are backbone layers. Our model achieves better performance
ground pixels, the edges occupy the smallest proportion than VividGraph [1]. More detailed comparison data are
of pixels. We set the weight of the background class to included for the ablation experiments.
0.8 and the weight of the edge class to 1.25; the weights
of other node classes remained 1. These hyperparame-
4.4 Semantic Parsing Module
ters are derived based on trial and error. This method
improves the detection performance of edge pixels while After obtaining the semantic map, we need to parse
maintaining node detection. the semantics to obtain complete information of the
The Dice loss [71] is based on the Dice coefficient, graph. To find node and edge candidates, we first extract
a metric that evaluates the similarity of two samples. connected components (CC) [72] from the semantic map,
Dice loss is a region-dependent loss, which is different as shown in Fig. 6(a).
from cross entropy loss. It helps the model segment the Second, we use mathematical morphology [73] on the
foreground from the image to prevent situations such CCs. We use the opening operation on node CC to
as the incomplete segmentation of the edge objects in eliminate small islands (e.g., Noise A and B in Fig. 6). We
Fig. 5(c). LDL is defined as follows: then use the closing operation on node CC to eliminate
small holes and fill gaps of the contours (e.g., Noise C in
X 2yi yi′ + γ

LDL = 1− 2 (4) Fig. 6). We also use the closing operation on edge CC to
i
yi + yi′2 + γ fuse narrow breaks and long thin gulfs (e.g., blue box

Edge Candidate Node Color Edge Color Traverse pixels around Node
(R, G, B) (R, G, B)
Gap Width
(Noise C) Height
Ellipse Candidate
(Noise A)
Coordinate Background Color Edge with only one Edge with two
Rectangle Node Candidate Edge Candidate
(x, y) confidence value (Noise) confidence values
(Noise B) (R, G, B)

(a) CC Analysis (b) Morphological Denoising (c) Relationship Calculation

Fig. 6. Semantic parsing process. In (a), we determine candidate CCs for edges and nodes by connected component
analysis and obtain (b) by morphological denoising. Thus, the visual attributes of the DNG are obtained through (b).
In (c), we traverse the pixels around the node to calculate the connection confidence and recover the connections.

in Fig. 4). Specifically, the opening operation is image 4.5 Large-Scale Graph Extraction
erosion followed by dilation with a disk structuring
element. The closed operation is the opposite. A disk The GraphDecoder supports general scale (node number
structuring element is similar to the convolution kernel, ≤ 30) DNGs. However, sometimes there are graphs with
and it is a small matrix p used for image processing. more than 30 nodes in the study, and these images often
We define its size as 31 Avg(Areaj ), where Avg(Areaj ) have a resolution greater than 1600×1600. This is a chal-
indicates the average area of all CCs. We obtain the lenging task for chart mining because the dimensions
attributes of edges and nodes from these candidates, as of the input in the semantic segmentation network are
shown in Fig. 6(b). The shape types of nodes (rectangle, stable, and high-resolution images will lose information
ellipse, diamond) are classified by our segmentation during the normalization process.
model. The color, coordinate and size are calculated by
CC. When the node color is different from the back-
ground color (e.g., Fig. 8), we consider the node to be Algorithm 1 Large-Scale Graph Mining Algorithm
solid. Conversely, (e.g., Fig. 2), we judge the node to be Input: CH×W : the large-scale graph image with high resolu-
a hollow node. tion
Output: y ′ : the semantic map of the large-scale graph
Third, we improved the node reconnection algorithm 1: Put CH×W into segmentation neural network
in VividGraph by imitating the way that humans per- ′
2: Obtain the semantic map yentire
ceive DNGs so that our pipelines can cope with non- 3: Cut CH×W into M pieces C m , m = 1, 2, ..., M
straight edges and nodes with various shapes. When 4: Put C m into segmentation neural network
′
5: Obtain the semantic map ym
people observe whether two nodes are connected, they ′ ′
6: Aggregate ym into ypiece
always look for a connecting edge around a node, so 7: Set α ∝ H × W
our heuristic algorithm mimics the process of human 8: for all Pixel
i ′do ′ ′
perception of connection. We consider the edges of three αypiece + (1 − α)yentire , ypiece (i) = 4
9: yi′ = ′ ′ ′
αyentire + (1 − α)ypiece , yentire (i) = 0, 1, 2, 3
different shapes: straight lines, polylines, and curves.
10: end for
They exhibit three common characteristics. First, they 11: return y ′ ;
are all connected to the sides of nodes. Second, they are
all continuous pixels rather than dashed lines. Third,
each edge connects two nodes. According to these com- To overcome this problem, we analyzed the charac-
mon characteristics, we design a general algorithm to teristics of the DNG. In the normalization process, edge
solve the topological analysis of DNGs. We traverse the information is easily lost because the edge occupies a
pixels around each node CC, as shown in Fig. 6. If a pixel few pixels, and the node information is still retained.
i belongs to a certain node, the confidence γpq that the Therefore, we can cut the image into pieces, extract
edge belongs to this node increases, where p, q represent them one by one, and then combine the semantic maps
Node p and Edge q. Since an edge always belongs to two of these pieces for semantic parsing. However, when
nodes, the two nodes with the highest confidence are cutting, the nodes are easily cut into different shapes,
connected. For Edge q, if max(γpq ) or submax(γpq ) is 0, we which causes errors in extraction. Therefore, we combine
consider the edge candidate to be noise, and we delete the semantic map obtained after the normalization of
it. We consider nodes p1 and p2 to be connected, where the whole image and the semantic map obtained after
p1 and p2 are the indices of max(γpq1 ) and submax(γpq2 ). cutting it into pieces for semantic parsing. We set a
We then assign the data obtained by the text detection confidence α ∈ [0.5, 1], which is proportional to the reso-
module to nodes or edges. For each detection result t in lution of the image. The confidence determines the bias
T extArrt with conf idence > 0.95, we find the nearest of the final semantic segmentation result. The details are
node p or edge q and assign t to it. shown in Algorithm 1, where H, W are the height and
The detailed process and pseudocode of the parsing weight of the image and y ′ represents the prediction of
module are attached in the appendix. the pixel category.

Duomo 8/1/17

ticket Date

8/18/17
Uffizi Florence
trip

mtb cycling activity Tuscany Zoe

sport Anna
swimming Holiday family

Julius

transportation

accommodation Finn
adults
plane
apartment
Tickets KLM

child
San Zanobi

(a) Input (b) Output1

(a) Input (b) Output
Duomo 8/1/17 Duomo 8/1/17

Fig. 7. The result of extracting data from the sketch. Uffizi

ticket

trip Florence
Date
8/18/17
Uffizi
ticket

trip Florence
Date
8/18/17

mtb cycling activity Tuscany Zoe cycling activity Tuscany Zoe

mtb

sport Anna sport Anna

Holiday family
swimming
Holiday family
swimming

Julius Julius

5 A PPLICATIONS adults
transportation

plane
accommodation Finn
adults
transportation

plane
accommodation

apartment
Finn

Tickets KLM apartment Tickets KLM

The graph decoder can be applied in many scenarios. child

San Zanobi
child
San Zanobi

In this section, we mainly introduce three applications: (c) Output2 (d) Output3
sketch transformation, raster image extraction and chart
redesign.

5.1 Sketch Transformation

The aim of visual engineers is to further reduce labor
costs in visual design; this is also the driving force
behind the development of visual analysis of production
links. In addition to the engineering cost, the design of (e) Output4 (f) Output5 (g) Output6
data visualization greatly affects the efficiency of data Fig. 8. The results of extracting the data from mind map
mining. and redesign. (b) is the extracted result. (c-d) are the
We used GraphDecoder to make some contributions results of recoloring. (e-g) are the results of relayout.
in this regard. Users can transfer hand-drawn visualiza-
tion works to our system, after which our system will
combine an intelligent layout and color map based on diagram can be obtained quickly, the time and labor
the algorithm and rules to quickly generate electronic costs can be reduced.
visualization works. This application can considerably Using graphs to represent algorithms is a way to
reduce the requirement for visual programming, which quickly understand them. The flowchart [2] is a com-
allows designers to free their hands and quickly realize monly used DNG to express algorithmic thinking. In-
ideas. formation on these algorithmic processes can be quickly
As shown in Fig. 7(a), the users can quickly pick up obtained through our system.
the paper and pen to draw the graph when an idea Fig. 1 shows some cases of extracting DNGs with
occurs to them. The users then only need to take a GraphDecoder, and more cases of E-R diagrams and
picture of the graph they have drawn, upload it to our algorithm flowcharts are shown in the appendix.
system, and obtain an editable and interactive DNG,
as shown in Fig. 7(b). This application is not only 5.3 Chart Redesign
convenient for professional visual designers, but also
enables users who do not have programming skills to An excellent layout and color scheme enable readers
create as much as they want. to quickly grasp information and features in the dia-
The current version of GraphDecoder only supports gram [33, 74, 75]. For example, mind map [4] is an ef-
graphical data transformation of sketches compared fective graphical tool for expressing divergent thinking.
with electronic images because the text detection mod- The mind map uses the characteristics of graphics and
ule cannot effectively recognize handwritten text. This texts, expresses the relationships of various topics with
limitation will also be discussed in Sec. 8. hierarchical diagrams, and establishes memory links
between topic keywords, graphics, colors, etc.
Fig. 8(a) is an image that depicts a family holiday
5.2 Raster Image Extraction arrangement. We input it into GraphDecoder to obtain
Visualization images spread on the internet generally its original data, which are generally output in the form
take the form of raster images, which can vividly express of a JSON file. We can directly redraw the mind map by
data characteristics. However, we always seek to obtain JSON to obtain the result shown in Fig. 8(b). In addition
their original data. For example, E-R diagrams [3] are to manually modifying this graph, we provide one-
very common among DNGs. After drawing the E-R click retargeting layouts and color schemes. The users
diagram, the database programmer needs to build the only need to input a static image, and then they can
database. At this time, if the original data of the E-R easily recolor the DNG with a default recommended

Color Position Node Connection Text

color scheme as shown in Fig. 8(c) or a dark theme as
shown in Fig. 8(d). They can also obtain a variety of Flowchart
visualization forms, as shown in Fig. 8(e-g).
The programmer of the database can quickly check the
Modeling
attributes of each entity and the connection relationship Graph
between the entities.
Mindmap

6 U SER S TUDY
Hand-drawn
Sketch
To expand the application of GraphDecoder, we con-
ducted a user study. We used system trials, user inter- 70 100 70 100 70 100 70 100 70 100
views and questionnaires to obtain user feedback, which (a) Scoring of extraction results
can improve our system. 2022/8/14 13:34

Procedure: Our user study includes five parts: (1) Manual Extraction Automatic Extraction
informed consent of users, (2) introduction of GraphDe-
Pair 1
coder, (3) system trials, (4) questionnaire, and (5) in-
terview. We first informed the participants that their
Pair 2
answers would be used for our research and then clas-
sified them by age and occupation. Then, we verbally Pair 3
introduced our system and methods to ensure that
every participant could understand the functions and Pair 4
usage of our system. We invited users to implement our
system for data extraction and transformation, then used Pair 5
questionnaires and interviews to obtain user feedback
−100 −50 0 50 100
and evaluate the actual effectiveness of our methods.
(b) Pairwise comparisons
Recruitment: We recruited 60 participants. Six of the Fig. 9. Questionnaire scoring results. (a) is the result of 8
participants were unable to understand our method scoring questions for the extraction results. (b) is the re-
due to capacity limitations, and we ultimately recov- sult of the 5 pairwise comparisons. Pairs 1-5 correspond
ered 54 valid user data (µage =25.9 years; 32 computer to Questions 12-16 in the questionnaire. Circles depict
professionals; 22 noncomputer professionals). Some of group averages (± 95% confidence intervals).
our participants were proficient with computers and
design, such as development engineers, visualization
designers, graphic designers and scientific researchers. Therefore, it is necessary to avoid traditional methods in
Other participants were noncomputer professionals, the pipeline as much as possible in the future.
such as teachers, middle school students, accountants, Pair Collection: Our pairwise comparisons neces-
and homemakers. sitated two groups. (1) Manual-extracted group: The
We recruited 5 additional designers (µage =25 years) designers used their skill for tools such as Photoshop,
working on visualization to collect the pairs in the PowerPoint, Processon, etc. to redraw and redesign
questionnaire for pairwise comparison. DNGs by observing the image. (2) Autoextracted group:
Questionnaire: We set up 13 questions in the ques- The designers used our system to extract and redesign
tionnaire, including 8 scoring questions for the extrac- DNGs.
tion results and 5 pairwise comparisons. We provided five designers with a DNG in bitmap
Scoring Questions: The hand-drawn sketches, mod- form and then asked them to redraw one DNG using
eling graphs, mind maps, and flowcharts each account their tools and another one using our system. We sup-
for a quarter of the scoring questions of the extraction plied a total of 5 DNGs and conducted a total of 25 trials.
results. Participants scored the extraction results based We calculated the time T , the number of left mouse
on five aspects: color, position, node (size, shape), con- clicks Nm , and the number of keyboard strokes Nk ,
nection relationship and text content, with a full score which were used for the two groups. The autoextracted
of 100. Since there are no handwritten data in our text group achieved T of 85 seconds, Nm of 32.4, and Nk
extraction model dataset, the hand-drawn sketch is not of 9. The manual-extracted group achieved T of 1165.8
scored for text. The final result is shown in Fig. 9(a). seconds, Nm of 465.2, and Nk of 424.8. This shows that
From the results, we can determine that our extraction our system reduces designer time for redesigning DNGs.
results can meet most of the needs of users. Most of the For each DNG given, we selected pairs with similar
average scores are above 90. The reason why the posi- and correct results and added them to the pair compar-
tion score is slightly lower than 90 is because we used a ison. There are five pairs in total.
small number of morphological methods in the semantic Pairwise Comparison: For the fairness of the com-
parsing module, which caused the node position to shift. parison, we randomly marked the images of the two

file:///E:/WeChat Files/ssc_55555555/FileStorage/File/2022-08/svg-scoring-result(2)(1).svg
© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: ASTAR. Downloaded on February 08,2023 at 04:50:38 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3225554
IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 10

and merging the same attributes of nodes and edges.

A B
These suggestions make our system friendlier and more
C durable.
For computer professionals, in addition to the first
two DNGs in their work, there are also many algo-
rithm flowcharts and modeling graphs. Even though
D
most of them have mastered programming and have a
certain design foundation, they primarily indicated that
E F
the interactive DNG extraction system can help them
complete their work faster. In the interview, three web
Fig. 10. The interface of our system. The left side of the
designers all expressed that the function of extracting
interface is the input, and the right side is the output.
DNGs and saving JSON files is very convenient. A data
Elements include (A) choose and upload, (B) save, (C)
engineer praised our redesign of the E-R diagram, which
interactive panel, (D) automatic layout option, (E) auto-
helped him understand the database structure more
matic color option, (F) mode switch.
quickly. It was said that our system should add more
automatic layout options and automatic color schemes
for different types of DNGs. We adopted the provided
groups as Picture 1 or Picture 2. The participants did not
opinions and increased the automatic layout options to
know which was the autoextracted result. We designed a
a total amount of ten. Two color matching strategies
rating slider with 100 on the far right and −100 on the
are provided. One traditional color matching strategy
far left. The larger the positive value was, the greater
is used to map the same color to the other color, and
the user preferred Picture 2 on the right. The larger
the other color matching strategy for modeling graphs
the absolute value of the negative number, the greater
and flowcharts is used to map nodes of the same shape
the user preferred Picture 1 on the left. Scoring sliders
to the same color.
are often employed for preference tasks [76] and have
Our revised system test version interface is shown in
high score reliability [77]. The slider was initially set
Fig. 10. Area A is the button for selecting and uploading
to 0, which made the questions unbiased [78]. When
the input image. Users can download the output in
quantifying the results, we set the score of the picture
area B. We provide three output formats, namely, PNG,
from the autoextracted group to a positive value. The
SVG, and JSON. Area C is our DNG interactive panel,
complete result is shown in Fig. 9(b). The scores are all
and users can customize almost all DNG attributes in
close to the neutral value in the five comparisons, indi-
this area. There are three modes: the default mode, the
cating that participants showed no preference between
node addition mode, and the edge addition mode, as
the two groups. This also shows that designers can use
shown in Area F. Area D and Area E are our options for
our systems to redesign DNGs similar to manual results
automatic color matching and automatic layout. More
in less time. The complete survey data related to the
demonstrations of our system have been attached in the
questionnaire have been attached in the appendix for
additional materials.
further research.
Interview: Each participant also experienced our beta
system. We then conducted interviews with users to 7 E VALUATION
improve our system. In the interview, we found that Our evaluation experiment includes two parts. First,
users of different professional groups have different we compared our segmentation module with a state-
requirements for the system. For noncomputer profes- of-the-art method [43] used in extracting timelines. We
sionals, the types of DNGs they are most concerned also conducted ablation experiments on our semantic
about are hand-drawn sketches and mind maps. Many segmentation network to prove the effectiveness of our
of them do not master visual programming, especially modifications to U-Net in VividGraph [1]. Second, we
older users. A woman who has been engaged in middle conducted evaluations of structural similarity and per-
school education for more than ten years said that ceptual similarity on the real corpus and the virtual
the data transformation of hand-drawn sketches saves corpus to prove the effectiveness of our method.
her a lot of time compared with the drawing module We deploy our pipeline on a PC with Intel Core i7-
in Microsoft Office. Another lady in the company’s 9700F, NVIDIA GeForce 1660 Ti, and 16 GB of memory.
financial management mentioned in the interview that The deep learning framework is implemented based on
mind maps are often used in daily life. However, those Keras [79].
who are not design professionals will generally search
for existing mind maps on the internet to imitate. Our
system can help to quickly modify the material in the 7.1 Segmentation Module Evaluation
form of raster images on the internet. Based on the feed- The accuracy of segmentation is a key component of
back we received, we have also improved the usability the accuracy of extracting DNG data. We designed a
of our tool, including by enlarging the text edit box special segmentation model for DNGs. Some work has

TABLE 1
Evaluation results of semantic segmentation model

Methods FIoU↑ MIoU↑ Back IoU↑ Rect IoU↑ Elli IoU↑ Dia IoU↑ Edge IoU↑
Chen et al. [43] 0.9529 0.5242 0.9773 0.4114 0.5612 0.5435 0.1276
VividGraph [1] 0.9895 0.9407 0.9944 0.9762 0.9859 0.9537 0.7831
Attention 0.9908 0.9479 0.9951 0.9801 0.9876 0.9659 0.8107
Lhybrid 0.9902 0.9441 0.9948 0.9791 0.9862 0.9585 0.8019
Backbone 0.9901 0.9439 0.9947 0.9821 0.9863 0.9659 0.7906
Ours Attention+Lhybrid 0.9920 0.9536 0.9957 0.9866 0.9910 0.9712 0.8236
Attention+Backbone 0.9904 0.9454 0.9948 0.9820 0.9855 0.9689 0.7959
Backbone+Lhybrid 0.9908 0.9465 0.9952 0.9793 0.9881 0.9627 0.8070
Attention+Backbone+Lhybrid 0.9924 0.9562 0.9959 0.9857 0.9908 0.9717 0.8367

also used segmentation to extract other types of graphs. which is a method widely used to compare the similar-
Chen et al. proposed a model [43] to extract the timeline. ity of two network structures. NetSimile is defined as
Since the model does not include the algorithm of the follows:
node connection, we only compare the segmentation n
i i

X Sigpre − Siggt
modules. For a fair comparison, we train this module N etSim(P re, Gt) = (6)
i + Sig i
Sigpre
on our training dataset. i=1 gt
We conducted ablation experiments on our dataset i
where Sigpre is the signature vector of the extracted
to prove the effectiveness of our contribution to the i
network structure and Siggt is the signature vector of
semantic segmentation model. We made three improve-
the ground truth. The signature vector contains 35-
ments to the model, including adding a module with an
dimensional features, including node in and out degree,
attention mechanism, using a hybrid loss function, and
clustering coefficient, and ego. This method measures
simplifying the backbone of the model. We evaluate our
the similarity of the network structure by comparing the
semantic segmentation model using intersection over
Canberra distance of two signature vectors. The lower
union (IoU), mean IoU (MIoU), and frequency-weighted
NetSimile is, the higher the structural similarity. The
IoU (FIoU), which are defined as follows:
 T ′ advantage of using this metric to evaluate structural
y y

 IoU = yii S yi′ similarity is that there is no need for one-to-one corre-
 i
spondence of node numbers. If a node is not detected,
 Pk
1 pii
M IoU = k+1 i=0
Pk
pij +
P k pji −pii (5)

 1
j=0
Pk j=0
pii
this method can also correctly measure the similarity
 F IoU = P P
 ,
k
i=0
k
j=0 pij i=0
Pk P k
j=0 pij + j=0 pji −pii
of the two networks. To demonstrate NetSimile more
intuitively, we compare the network structure of the
where pij represents the situation. The ground truth
ground truth with the network structure of the same
is i, and the prediction is j. k is the number of pixel
number of nodes but with no edges to obtain the largest
classifications. We set k = 5 in our task.
NetSimile N etSimmax . Then, the structural similarity
IoU represents the overlap area ratio of the ground-
StruSim is defined as follows:
truth bounding box and predicted bounding box, which
is usually employed for evaluation [80, 81]. Due to N etSim(P re, Gt)
StruSim(P re, Gt) = (1− )×100% (7)
the characteristics of the graphs, the background pixels N etSimmax
account for a large proportion, so MIoU is more suitable
where StruSim ranges from 0 to 100%. StruSim =
for evaluating our model than is FIoU. The experimental
100% means that the two network structures are the
results are shown in Table 1. From the results, we can
same.
determine that the IoU of each category has improved,
The other method is to compare the visual saliency
especially the IoU of the edge. Therefore, our modi-
map of the input images and the redrawn images. The
fications to the model have effectively improved the
information of nodes and links will affect the metrics,
semantic segmentation capabilities of graphs (especially
including size, location, color, etc. We used AntV to
edges). This is because the attention mechanism we
redraw the extracted data into a DNG image. We then
introduced can solve the oversegmentation and under-
used the visual saliency model proposed by Bylinskii
segmentation problems caused by DNG features, as
et al. [23] to output the visual saliency map of the two
shown in Fig. 4.
images that need to be compared. This visual saliency
model can output the important areas of each graph
7.2 Structural and Perceptual Evaluation and assign all pixels a score from 0 to 255. We evaluate
We propose two methods to evaluate the effectiveness of the perceptual similarity by comparing the difference
our extracted results. The first method is NetSimile [22], in the average scores of the two images. Similarly,

© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: ASTAR. Downloaded on February 08,2023 at 04:50:38 UTC from IEEE Xplore. Restrictions apply.
G 教程 andAPI
搜索… This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited 图
content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3225554
搜索
小提琴图
JavaScript Data
IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 12
312 // 小提琴图用 KDE transform 提供数据

场景案例 313
314
const dv1 = new DataView().source(data).transform({
type: 'kde',
OGER with
GraphDecoder VividGraph 315 extent: [-1, 2], // 采样范围
交互语法 Semantic Parsing
316
system does not have a gradient color option, and the
fields,
150%
317 graph is filled with a single color. However, we can find
as: ['key', 'y', 'size'], // 生成的统计字段：字段名、采样点、采样点对应

折线图 100%
318
319
groupBy: ['Species'],
that our methods still correctly identify the connection
minSize: 0.01, // 小于这个值的概率将被忽略
320 });
面积图 50% 321 relationships of these DNGs from StruSim. The graph
// 需要根据分组提供分位值等统计数据，所以提前拍平数据
322 const dv2 = new DataView().source(data);
decoder can handle most DNGs, especially when DNGs
柱状图 0%
323
324
dv2.transform({
include noise, nonsolid nodes, and different edge sizes,
type: 'fold',
325 fields,
条形图 -50% 326 which are all difficult for morphological methods.
});
StruSim PerSim 327 // 计算 95% 分位值，用于画 95% 分位线
In addition, we found that image resolution and struc-
饼图 328 const dv3 = new DataView().source(dv2).transform({
tural complexity are important factors affecting extrac-
Fig. 11. The results of evaluation experiments on a real
点图 data corpus. The circles show the median values. Error
tion accuracy. Therefore, we constructed an additional
set of evaluation data corpora with different resolutions
雷达图 bars depict the interquartile range. and numbers of nodes. To ensure that the evaluation
漏斗图 corpora and the training dataset do not overlap, we
chose AntV as the visualization tool for the additional
热力图 we also divide the difference between this score and
corpus. Each image in the corpus has a different layout,
the ground truth score into a perceptual similarity
箱型图 P erSim ∈ [0, 100%]. P erSim is defined as follows:
different colors, and different node sizes and edge types.
We chose three resolutions of 640 × 640, 960 × 960,
烛形图 |Salipre − Saligt | and 1280 × 1280 and three types of node numbers
P erSim(P re, Gt) = 100% − (8)
仪表盘 Saligt around 10, 20, and 30. We conducted experiments for
comparison with OGER and VividGraph on nine sets of
地图 Saligt is the visual saliency map score of the ground corpora. The experimental data are shown in Fig. 12.
truth, and Salipre is the visual saliency map score of
分面 Our method achieves state-of-the-art results in terms
our extraction results. of all three metrics. OGER with semantic parsing is a
关系图 We collected 100 images from E-Charts, D3, AntV, complete heuristic method, so its results change with
Google, and Xmind, and user hand-drawn sketches
组件使用 the resolution and the number of nodes. Although we
were used as our evaluation dataset. We compared set optimal thresholds for each set of OGER to enable
其他图表 our method with VividGraph [1] and OGER [18] (an smooth binarization and skeletonization, our results are
improved version of OGR [17]). VividGraph does not
大数据量 also superior to it. This is because OGER’s segmentation
have a text extraction module, so we use our text method does not work for hollow nodes. The segmen-
其他图表 detection module to extract texts for fair comparison. tation model of VividGraph causes the phenomena of
VividGraph
2 uses an untuned segmentation model, and oversegmentation and undersegmentation, and the node
its reconnection algorithm is not suitable for the con- reconnection algorithm can only recover the connections
nection of nonstraight edges. Our real dataset contains of straight edges, so StruSim is lower. These results
many DNGs with hollow nodes and nonstraight edges confirm that our methods can extract DNGs of different
(as shown in Fig. 1), which results in the poor StruSim scales.
achieved by VividGraph. OGER has many limitations.
OGER cannot extract texts and classify nodes. It cannot
analyze the shape of nodes and edges, nor can it match 7.3 Time Performance
textual data and graphical data. We added our semantic Our method also consumes less time than OGER and
parsing module to enable a fair comparison. This also VividGraph. We first focus on real-world datasets.
enables OGER to handle connections of nonstraight GraphDecoder spent an average of 3.115 (95% CI:
edges in the dataset. However, compared with end-to- 3.108 − 3.122, p<0.01) seconds on this corpus, while
end methods, OGER still has many limitations in com-
antv.vision/zh/examples/other/other#violin OGER spent an average of 6.585 (95% CI: 6.507 − 6.663,
parison. Since OGER is entirely a heuristic method, we p<0.01) seconds; VividGraph spent an average of 3.184
need to adjust the segmentation threshold for different (95% CI: 3.157−3.210, p<0.01) seconds. Our methods are
resolutions and different scales of network graphs and faster than OGER because our segmentation uses deep
binarization parameters for different colormaps. We also learning methods. This time gap increases with the im-
resize the images to a suitable resolution and set the age resolution. Our time performance is also improved
proper threshold. These operations are not counted in over VividGraph because our semantic parsing module
the time of the OGER. traverses each node: the corresponding time complexity
The evaluation results are shown in Fig. 11. Compared is O(N ). VividGraph traverses every two nodes, with
with VividGraph and OGER, our method achieves the complexity O(N 2 ).
best results for both the StruSim and P erSim metrics. We also evaluated the time performance based on
The results show that our method can extract data additional datasets. As shown in Fig. 12(c), we found
from DNGs in the real world with better quality than that at the same resolution, when the number of nodes
other state-of-the-art methods. The error of P erSim is increases, the average time spent by all three meth-
primarily due to the rich styles of many DNGs in the real ods increases slightly. However, when the resolution
world, such as gradient colors. When we redraw, our increases, the average time consumption of the OGER

Number
Method Resolution of Nodes Structural Similarity (a) Perceptual Similarity (b) Time Performance (c)
Node≤10 98.65% 94.00% 2.99s
640×640 Node≤20 98.47% 95.83% 3.06s
Node≤30 98.32% 96.81% 3.14s
Node≤10 99.08% 94.62% 3.04s
GraphDecoder 960×960 Node≤20 99.74% 96.32% 3.02s
Node≤30 99.90% 97.66% 3.11s
Node≤10 98.10% 95.17% 3.03s
1280×1280 Node≤20 99.63% 97.29% 3.10s
Node≤30 99.90% 98.09% 3.11s

Node≤10 62.75% 55.12% 2.86s

640×640 Node≤20 69.13% 70.79% 2.98s
Node≤30 71.68% 81.79% 3.10s
Node≤10 59.83% 57.65% 6.52s
OGER with 960×960 Node≤20 59.31% 71.79% 6.46s
Semantic Parsing Node≤30 58.87% 83.85% 6.54s
Node≤10 59.75% 54.29% 12.33s
1280×1280 Node≤20 61.93% 70.41% 11.35s
Node≤30 63.38% 81.13% 13.18s

Node≤10 48.41% 46.50% 3.37s

640×640 Node≤20 28.82% 67.20% 3.42s
Node≤30 15.86% 71.30% 3.57s
Node≤10 49.72% 46.91% 3.47s
VividGraph 960×960 Node≤20 30.06% 64.18% 3.50s
Node≤30 22.76% 68.77% 3.52s
Node≤10 48.88% 45.70% 3.35s
1280×1280 Node≤20 31.96% 63.58% 3.62s
Node≤30 24.09% 67.86% 3.64s

0% 25% 50% 75% 100% 0% 25% 50% 75% 100% 0 5 10 15

Fig. 12. The results of evaluation experiments on additional sets of evaluation data corpora. In (a), we used structural
similarity to evaluate the extraction accuracy of relationships. In (b), we used perceptual similarity to evaluate the
extraction accuracy of various visual attributes. In (c), we show the time spent by each method. The white values
represent the mean values. Error bars depict ± 95% CIs for mean values.

increases significantly, while the average time consump- In our semantic parsing module, we use heuristic
tion of the others remains stable. This illustrates the time methods to analyze semantic maps. We did not use
performance advantage of the deep learning module. deep learning [11, 41, 42] to extract relationships for
In addition, our time performance is better than that two reasons. First, the heuristic method can already
of VividGraph under the same conditions. This again effectively analyze the relationship, as shown in the
proves that our method is more efficient than Vivid- structural evaluation in section 7. Second, we extract
Graph in computing node relations. many attribute values from the semantic maps, includ-
ing relationships, positions, colors, sizes, and shapes. A
8 D ISCUSSION single deep learning model cannot accurately regress too
many attribute values. If we design a model for each
8.1 Deep Learning or Heuristic Methods
attribute, the pipeline will be redundant. For this part,
The popularity of deep learning brings tremendous the heuristic method is simple and effective.
changes to visualization research [82]. We believe that
deep learning should be used in scenarios for which
heuristic methods have limitations. If heuristic methods 8.2 Visualizations or Natural Images
work well, then deep learning is of little significance. There are works [6, 43] which propose chart extraction in
We designed a deep learning model to segment nodes relation to the difference between graphic and natural el-
and edges. Compared with heuristic methods such as ements. Most of the existing CNN models and attention
OGR [17] and OGER [18], deep learning models have mechanisms are designed based on natural images. On
better robustness and time performance. First, heuristic the one hand, they are concerned with large objects such
methods need a suitable threshold for binarization, but as people, trees, and vehicles. On the other hand, they
there is no perfect threshold for the various colormaps segment low-level semantics. For example, a person is
of real-world DNGs to distinguish all targets and back- recognized, but the result is slightly wider and shorter
grounds. Second, deep learning solves the limitation of than the ground truth. This has no effect on the desired
the heuristic method for hollow nodes because heuristic outcome. However, the situation is different for charts,
methods encounter difficulty distinguishing the edges especially DNGs.
and outlines of hollow nodes. Third, existing heuristic Visualization images should be more focused on the
methods for segmenting network graphs do not include ambiguity of basic geometric shapes. DNGs have many
node classification modules. They are only suitable for edges and nodes with various shapes. Since most of
graphs with circular solid nodes. If we attempt to the targets of DNG are basic geometric shapes, they are
complete the node classification, heuristic segmentation easily confused with hollow nodes due to the ambiguity
methods need to add more complex thresholds and caused by edges during segmentation. When design-
rules. However, our deep learning model has integrated ing a segmentation model for visualization images, we
these. suggest paying attention to the undersegmentation and
file:///E:/WeChat Files/ssc_55555555/FileStorage/File/2022-08/svg-performance(1)(1).svg

oversegmentation categories of the attention region and

loss function. In DNGs, both of these categories are
edges.
Additionally, due to the difficulty of labeling and
the lack of public corpora, the training sets for most
data-driven visualization work [1, 11, 83] are synthetic
datasets. The scale of such datasets is smaller than that
of common datasets of natural images (e.g., COCO [84]). (a) Input (b) Segmentation Result
Since medical image segmentation also requires high-
Fig. 13. A case of segmentation fault. The boundaries of
level semantics and is implemented with the character-
the red nodes are incorrectly identified as ellipses.
istics of small datasets, we found that some methods can
be borrowed for visualization image segmentation.
The recovery of various visual attributes from visual-
izations is actually a multiple perception task [47, 85]. an ellipse [2]. This phenomenon indicates that our seg-
Bar graphs require the perception of area or length, mentation model not only learns local features but also
pie charts require perception of angle, and line graphs learns global features of various DNGs. We can exploit
require the perception of the direction and position this limitation to discover irregular DNG images in the
of lines. DNGs necessitate more tasks, including the future.
determination of area, shape, color, text, relationship Third, we retained the heuristic method in the se-
and position. Every perception task requires high-level mantic parsing module. We attempted to transfer the
semantics. end-to-end method [11] of other chart types to DNGs
To solve complex perceptual tasks, we believe that but found that it did not function successfully. This
imitating the process of human understanding visual- is because the DNGs contain excessive data attributes.
ization can be considered. The models of Chen et al. [43] The heuristic semantic parsing module ensures the ro-
focus on global information and local information of bustness and correctness of the original data obtained
timelines, respectively. Our pipeline also divides the from the semantic map but also increases the time
complex perception task of DNG into two steps: first consumption. At the same time, the heuristic module
obtaining global information through a segmentation cannot effectively solve the ambiguity of cross-edges.
model and then recovering local information through We can use the direction vector judgment [18] to solve
semantic parsing. a part of the cross-edge problem, but the overall effect
is not good enough. We plan to improve the semantic
parsing module.
8.3 Limitations and Future Work
Finally, although the current OCR system is mature,
The current version of GraphDecoder also has certain there are still some limitations. For example, it is difficult
limitations. to extract hand-drawn text, which means that we can
First, the data-driven segmentation module also only handle hand-drawn DNGs without text. However,
makes the model constrained by the training set. The this is a common problem in the OCR field. We will
semantic manual annotation of DNGs is a complicated replace the text detection module with a better OCR
task, so we designed a synthetic dataset as a training module in the future.
set. We have strived to cover various elements of dif-
ferent DNGs, including visualization tools, node shape,
node size, node distribution, edge type, edge thickness,
colormap, and text. The DNGs that the current ver- 9 C ONCLUSION
sion of GraphDecoder can handle represent 85.19% of
participants in our user study. However, the styles of We proposed a method to extract the original data
DNG in the real world are endless. We introduce noise of DNG images. Our system, GraphDecoder, can be
into the training set via data augmentation, but with no applied for sketches, raster image extraction and chart
elements (e.g., chart title, chart annotation) other than design. Our method is suitable for mind maps, E-R di-
the DNG content. For the removal of these elements, agrams, flowcharts, and tree diagrams. We designed an
known methods [24, 33] can be used in combination. attention-aware semantic segmentation neural network
We plan to make the code of DNG generation and our to improve the ability to extract DNGs. Our evalua-
datasets available for the visualization community and tion of a real data corpus and additional data corpora
construct a larger dataset of DNGs in the future. demonstrates that GraphDecoder can efficiently and cor-
Second, a surprising finding is that rectangles or rectly extract DNGs of different scales, resolutions, and
rounded rectangles may be misidentified as ellipses types. We also shared our experience in data extraction
(Fig. 13) when the input DNG is a flowchart because from advanced and complex graphs such as DNGs and
the flowcharts in the training set are regular, which discussed how to improve and optimize our work in the
means that the first node of a standard flowchart is future.

R EFERENCES in Proceedings of the 30th Annual ACM symposium on user interface

software and technology, 2017, pp. 57–69.
[1] S. Song, C. Li, Y. Sun, and C. Wang, “Vividgraph: Learning to [24] J. Chen, M. Ling, R. Li, P. Isenberg, T. Isenberg, M. Sedlmair,
extract and redesign network graphs from visualization images,” T. Moller, R. S. Laramee, H.-W. Shen, K. Wunsche et al., “Vis30k: A
IEEE Transactions on Visualization and Computer Graphics, 2022. collection of figures and tables from ieee visualization conference
[2] I. Nassi and B. Shneiderman, “Flowchart techniques for struc- publications,” IEEE Transactions on Visualization and Computer
tured programming,” ACM Sigplan Notices, vol. 8, no. 8, pp. 12– Graphics, 2021.
26, 1973. [25] A. Balaji, T. Ramanathan, and V. Sonathi, “Chart-text: A fully au-
[3] P. P.-S. Chen, “The entity-relationship model—toward a unified tomated chart image descriptor,” arXiv preprint arXiv:1812.10636,
view of data,” ACM transactions on database systems (TODS), vol. 1, 2018.
no. 1, pp. 9–36, 1976. [26] W. Dai, M. Wang, Z. Niu, and J. Zhang, “Chart decoder: Gener-
[4] T. Buzan, Use both sides of your brain. EP Dutton New York, 1983. ating textual and numeric information from chart images auto-
[5] Y. Liu, X. Lu, Y. Qin, Z. Tang, and J. Xu, “Review of chart recogni- matically,” Journal of Visual Languages & Computing, vol. 48, pp.
tion in document images,” in Visualization and Data Analysis 2013, 101–109, 2018.
vol. 8654. International Society for Optics and Photonics, 2013, [27] R. A. Al-Zaidy and C. L. Giles, “A machine learning approach for
pp. 384–391. semantic structuring of scientific charts in scholarly documents.”
in AAAI, 2017, pp. 4644–4649.
[6] K. Davila, S. Setlur, D. Doermann, U. K. Bhargava, and V. Govin-
daraju, “Chart mining: a survey of methods for automated chart [28] J. Choi, S. Jung, D. G. Park, J. Choo, and N. Elmqvist, “Visual-
analysis,” IEEE transactions on pattern analysis and machine intelli- izing for the non-visual: Enabling the visually impaired to use
gence, 2020. visualization,” in Computer Graphics Forum, vol. 38, no. 3. Wiley
Online Library, 2019, pp. 249–260.
[7] P. Zhang, C. Li, and C. Wang, “Viscode: Embedding information
[29] X. Liu, D. Klabjan, and P. NBless, “Data extraction from charts
in visualization images using encoder-decoder network,” IEEE
via single deep neural network,” arXiv preprint arXiv:1906.11906,
Transactions on Visualization and Computer Graphics, 2020.
2019.
[8] J. Fu, B. Zhu, W. Cui, S. Ge, Y. Wang, H. Zhang, H. Huang,
[30] M. K. I. Molla, K. H. Talukder, and M. A. Hossain, “Line
Y. Tang, D. Zhang, and X. Ma, “Chartem: Reviving chart images
chart recognition and data extraction technique,” in International
with data embedding,” IEEE Transactions on Visualization and
Conference on Intelligent Data Engineering and Automated Learning.
Computer Graphics, 2020.
Springer, 2003, pp. 865–870.
[9] M. Savva, N. Kong, A. Chhajta, L. Fei-Fei, M. Agrawala, and
[31] V. K. Reddy and C. Kaushik, “Image processing based data ex-
J. Heer, “Revision: Automated classification, analysis and re-
traction from graphical representation,” in 2015 IEEE International
design of chart images,” in Proceedings of the 24th annual ACM
Conference on Computer Graphics, Vision and Information Security
symposium on User interface software and technology, 2011, pp. 393–
(CGVIS). IEEE, 2015, pp. 190–194.
402.
[32] K. Mao, X. Chen, K. Zhu, D. Hu, and Y. Li, “A method to extract
[10] J. Yuan, C. Chen, W. Yang, M. Liu, J. Xia, and S. Liu, “A survey of essential information from meteorological facsimile charts,” In-
visual analytics techniques for machine learning,” Computational ternational Journal of Pattern Recognition and Artificial Intelligence,
Visual Media, vol. 7, no. 1, pp. 3–36, 2021. vol. 33, no. 01, p. 1954001, 2019.
[11] F. Zhou, Y. Zhao, W. Chen, Y. Tan, Y. Xu, Y. Chen, C. Liu, and [33] J. Poco, A. Mayhua, and J. Heer, “Extracting and retargeting
Y. Zhao, “Reverse-engineering bar charts using neural networks,” color mappings from bitmap images of visualizations,” IEEE
Journal of Visualization, pp. 491–435, 2021. transactions on visualization and computer graphics, vol. 24, no. 1,
[12] R. Burns, S. Carberry, and S. Elzer Schwartz, “An automated pp. 637–646, 2017.
approach for the recognition of intended messages in grouped [34] R.-Q. Wu, X.-R. Cheng, and C.-J. Yang, “Extracting contour
bar charts,” Computational Intelligence, vol. 35, no. 4, pp. 955–1002, lines from topographic maps based on cartography and graphics
2019. knowledge,” Journal of Computer Science and Technology, vol. 9,
[13] P. De, “Automatic data extraction from 2d and 3d pie chart im- no. 02, pp. 58–64, 2009.
ages,” in 2018 IEEE 8th International Advance Computing Conference [35] A. Flower, J. W. McKenna, and G. Upreti, “Validity and reliability
(IACC). IEEE, 2018, pp. 20–25. of graphclick and datathief iii for data extraction,” Behavior
[14] D. Jung, W. Kim, H. Song, J.-i. Hwang, B. Lee, B. Kim, and J. Seo, modification, vol. 40, no. 3, pp. 396–413, 2016.
“Chartsense: Interactive data extraction from chart images,” in [36] G. G. Méndez, M. A. Nacenta, and S. Vandenheste, “ivolver:
Proceedings of the 2017 chi conference on human factors in computing Interactive visual language for visualization extraction and re-
systems, 2017, pp. 6706–6717. construction,” in Proceedings of the 2016 CHI Conference on Human
[15] N. Siegel, Z. Horvitz, R. Levin, S. Divvala, and A. Farhadi, “Fig- Factors in Computing Systems, 2016, pp. 4073–4085.
ureseer: Parsing result-figures in research papers,” in European [37] A. J. Kadic, K. Vucic, S. Dosenovic, D. Sapunar, and L. Puljak,
Conference on Computer Vision. Springer, 2016, pp. 664–680. “Extracting data from figures with software was faster, with
[16] J. Luo, Z. Li, J. Wang, and C.-Y. Lin, “Chartocr: Data extraction higher interrater reliability than manual extraction,” Journal of
from charts images via a deep hybrid framework,” in Proceedings clinical epidemiology, vol. 74, pp. 119–123, 2016.
of the IEEE/CVF Winter Conference on Applications of Computer [38] “Dagra data digitizer,” https://www.datadigitization.com/.
Vision, 2021, pp. 1917–1925. [39] M. Mitchell, B. Muftakhidinov, T. Winchen, A. Wilms, B. van
[17] C. Auer, C. Bachmaier, F. J. Brandenburg, A. Gleißner, and J. Reisl- Schaik, badshah400, Mo-Gul, T. G. Badger, Z. Jedrzejewski-
huber, “Optical graph recognition,” in International Symposium on Szmek, kensington, and kylesower, “markummitchell/engauge-
Graph Drawing. Springer, 2012, pp. 529–540. digitizer: Nonrelease,” Jul. 2020. [Online]. Available: https:
[18] R. Opmanis, “Optical graph edge recognition.” in VISIGRAPP (3: //doi.org/10.5281/zenodo.3941227
IVAPP), 2018, pp. 184–191. [40] M. Oldenhof, A. Arany, Y. Moreau, and J. Simm, “Chemgrapher:
[19] E. Brynjolfsson and K. McElheran, “The rapid adoption of data- optical graph recognition of chemical compounds by deep learn-
driven decision-making,” American Economic Review, vol. 106, ing,” Journal of Chemical Information and Modeling, vol. 60, no. 10,
no. 5, pp. 133–39, 2016. pp. 4506–4517, 2020.
[20] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional [41] A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and
networks for biomedical image segmentation,” in International A. Farhadi, “A diagram is worth a dozen images,” in European
Conference on Medical image computing and computer-assisted inter- conference on computer vision. Springer, 2016, pp. 235–251.
vention. Springer, 2015, pp. 234–241. [42] D. Kim, Y. Yoo, J.-S. Kim, S. Lee, and N. Kwak, “Dynamic
[21] “Antv,” https://g6.antv.vision/. graph generation network: Generating relational knowledge from
[22] M. Berlingerio, D. Koutra, T. Eliassi-Rad, and C. Faloutsos, diagrams,” in Proceedings of the IEEE Conference on Computer Vision
“Netsimile: A scalable approach to size-independent network and Pattern Recognition, 2018, pp. 4167–4175.
similarity,” arXiv preprint arXiv:1209.2684, 2012. [43] Z. Chen, Y. Wang, Q. Wang, Y. Wang, and H. Qu, “Towards au-
[23] Z. Bylinskii, N. W. Kim, P. O’Donovan, S. Alsheikh, S. Madan, tomated infographic design: Deep learning-based auto-extraction
H. Pfister, F. Durand, B. Russell, and A. Hertzmann, “Learning of extensible timeline,” IEEE transactions on visualization and
visual importance for graphic designs and data visualizations,” computer graphics, vol. 26, no. 1, pp. 917–926, 2019.

[44] C. Rother, V. Kolmogorov, and A. Blake, “” grabcut” interactive [67] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-
foreground extraction using iterated graph cuts,” ACM transac- time object detection with region proposal networks,” Advances
tions on graphics (TOG), vol. 23, no. 3, pp. 309–314, 2004. in neural information processing systems, vol. 28, pp. 91–99, 2015.
[45] J. Wu, C. Wang, L. Zhang, and Y. Rui, “Offline sketch parsing [68] B. Shi, X. Bai, and C. Yao, “An end-to-end trainable neural net-
via shapeness estimation,” in Twenty-Fourth International Joint work for image-based sequence recognition and its application to
Conference on Artificial Intelligence, 2015. scene text recognition,” IEEE transactions on pattern analysis and
[46] X.-L. Yun, Y.-M. Zhang, J.-Y. Ye, and C.-L. Liu, “Online hand- machine intelligence, vol. 39, no. 11, pp. 2298–2304, 2016.
written diagram recognition with graph attention networks,” in [69] F. Böschen, T. Beck, and A. Scherp, “Survey and empirical com-
International Conference on Image and Graphics. Springer, 2019, pp. parison of different approaches for text extraction from scholarly
232–244. figures,” Multimedia Tools and Applications, vol. 77, no. 22, pp.
[47] D. Haehn, J. Tompkin, and H. Pfister, “Evaluating ‘graphical per- 29 475–29 505, 2018.
ception’with cnns,” IEEE transactions on visualization and computer [70] P.-T. De Boer, D. P. Kroese, S. Mannor, and R. Y. Rubinstein,
graphics, vol. 25, no. 1, pp. 641–650, 2018. “A tutorial on the cross-entropy method,” Annals of operations
[48] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based research, vol. 134, no. 1, pp. 19–67, 2005.
learning applied to document recognition,” Proceedings of the [71] X. Li, X. Sun, Y. Meng, J. Liang, F. Wu, and J. Li, “Dice loss
IEEE, vol. 86, no. 11, pp. 2278–2324, 1998. for data-imbalanced nlp tasks,” in Proceedings of the 58th Annual
[49] K. Simonyan and A. Zisserman, “Very deep convolutional Meeting of the Association for Computational Linguistics, 2020, pp.
networks for large-scale image recognition,” arXiv preprint 465–476.
arXiv:1409.1556, 2014. [72] R. L. Wilder, “Evolution of the topological concept of “con-
[50] F. Chollet, “Xception: Deep learning with depthwise separable nected”,” The American Mathematical Monthly, vol. 85, no. 9, pp.
convolutions,” in Proceedings of the IEEE conference on computer 720–726, 1978.
vision and pattern recognition, 2017, pp. 1251–1258. [73] R. M. Haralick, S. R. Sternberg, and X. Zhuang, “Image analysis
[51] L. Giovannangeli, R. Bourqui, R. Giot, and D. Auber, “Toward using mathematical morphology,” IEEE transactions on pattern
automatic comparison of visualization techniques: Application analysis and machine intelligence, no. 4, pp. 532–550, 1987.
to graph visualization,” Visual Informatics, 2020. [74] B. Alper, B. Bach, N. Henry Riche, T. Isenberg, and J.-D. Fekete,
[52] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional net- “Weighted graph comparison techniques for brain connectivity
works for semantic segmentation,” in Proceedings of the IEEE analysis,” in Proceedings of the SIGCHI conference on human factors
conference on computer vision and pattern recognition, 2015, pp. 3431– in computing systems, 2013, pp. 483–492.
3440. [75] K.-P. Yee, D. Fisher, R. Dhamija, and M. Hearst, “Animated
[53] V. Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep exploration of graphs with radial layout,” in Proc. IEEE InfoVis
convolutional encoder-decoder architecture for image segmenta- 2001, 2001, pp. 43–50.
tion,” IEEE transactions on pattern analysis and machine intelligence, [76] C. C. Gramazio, D. H. Laidlaw, and K. B. Schloss, “Colorgorical:
vol. 39, no. 12, pp. 2481–2495, 2017. Creating discriminable and preferable color palettes for informa-
[54] X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural tion visualization,” IEEE transactions on visualization and computer
networks,” in Proceedings of the IEEE conference on computer vision graphics, vol. 23, no. 1, pp. 521–530, 2016.
and pattern recognition, 2018, pp. 7794–7803. [77] C. Cook, F. Heath, R. L. Thompson, and B. Thompson, “Score
[55] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” reliability in webor internet-based surveys: Unnumbered graphic
in Proceedings of the IEEE conference on computer vision and pattern rating scales versus likert-type scales,” Educational and Psycholog-
recognition, 2018, pp. 7132–7141. ical Measurement, vol. 61, no. 4, pp. 697–706, 2001.
[56] O. Oktay, J. Schlemper, L. L. Folgoc, M. Lee, M. Heinrich, [78] M. Liu and F. G. Conrad, “Where should i start? on default values
K. Misawa, K. Mori, S. McDonagh, N. Y. Hammerla, B. Kainz for slider questions in web surveys,” Social Science Computer
et al., “Attention u-net: Learning where to look for the pancreas,” Review, vol. 37, no. 2, pp. 248–269, 2019.
in Proceedings of the 1st Conference on Medical Imaging with Deep [79] F. Chollet et al., “Keras,” https://github.com/keras-team/keras,
Learning (MIDL 2018), 2018, pp. 1–10. 2015.
[57] L. Wang, J. Giesen, K. T. McDonnell, P. Zolliker, and K. Mueller, [80] D. Zhou, J. Fang, X. Song, C. Guan, J. Yin, Y. Dai, and R. Yang,
“Color design for illustrative visualization,” IEEE Transactions on “Iou loss for 2d/3d object detection,” in 2019 International Con-
Visualization and Computer Graphics, vol. 14, no. 6, pp. 1739–1754, ference on 3D Vision (3DV). IEEE, 2019, pp. 85–94.
2008. [81] M. A. Rahman and Y. Wang, “Optimizing intersection-over-union
[58] B. Caldwell, M. Cooper, L. G. Reid, G. Vanderheiden, in deep neural networks for image segmentation,” in International
W. Chisholm, J. Slatin, and J. White, “Web content accessibility symposium on visual computing. Springer, 2016, pp. 234–244.
guidelines (wcag) 2.0,” WWW Consortium (W3C), 2008. [82] Q. Wang, Z. Chen, Y. Wang, and H. Qu, “A survey on ml4vis:
[59] R. Rossi and N. Ahmed, “The network data repository with Applying machinelearning advances to data visualization,” IEEE
interactive graph analytics and visualization,” in Twenty-Ninth Transactions on Visualization and Computer Graphics, 2021.
AAAI Conference on Artificial Intelligence, 2015. [83] L. Yuan, W. Zeng, S. Fu, Z. Zeng, H. Li, C.-W. Fu, and H. Qu,
[60] D. H. Kim, E. Hoque, and M. Agrawala, “Answering questions “Deep colormap extraction from visualizations,” IEEE Transac-
about charts and generating visual explanations,” in Proceedings tions on Visualization and Computer Graphics, pp. 1–1, 2021.
of the 2020 CHI Conference on Human Factors in Computing Systems, [84] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan,
2020, pp. 1–13. P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects
[61] C. Lai, Z. Lin, R. Jiang, Y. Han, C. Liu, and X. Yuan, “Automatic in context,” in European conference on computer vision. Springer,
annotation synchronizing with textual description for visualiza- 2014, pp. 740–755.
tion,” in Proceedings of the 2020 CHI Conference on Human Factors [85] W. S. Cleveland, The elements of graphing data. Wadsworth Publ.
in Computing Systems, 2020, pp. 1–13. Co., 1985.
[62] B. Saleh, M. Dontcheva, A. Hertzmann, and Z. Liu, “Learning
style similarity for searching infographics,” in Proceedings of the
41st Graphics Interface Conference, 2015, pp. 59–64.
[63] M. Bostock, V. Ogievetsky, and J. Heer, “D3 data-driven docu- Sicheng Song received his B.Eng. from
ments,” IEEE transactions on visualization and computer graphics, Hangzhou Dianzi University, China, in 2019.
vol. 17, no. 12, pp. 2301–2309, 2011. He is working toward the Ph.D. degree with
[64] J. D. Hunter, “Matplotlib: A 2d graphics environment,” IEEE East China Normal University, Shanghai, China.
Annals of the History of Computing, vol. 9, no. 03, pp. 90–95, 2007. His main research interests include information
[65] S. Van der Walt, J. L. Schönberger, J. Nunez-Iglesias, F. Boulogne, visualization and visual analysis.
J. D. Warner, N. Yager, E. Gouillart, and T. Yu, “scikit-image:
image processing in python,” PeerJ, vol. 2, p. e453, 2014.
[66] Z. Tian, W. Huang, T. He, P. He, and Y. Qiao, “Detecting text
in natural image with connectionist text proposal network,” in
European conference on computer vision. Springer, 2016, pp. 56–72.

Chenhui Li received Ph.D. from the Depart-

ment of Computing at Hong Kong Polytechnic
University, in 2018. He is an associate profes-
sor with the School of Computer Science and
Technology at East China Normal University.
He received ICCI*CC Best Paper Award (2015)
and SIGGRAPH Asia Sym. Vis. Best Paper
Award (2017). He has served as a local chair
in VINCI2019. He works on the research of in-
formation visualization and computer graphics.

Dong Li received her B.Eng. from Zhejiang Uni-

versity of Technology, in 2019. He received his
Master degree from East China Normal Univer-
sity, in 2022. His main research interests include
information visualization and visual analysis.

Juntong Chen received his B.Eng. from East

China Normal University, in 2022. He is work-
ing toward the Master degree with East China
Normal University, Shanghai, China. His main
research interests include information visualiza-
tion and visual analysis.

Changbo Wang is a professor with the School

of Computer Science and Technology, East
China Normal University. He received his Ph.D.
degree at the State Key Lab of CADCG of Zhe-
jiang University in 2006. He was a post-doctor
of the State University of New York in 2010.
His research interests mainly include computer
graphics, information visualization, visual Ana-
lytics, etc. He is serving as the Young AE of
Frontiers of Computer Science, and PC member
for several international conferences.

TE and TE Agent Installation Guide
No ratings yet
TE and TE Agent Installation Guide
15 pages
VividGraph Learning To Extract and Redesign Network Graphs From Visualization Images
No ratings yet
VividGraph Learning To Extract and Redesign Network Graphs From Visualization Images
14 pages
A Survey of Graph Neural Networks in Various Learning Paradigms Methods, Applications, and Challenges
No ratings yet
A Survey of Graph Neural Networks in Various Learning Paradigms Methods, Applications, and Challenges
70 pages
GNNS
No ratings yet
GNNS
7 pages
Graph Neural Networks
100% (1)
Graph Neural Networks
27 pages
A Comprehensive Survey On Graph Neural Networks
No ratings yet
A Comprehensive Survey On Graph Neural Networks
22 pages
lee2019
No ratings yet
lee2019
6 pages
Improving Global Awareness of Linkset Predictions Using Cross-Attentive Modulation Tokens
No ratings yet
Improving Global Awareness of Linkset Predictions Using Cross-Attentive Modulation Tokens
17 pages
Gcn
No ratings yet
Gcn
23 pages
Yang 20 A
No ratings yet
Yang 20 A
16 pages
Papers Papers PDF
No ratings yet
Papers Papers PDF
48 pages
F G R L P T G: AST Raph Epresentation Earning With Y Orch Eometric
No ratings yet
F G R L P T G: AST Raph Epresentation Earning With Y Orch Eometric
9 pages
Suevey On GNN
No ratings yet
Suevey On GNN
31 pages
project paper
No ratings yet
project paper
10 pages
Bacciu 2020
No ratings yet
Bacciu 2020
62 pages
Thura2022-06-20 (Research Plan)
No ratings yet
Thura2022-06-20 (Research Plan)
26 pages
A Comprehensive Survey of Graph Neural Networks PDF
No ratings yet
A Comprehensive Survey of Graph Neural Networks PDF
22 pages
Thura2022-05-30 - Geometric Deep Learning
No ratings yet
Thura2022-05-30 - Geometric Deep Learning
33 pages
Luo ChartOCR Data Extraction From Charts Images Via A Deep Hybrid WACV 2021 Paper
No ratings yet
Luo ChartOCR Data Extraction From Charts Images Via A Deep Hybrid WACV 2021 Paper
9 pages
Approximation- and Quantization-Aware Training for Graph Neural Networks
No ratings yet
Approximation- and Quantization-Aware Training for Graph Neural Networks
14 pages
Deep Learning On Graphs: A Survey: Ziwei Zhang, Peng Cui and Wenwu Zhu, Fellow, IEEE
No ratings yet
Deep Learning On Graphs: A Survey: Ziwei Zhang, Peng Cui and Wenwu Zhu, Fellow, IEEE
24 pages
A Comparison Between Recursive Neural Networks and Graph Neural Networks
No ratings yet
A Comparison Between Recursive Neural Networks and Graph Neural Networks
8 pages
A Gentle Introduction To Graph Neural Networks
No ratings yet
A Gentle Introduction To Graph Neural Networks
14 pages
The Graph Neural Network Model
No ratings yet
The Graph Neural Network Model
20 pages
CAAI Trans on Intel Tech - 2024 - Sharma - Image and video analysis using graph neural network for Internet of Medical
No ratings yet
CAAI Trans on Intel Tech - 2024 - Sharma - Image and video analysis using graph neural network for Internet of Medical
15 pages
A Diagram Is Worth A Dozen Images
No ratings yet
A Diagram Is Worth A Dozen Images
17 pages
Computing Graph Neural Networks: A Survey From Algorithms To Accelerators
No ratings yet
Computing Graph Neural Networks: A Survey From Algorithms To Accelerators
38 pages
2016 - Alex Pappachen James - Edge Detection For Pattern Recognition A Survey
No ratings yet
2016 - Alex Pappachen James - Edge Detection For Pattern Recognition A Survey
33 pages
Masked Attention Is All You Need For Graphs: Duvenaud Et Al. 2015 Kearnes Et Al. 2016 Gilmer Et Al. 2017
No ratings yet
Masked Attention Is All You Need For Graphs: Duvenaud Et Al. 2015 Kearnes Et Al. 2016 Gilmer Et Al. 2017
15 pages
Unit - 3 - Object Recognition
No ratings yet
Unit - 3 - Object Recognition
12 pages
2404.18144v1-pages-7
No ratings yet
2404.18144v1-pages-7
10 pages
Graph_Learning_A_Survey
No ratings yet
Graph_Learning_A_Survey
19 pages
Learning Methods
No ratings yet
Learning Methods
70 pages
Gated Attention Networks For Learning On Large
No ratings yet
Gated Attention Networks For Learning On Large
11 pages
Graph Neural Networks: A Review of Methods and Applications
No ratings yet
Graph Neural Networks: A Review of Methods and Applications
22 pages
Graph Neural Networks: A Review of Methods and Applications
No ratings yet
Graph Neural Networks: A Review of Methods and Applications
20 pages
Adaptive Classifier Construction An Approach To Ha
No ratings yet
Adaptive Classifier Construction An Approach To Ha
9 pages
23 - AAAI - Substructure Aware Graph Neural Networks
No ratings yet
23 - AAAI - Substructure Aware Graph Neural Networks
9 pages
3516 Graph Neural Networks With Ada
No ratings yet
3516 Graph Neural Networks With Ada
13 pages
document
No ratings yet
document
12 pages
Networkx Tutorial
No ratings yet
Networkx Tutorial
8 pages
Geometric Deep Learning On Graphs and Manifolds Using Mixture Model Cnns
No ratings yet
Geometric Deep Learning On Graphs and Manifolds Using Mixture Model Cnns
13 pages
Graph Neural Networks For Image Understanding Based On Multiple Cues
No ratings yet
Graph Neural Networks For Image Understanding Based On Multiple Cues
10 pages
DGCNN
No ratings yet
DGCNN
8 pages
SDFSDFSDFCVXC 543
No ratings yet
SDFSDFSDFCVXC 543
6 pages
2022 - Chuan Shi, Xiao Wang, Cheng Yang - Advances in Graph Neural Networks-Springer
No ratings yet
2022 - Chuan Shi, Xiao Wang, Cheng Yang - Advances in Graph Neural Networks-Springer
207 pages
Networkx Tutorial
100% (1)
Networkx Tutorial
8 pages
Hierarchical Graph Neural Networks
No ratings yet
Hierarchical Graph Neural Networks
14 pages
paper to convert in box data
No ratings yet
paper to convert in box data
15 pages
2019 - Medium - Tutorial On Graph Neural Networks For Computer Vision and Beyond - by Boris Knyazev
No ratings yet
2019 - Medium - Tutorial On Graph Neural Networks For Computer Vision and Beyond - by Boris Knyazev
21 pages
Original GNN
No ratings yet
Original GNN
22 pages
OuterplanarGraphs_V5 (1)
No ratings yet
OuterplanarGraphs_V5 (1)
68 pages
Directed Graphs: Digraph Digraph - Out - Edges Digraph - in - Degree Digraph - Predecessors Digraph - Successors
No ratings yet
Directed Graphs: Digraph Digraph - Out - Edges Digraph - in - Degree Digraph - Predecessors Digraph - Successors
3 pages
DL UNIT-5
No ratings yet
DL UNIT-5
34 pages
10.6 Graph Drawing Options (Part 1)
No ratings yet
10.6 Graph Drawing Options (Part 1)
6 pages
Do Transformers Really Perform Bad For Graph Representation?
No ratings yet
Do Transformers Really Perform Bad For Graph Representation?
19 pages
graphormer-2021-neurIPS
No ratings yet
graphormer-2021-neurIPS
12 pages
SplineCNN-Fast Geometric Deep Learning with Continuous B-Spline Kernels
No ratings yet
SplineCNN-Fast Geometric Deep Learning with Continuous B-Spline Kernels
9 pages
Getting Started With Graphviz and Python
No ratings yet
Getting Started With Graphviz and Python
9 pages
Learning Structure Perception Mlps On Graphs: A Layer Wise Graph Knowledge Distillation Framework
No ratings yet
Learning Structure Perception Mlps On Graphs: A Layer Wise Graph Knowledge Distillation Framework
16 pages
Scanline Rendering: Exploring Visual Realism Through Scanline Rendering Techniques
From Everand
Scanline Rendering: Exploring Visual Realism Through Scanline Rendering Techniques
Fouad Sabry
No ratings yet
ChatGPT For Cybersecurity 4 1674227454
No ratings yet
ChatGPT For Cybersecurity 4 1674227454
63 pages
Laporan Kerjasama Dengan Puskesmas
No ratings yet
Laporan Kerjasama Dengan Puskesmas
183 pages
aq-300-operators-manual-en1-0
No ratings yet
aq-300-operators-manual-en1-0
20 pages
NDS TST 02 Operating - Manual - V3.0 20110712 - EN
No ratings yet
NDS TST 02 Operating - Manual - V3.0 20110712 - EN
107 pages
Salesforce Developer Tasks
No ratings yet
Salesforce Developer Tasks
9 pages
SoMachine - Modbus and ASCII Read - Write Functions - BERGER ...
No ratings yet
SoMachine - Modbus and ASCII Read - Write Functions - BERGER ...
82 pages
Aspack.v2.12 DLL-OEPFinder
No ratings yet
Aspack.v2.12 DLL-OEPFinder
2 pages
Snowl Manual (Rev 1.1)
No ratings yet
Snowl Manual (Rev 1.1)
15 pages
Nandini - Gupta - Resume PDF
No ratings yet
Nandini - Gupta - Resume PDF
2 pages
Software Project Planning
No ratings yet
Software Project Planning
206 pages
KV 2322
No ratings yet
KV 2322
1 page
Hpaji Ningli Laika A Karai Mungga
No ratings yet
Hpaji Ningli Laika A Karai Mungga
4 pages
Installation Instructions For The Standard & Premium Versions of Kwikpop For Multicharts
No ratings yet
Installation Instructions For The Standard & Premium Versions of Kwikpop For Multicharts
10 pages
b_ise_31_RN
No ratings yet
b_ise_31_RN
82 pages
LIMS Laboratory Information Management System (LIMS) Assessment
100% (1)
LIMS Laboratory Information Management System (LIMS) Assessment
9 pages
May 2023 IS1060 Examination Questions
No ratings yet
May 2023 IS1060 Examination Questions
3 pages
Fritzwlan-Repeater-Dvb-C Man en GB
No ratings yet
Fritzwlan-Repeater-Dvb-C Man en GB
51 pages
1 Types of Parsers in Compiler Design
100% (1)
1 Types of Parsers in Compiler Design
4 pages
Chapter 4 LL - Requirement Engineering
No ratings yet
Chapter 4 LL - Requirement Engineering
5 pages
2
100% (1)
2
21 pages
Guide To SQL 9th Edition Pratt Solutions Manual 1
100% (64)
Guide To SQL 9th Edition Pratt Solutions Manual 1
36 pages
Chapter 12
No ratings yet
Chapter 12
34 pages
SACS Utilities
No ratings yet
SACS Utilities
45 pages
Database Programming
No ratings yet
Database Programming
6 pages
4.2 SoftEngg Unit-IV Coding and Software Testing
No ratings yet
4.2 SoftEngg Unit-IV Coding and Software Testing
72 pages
Summer Training Report: Submitted in Partial Fulfillment For The Second Year Summer Internship of
No ratings yet
Summer Training Report: Submitted in Partial Fulfillment For The Second Year Summer Internship of
27 pages
Can-Bus Shield - Wiki
No ratings yet
Can-Bus Shield - Wiki
9 pages
PalletTech-Parser V34 en
No ratings yet
PalletTech-Parser V34 en
120 pages
Create Windows - Linux Bootable USB Using Etcher - ComputingForGeeks
No ratings yet
Create Windows - Linux Bootable USB Using Etcher - ComputingForGeeks
5 pages