See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/338050378

Vibration-based semantic damage segmentation for large-scale structural

health monitoring

Article  in  Computer-Aided Civil and Infrastructure Engineering · June 2020

DOI: 10.1111/mice.12523

CITATIONS READS

3 1,010

2 authors:

Seyed Omid Sajedi Xiao Liang

University at Buffalo, The State University of New York University at Buffalo, The State University of New York
10 PUBLICATIONS   15 CITATIONS    34 PUBLICATIONS   180 CITATIONS   

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Xiao Liang on 16 January 2020.

The user has requested enhancement of the downloaded file.

DOI: 10.1111/mice.12523

ORIGINAL ARTICLE

Vibration-based semantic damage segmentation for large-scale

structural health monitoring

Seyed Omid Sajedi Xiao Liang

Department of Civil, Structural and

Environmental Engineering, University at
Abstract
Buffalo,The State University of New York, Toward reduced recovery time after extreme events, near real-time damage diagnosis
NY, USA of structures is critical to provide reliable information. For this task, a fully convo-
Correspondence lutional encoder–decoder neural network is developed, which considers the spatial
Xiao Liang, 242 Ketter Hall, Department of correlation of sensors in the automatic feature extraction process through a grid envi-
Civil, Structural and Environmental Engineer-
ronment. A cost-sensitive score function is designed to include the consequences of
ing, University at Buffalo, Buffalo, NY 14260,
USA. misclassification in the framework while considering the ground motion uncertainty
Email: [email protected]. in training. A 10-story-10-bay reinforced concrete (RC) moment frame is modeled
to present the design process of the deep learning architecture. The proposed models
achieve global testing accuracies of 96.3% to locate damage and 93.2% to classify 16
damage mechanisms. Moreover, to handle class imbalance, three strategies are inves-
tigated enabling an increase of 16.2% regarding the mean damage class accuracy. To
evaluate the generalization capacities of the framework, the classifiers are tested on
1,080 different RC frames by varying model properties. With less than a 2% reduction
in global accuracy, the data-driven model is shown to be reliable for the damage diag-
nosis of different frames. Given the robustness and capabilities of the grid environ-
ment, the proposed framework is applicable to different domains of structural health
monitoring research and practice to obtain reliable information.

1 I N T RO D U C T I O N In the last few years, there have been giant advances

in computer vision for the tasks of image classification
Proper recovery after extreme events such as earthquakes and object detection. Deep learning in computer vision has
is an essential characteristic of resilient communities. The shown impressive success in dealing with real-life images
ability to manage postdisaster consequences requires reli- (e.g., Badrinarayanan, Kendall, & Cipolla, 2015; Koziarski
able information about the impact of seismic events such & Cyganek, 2017; Molina-Cabello, Luque-Baena, López-
that existing “resources and skills” can be allocated properly Rubio, & Thurnhofer-Hemsi, 2018; Ren, He, Girshick, &
(Comfort, 1999). Bruneau et al. (2003) defined the reduced Sun, 2015; Simonyan & Zisserman, 2014; Wang & Bai,
time to recovery as one of the principal features of a resilient 2018) accompanied with the development of computation-
system. Hence, rapid condition monitoring of civil infrastruc- ally efficient methods that handle big data (Ortega-Zamorano,
tures is necessary while the need for near real-time assessment Jerez, Gómez, & Franco, 2017; Torres, Galicia, Troncoso,
of structural integrity is highlighted. That being said, struc- & Martínez-Álvarez, 2018). The progress has motivated
tural health monitoring (SHM) has been an effective tool for researchers to apply such algorithms in civil engineering (e.g.,
such evaluations with an emphasis on damage diagnosis. Oh, Kim, Kim, Park, & Adeli, 2017; Rafiei, Khushefati,

© 2019 Computer-Aided Civil and Infrastructure Engineering

Comput Aided Civ Inf. 2019;1–18. wileyonlinelibrary.com/journal/mice 1

2 SAJEDI AND LIANG

Demirboga, & Adeli, 2017) and more specifically automat- Kim, & Chon, 2014; Chun, Yamashita, & Furukawa, 2015;
ing the structural inspection process (Koch, Georgieva, Khodabandehlou, Pekcan, & Fadali, 2019; Liang, Mosalam,
Kasireddy, Akinci, & Fieguth, 2015). This automation & Muin, 2018; Mita & Hagiwara, 2003; Rafiei & Adeli, 2017;
includes structural component detection (Narazaki, Hoskere, Sajedi & Liang, 2019b, 2019c).
Hoang, & Spencer, 2018) and damage identification. Classi- Given the presented literature, machine learning algo-
fication of damage in the corresponding literature is mainly rithms for the vibration-based SHM have shown promising
limited to visible cracks (e.g., Cha, Choi, & Büyüköztürk, performance for the purpose of near real-time implementa-
2017; Xue & Li, 2018; Zhang et al., 2017), corrosion, spalling, tion. However, to the authors’ knowledge, the existing data-
delamination, rebar exposure, cavity, or a subset of previ- driven frameworks for damage diagnosis have the following
ous damage types (e.g., Cha, Choi, Suh, Mahmoudkhani, & deficiencies: (a) the number of sensor placements (affecting
Büyüköztürk, 2018; Hoskere, Narazaki, Hoang, & Spencer, the input size) is usually limited due to the computational
2018; R. Li, Yuan, Zhang, & Yuan, 2018). Some researchers demand; (b) enlarging the number of potential output dam-
have presented extended frameworks that perform damage age locations might reduce the model accuracy and its com-
detection and classification in different global and local fail- putational efficiency; and (c) information regarding the geo-
ure levels (e.g., Gao & Mosalam, 2018; Liang, 2019; Sajedi metric location of sensors is not included in the input data
& Liang, 2019a; Yeum, Dyke, Ramirez, & Benes, 2016). A structure.
pixel-wise description of damage may contain valuable infor- In this article, to resolve or alleviate the aforementioned
mation. However, current vision-based models are unable or limitations, we propose a grid environment framework that
have very limited capability to provide inference about struc- provides a practical, yet computationally efficient solution.
tural performance in terms of collapse capacity, residual drift, A fully convolutional encoder–decoder neural network is
etc. The only source of information for vision-based inspec- designed to perform semantic damage segmentation. This
tions are images that may not be available for critical structural framework is capable of providing near real-time and fine-
elements. In addition, images only provide information about detailed SHM for large-scale civil infrastructure. Different
the deterioration on the surface while internal layers might site hazard uncertainties and consequences of misclassifi-
be heavily damaged (Kashif Ur Rehman, Ibrahim, Memon, & cation can be considered while being able to modify the
Jameel, 2016). model’s performance in favor of predefined locations in a
Vibration records are another source of information uti- structure.
lized in SHM with the underlying assumption that dynamic The remainder of the article is organized as follows. In Sec-
properties and responses will be changed when damage is tion 2, the damage diagnosis framework is introduced and the
present. Corresponding model-based SHM methods have necessary formulations are provided. The data-driven frame-
been investigated with experimental data from existing work and the idea of gird environments are described through
structures (e.g., Amezquita-Sanchez, Park, & Adeli, 2017; a case study in Section 3. Furthermore, a step-by-step design
Behmanesh & Moaveni, 2015; Z. Li, Park, & Adeli, 2017; process is presented in Section 4 to optimize the deep learning
Perez-Ramirez et al., 2016; Yu et al., 2017). Interpretation of architecture, hyperparameters, and to investigate class imbal-
such variations as damage commonly requires expertise and ance. In Section 5, a second test data set is generated to
is challenging for the large-scale infrastructure. Moreover, evaluate the generalization of the deep learning framework
these methods could be computationally expensive, which for reinforced concrete (RC) frames with different proper-
makes their real-time implementation difficult. ties. Detailed performance evaluation of the frameworks is
Data-driven methods in SHM are relatively more recent provided in Section 6 followed by the conclusions given in
developments compared to the model-based approaches. Sta- Section 7.
tistical learning and pattern recognition algorithms can pro-
vide alternatives in damage diagnosis of civil infrastruc-
ture. Significant efforts have been dedicated to the task of 2 DAM AG E DIAGNOS IS
anomaly detection for identifying the existence of damage, F RAM EWO RK
mainly utilizing unsupervised algorithms. Proposed meth-
ods in this area include machine learning algorithms, such Semantic segmentation methods are rapidly growing for the
as clustering (Kesavan & Kiremidjian, 2012) and kernel- task of scene understanding. Different algorithms are utilized
based methods (Gui, Pan, Lin, Li, & Yuan, 2017; Santos, to assign a class label to the pixels in an image, competing
Figueiredo, Silva, Sales, & Costa, 2016). More sophisti- in robustness and efficiency. One recent model for seman-
cated approaches are proposed for identifying damage loca- tic segmentation is SegNet (Badrinarayanan et al., 2015)
tion (Abdeljaber et al., 2018; Abdeljaber, Avci, Kiranyaz, that outputs detailed pixel segmentation using a fully con-
Gabbouj, & Inman, 2017; Lin, Nie, & Ma, 2017; Rafiei & volutional encoder–decoder architecture. In damage diag-
Adeli, 2018; Zhou, Ni, & Zhang, 2014) and severity (Chong, nosis, especially for large-scale SHM, detailed information
SAJEDI AND LIANG 3

especially in a large network of multiple sensors. Moreover,

different input records will need to have an identical num-
ber/duration of time steps, which rarely occurs for ground
motion records. In this case, 𝐼 𝜂 provides a superior compu-
tational advantage. Note that the dimensionality of a sensor
record, after integration, will be reduced to one, regardless
of 𝑡g .
For a 2D grid environment, a third-order tensor is con-
sidered as the input. Hence, a single input observation will
have the dimensions 𝑛hg × 𝑛vg × (𝑛𝜂 + 1) where 𝑛hg and 𝑛vg
are the numbers of horizontal and vertical grid nodes, and
𝑛𝜂 is the number of different 𝜂 values expressed in Equa-
FIGURE 1 An example of a 2D grid environment tion (1). The third dimension represents a similar concept
like the RGB channels in colored images but with cumula-
tive intensity measures in different 𝜂 channels. The (𝑛𝜂 + 1)th
regarding the potential defect locations and corresponding
channel includes a binary value of 0 or 1 depending on
damage severities is essential to the stakeholders. To address
the physical existence of a sensor in a specific grid loca-
the previously discussed limits of existing data-driven mod-
tion (the corresponding 𝐼 𝜂 values are assumed zero for null
els, we propose a grid environment where the information
nodes).
regarding the geometric location of sensors is considered in
the process of automatic feature extraction (see Figure 1). This
idea not only enhances the model’s robustness but also pro- 2.2 Grid environment and null nodes
vides an efficient computational foundation to simultaneously
Introducing the null concept is one of the novelties in this
monitor damage in a relatively large number of grid nodes.
article, which provides several benefits. Most importantly,
This efficiency stems from the structure of input data and
the sophisticated geometric arrangement of accelerometers is
the fact that convolution is used to segment damage in differ-
considered in the grid environments.
ent nodes. The following subsection will provide more detail
Most neural networks are formulated based on tensor
on this claim. Throughout the article, this prediction process
(matrix) operations, thus the input and output should also
will be mentioned as semantic damage segmentation (SDS).
be tensors. Assuming a 2D grid environment, both input and
The term semantic stems from the fact that the condition of
the predicted output of a grid node are vectors. It is possible
each grid node is explained with the corresponding class of
to stack all the vectors from the physical sensor nodes, and
damage.
then reshape the combined tensor into an image-like structure
without any null nodes. However, in the presence of irregu-
2.1 Vibration-based input tensor larity, such as the structure in Figure 1, information on the
In this study, we use cumulative intensity measures (𝐼 𝜂 ) as relative geometric location of sensors will be lost. In contrast,
the damage feature in the input tensor. Previous studies have by introducing null nodes and hard coding their input–output
shown that specific forms of 𝐼 𝜂 (e.g., arias intensity and values, this geometric information is implicitly imported into
cumulative absolute velocity) correlate well with the nonlin- the data structure. For example, as shown in Figure 1, sliding
ear structural response. Mainly in the earthquake engineer- kernels in a convolutional neural network (CNN) will differ-
ing community, these features are used as damage indica- entiate between the location of sensor node 1 (corner struc-
tors (Campbell & Bozorgnia, 2012; Liang et al., 2018; Muin tural joint type) and node 2 (middle structural joint type) with
& Mosalam, 2017). The mathematical expression for 𝐼 𝜂 is respect to the adjacent null nodes. This way, the geometric
expressed as follows: information is also used in the process of automatic feature
extraction.
𝑡g Null nodes may also be used to extend the boundaries of the
𝐼𝜂 = |𝑎|𝜂 𝑑𝑡 (1)
∫0 grid environment such that a unique deep learning architec-
ture could be used for other structures or sensor arrangements
where a is the acceleration value per time step t, integrated within the boundaries of that grid environment. This high-
over the duration of excitation event 𝑡g , and 𝜂 is a hyperpa- lights another advantage of the grid environment compared
rameter. Note that directly importing raw acceleration histo- with the reshape strategy mentioned earlier. As long as the
ries into a data-driven model has two disadvantages. First, the grid environment can fit sensors within the respected range
input size from a single accelerometer may be in the order of 𝑛hg and 𝑛vg , the input–output dimensions will not change,
of thousands. This issue will cause computational challenges, even with a different sensor array.
4 SAJEDI AND LIANG

We later show that this concept also facilitates encoder– except that last block with softmax (Goodfellow et al., 2016)
decoder mapping for the task of segmentation. It should activation.
be noted that there is a difference between introducing null
nodes and the “padding” operation (Goodfellow, Bengio, & 2.4 Loss function for backpropagation
Courville, 2016). The latter is only applied to modify (or pre-
serve) the extracted feature dimensionality after convolution To calculate the losses in each individual grid node, the cross-
whereas the null nodes are created as a part of input data pre- entropy loss function is considered.
processing. From here forth, we will describe and validate the
𝑛hg ×𝑛vg 𝑝+1
models for 2D environments. Note that the input tensor would ∑ ∑
𝐿𝑖 = −𝑃𝑖 𝜓𝑗 𝑢𝑖𝑗 𝑦𝑖𝑗𝑘 log 𝑞𝑗𝑘
𝑖
(2)
be a fourth-order tensor for 3D grid environments. The deep
𝑗=1 𝑘=1
learning architecture shall be adjusted accordingly to consider
a 3D input in that regard. For example, instead of a 2D con- where 𝐿𝑖 is the loss value for the observation i (e.g., a seismic
volution, a 3D version will be used. Therefore, the framework excitation event), and index j counts all the nodes in the grid
and the concepts studied in this article are still applicable to a environment. The loss of an observation is the sum of losses
3D physical model. for all nodes. 𝑦𝑖𝑗𝑘 is an element for the one-hot encoding of
output probability of class k associated with node j in obser-
2.3 Deep learning architecture vation i (a vector containing the value of 1 for the true class
and 0 otherwise), and 𝑞𝑗𝑘 𝑖 is the corresponding softmax proba-
After processing the input–output data for each observation, bilities. 𝑃𝑖 in this formulation is the probability of observation
the architecture of a neural network can be designed. A fully i that can be obtained from ground motion attenuation models
convolutional encoder–decoder network is developed which (e.g., Campbell & Bozorgnia, 2014). This factor is to account
enables end-to-end training. Before proceeding to the design for seismic hazard uncertainties in the proposed framework.
and optimization of the neural network, it is necessary to The factor 𝜓𝑗 is introduced to highlight the importance of cer-
define the building blocks of deep learning models. The pro- tain nodes, with respect to the others. This definition is useful
posed CNN includes encoder and decoder blocks. An encoder when there exist locations that contain critical facilities with
block is comprised of a sequence of following operations: con- higher importance compared to the other locations in the grid
volution, batch normalization (Ioffe & Szegedy, 2015), acti- environment. In this case, each node can be assigned an impor-
vation, and max-pooling. The sliding filter size (ks ) and the tance factor regardless of its potential damage.
number of filters (nf ) are the main hyperparameters of the The coefficients 𝑢𝑖𝑗 are hyperparameters that tune the per-
convolution operator. It is assumed that sliding filters relo- formance of algorithms in favor of the true class label. In the
cate with the stride of 1. Moreover, padding is performed to vision-based algorithms, these coefficients are used to deal
keep input and output dimensions equal after each convolu- with imbalanced data sets (Eigen & Fergus, 2015), which is
tion. Followed by batch normalization and activation, max- also a critical issue in data-driven SHM where achieving a
pooling is performed with window sizes of 2 × 2 and nonover- uniform distribution of damage classes is challenging and per-
lapping strides of 2. Decoders are similar to their peer encoder haps even impossible. For example, depending on the struc-
block with the exception that max-pooling is replaced with tural properties, some damage mechanisms are more likely to
up-sampling, which is moved to the beginning of the decoder dominate others. A collapse mechanism is commonly formed
block. It should be noted that the input–output dimensions of by the failure of certain structural components (e.g., a soft
each encoder block should be consistent with the correspond- story mechanism). Therefore, the majority of structural ele-
ing decoder. This is necessary to preserve the dimensions of ments might remain locally undamaged. The value of 𝑢𝑖𝑗
the grid environment for SDS. The final output of SDS will depends only on the true class of node j in observation i.
be 𝑛hg × 𝑛vg × (𝑝 + 1) dimensional, where p is the number of Therefore, 𝑢𝑖𝑗 can have 𝑝 + 1 possible distinct values (𝑢𝑖𝑗 ∈
possible damage scenarios in each node (plus the null class). {𝑢̂ 0 , 𝑢̂ 1 , … , 𝑢̂ 𝑝 }). Later in Section 4, the strategies to select
To achieve this output shape, after the last decoder block, a these coefficients are investigated. In the following, we will
convolution layer with 𝑘s = 1 and 𝑛f = 𝑝 + 1 is included in assume that attributes associated with a null class (NC) are
the architecture. The whole process for the proposed SDS indexed with 0 and for the sensor nodes, indices 1 through p
framework is summarized in Figure 2. The shaded areas in are assigned depending on the damage classes (DC).
this figure indicate that the number of encoder–decoder pairs
(denoted as 𝑛b ) may be subject to change. Moreover, the last
encoder or decoder blocks differ from their adjacent peers
2.5 Cost-sensitive objective function
in that they, respectively, do not have a max-pooling or up- Modeling the SDS framework based on minimizing the loss
sampling layer. Rectified linear unit (ReLU) activation func- function in Equation (2) will increase the global accuracy.
tion is uniformly used throughout all computational blocks Nonetheless, it is of great benefit in SHM to consider the
SAJEDI AND LIANG 5

FIGURE 2 The proposed deep learning architecture for damage segmentation

consequences of misclassification. Noting that the data-driven age is predicted as less critical compared to the real condition.
models may have inevitable errors in prediction, a decision In this concept, the global accuracy (𝐴p ) can be measured
maker is more likely to favor a conservative false alarm over as the mean of prediction accuracy for individual nodes (𝐴𝑗 )
the opposite. The cross-entropy loss, though computation- expressed as follows:
ally convenient, is formulated only to maximize the proba- ∑𝑝+1
bility associated with the true class (hence increasing accu- 𝑠𝑗
𝑚=1 𝑚𝑚
𝑗
𝐴 = ∑𝑝+1 ∑𝑝+1 𝑗 (5)
racy) although it does not differentiate how misclassification 𝑠
𝑚=1 𝑛=1 𝑚𝑛
occurs. It is necessary to define a metric that emphasizes
on the accuracy while penalizing misclassifications based on Based on the previous definitions, a score quantity (𝐶𝑗 ) for
their consequences. We will later use this metric as the objec- each node can be expressed as:
tive function to tune the hyperparameters. For this purpose, a
global score function is defined. In this definition, each sen- 𝐶𝑗 = 𝑤1 𝑆D − 𝑤2 𝑆LT − 𝑤3 𝑆UT (6)
sor node has a confusion score matrix (CSM) in which rows
where
(m) and columns (n), respectively, correspond to the true and
predicted damage labels. The elements of the CSM (𝑠𝑗𝑚𝑛 ) for 𝑝+1
∑
the node j can be expressed as: 𝑆D𝑗 = 𝑠𝑗𝑚𝑚 (7)
𝑚=1
𝑛obs
∑
𝑠𝑗𝑚𝑛 = 𝑖𝑗
𝑃𝑖 𝐼𝑚𝑛 (3) 𝑝+1 𝑚−1
𝑖=1 𝑗
∑ ∑
𝑆LT = 𝜆𝑚𝑛 𝑠𝑗𝑚𝑛 (8)
𝑖𝑗 𝑚=2 𝑛=1
where 𝐼𝑚𝑛 is an identity function defined as:

⎧1 If node 𝑗 in observation i belongs to the 𝑝+1 ∑

∑ 𝑛−1
𝑖𝑗 ⎪ 𝑗
𝑆UT = 𝜆𝑚𝑛 𝑠𝑗𝑚𝑛 (9)
𝐼𝑚𝑛 = ⎨ class 𝑚 but is predicted as the class 𝑛 (4)
⎪0 Otherwise 𝑛=2 𝑚=1
⎩
Equations (8) and (9) are, respectively, the sum of penal-
Note that the diagonal elements (𝑠𝑚𝑚 ) imply accurate pre- ized values for lower and upper diagonal elements. 𝑤1 , 𝑤2 ,
dictions. Assuming that the p damage levels can be sorted and 𝑤3 are predefined constants that are designed to modify
(e.g., based on severity, consequence), upper triangular ele- the weight of each term in Equation (6). It is reasonable that
ments represent the conservative predictions whereas the exis- a decision maker set 𝑤2 > 𝑤3 with more emphasis on reduc-
𝑗
tence of 𝑠𝑚𝑛 arrays in the lower diagonal (𝑚 > 𝑛) implies dam- ing underestimated prediction scores (𝑆LT ) compared to the
6 SAJEDI AND LIANG

Prediction class considered in the score measurements. However, if an exist-

ing sensor node is misclassified as null, it will affect the final
1 2 3 4 5
loss and score values in Equations (2) and (11).
1 1 22.1 32.1 42.1 52.1
True class 2 22.2 1 22.2 32.2 42.2 2.6 On the selection of hyperparameters
3 32.3 22.3 1 22.3 32.3 The proposed SDS framework is tailored with two features:
(a) considering the observation probability of occurrence
4 42.4 32.4 22.4 1 22.4
in training and (b) controlling the misclassification conse-
5 52.5 42.5 32.5 22.5 1 quences. That being said, the hyperparameters in this arti-
cle are divided into two different categories. The first type
FIGURE 3 An example selection of the penalty matrix (𝝀) for includes a set of hyperparameters that are tuned to obtain
five levels of damage setting: 𝜏 = 2 and 𝜅 = 0.1 a robust yet computationally efficient classifier. A detailed
design and investigation process is given in the following sec-
𝑗 tions that provide insight for the future implementations of the
conservative upper diagonal values (𝑆UT ). Furthermore, the
elements of the CSM corresponding to misclassification are SDS framework.
penalized with a factor 𝜆𝑚𝑛 . The following function can be The second set of hyperparameters is used to adjust the per-
used for this purpose: formance of the classifier based on the preferences of the deci-
sion maker. A summary is provided in Table 1 to aid in select-
𝜆𝑚𝑛 = |𝑚 − 𝑛 + 1|𝜏+𝜅𝑚 (10) ing this type of hyperparameters. It should be emphasized that
the SDS framework can still work regardless of these factors.
where 𝜏 and 𝜅 are constants to tune the intensity of the penalty For special cases of damage diagnosis, the decision maker
coefficients. The different effects of these two factors are best can benefit from both features. However, in the absence of
explained through an example. One may consider five damage such knowledge or preference, one may easily eliminate such
levels (1–5 in the order of severity). The predefined hyper- hyperparameters or use the default/recommended values in
parameter 𝜏 will intensify the misclassification penalty for Table 1. Despite this fact, we have shown a case study in the
a given damage class. For example, predicting level 5 as 1 article that benefits from this methodology with an outlook on
should be more harshly penalized compared to the case where performance-based earthquake engineering.
damage class 5 is predicted as 4. However, merely using 𝜏,
there is no difference between the consequences of misclas-
sifying damage level 5 as 4, or 3 as 2. Using another hyper-
3 RC F RAM E CA S E STUDY
parameter, 𝜅, the penalty factor would be higher for a mis-
take in a more critical damage state. Increasing 𝜅 will make
In this section, a grid environment is generated while the mod-
the performance metric more sensitive to the prediction errors
eling of an RC moment frame and simulation process are
associated with severe damage levels. The penalty factors can
described. The following information is used to build a data
be combined into a matrix (𝝀) to facilitate the computations.
set of training, validation, and testing for the SDS framework.
Based on this example, 𝝀 is given in Figure 3. The following
Note that later in Section 5, a different test data set will be
process of assigning penalty factors is merely a recommenda-
introduced for further evaluation of the SDS framework.
tion and the decision maker can manually modify 𝝀 without
further changes to the rest of the framework.
Building upon Equation (6) and considering the impor- 3.1 Physical model description
tance of nodes with respect to each other, a single-valued A 2D, 10-story-10-bay RC moment frame with uniform span
global objective function, considering all sensor nodes, can lengths (24 ft) and story heights (14 ft) is considered in this
be expressed as: study (Figure 4). Structural elements are loaded and designed
∑ based on the existing codes of practice in the United States
𝐺𝑆 = 𝜓𝑗 𝐶𝑗 (11)
(ACI, 2014; ASCE/SEI, 2016). Static live and superimposed
𝑗≠null
dead loads are, respectively, set to 40 and 50 psf., in addi-
The global score objective function (𝐺𝑆) can be maxi- tion to the dead weight introduced by a 6-inches thick flat
mized by tuning 𝑢̂ 0 , 𝑢̂ 1 , … , 𝑢̂ 𝑝 for the desired performance. It slab. It is assumed that the tributary width of each span is
should be noted that in this study, 𝑢̂ 0 is set to zero in the train- 24 ft. Two sets of different sections are considered for beams
ing process because the correct prediction of virtual nodes as (B1 & B2) and columns (C1 & C2) while proportionate self-
null does not matter (users already have this information with- weights are assigned to each structural element. Other design
out deep learning). Therefore, null nodes’ prediction is not properties are summarized as follows: concrete’s compressive
SAJEDI AND LIANG 7

TABLE 1 Summary of user-defined hyperparameters in the SDS framework

Hyperparameter(s) Description Default/recommended values
𝑤1 , 𝑤2 , 𝑤3 Coefficients governing the weight of accurate, 𝑤1 = 1 and 𝑤2 , 𝑤3 = 0 yields the model with
underestimated, and conservative prediction the highest accuracy, regardless of
scores in the optimization process misclassification consequences.
𝑃𝑖 Probability of occurrence for the observation i, For earthquakes, such values can be obtained
which can be used to train a deep learning from the existing attenuation models (see
classifier that is more accurate for the more Section 3.2). 𝑃𝑖 = 1 can be used in the
probable observations. absence of such information.
𝜓𝑗 The coefficient that can be used to model the A default value of 𝜓𝑗 = 1 can be considered.
importance of correct predictions with respect Higher factors can be set for specific
to different physical nodes in the grid structural joints that are relatively more
environment. important compared to other locations of
interest in the grid environment.
𝜏 Coefficient used to model the consequence of 𝜏 = 2 (may require adjustment depending on
predicting class i as j the number of damage classes)
𝜅 Coefficient used to intensify the consequence of 𝜅 = 1∕6 (may require adjustment depending
misclassification for more critical damage on the number of damage classes)
levels
𝑢𝑖𝑗 Factors to improve the prediction accuracy of Calculated based on the recommendation of
less frequent observations in the training data. Eigen and Fergus (2015) (see Section 4.3)

(a) (b)

(c)

FIGURE 4 Description of case study. (a) BeamWithHinges element, (b) RC sections, (c) the grid environment

strength,𝑓𝑐′ = 4 ksi; concrete’s modulus of elasticity, 𝐸c = where distributed plasticity can be modeled with fiber sec-
3604 ksi; and grade 60 (A615) rebar for the reinforcement. tions at the ends. The lengths of fiber elements are assigned
For a more realistic assessment of damage, incremental based on section design and geometric properties (Bae &
nonlinear time-history analyses (NTHA) are performed in Bayrak, 2008). Lumped masses are assigned to each node
OpenSees (McKenna, Fenves, & Filippou, 2010). BeamWith- assuming 25% live load contribution. Note that in this study,
Hinges element (Scott & Fenves, 2006) is used in this model rigid diaphragms are not considered on purpose. We intend
8 SAJEDI AND LIANG

the physical model to show more complex behavior both in

dynamic response and damage patterns. Material properties
of confined (core) and unconfined (cover) concrete are esti-
mated based on the existing models (Mander, Priestley, &
Park, 1988; Todeschini, Bianchini, & Kesler, 1964). As a
result, Concrete02 (Mohd, 1994) and Steel02 (Filippou, Bert-
ero, & Popov, 1983) from the OpenSees material library are,
respectively, used to model the nonlinear behavior of concrete
and reinforcement bar elements in a section.
It should be noted that the results in this article are obtained
from a series of numerical simulations to consider various
FIGURE 5 Example of distorting input record (NSR = 0.8)
damage scenarios. The overall accuracy can be affected by
modeling accuracy. We have used state-of-the-art methods in
performing NTHAs based on the recommendations of exist- cess is presented in Figure 5. The processes of training, vali-
ing literature. However, for specific industrial applications, it dation, and testing in the article are based on the distorted sig-
is highly recommended to calibrate models with prototype nals. It should be emphasized that the distortions only affect
laboratory tests to enhance the reliability of the nonlinear the deep learning input and the NTHAs are based on the actual
analyses. ground motion records. For this case study, the horizontal
acceleration records for each node are used.
3.2 Ground motion selection and signal
distortion 3.3 Damage classes and data splitting
NTHA simulations are performed considering an M7 earth- The structural damage assessment can be done with several
quake scenario. The GM events are obtained from the PEER metrics such as the maximum story drift ratio. However, as
NGA-west2 database (Ancheta et al., 2014) with the follow- implied earlier, the proposed SDS framework is intended to
ing properties: rupture distance (𝑅RUP ) within 20 km; average identify the damage location and severity among grid nodes.
shear-wave velocity for the upper 30 m depth (𝑉s30 ) greater That being said, the damage condition of each joint is moni-
than 180 m/s; and magnitude ranging from 6.5–7.5. These tored based on the magnitude of plastic rotations from beams
filters result in 208 GM records. However, events with long and columns. Because one of the main purposes of SHM is to
durations (greater than 200 seconds) are discarded due to provide reliable information for decision making, we consider
insignificant damage to the model but huge computational 16 discrete damage mechanisms based on the plastic rotations
demand. This yields to 180 recorded events that are used to of beams and columns.
generate the data set in this study. Incremental dynamic anal- Based on the recommendations of FEMA (Prestandard,
yses are performed based on peak ground velocity (PGV) FEMA, 2000), four performance levels can be assigned to
where scale factors are found in a similar fashion to Liang individual beams and columns: immediate occupancy (IO),
and Mosalam (2016). The probability distribution for seis- life safety (LS), collapse prevention (CP), and beyond collapse
mic hazard has been obtained from the CB-2014 attenuation prevention (E). Details regarding these performance criteria
model (Campbell & Bozorgnia, 2014) where 30 target PGV can be found in chapters 1 and 6 of FEMA 356 (2000). We
values are sampled. Using this attenuation model, the proba- denote these damage classes with indices 0 to 3 depending on
bility of occurrence (𝑃𝑖 ) for observations can be obtained for the extent of damage severity. Considering four possible dam-
the training and evaluation of the SDS framework. With two age severities in beam and column element types, 16 possible
principal horizontal directions, 30 scale factors, and 180 GM damage classes (DCs) are defined. Examples for this type of
events, 10,800 NTHA simulations are performed. Simulations damage in real buildings are shown in Figure 6.
are conducted in six parallel processes on an 8th gen Intel core There is a maximum of two beams and two columns inter-
i7 8700K CPU @ 3.70 GHz. connected to a joint. In such nodes, the plastic hinge rotations
For applications during the earthquake events, the damage in the two beams may result in different DCs. Moreover, some
detection and assessment algorithm should be robust to the nodes have only one or none of one element type (e.g., ground-
measurement noise in sensor recordings. Although simula- level nodes do not have beams). It is intended to generalize the
tions are performed assuming raw GM data, we artificially definition of damage regardless of node location. Hence, for
distort the input and response acceleration records. Random each element type (beam or column), the maximum plastic
Gaussian noise with a noise to signal ratio (NSR) of 0.8 (ratio hinge rotations are considered. The logic behind this is that,
between their variances) is assumed to add artificial noise for for near real-time applications, the decision maker is more
individual (event, scale, node) tuples. An example of this pro- interested in obtaining information on critical state rather than
SAJEDI AND LIANG 9

damage level 3 and the other with 0, the more critical value
(3 in this joint) is selected to label the beam damage level (the
same rule applies to the columns). Based on the critical dam-
age level of beam and column, a DC-ID is selected in accor-
dance with Table 2. If a given element type does not exist, we
simply assume that its damage level is zero. For instance, the
ground truth label for the base level nodes is limited to 1, 3,
6, and 10 damage classes.
FIGURE 6 Plastic hinge formation in columns (Feng, Qiang, As shown in Figure 4, the grid environment considered in
Qin, & Gao, 2017) and beams (Pampanin, 2012) this study is 16 × 16 where the first 11 grid nodes in the two
horizontal directions are considered as sensor nodes and the
rest are null. For each scale factor, the 180 ground motion
TABLE 2 Definition of damage class in grid nodes
events are randomly shuffled. For the purpose of training,
Damage level
validation, and testing, the data set is, respectively, divided
DC-ID Column Beam Severity index Frequency
into the splits of 80%, 10%, and 10%. We propose two differ-
1 0a 0 0.0 757905 ent damage segmentation frameworks, SDS-B and SDS-16.
2 0 1b 1.0 265619 SDS-B outputs a binary decision for individual sensor nodes,
3 1 0 1.2 13780 to identify the potential damage locations (anything beyond
4 0 2c 2.0 219669 DC-1). In an extended version of this framework, SDS-16
5 1 1 2.2 2526 not only locates the potential existence of damage but also
6 2 0 2.4 1331 classifies its severity into one of the 16 classes defined in
7 0 3d 3.0 35713 Table 2.
8 1 2 3.2 5013
9 2 1 3.4 97
10 3 0 3.6 3667
11 1 3 4.2 1028
4 ARCHITECTURE DESIGN
12 2 2 4.4 170
The proposed deep learning model, similar to most data-
13 3 1 4.6 61
driven methods, has several hyperparameters that affect the
14 2 3 5.4 43 performance of the classifier. The performance in this context
15 3 2 5.6 122 can be regarded as robustness and computational efficiency.
16 3 3 6.6 56 Although the existing deep learning models in the literature
a
Immediate occupancy (IO). are commonly optimized for image classification, signal pro-
b
Life safety (LS). cessing is a completely new domain where input and output
c Collapse prevention (CP).
d arrays have different physical interpretations and dimension-
Beyond CP (E).
ality. That being said, some predefined settings of the vision-
based models are unlikely to be appropriate for the task of
the unnecessary details (e.g., if damage occurs in the left beam damage diagnosis. For example, we will show that damage
or the right one in a joint). The definition of damage mecha- localization can be effectively performed with relatively shal-
nisms (combinations of critical beam and column damage lev- lower neural networks. This issue is investigated in this sec-
els) and corresponding observation frequencies are presented tion, where we propose a systematic procedure to tune the
in Table 2. SDS framework.
The DCs are sorted based on their consequences as men- Model training is accelerated by running epochs on an
tioned in Section 2. This task could be challenging and NVIDIA GeForce GTX 1080 GPU @ 8 GHz with 2560
requires domain expertise. In this article, based on the seis- CUDA cores and mini-batch sizes of 256. Keras API (Chollet,
mic design concept known as “weak beam, strong column,” 2015) is used in this study to train the deep learning mod-
we consider a damage severity index, which is obtained by els. Before proceeding to the design process, it is necessary
the summation of critical damage index of beams and 1.2 to build a baseline model while we present a step-by-step
times the damage index of columns. The logic behind this design process. At this point, the predefined coefficients of
selection is inspired by provision 18.7.3.2 in ACI 318-14 the SDS framework can be summarized as: 𝑢̂ 0 = 0, 𝑢̂ 1,…,𝑝 = 1,
(2014). 𝜓𝑗 = 1, 𝜏 = 2, 𝜅 = 0.1, 𝑤1 = 10, 𝑤2 = 3, 𝑤3 = 1. Noisy raw
To better illustrate the labeling process, an example is pro- acceleration values are preprocessed to generate input obser-
vided. If there are two beams connected to a joint, one with the vations assuming 𝜂 ∈ {0.4, 0.8, 1.2, 1.6} in four acceleration
10 SAJEDI AND LIANG

(a) (b)

FIGURE 7 Validation loss of SDS-16 using (a) default hyperparameters and (b) hyperparameters optimized by BO. Pairs of (a,b) in the legend
denote validation accuracy (%) for the two sets of hyperparameters

channels and one null data channel. This leads to an input 4.2 Hyperparameters of the CNN
dimension of 16 × 16 × 5. Considering that the number of architecture
filters (𝑛f ) can vary in different encoder–decoder blocks, for
The architecture design is the second step in building the SDS
simplicity, we imply this attribute of the network architecture
framework. It is noted that the total number of individual pix-
with the hyperparameter 𝛼. Therefore, the first encoder (or
els in images is usually significantly greater than that of the
last decoder) block has 2𝛼 filters and the last encoder block
sensor placements in grid environments. For example, Seg-
(or the first decoder) has 𝑛f = 2𝛼+𝑛b −1 (refer to Figure 2). For
Net (𝛼 = 6, 𝑛b = 4) takes 360 × 480 input images, whereas
the deep learning model at this stage: 𝛼 = 5 and 𝑛b = 3, that is,
our grid environment is 16 × 16. For near real-time dam-
the model is assumed to have three pairs of encoder–decoder
age diagnosis, it is reasonable to select the depth and other
blocks with, respectively, 25 , 26 , and 27 filters and 3 × 3 slid-
network dimensions accordingly. Another reason for choos-
ing kernels (𝑘s = 3).
ing a 16 × 16 grid environment is for computational con-
sideration. After performing the max-pooling operation, the
4.1 Weight optimizers dimensions corresponding to grid height and width will be
reduced by half. To properly decode the extracted features
The first step of SDS design is to select a proper optimizer and map them to the same grid environment (𝑛hg × 𝑛vg ), it
for learnable parameters. Stochastic gradient descent (SGD), is necessary to maintain the corresponding dimensions divis-
RMSprop, Adagrad, Adadelta, Adam, Adamax, and Nadam ible by two. Therefore, having a grid size of 16 in both direc-
are investigated for training. Interested readers are referred to tions makes it possible to have a maximum of four encoder
the Keras documentation (Chollet, 2015) for the existing liter- blocks.
ature on these optimizers. The model that was described ear- A parametric study is performed to evaluate different archi-
lier is trained for two sets of hyperparameters. Initially, SDS- tecture designs for robustness and computational efficiency.
16 is trained for 200 epochs with default settings. Then, lever- The following pools of design parameters are considered:
aging Bayesian optimization (BO), hyperparameters (e.g., 𝑛b ∈ {2, 3, 4}, 𝛼 ∈ {2, 3, … , 8}, and 𝑘s ∈ {2, 3, 4, 5, 6}. Lim-
learning rate and decay) are optimized to maximize the score iting the maximum number of network parameters (𝑁par ) to
objective function proposed in Equation (11). The loss func- 20 million, 90 possible combinations can be considered for
tion in Equation (2) is monitored per epochs of training for SDS-B and SDS-16. It should be noted that the architecture
the validation set. It should be noted that due to different ran- of SDS-B differs from SDS-16 in the last convolution layer
dom weight initializations for each instance of training, the due to different output shapes. Very deep or shallow encoder–
final accuracy and loss value will slightly vary. The results are decoder neural networks might be prone to under- or over-
presented in Figure 7 where the validation accuracy for each fitting. It will be shown that multiple designs have similar
optimizer is also included. The default settings for SGD lead performance in terms of the global score (GS). That being
to relatively poor results without Bayesian optimization. After said, the results of the optimal architecture design are pre-
conducting Bayesian optimization, it is observed that using sented based on two criteria: the global score defined in Equa-
learning rates greater than 0.01 leads to faster convergence for tion (11) and computational efficiency. The ones presented
SGD. The other methods had similar performance or insignif- in the second category are selected based on a compromise
icant improvements after optimization. Overall, Nadam pro- in the global score but offering significantly reduced cost of
vides more accurate and stable output and is used in further computation. Because the real-time performance of the model
tuning of the models. may also depend on software and hardware configuration, we
SAJEDI AND LIANG 11

characterize such models based on 𝑁par in the deep learning inant where damage is localized in the elements of a cer-
framework. To do so, the top 10% high score models are sorted tain story level. Although the design engineer may be able
based on the number of parameters and the ones with min- to avoid certain undesirable patterns, SHM mainly deals with
imum 𝑁par are selected as efficient models. The results are existing structures that may have not been designed to satisfy
summarized in Tables 3 and 4. the current code requirements. Moreover, uncertainties asso-
For both models, it is evident that there is a significant dif- ciated with extreme events (e.g., earthquakes) turn the near
ference between the number of network parameters in two real-time SHM of modern structures into a challenging task,
categories. It is also worthy of mention that, compared to where there exists a significant imbalance between different
SDS-16, SDS-B can achieve the optimal performance with damage classes. In the previous section, we monitored the
relatively simpler (shallower) architectures. Considering the model’s performance based on global score values and global
number of parameters, GS, and global accuracy, we will use accuracy (𝐴p ). It is mathematically evident that 𝐴p is highly
a deep learning architecture with three pairs of encoder– dominated by the prediction outcome of undamaged nodes
decoder blocks, 𝛼 = 5, and a kernel size (𝑘s ) of 5 for both given their frequency (see Table 2). This imbalance might
SDS-16 and SDS-B. result in a damage diagnosis model that has superior per-
formance to correctly diagnose the undamaged nodes while
4.3 Damage class imbalance the accuracy for less frequent scenarios is poor. That being
said, there is equal or greater interest in the correct prediction
In large-scale structures, the occurrence and intensity of dam- of damaged classes in SHM. In what follows, we investigate
age are commonly not uniformly distributed among structural methods to improve average class accuracies concerning this
elements. On the contrary, it usually follows specific patterns issue.
that could lead to certain collapse mechanisms. For exam- Class imbalance is also an issue in computer-vision for
ple, in a moment frame, soft-story mechanism may be dom- semantic segmentation where the less important background
pixels dominate object classes in terms of pixel frequency.
TABLE 3 SDS-B architecture design summary Tuning the training weights (𝑢) ̂ can be used as a means to
𝒏𝐛 𝜶 𝒌𝐬 GS 𝑨𝐩 (%) 𝑵𝐩𝐚𝐫 modify loss function to favor certain classes in the training
High score models 3 7 3 40808.42 96.18 5323523 process. Median frequency weight (MFW) assignment has
2 8 5 40737.20 96.23 13147651
been adopted in vision semantic segmentation research where
a class imbalance weight is obtained by dividing the median
2 8 6 40727.33 95.86 18928899
class frequency by the corresponding number of observations
3 5 5 40699.17 96.10 927939
in that class (Eigen & Fergus, 2015). Note that this strategy
3 5 4 40685.09 96.26 594723
will only affect the training process (loss values) of the deep
Efficient models 3 2 5 37740.39 91.71 15195
learning model. At the time of preparing this article, a new
3 3 3 37801.41 91.45 21683 approach was proposed by Chan, Rottmann, Hüger, Schlicht,
3 2 6 37624.94 92.01 21751 and Gottschalk (2019) in which the uniform weights (UW) are
4 2 3 37725.42 91.21 22107 utilized in training but the decision rule is changed by weight-
2 4 4 37252.93 91.38 34579 ing posterior class probabilities. In other words, besides train-
ing, the main difference between the two approaches is that the
first one uses the maximum a posteriori probability (MAP),
TABLE 4 SDS-16 architecture design summary whereas the latter utilizes the maximum likelihood (ML) to
𝒏𝐛 𝜶 𝒌𝐬 GS 𝑨𝐩 (%) 𝑵𝐩𝐚𝐫 modify class prediction probabilities.
High score 4 6 4 29274.40 93.27 9715153 In this article, we compared the baseline model (UW-
models MAP) with the two proposed approaches (UW-ML, MFW-
2 8 6 28882.31 93.65 18932497 MAP) while also experimenting with the combination of two
3 5 4 28244.72 93.18 595185 strategies (MFW-ML). In ML decision making, softmax prob-
3 6 5 28189.61 93.68 3699985 abilities are divided by prior class probabilities obtained from
2 8 4 28084.34 93.37 8421137 damage class frequencies in the training set. These prior prob-
Efficient 3 4 5 26815.23 92.91 233809 abilities are independently obtained for sensor nodes in the
models grid environment based on their location. Damage classes that
3 5 3 26351.43 93.39 336017 do not exist in the training data for a single node are elim-
3 5 4 28244.72 93.18 595185
inated from softmax output probabilities. The logic behind
this elimination stems from the numerical issue of division
3 5 5 27216.08 93.80 928401
by zero class probabilities. Also, we intend to avoid our
4 4 5 26495.12 93.34 951889
deep learning framework to predict classes beyond what it is
12 SAJEDI AND LIANG

TABLE 5 The comparison of imbalance handling strategies on observation from Table 5 is that the undamaged class accuracy
SDS-16 (UCA) is reduced after increasing 𝜅 as the SDS framework is
𝜿 = 1/6 𝜿 = 1/10 penalized less intensely for misclassifying undamaged node
𝑨𝐩 MCA 𝑼 𝑪𝑨 𝑨𝐩 𝑴𝑪𝑨 𝑼 𝑪𝑨 observations. The adoption of either modifying training class
Models (%) (%) (%) (%) (%) (%) weights or altering the decision rule contributes to improv-
UW-MAP 93.23 45.44 95.87 92.85 42.23 96.16 ing MCA. We observe that the combination of two strategies
UW-ML 91.34 55.77 94.16 92.22 49.97 94.66 results in the best model regarding MCA.
MFW-MAP 82.06 60.19 81.75 77.82 55.44 85.20 Based on the presented results and setting 𝜅 = 1∕6, a
MFW-ML 71.01 61.66 64.97 73.45 58.44 75.53 detailed report of the class accuracies is provided in Table 6.
It can be seen that the prediction accuracy is significantly
trained for. For example, in the ground level, only one of the improved for less frequent damage classes and the strategy of
damage labels 1, 3, 6, and 10 is physically possible because MFW-ML yields the optimal MCA. In this article, all types of
plastic hinges can only form in the columns as no beam exists accuracies are calculated considering the probability of each
there. GM event. For example, in DC 16, only one of the six test
In the following, these strategies are investigated for two observations is correctly predicted but the correct prediction
different values of 𝜅 ∈ {1∕6, 1∕10} in SDS-16. Because the belongs to a less likely seismic hazard level. Hence, instead
decision rule and weights are different in these experiments, of 16.7%, class accuracy is reported as 2.4%.
Bayesian optimization is utilized to tune Nadam hyperparam- The SDS-B framework, which is designed to identify the
eters to improve the performance of each model. It is observed location of damage, is also investigated using the same balanc-
that the models using the MFW strategy require longer train- ing strategies. It should be noted that for this model, the distri-
ing to converge. Therefore, 200 epochs for UW and 400 bution of the damage classes is relatively balanced compared
epochs for MFW models are used in the training process. Note to that of SDS-16. As a result, the performance of the frame-
that the objective function of Bayesian optimization is calcu- work to identify damage location is more robust while the
lated on the validation set while we compare the models based strategy of MFW-MAP gives the best performance. A sum-
on unseen test data. Given this description, mean class accu- mary of the results is presented in Table 7.
racies (MCA) of 16 possible labels are recorded in Table 5.
For a specific DC, class accuracy is used as a metric to mon-
itor the weighted percentage of correct predictions among all 5 STOCHAST IC TEST DATA SET
nodes. It is observed that by increasing 𝜅, as intended, the
models are more sensitive to the misclassifications in more In the previous sections, 10,800 simulations are per-
severe damage levels, which results in greater MCA. Another formed and the data set is randomly shuffled for training,

TABLE 6 SDS-16 class accuracy report with 𝜅 = 1/6

Frequency Test set class accuracies (%)
DC-ID Training set Test set UW-MAP UW-ML MFW-MAP MFW-ML
1 604056 77419 95.87 94.16 81.75 64.97
2 212504 26791 88.66 86.67 80.82 78.36
3 11243 1194 80.65 77.69 81.99 79.04
4 177703 20305 91.29 88.12 85.18 82.70
5 2031 254 64.25 70.23 86.32 89.18
6 1059 130 21.11 79.14 42.59 72.29
7 28719 3516 89.87 94.49 93.61 94.05
8 3988 482 70.34 75.15 86.38 90.15
9 62 19 0.00 26.04 23.31 37.69
10 2902 414 82.12 74.43 95.76 74.45
11 831 101 43.08 65.93 83.47 95.00
12 136 22 0.00 53.82 47.67 54.45
13 47 9 0.00 0.00 32.18 32.18
14 31 2 0.00 0.00 0.00 0.00
15 88 16 0.00 6.55 39.65 39.65
16 40 6 0.00 0.00 2.40 2.40
SAJEDI AND LIANG 13

TABLE 7 SDS-B class accuracy report (𝜅 = 1/6)

Frequency Test set class accuracies (%)
Target class Training set Test Set UW-MAP UW-ML MFW-MAP MFW-ML
Damage 604056 77419 97.26 95.37 96.12 97.00
No damage 441384 53261 94.33 96.39 96.16 94.72

(a) (b)

FIGURE 8 Stochastic sampling for the test observations

validation, and testing splits. It should be emphasized that all from the adjacent elements in the frame for a single obser-
the observations are obtained using the same physical model vation. Nonetheless, a specific element (e.g., the beam in the
but for different ground motion inputs. In other words, the third bay of the second level) has different sampled properties
SDS framework is tested for a series of unseen excitations but in different observations (unlike the ideal FE model). We will
on the same building model. No matter how sophisticated is denote the second data set as the stochastic test set thereafter
the numerical simulation, the structural properties of a real in the article. This data set includes 1,080 unique frames sub-
building can be different from the ideal finite element (FE) jected to 1,080 unseen input motions. The proposed stochastic
model. Such properties are not deterministic. With this mind- sampling is illustrated in Figure 8b.
set, a second test data set is generated while considering that
some principal parameters of the FE model are treated as ran-
dom variables (RVs). 6 P ERFO RM ANCE EVALUATION
The compressive strength of concrete (𝑓𝑐′ ) and rebar’s yield
strength (𝐹𝑦𝑟 ) are subject to variations due to the manufac- In this section, the SDS classifiers obtained from Section 4
turing and construction quality. Moreover, due to formwork will be further tested on the stochastic data set and the results
imperfections, the as-built dimensions of a section (height and from the ideal and stochastic test data sets are compared with
width) are commonly different from the design blueprints. It is each other. Later in this section, a detailed investigation of
assumed that these four model properties are randomly sam- models’ robustness is given through CSMs.
pled from normal distributions. This sampling is performed
for all the elements of a frame and independently for each test 6.1 Comparison of the ideal and stochastic
observation. test sets
The mean value of each distribution is obtained from the
ideal FE model (see Section 3.1). Standard deviations are GA and MCA metrics of the trained models in Section 4 are
selected from the statistical data in the existing literature (e.g., compared for the two test data sets. For the SDS-B model, the
Arafah, 1997) and standards (ACI, 2006, 2014) for a reason- results are summarized in Table 8. Similarly, the performance
able estimate. Figure 8a provides more details regarding the of the SDS-16 is expressed in Table 9. In the stochastic test
Gaussian distribution of each RV. data set, GA has slightly decreased for both SDS-B and SDS-
It should be emphasized that, for each observation, a 16 although the performance is still promising. An important
new stochastic physical model is built for the corresponding point to consider here is that the frames tested in the stochas-
NTHA. In this way, the properties of an element are different tic data set are different from the ideal FE model, resulting in
14 SAJEDI AND LIANG

TABLE 8 Comparison of two test sets on SDS-B (a)

GA (%) MCA (%)
Models Ideal Stochastic Ideal Stochastic
UW-MAP 96.09 94.89 95.80 94.55
UW-ML 95.78 94.64 95.88 94.70
MFW-MAP 96.13 94.84 96.14 94.80
MFW-ML 96.09 94.65 95.86 94.31

TABLE 9 Comparison of two test sets on SDS-16

GA (%) MCA (%)
Models Ideal Stochastic Ideal Stochastic
UW-MAP 93.23 91.32 45.45 46.05
(b) (c)
UW-ML 91.34 89.78 55.78 56.07
MFW-MAP 82.06 81.50 60.19 66.35
MFW-ML 71.01 70.72 61.66 65.95

observations that are different in both input (building accel-

eration response) and target/output values (damage distribu-
tion). Therefore, the frequency of damage classes will not be
the same in the two test data sets. As a result, one may see an
increase in the MCA values for the stochastic model. FIGURE 9 The detailed nodal performance of SDS-B for
damage localization (𝜅 = 1∕6, MFW-MAP)
6.2 Confusion score matrices
The performance of the proposed framework can be bet- the joint in the level 2, pier 2 (Figure 9c), which is the base
ter investigated utilizing the concept of CSM as previously node in the middle column of the RC moment frame. The rel-
defined in Section 2. Unlike the tabular results reported ear- atively lower score values in the second row (compared to the
lier, the CSM not only provides valuable information about undamaged case, that is, first row) imply that for the majority
the diagnosis framework but also provides a statistical sum- of test observations this node remains undamaged. The more
mary of structural performance. For example, it is possible to complex level of damage diagnosis, identifying the damage
observe what are the more frequent damage classes in each severity, is presented in the following. CSMs are provided for
node. To gain a better insight on correct predictions and mis- the SDS-16 model following the MFW-ML strategy that was
classifications for each damage level, the arrays of the CSM presented in Section 4. Similar to the way of presenting in
are normalized by the maximum score value in each row and Figure 9, the performance of SDS-16 with 𝜅 = 1∕6 is given in
assigned a color scale accordingly. The color scales make it Figure 10.
possible to obtain a general insight into how the SDS frame- Given the total number of possible classes, the correspond-
work identifies damage in different nodes compared to each ing CSMs are 16 × 16 matrices. However, for some nodes,
other. All the graphs are generated from the stochastic test set certain classes are not observed in the data set. For a more
described in Section 5. effective graphical representation, the damage classes with
For damage localization, the SDS-B model trained and all-zero rows and columns are eliminated from CSMs. More-
optimized with respect to the MFW-MAP strategy is pre- over, the elements of the CSM with insignificant score values
sented. For each pier–level combination, the CSM is given (i.e., less than 0.01) are denoted as 𝜀. Figure 10a summarizes
on the frame joints (Figure 9a) while two sample CSMs are the global pier–level nodal CSMs based on the color scale
taken out and investigated (Figures 9b and c). The model’s technique described earlier. It can be seen that the prediction
robustness is verified by observing that, the higher score val- for the majority of nodes is reasonably accurate. Figures 10b
ues are mainly distributed in the main diagonal for almost and 10c are examples of the typical CSMs obtained from the
all nodes. The CSM for the joint in level 7 and pier 5 test observations. Note that, for all the three CSMs shown
(Figure 9b) provides a representative example in which the in detail, the sum of the score values in the lower diagonal
distribution of damaged and undamaged classes is rather bal- is significantly smaller compared to that of the correspond-
anced. In this case, 95.6% accuracy is obtained for the pre- ing upper diagonal values. This observation supports the
diction of the damaged class. The other CSM is provided for fact that the proposed model has been designed to minimize
SAJEDI AND LIANG 15

(a)

(b)

(d)

(c)

FIGURE 10 The detailed nodal performance of SDS-16 (𝜅 = 1∕6, MFW-ML)

16 SAJEDI AND LIANG

underestimated damage predictions. Although, some nodes • The concept of a grid environment makes it possible to
may still have lower diagonal CSM elements. Moreover, there feed information from a variable number of sensors with-
exist dark-colored arrays denoted as 𝜀. The dark color in such out changing the deep learning architecture. This is made
arrays results from the limited number of observations in nor- possible by introducing null nodes.
malization. For example, if there is only one test observation • The SDS framework is capable of using the relative geo-
in a CSM row, it will be normalized by itself. However, given metric location of sensors in the automatic feature extrac-
the insignificant score value, such observations are less likely tion process, which leads to superior robustness in damage
(e.g., CSM elements in for classes 7, 10, 13, and 15 of CSM diagnosis models.
in Figure 10d).
• The global accuracies of 96.3% and 93.2% are, respec-
Figure 10b implies a relatively successful prediction with
tively, achieved for the identification of damage location
limited misclassifications in which the framework is capable
and severity in the test set with the ideal FE model.
of correctly predicting even the less frequent damage classes
(e.g., DC 8 and 11). Some nodes (e.g., the CSM in Figure 10d) • Three strategies are investigated to handle the highly imbal-
have relatively high rates of misclassification for the undam- anced data set for SDS, which gives a 16.2% improvement
aged class (DC 1), which is a result of the trade-off from regarding mean class accuracy for the SDS-16 model.
increasing the mean damage class accuracies through adopt- • A stochastic test data set is generated by simulating 1,080
ing MFW-ML approach and increasing 𝜅 (see Table 6). This different RC frames than the ideal FE model. The drop in
issue will also result in all-zero rows in some CSMs. For global accuracy is shown to be at most 2% in the stochastic
example, in Figure 10d, there is no ground truth label for DCs test set. The results from the stochastic test set indicate the
9, 11, and 14; however, the misclassifications of other classes generalization capabilities and potential robustness of the
cause the existence of these CSM rows. Nonetheless, for the SDS framework.
majority of nodes, the CSMs remain diagonal.

REFERENCES
7 CONC LU SI ON S Abdeljaber, O., Avci, O., Kiranyaz, M. S., Boashash, B., Sodano, H., &
Inman, D. (2018). 1-D CNNs for structural damage detection: Verifi-
Data-driven SHM has been the focus of many studies for cation on a structural health monitoring benchmark data. Neurocom-
more than a decade. However, SHM using a large sensor net- puting, 275, 1308–1317.
work remains a challenge. Different from images that can be Abdeljaber, O., Avci, O., Kiranyaz, S., Gabbouj, M., & Inman, D. J.
cropped or resized without significant loss of information, the (2017). Real-time vibration-based structural damage detection using
one-dimensional convolutional neural networks. Journal of Sound
effects of performing similar operations on accelerometer data
and Vibration, 388, 154–170.
are unknown. Therefore, the architecture of the deep learning
ACI. (2006). Specifications for tolerances for concrete construction and
models for SHM may have to be redesigned for a different materials and commentary (ACI 117-06). Farmington Hills, MI:
geometry as the input size may vary. Moreover, the number of American Concrete Institute.
potential damage locations will change under different exci- ACI. (2014). Building code requirements for structural concrete (ACI
tations. Considering these limitations, a grid environment is 318-14) and commentary. Farmington Hills, MI: American Concrete
proposed as a part of a fully convolutional encoder–decoder Institute.
neural network for near real-time SDS. The main highlights Amezquita-Sanchez, J. P., Park, H. S., & Adeli, H. (2017). A novel
methodology for modal parameters identification of large smart
of the proposed framework are summarized as follows:
structures using MUSIC, empirical wavelet transform, and Hilbert
transform. Engineering Structures, 147, 148–159.
Ancheta, T. D., Darragh, R. B., Stewart, J. P., Seyhan, E., Silva, W.
• A cost-sensitive objective function is designed and opti- J., Chiou, B. S.-J., … Donahue, J. L. (2014). NGA-West2 database.
mized such that the different consequences of misclassifi- Earthquake Spectra, 30(3), 989–1005.
Arafah, A. (1997). Statistics for concrete and steel quality in Saudi Ara-
cations can be considered.
bia. Concrete Research, 49(180), 185–193.
• The probability of occurrence for different observations is ASCE/SEI. (2016). Minimum design loads for buildings and other struc-
implemented in the training and evaluation process of the tures (Vol. 7). Reston, VA: American Society of Civil Engineers.
damage diagnosis model. Badrinarayanan, V., Kendall, A., & Cipolla, R. (2015). SegNet: A deep
convolutional encoder-decoder architecture for image segmentation.
• A systematic design method is explained for the optimiza-
arXiv preprint arXiv:1511.00561.
tion of deep learning architecture. The presented design Bae, S., & Bayrak, O. (2008). Plastic hinge length of reinforced concrete
results are applicable to future implementation of this columns. ACI Structural Journal, 105(3), 290–300.
framework in different domains of SHM (e.g., different Behmanesh, I., & Moaveni, B. (2015). Probabilistic identification of
structural systems). simulated damage on the Dowling Hall footbridge through Bayesian
SAJEDI AND LIANG 17

finite element model updating. Structural Control & Health Moni- Ioffe, S., & Szegedy, C. (2015). Batch normalization: Accelerating deep
toring, 22(3), 463–483. network training by reducing internal covariate shift. arXiv preprint
Bruneau, M., Chang, S. E., Eguchi, R. T., Lee, G. C., O’Rourke, T. D., arXiv:1502.03167.
Reinhorn, A. M., … von Winterfeldt, D. (2003). A framework to Kashif Ur Rehman, S., Ibrahim, Z., Memon, S. A., & Jameel, M. (2016).
quantitatively assess and enhance the seismic resilience of commu- Nondestructive test methods for concrete bridges: A review. Con-
nities. Earthquake Spectra, 19(4), 733–752. struction and Building Materials, 107, 58–86.
Campbell, K. W., & Bozorgnia, Y. (2012). Cumulative absolute veloc- Kesavan, K. N., & Kiremidjian, A. S. (2012). A wavelet-based damage
ity (CAV) and seismic intensity based on the PEER-NGA database. diagnosis algorithm using principal component analysis. Structural
Earthquake Spectra, 28(2), 457–485. Control and Health Monitoring, 19(8), 672–685.
Campbell, K. W., & Bozorgnia, Y. (2014). NGA-West2 ground motion Khodabandehlou, H., Pekcan, G., & Fadali, M. S. (2019). Vibration-
model for the average horizontal components of PGA, PGV, and 5% based structural condition assessment using convolution neu-
damped linear acceleration response spectra. Earthquake Spectra, ral networks. Structural Control and Health Monitoring, 26(2),
30(3), 1087–1115. e2308.
Cha, Y. J., Choi, W., & Büyüköztürk, O. (2017). Deep learning- Koch, C., Georgieva, K., Kasireddy, V., Akinci, B., & Fieguth, P. (2015).
based crack damage detection using convolutional neural networks. A review on computer vision based defect detection and condition
Computer-Aided Civil and Infrastructure Engineering, 32(5), 361– assessment of concrete and asphalt civil infrastructure. Advanced
378. Engineering Informatics, 29(2), 196–210.
Cha, Y. J., Choi, W., Suh, G., Mahmoudkhani, S., & Büyüköztürk, O. Koziarski, M., & Cyganek, B. (2017). Image recognition with deep neu-
(2018). Autonomous structural visual inspection using region-based ral networks in presence of noise: Dealing with and taking advantage
deep learning for detecting multiple damage types. Computer-Aided of distortions. Integrated Computer-Aided Engineering, 24(4), 337–
Civil and Infrastructure Engineering, 33(9), 731–747. 349.
Chan, R., Rottmann, M., Hüger, F., Schlicht, P., & Gottschalk, H. (2019). Li, R., Yuan, Y., Zhang, W., & Yuan, Y. (2018). Unified vision-based
Application of decision rules for handling class imbalance in seman- methodology for simultaneous concrete defect detection and geolo-
tic segmentation. arXiv preprint arXiv:1901.08394. calization. Computer-Aided Civil and Infrastructure Engineering,
Chollet, F. (2015). Keras. Retrieved from https://github.com/keras- 33(7), 527–544
team/keras Li, Z., Park, H. S., & Adeli, H. (2017). New method for modal identi-
Chong, J. W., Kim, Y., & Chon, K. H. (2014). Nonlinear multiclass sup- fication of super high-rise building structures using discretized syn-
port vector machine–based health monitoring system for buildings chrosqueezed wavelet and Hilbert transforms. The Structural Design
employing magnetorheological dampers. Journal of Intelligent Mate- of Tall Special Buildings, 26(3), e1312.
rial Systems and Structures, 25(12), 1456–1468. Liang, X. (2019). Image-based post-disaster inspection of reinforced
Chun, P., Yamashita, H., & Furukawa, S. (2015). Bridge damage severity concrete bridge systems using deep learning with Bayesian optimiza-
quantification using multipoint acceleration measurement and artifi- tion. Computer-Aided Civil and Infrastructure Engineering, 34(5),
cial neural networks. Journal of Shock and Vibration, 2015, 1–11. 415–430.
Comfort, L. (1999). Shared risk: Complex systems in seismic response. Liang, X., & Mosalam, K. M. (2016). Performance-based robust non-
New York: Pergamon. linear seismic analysis with application to reinforced concrete high-
Eigen, D., & Fergus, R. (2015). Predicting depth, surface normals and way bridge systems. Berkeley, CA: University of California, Pacific
semantic labels with a common multi-scale convolutional architec- Earthquake Engineering Research Center.
ture. Proceedings of the IEEE International Conference on Computer Liang, X., Mosalam, K. M., & Muin, S. (2018). Simulation-based data-
Vision. driven damage detection for highway bridge systems. Proceedings of
Feng, P., Qiang, H., Qin, W., & Gao, M. (2017). A novel kinked rebar the 11th National Conference on Earthquake Engineering (NCEE).
configuration for simultaneously improving the seismic performance Lin, Y. Z., Nie, Z. H., & Ma, H. W. (2017). Structural damage detection
and progressive collapse resistance of RC frame structures. Engineer- with automatic feature-extraction through deep learning. Computer-
ing Structures, 147, 752–767. Aided Civil and Infrastructure Engineering, 32(12), 1025–1046.
Filippou, F. C., Bertero, V. V., & Popov, E. P. (1983). Effects of bond Mander, J. B., Priestley, M. J., & Park, R. (1988). Theoretical stress-
deterioration on hysteretic behavior of reinforced concrete joints strain model for confined concrete. Journal of Structural Engineer-
(UCB/EERC-83/19). Berkeley, California: Earthquake Engineering ing, 114(8), 1804–1826.
Research Center. McKenna, F., Fenves, G. L., & Filippou, F. C. (2010). The open system
Gao, Y., & Mosalam, K. M. (2018). Deep transfer learning for image- for earthquake engineering simulation. Berkeley, CA: University of
based structural damage recognition. Computer-Aided Civil and California.
Infrastructure Engineering, 33(9), 748–768. Mita, A., & Hagiwara, H. (2003). Quantitative damage diagnosis of shear
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. Cam- structures using support vector machine. KSCE Journal of Civil Engi-
bridge, MA: MIT Press. neering, 7(6), 683–689.
Gui, G., Pan, H., Lin, Z., Li, Y., & Yuan, Z. (2017). Data-driven support Mohd, Y. M. (1994). Nonlinear analysis of prestressed concrete struc-
vector machine with optimization techniques for structural health tures under monotonic and cyclic loads (PhD dissertation). Univer-
monitoring and damage detection. KSCE Journal of Civil Engineer- sity of California at Berkeley.
ing, 21(2), 523–534. Molina-Cabello, M. A., Luque-Baena, R. M., López-Rubio, E., &
Hoskere, V., Narazaki, Y., Hoang, T., & Spencer, B. F., Jr. (2018). Vision- Thurnhofer-Hemsi, K. (2018). Vehicle type detection by ensembles
based structural inspection using multiscale deep convolutional neu- of convolutional neural networks operating on super-resolved images.
ral networks. arXiv preprint arXiv:1805.01055. Integrated Computer-Aided Engineering, 25(4), 321–333.
18 SAJEDI AND LIANG

Muin, S., & Mosalam, K. M. (2017). Cumulative absolute velocity as a Santos, A., Figueiredo, E., Silva, M., Sales, C., & Costa, J.
local damage indicator of instrumented structures. Earthquake Spec- (2016). Machine learning algorithms for damage detection: Kernel-
tra, 33(2), 641–664. based approaches. Journal of Sound and Vibration, 363, 584–
Narazaki, Y., Hoskere, V., Hoang, T. A., & Spencer, B. F., Jr. 599.
(2018). Automated vision-based bridge component extraction Scott, M. H., & Fenves, G. L. (2006). Plastic hinge integration methods
using multiscale convolutional neural networks. arXiv preprint for force-based beam–column elements. Journal of Structural Engi-
arXiv:1805.06042. neering, 132(2), 244–252.
Oh, B. K., Kim, K. J., Kim, Y., Park, H. S., & Adeli, H. (2017). Evolu- Simonyan, K., & Zisserman, A. (2014). Very deep convolutional
tionary learning based sustainable strain sensing model for structural networks for large-scale image recognition. arXiv preprint
health monitoring of high-rise buildings. Applied Soft Computing, arXiv:1409.1556.
58, 576–585. Todeschini, C. E., Bianchini, A. C., & Kesler, C. E. (1964, June). Behav-
Ortega-Zamorano, F., Jerez, J. M., Gómez, I., & Franco, L. (2017). Layer ior of concrete columns reinforced with high strength steels. ACI
multiplexing FPGA implementation for deep back-propagation learn- Journal Proceedings, 61(6), 701–716.
ing. Integrated Computer-Aided Engineering, 24(2), 171–185. Torres, J. F., Galicia, A., Troncoso, A., & Martínez-Álvarez, F. (2018).
Pampanin, S. (2012). Reality-check and renewed challenges in earth- A scalable approach based on deep learning for big data time series
quake engineering: Implementing low-damage systems—From the- forecasting. Integrated Computer-Aided Engineering, 25(4), 1–14.
ory to practice. Bulletin of the New Zealand Society for Earthquake Wang, P., & Bai, X. (2018). Regional parallel structure based CNN
Engineering, 45, 137–160. for thermal infrared face identification. Integrated Computer-Aided
Perez-Ramirez, C. A., Amezquita-Sanchez, J. P., Adeli, H., Valtierra- Engineering, 25(3), 247–260.
Rodriguez, M., Romero-Troncoso, R. D. J., Dominguez-Gonzalez, Xue, Y., & Li, Y. (2018). A fast detection method via region-based
A., & Osornio-Rios, R. A. (2016). Time-frequency techniques for fully convolutional neural networks for shield tunnel lining defects.
modal parameters identification of civil structures from acquired Computer-Aided Civil and Infrastructure Engineering, 33(8), 638–
dynamic signals. Journal of Vibroengineering, 18(5), 3164–3185. 654.
Prestandard, FEMA. (2000). Commentary for the seismic rehabilita- Yeum, C. M., Dyke, S. J., Ramirez, J., & Benes, B. (2016). Big visual
tion of buildings (FEMA-356). Washington, DC: Federal Emergency data analytics for damage classification in civil engineering. In
Management Agency, 7. Transforming the Future of Infrastructure through Smarter Informa-
Rafiei, M. H., & Adeli, H. (2017). A novel machine learning-based algo- tion: Proceedings of the International Conference on Smart Infras-
rithm to detect damage in high-rise building structures. The Struc- tructure and Construction, June 27–29, pp. 569–574.
tural Design of Tall Special Buildings, 26(18), e1400. Yu, H., Mohammed, M. A., Mohammadi, M. E., Moaveni, B., Barbosa,
Rafiei, M. H., & Adeli, H. (2018). A novel unsupervised deep learning A. R., Stavridis, A., & Wood, R. L. (2017). Structural identification
model for global and local health condition assessment of structures. of an 18-story RC building in Nepal using post-earthquake ambient
Engineering Structures, 156, 598–607. vibration and lidar data. Frontiers in Built Environment, 3, 11.
Rafiei, M. H., Khushefati, W. H., Demirboga, R., & Adeli, H. (2017). Zhang, A., Wang, K. C., Li, B., Yang, E., Dai, X., Peng, Y., … Chen,
Supervised deep restricted Boltzmann machine for estimation of con- C. (2017). Automated pixel-level pavement crack detection on 3D
crete compressive strength. ACI Materials Journal, 114(2), 237–244. asphalt surfaces using a deep-learning network. Computer-Aided
Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster R-CNN: Civil and Infrastructure Engineering, 32(10), 805–819.
Towards real-time object detection with region proposal networks. Zhou, X., Ni, Y., & Zhang, F. (2014). Damage localization of cable-
In Advances in neural information processing systems (pp. 91–99). supported bridges using modal frequency data and probabilistic neu-
Sajedi, S. O., & Liang, X. (2019a). A convolutional cost-sensitive crack ral network. Mathematical Problems in Engineering, 2014, 1–10.
localization algorithm for automated and reliable RC bridge inspec-
tion, in risk-based bridge engineering. London, UK: CRC Press, Tay-
lor & Francis Group.
Sajedi, S. O., & Liang, X. (2019b). A data-driven framework for How to cite this article: Sajedi SO, Liang X.
near real-time and robust damage diagnosis of building struc- Vibration-based semantic damage segmentation
tures. Structural Control & Health Monitoring, 2019, e2488. for large-scale structural health monitoring. Com-
https://doi.org/10.1002/stc.2488
put Aided Civ Inf. 2019;1–18. https://doi.org/10.
Sajedi, S. O., & Liang, X. (2019c). Intensity-based feature selection for
1111/mice.12523
near real-time damage diagnosis of building structures. 2019 IABSE
Congress, New York City, NY, September 4–6.

View publication stats