A Robotic System Towards Concrete Structure Spalling and Crack Database
A Robotic System Towards Concrete Structure Spalling and Crack Database
Liang YANG1,2 , Bing LI1 , Wei LI1 , Zhaoming LIU2 , Guoyong YANG2 , Jizhong XIAO 1,2 *
Abstract— Concrete spalling and crack inspection is a labor The concept of using deep convolutional neural networks
intensive and routine task. However, it plays an important role (CNN) for tunnel visual inspection was presented in [6],
in structure health monitoring (SHM) of civil infrastructures. where related techniques and descriptions were elaborated.
Autonomous inspection with robots has been regarded as one of
the best ways to reduce both error and cost. This paper presents Authors in [6] mainly used the CNN for feature extraction
an automated approach using Unmanned Aerial Vehicle(UAV) and Multi-Layer Perceptron (MLP) for classification. The
and towards a Concrete Structure Spalling and Crack database paper didn’t make it clear that what is the database, and
(CSSC), which is by far the first released database for deep what kind of defection can be detected by the system.
learning inspection. We aim locate the spalling and crack Crack detection is discussed in [7], and it is said to have
regions to assist 3D registration and visualization. For deep
inspection, we provide a complete procedure of data searching, a better accuracy over SVM and boosting without notifying
labeling, training, and post processing. We further present the corresponding database and field tests. Authors in [8] did
a visual Simultaneously Localization and Mapping(SLAM) a good job for providing a database for crack inspection and
approach for localization and reconstruction. Comparative labeling, and evaluation of the performance.
experiments and field tests are illustrated, results show that To automate the concrete structure crack inspection, var-
we can achieve an accuracy over 70% for field tests, and more
than 93% accuracy with CSSC database. ious robotic systems have been explored. Authors in [9]
developed an early vision-based inspection vehicle robot
I. INTRODUCTION which used batch processes of crack detecting algorithm-
s and data storage in bridge management system (BMS)
Manual inspection and evaluation is a common procedure database. Vision-based UAV robots were deployed for bridge
for structural health monitoring (SHM). The human inspec- inspection and condition assessment in [10], and conven-
tors need to periodically detect the visible surface defects tional image processing approaches were applied for crack
(such as cracks) and the interior subsurface defects (such identification. Therefore, based on our previous research
as delamination or voids) using non-destructive evaluation on non-destructive evaluation (NDE) robotics for concrete
(NDE) devices such as high resolution cameras, impact echo deck inspection [11], it is very promising to deploy UAV
(IE) [1] and ground penetration radar (GPR) [2] to assess the inspection robot for low-cost and omnibearing detection.
serviceability conditions of civil infrastructures. However, By automatically collecting increasing amount of inspection
such manual inspections are time consuming, labor intensive, data along with corresponding spatial information, data-
and often require the setup of scaffolding equipment to driven based classification and recognition approaches like
access hard to reach places, which is costly and leads to CNN shows the potential to provide more robust inspection
safety concerns. detection result for SHM assessment than conventional ap-
Machine vision and automated inspection and evaluation proaches.
procedures have become very popular for detecting surface This paper present a low-cost automated inspection ap-
flaws in civil engineering communities in the past decade proach using UAV equipped with stereo-vision system. Our
[3]. Conventional image processing methods are applied for contributions are two folds. First, we build a database by
detection by emphasizing the high contrast distinct visual collecting and labelling images of spalling and cracks on
features. Edge detection algorithms, such as fast Haar trans- concrete structures for deep training purpose, and we eval-
form (FHT), fast Fourier transform (FFT), Sobel operator uate the deep learning approach performance for inspection
and Canny edge detector, have been deployed for crack based a proposed cascading shotting approach. Second, we
identification [4]. These algorithms are also used with other leveraging our previous work on UAV navigation and SLAM
algorithms such as image segmentation, image thresholding [12], [13], [14], [15]. We further developed a novel method
(like OSTU) and morphology operations [5], and work to project the labelled 2D flaws to point cloud model for 3D
well on uniform background. But these are still relying on visualization. To the best of our knowledge, this paper firstly
segmentation and handcrafted feature detectors (like edge, utilizes the robotics automated UAV for deep learning based
line) for training, and lead to failures for various defect types. visual inspection on concrete structure.
*This work was supported in part by U.S. Army Research Office under II. S YSTEM A RCHITECTURE
grant No. W911NF-09-1-0565, and NSFC under grant No. 61528303.
Jizhong Xiao is the corresponding Author. As shown in Fig.1, the inspection system consists of three
1 CCNY Robotics Lab, The City College, City University of New York,
subsystems, including: 1) Control and Mission system (CM-
NY 10031 USA jxiao,lyang1,bli,[email protected]
2 University of Chinese Academy of Sciences, Beijing, China, 100049. S): a quadrotor UAV with mission planner and navigation
gyyang,[email protected] control system [13], [12]. It fuses visual odometry with
Authorized licensed use limited to: SHENZHEN UNIVERSITY. Downloaded on March 31,2023 at 07:57:12 UTC from IEEE Xplore. Restrictions apply.
Fig. 1. The inspection system consists of CityFlyer UAV with mission planner and navigation control system, Deep-Inspector software for spalling and
crack detection and labelling, and VO-based positioning and mapping module. The final 3D map can be merged into GIS system, which can help build
seamless data net for monitoring purpose.
XIMU = [W ℜTI WV T
I
W aT
I
I
Wq ba bg ] (1) III. D EEP BASED D ETECTION
The core of the system is the deep inspection system
Where W ℜTI denotes the position of IMU in the world with database with data-collection ability. To our knowledge,
frame W . WI q is unit quaternion that represents the rotation there does not exist any database containing this kind of
from the world frame W to the IMU frame I. W VIT and W aTI information, especially for both spalling and crack. With web
are the IMU velocity and acceleration with respect to the search and real data collection in Manhattan, New York), we
world. ba and bg denote the the biases affecting the gyroscope built the CSSC database. We also keep the field test data for
and accelerometer measurements.The system derivative form public. In this section, we discuss the complete procedure
can be partially presented as following using East-North- of data preparation, labeling, training and post processing to
Up(ENU) coordinate system (partly referred in [16], [17], vote defective region for 3D registration.
Authorized licensed use limited to: SHENZHEN UNIVERSITY. Downloaded on March 31,2023 at 07:57:12 UTC from IEEE Xplore. Restrictions apply.
9** 9**
6L]HGHFLGHGE\ &RQYBBOD\HU
&RQYBOD\HUWR&RQYBBOD\HU
GLVWDQFHJDLQ WR&RQYBBOD\HU
)L[HG
)9** LPDJH
IF IF
IF
Fig. 2. Architecture of the proposed Fine-Tuned VGG-16 (FVGG) model. FVGG is robust to input resolution to enable precise detection.
Authorized licensed use limited to: SHENZHEN UNIVERSITY. Downloaded on March 31,2023 at 07:57:12 UTC from IEEE Xplore. Restrictions apply.
where p(xI ) denotes the true probability of input xI , q(xI ) (K Pi∗ ,K Pj∗ ) = arg min(K P̂j − Ti, j ·K P̂i )T Ωi, j (K P̂j − Ti, j ·K P̂i )
KP
i
is the predicted probability. (6)
Where Ωi, j denotes the information matrix, ∗ denotes the
C. Post Processing For Defects Labeling
optimized pose, ∧ denotes the estimation. After each pose
After the model is well trained, we utilize the model to optimizing process, we only consider the errors of the last
do detection. In Section III.B, we discussed that the input key frame to be corrected, and we have the error with,
images are varied with the distance. The distance factor K ∗ K −1
error = Plast · Plast . Then, we can correct the current VO
KT
can be easily achieved with stereo camera. For a constant ∗
output with Pcurrent =K Terror · Pcurrent .
sub-image size Is , we use a sliding window to do detection
through the whole image. Then each image will be labeled B. Spalling and Crack Alignment
with crack true, or spalling true, or no defects. Since the For spalling the area and depth information is interesting
accuracy of the model cannot be guaranteed to be 100%, we to engineers. The deep detection tells the regions, it enables
store all the detection results of the same place and the final marking of the region with special color in the 3D model,
decision is made through the averaging the probability. which in turn provides much easier way for evaluation. For
crack, engineers are more interested in the tiny cracks with
IV. V ISUAL P OSITIONING A ND I NSPECTION
width information. The regions can give more information
R EGISTRATION
to do post detection for crack and spalling distribution.
3D representation of the construction is commonly used
for reverse engineering applications and structure analysis in V. E XPERIMENTS
civil engineering, besides 3D model can help by providing Since the robustness of the CityFlyer control system has
depth information as well as the distribution for better post been demonstrated in [12], [13], we skip this part. The
assessment [21]. In this work, the 3D mapping system is built simulation is carried out to detect the region of spalling and
based on the visual-SLAM, then the deep based detection crack in CSSC. We provide a detailed analysis for success
result helps to register the spalling and crack region in the and failure. The two field tests were performed in Manhattan
3D map. with CityFlyer, both under bridge area.
A. Loop closing Aided VO A. Deep Network Based Detection with CSSC
Visual Odomery cannot eliminate the long term drift The initial CSSC database is divided into three parts, that
without adopting the loop closing to correct the error. For is, 70% for training, 10% for validation, and 20% for testing.
SLAM, we just record the key frames F K = {Ii , Pi |i ∈ Some results are illustrated in Fig.5. For test, we achieved a
{1, 2, ..., m}} (i.e vertex) based on detection of threshold 93.36% mean accuracy(see in Table. I). The failure detection
movement, where Ii is the image, Pi is the pose. We know is mainly caused by image blur or low contrast light.
that the step transformation between consecutive two frames, In Fig.5, images #1, #2, #3 are spalling detection results,
and the transformation Ti, j between any two frames i, j can and images #4, #5 are crack detection results. For an expert
Authorized licensed use limited to: SHENZHEN UNIVERSITY. Downloaded on March 31,2023 at 07:57:12 UTC from IEEE Xplore. Restrictions apply.
Fig. 5. Results generated by our deep based detection model with CSSC database, where the green rectangle denotes the region of spalling and crack.
In images #6 and #7, the red features denote the post-labeling of the crack for evaluation, where the width of the crack can be achieved via stereo-vision.
Fig. 7. The detection results achieved by CityFlyer in field test 1. Image #1 denotes the trajectory of the CityFlyer, #2 and #3 are detected results, and
#4 is the 3D registered model.
Fig. 8. The detection results achieved by CityFlyer in field test 2. Image #1 denotes the trajectory of the CityFlyer, #2 and #3 are detected results, and
#4 is the 3D registered model.
TABLE I
Q UANTIFIED R ESULT O F D ETECTION W ITH CSSC DATABASE
Database Average Precision (%) Partial Incomplete Detection (%) Total Image
CCNY-CSSC 93.36 6.64 1232
TABLE II
F IELD T EST R ESULT AT M ANHATTAN 155 S T
Test No. Average Precision (%) Blurred Image (frames) Average Precision Without Blur(%) Over Estimated (%) Total Image
No.1 72.45 149 76.73 97.18 4998
No.2 67.65 55 71.19 24.3 2650
Further Tuned With Field Data
No. 1 83.69 149 87.97 93.34 4998
No. 2 81.38 55 84.92 33.57 2650
engineer, the most difficult part is not to find the spalling, but B. Field Tests in Manhattan
to find the crack(especially minor crack). Thus, our method
adopts a two steps approach: 1) deep network based detection The field tests were carried out at a bridge located at upper
to the region containing the crack;2) feature detection to find west side of 155th Broadway in Manhattan. The ground
the crack distribution. Then, we can see in Fig.5, images station for monitoring and deep CNN processing was a Dell
#6, #7 with red labeling indicates the distribution of the XPS 15 laptop, with a Nvidia 960M graphic card. Two sets
crack. Mis-detection is presented in Fig.6, where image #1 of field tests were implemented with different scenario.
dneotes the case of image blur, and image #2 denotes the 1) Field Test 1: The first experiment was performed at
case of low contrast. the middle part of the bridge where a darker light affected
Authorized licensed use limited to: SHENZHEN UNIVERSITY. Downloaded on March 31,2023 at 07:57:12 UTC from IEEE Xplore. Restrictions apply.
the inspection. The flying duration was 210s with a total of R EFERENCES
4,998 images captured. It is presented in Fig.7. The purple [1] Sansalone, M.J., Streett, W.B.: Impact-echo. nondestructive evaluation
line in image #1 denotes the trajectory of CityFlyer with the of concrete and masonry. (1997)
goal to capture the spalling and crack at the right angle. [2] Daniels, J.J., et al.: Ground penetrating radar fundamentals. Prepared
as an appendix to a Report to the US EPA, Region V (2000) 1–21
The expected spalling and crack region number is two, [3] Zaurin, R., Catbas, F.: Integration of computer imaging and sensor
which is shown in Fig.7 and in image #2 and a closer look of data for structural health monitoring of bridges. Smart Materials and
right part is shown in image #3. For all the detection results Structures 19(1) (2009) 015019
[4] Abdel-Qader, I., Abudayyeh, O., Kelly, M.E.: Analysis of edge-
we found so far, shows 76.73% accuracy without image detection techniques for crack identification in bridges. Journal of
blur (see in Table.II). Our 3D registration has an advantage Computing in Civil Engineering 17(4) (2003) 255–263
to do 3D visualization, and it enables professional health [5] Sinha, S.K., Fieguth, P.W.: Automated detection of cracks in buried
concrete pipe images. Automation in Construction 15(1) (2006) 58–72
evaluation. It is shown in image #4, our method can label [6] Makantasis, K., Protopapadakis, E., Doulamis, A., Doulamis, N.,
the region in 3D space for inspection. Loupos, C.: Deep convolutional neural networks for efficient vision
Simulations with CSSC show that image blur can affect based tunnel inspection. In: In Proc. Of IEEE International Conference
on Intelligent Computer Communication and Processing, IEEE (2015)
the detection, which is also found in field test as illustrated 335–342
in Fig.9. The blue rectangles denote the missed detection. [7] Zhang, L., Yang, F., Zhang, Y.D., Zhu, Y.J.: Road crack detection using
By analyzing all the images, we found that the image deep convolutional neural network. In: Image Processing (ICIP), 2016
IEEE International Conference on, IEEE (2016) 3708–3712
brightness, image contrast, and blurred degree affect the [8] Eisenbach, M., Stricker, R., Debes, K., Gross, H.M.: Crack detection
detection accuracy, which then should be considered to be with an interactive and adaptive video inspection system
pre-processed to improve the accuracy. [9] Oh, J.K., Jang, G., Oh, S., Lee, J.H., Yi, B.J., Moon, Y.S., Lee,
J.S., Choi, Y.: Bridge inspection robot system with machine vision.
2) Field Test 2: The second experiment was carried out in Automation in Construction 18(7) (2009) 929–941
the entrance of the under bridge area, where a brighter light [10] Khan, F., Ellenberg, A., Mazzotti, M., Kontsos, A., Moon, F., Pradhan,
exists. Fig.8 shows the CityFlyer’s trajectory and several rep- A., Bartoli, I.: Investigation on bridge assessment using unmanned
aerial systems. In: Structures Congress 2015. (2015) 404–413
resentative results, and the 3D registered model is presented [11] Xiao, J., Agrawal, A.: Robotic inspection of bridges using impact-echo
in image #4. CityFlyer was set at the left side of the spalling technology. UTRC-RITA Project Final Report (2015)
and crack region, then it tried to move toward the front angle [12] Dryanovski, I., Valenti, R.G., Xiao, J.: An open-source navigation
system for micro aerial vehicles. Autonomous Robots 34(3) (2013)
of spalling and crack for inspection (see in image #1). This 177–188
site contains two spalling and crack regions as illustrated in [13] Valenti, R.G., Jian, Y.D., Ni, K., Xiao, J.: An autonomous flyer
images #2 and #3. For all the images collected, we have an photographer. In: In Proc. Of IEEE International Conference on Cyber
Technology in Automation, Control, and Intelligent Systems, IEEE
average accuracy of 71.19% without blurred image. (2016) 273–278
Compared to Test 1, the brighter image gave better in- [14] Yang, L., Qi, J., Jiang, Z., Song, D., Han, J., Xiao, J.: Guiding attrac-
spection results. It is shown in Table.II, the over detection is tion based random tree path planning under uncertainty: Dedicate for
uav. In: In Proc. Of IEEE International Conference on Mechatronics
represented by the orange rectangles in Fig.9, image #1, #2. and Automation (ICMA), IEEE (2014) 1182–1187
Over detection is casue by darker light, which lowers the [15] Yang, L., Qi, J., Cao, Y., He, Y., Han, J., Xiao, J.: Uav path planning
feature comparison between palling and crack region with framework under kinodynamic constraints in cluttered environments.
In: International Conference on Intelligent Robotics and Applications,
other regions. Thus, Test 1 has a 97.18% over estimation Springer (2015) 248–259
compared to 24.3% of brighter images. [16] Mourikis, A.I., Roumeliotis, S.I.: A multi-state constraint kalman filter
As discussed in simulation and experiment 1, image blur for vision-aided inertial navigation. In: In Proc. Of IEEE international
conference on Robotics and automation, IEEE (2007) 3565–3572
and brightness affect the detection a lot. Besides, we further [17] Armesto, L., Tornero, J., Vincze, M.: Fast ego-motion estimation with
fine-tuned the model with field test image data with the same multi-rate fusion of inertial and vision. The International Journal of
process as discussed in Section.III. The comparative results Robotics Research 26(6) (2007) 577–589
[18] Prasanna, P., Dana, K.J., Gucunski, N., Basily, B.B., La, H.M., Lim,
are illustrated in Table.II, we can see that we got 11.14% R.S., Parvardeh, H.: Automated crack detection on concrete bridges.
and 13.73% improvement after fine-tuned with filed images. IEEE Transactions on Automation Science and Engineering 13(2)
(2016) 591–599
VI. CONCLUSIONS [19] Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y.,
Berg, A.C.: Ssd: Single shot multibox detector. In: European
The paper proposed a automated UAV-based system for conference on computer vision, Springer (2016) 21–37
concrete structure inspection with the ability to locate s- [20] Simonyan, K., Zisserman, A.: Very deep convolutional networks for
large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
palling and crack area and then to provide depth and width [21] Liu, W., Chen, S.e.: Reliability analysis of bridge evaluations based
information for evaluation. Inspection system consists of C- on 3d light detection and ranging data. Structural Control and Health
ityFlyer for image capturing with a visual-inertial odometry, Monitoring 20(12) (2013) 1397–1409
[22] Kümmerle, R., Grisetti, G., Strasdat, H., Konolige, K., Burgard, W.: g
deep based spalling and crack detection, and 3D registration 2 o: A general framework for graph optimization. In: In Proc. Of IEEE
for visualization. The system can significantly lower the labor International Conference on Robotics and Automation, IEEE (2011)
costs. Moreover, it has an advantage of accurate automated 3607–3613
detection. The system shows a 93.36% detection accuracy
with CSSC database, and over 70% accuracy with field tests.
The system supports real-time localization with detection
processing and can be used as a reliable system for concrete
structure health monitoring.
Authorized licensed use limited to: SHENZHEN UNIVERSITY. Downloaded on March 31,2023 at 07:57:12 UTC from IEEE Xplore. Restrictions apply.