Computer-Aided Civil and Infrastructure Engineering 30 (2015) 759–770

Vision-Based Automated Crack Detection for Bridge

Inspection
Chul Min Yeum
The School of Civil Engineering, Purdue University, West Lafayette, IN, USA

&

Shirley J. Dyke*
The Schools of Mechanical Engineering and Civil Engineering, Purdue University, West Lafayette, IN, USA

Abstract: Visual inspection of bridges is customarily which are regularly scheduled inspection every 2 years
used to identify and evaluate faults. However, current (Phares et al., 2001). The study showed that there is a
procedures followed by human inspectors demand long great discrepancy in the results when the same structure
inspection times to examine large and difficult to access is inspected by several inspectors. Factors include
bridges. Also, highly relying on an inspector’s subjective accessibility, light intensity, lack of specialized knowl-
or empirical knowledge induces false evaluation. To ad- edge, perception of maintenance, and visual acuity and
dress these limitations, a vision-based visual inspection color vision. Although some factors are attributed to
technique is proposed by automatically processing and carelessness or improper training of inspectors, most of
analyzing a large volume of collected images. Images them cannot be physically overcome by a human in the
used in this technique are captured without controlling current visual inspection process.
angles and positions of cameras and no need for prelim- To tackle this issue, initially, visual information from
inary calibration. As a pilot study, cracks near bolts on the bridge should be remotely accessed and collected
a steel structure are identified from images. Using im- automatically according to established standardized
ages from many different angles and prior knowledge procedures. In the literature, many researchers have
of the typical appearance and characteristics of this class proposed remote access of image acquisition systems for
of faults, the proposed technique can successfully detect capturing images under or over bridges. A visual moni-
cracks near bolts. toring system was proposed by controlling several cam-
eras mounted on bridges to collect images (Jahanshahi
1 INTRODUCTION et al., 2011). Using these images, the scenes of bridges
are periodically constructed to evaluate the evolution
Visual inspection is the primary method used to of cracks or corrosion. Another approach is to develop
evaluate bridge condition. Most decisions relating to equipment for improving accessibility to large bridges.
bridge maintenance are based on assessments from U-BIROS (Ubiquitous Bridge Inspection Robot Sys-
visual inspections. However, current visual inspection tem) is a robotic image acquisition system, which scans
conducted by human inspectors has several limitations. bridges using a robotic arm equipped with cameras
The study, conducted by the U.S. Federal Highway Ad- (Lee et al., 2011). This system is similar to an under-
ministration’s Nondestructive Evaluation Validation bridge inspection vehicle but replaces a bucket with
Center (NDEVC) in 2001, investigated accuracy and cameras. The California Department of Transportation
reliability of routine and in-depth visual inspections, (Caltrans) bridge inspection project developed a wired
aerial robotic platform for close inspection of bridges
∗ Towhom correspondence should be addressed. E-mail: sdyke@ or other elevated highway structures (Moller, 2008;
purdue.edu. Miller, 2004). The vehicle is capable of vertical takeoff

C 2015 Computer-Aided Civil and Infrastructure Engineering.

DOI: 10.1111/mice.12141
 14678667, 2015, 10, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/mice.12141 by <Shibboleth>[email protected], Wiley Online Library on [27/02/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
760 Yeum & Dyke

Fig. 1. Images of a fatigue crack on a steel beam, captured from different viewpoints. Note that the fatigue crack in the images is
initiated from the tip of the artificial notch near the bolt, which is to simulate initial discontinuity of the circumference of the
bolt hole.

and landing, translation to horizontal movement and inspectors. The system automatically generates an
orienting a video camera, all controlled by operating inspection report to help expert visual inspectors make
personnel on the ground. Recently, a multi-rotor decisions whether the tested bridge requires further
helicopter became commercially available to wirelessly inspection or prompt maintenance. By preserving and
take pictures or videos for inspection of structures archiving such reports and decisions over the lifetime of
such as pipelines, power lines, or dams (Angel Aerial the structure, the system would continue updating to be
Survey, 2014; US Aerial Video, 2014). more robust and smarter in the inspection of damage,
Once a suitable collection of visual information is reducing false alarms or misdetections.
collected from a bridge, robust inspection techniques This study was begun by exploring the question:
should automatically perform the visual inspection Given a large volume of images from the UAV, would it
tasks set forth in the manual. This autonomous in- be feasible to detect damage in a realistic structure using
spection is not a new concept and is widely developed currently available vision-based damage detection tech-
and used for civil, mechanical, or aerospace structures niques? To answer this question, multiple photographs
(Abdel-Qader et al. 2003; Shen et al., 2013; Zhu et al., were taken of a steel beam having a real fatigue crack
2011; Malekian et al., 2012; O’Byrne et al., 2013, 2014; that initiated from one of bolt holes. Analysis of the data
Jiang and Adeli, 2007). In this study, the focus is on was performed using standard available methods found
cracks because it is a common type of damage in civil in the literature, such as edge detectors or morpholog-
structures and is typically visually clear in appearance. ical detectors (Jahanshahi et al., 2009). As a result, we
A review of vision-based crack detection techniques is identified two major issues that need to be addressed
available in references Jahanshahi et al. (2009, 2013) for vision-based automated inspection, which previous
and Torok et al. (2013). researchers may or may not have observed. First,
Futuristic visual inspection is motivated and imag- searching for cracks over the entire area of an image
ined as follows. An unmanned aerial vehicle (UAV) generates many false-positive alarms and misdetec-
equipped with high resolution cameras arrives at a tions. In Figure 1a, there are many crack-like features
candidate bridge for inspection. Following preliminary such as structure boundaries, wires, or corrosion edges,
designed flying paths or navigating automatically, the causing either incorrect detection or a failure to detect
UAV collects and records images. The flying path real cracks due to its narrow width. However, in these
is periodically updated based on previous inspection cases detection of the real crack by human inspectors
records, for example, taking more images in damaged is not easy but is still possible. This detection is because
areas as detected in previous flights. The UAV trans- they have prior knowledge about the crack’s typical
mits collected images to a base station. At the base appearance and characteristics. In this case, the rel-
station, processing takes place on the large volume evant information is that cracks on a steel structure
of images, and damage on the structure is detected, have thin and shiny edges and are often initiated and
localized, and quantified automatically without human propagated from bolt holes (Indiana Department
 14678667, 2015, 10, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/mice.12141 by <Shibboleth>[email protected], Wiley Online Library on [27/02/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Vision-based automated crack detection for bridge inspection 761

of Transformation, 2013). These features draw their few images taken from set positions where cracks are
attention to bolts and nearby areas, facilitating crack visually clear. However, automatic image collection
detection. A second issue observed is that the crack using aerial cameras or other equipment would not
may be visible or invisible depending on the viewpoint. guarantee that favorable images would be obtained
Figure 1b shows identical scenes of a fatigue crack due to the uncertainty in crack’s location and direction.
but from two different viewpoints. Comparing the Instead, the proposed technique begins by searching
white dotted boxes in both images, Figures 1a and damage sensitive areas from a large pool of images. By
b, the crack is hardly observed in the second image. detecting these areas from many different viewpoints,
Therefore we conclude that images of same scene from detectability of damage can be dramatically increased
many different viewpoints are needed to detect the even if it is small, and false-positive alarms can be
crack without knowing how and where it is created and reduced by limiting searching areas. To the best knowl-
propagated. Most previous research, of course, has un- edge of the authors, there is no literature developing
consciously considered these two issues as they collect vision-based visual inspection with this concept.
images through controlling the circumstances, such as The remainder of this article is organized as follows.
camera positions or angles depending on appearance Section 2 starts from the brief overview of the proposed
and location of cracks. However, in reality, it is hard to approach and provides technical details about image
obtain sufficiently good images taken under the “best” acquisition, object detection and grouping, and crack
conditions because the crack location, crack direction, detection. Experimental descriptions and results are
and lighting direction cannot be known in advance, presented in Section 3. Section 4 includes summary and
and also it is hard to precisely and continuously control conclusion.
camera positions and angles installed in the UAV.
In an attempt to consider the above two findings,
we build the framework for the proposed technique. 2 METHODOLOGY
In this study, an automated crack detection technique
is proposed using images collected under uncontrolled The overview of the proposed technique is shown in
circumstances. Rather than searching for cracks on Figure 2. First, in Figure 2a, images of the structure
entire images, objects which have areas susceptible from many angles are collected using image acquisition
to cracks (bolts in this study) are first detected in the equipment. This step may involve one or more possible
images. This step greatly increases detectability of ways of image collection such as aerial cameras or in-
cracks by narrowing down searching areas and scales in spection robots. Second, in Figure 2b, target structural
acquired images. Object detection and grouping tech- components called objects, which are susceptible to
niques used in computer vision areas are implemented cracking, are detected and extracted from the images.
to extract, match, and group the same objects from The object patch indicates one such object and its
many angles across images. Crack-like edges, which are nearby area where cracking is more likely to present.
similar in appearance to real cracks, are first detected Third, in Figure 2c, common object patches (corre-
from images of object areas using image processing sponding to the same object) across the collection of
techniques. Then based on prior knowledge of crack’s images are matched and grouped. Finally, in Figure 2d,
typical appearance and characteristics, a decision is the proposed crack detection technique diagnoses that a
made whether crack-like edges are true cracks or not. crack exists in the structural components. In this article,
In this article, as one of visual inspection tasks, cracks fatigue cracks initiated from bolt holes are chosen as tar-
occurring near bolts on a steel structure are detected. get damage. Thus, the terms of “structure” and “object”
However, users can extend the proposed visual inspec- in this section indicate the bridge and bolt, respectively.
tion framework to conduct other types of visual in- However, the proposed technique can be easily gener-
spection. For example, suppose that corrosion or crack alized to detect cracks from any structural components
damage in gusset plates is damage of interest. Gusset such as joints or welded areas and on any structure, not
plates become “objects” in this study and techniques limited to bridges. The latest techniques in computer
suggested in Sections 2.1 to 2.3 can be applied to extract vision are implemented to increase the quality of object
images of individual gusset plate from different angles. detection, object grouping, and crack detection.
Users analyze images of all gusset plates in a test bridge
by applying a corresponding crack detection technique.
2.1 Image acquisition
The major contribution of the proposed technique
is to propose a new approach to automated visual in- For image acquisition, the UAV flies under or over
spection using a large volume of images. Many previous bridges by following a predetermined flying path,
researchers have focused on detecting cracks from a and cameras installed capture scenes of bridges
 14678667, 2015, 10, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/mice.12141 by <Shibboleth>[email protected], Wiley Online Library on [27/02/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
762 Yeum & Dyke

Fig. 2. Overview showing the procedure for the proposed damage detection technique: (a) acquisition of images from multiple
angles, (b) detection of object patches which may include damage, (c) grouping of same objects across images, and (d) detection
of crack damage initiated from objects.

consecutively. Some guidelines for image acquisition level of perception of a human, but many researchers
are suggested here for the best performance of the have improved detectability of objects such as faces or
proposed technique, but are not required: (1) Images pedestrians (Viola and Jones, 2001; Dalal and Triggs,
are not blurred and have sufficient resolution so that 2005; Dollár et al., 2009; Felzenszwalb et al., 2010). In
objects and cracks are clearly shown in images. (2) this study, modifications to established object detection
Angles between the camera and bridge, called tilt or algorithms are made for our purpose.
perspective angle, should not be large. If interest ob- In this study, an integral channel-based sliding
jects on the bridge are presented within a short distance window technique is applied over multiple scales of
of each other, like bolts, their scenes may overlap on the images. The sliding window technique uses a fixed
images under large perspective angle, making object rectangular window that slides over the images to
and crack detection difficult. However, angle variation decide whether the window contains an object or not.
of the camera is necessary due to the dependency of To make this judgment, features are extracted from
the crack’s appearance on the angle of the images. (3) each window. Here, the channel image indicates linear
Distance between the UAV and bridge stays roughly and non-linear transformation of the original image
constant. The number of scale images that need to be to help discriminate the object from non-objects as a
searched can be reduced by known approximate dis- preliminary process of feature extraction. A total of 11
tances and physical bolt sizes (Jahanshahi et al., 2013). types of image channels are used in this study: H and S
This results in increasing object detection rates or components in HSV color space, U and V components
decreasing false-positive errors with low computation in LUV color space, gradient magnitude, and histogram
time, and (4) GPS data of each image are recorded and of gradient with 6 orientations (HOG). The details of
saved for approximated estimating crack locations. HSV and LUV color formats can be seen in reference
Schanda (2007). HSV and LUV color spaces can sep-
2.2 Object detection arate colors from brightness which is not robust under
lighting variation. Thus, the V component in HSV and
Object detection is challenging because an object’s posi- the L component in LUV, which are brightness terms,
tion, pose, scales, lighting, and background vary relative are ignored from channel images. To consider the
to camera angles and positions. The key of object detec- object shape, a gradient magnitude and HOG are used.
tion is the selection of robust features that uniquely rep- These gradient features have been proven in many ap-
resent the object without affecting the above variations. plications for object detection (Dalal and Triggs, 2005;
There are no perfect solutions that achieve the high Dollár et al., 2009; Felzenszwalb et al., 2010). Figure 3
 14678667, 2015, 10, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/mice.12141 by <Shibboleth>[email protected], Wiley Online Library on [27/02/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Vision-based automated crack detection for bridge inspection 763

Fig. 3. Examples of channel images of a bolt: (a) original RGB (herein grayscale), (b, c) U and V components in LUV, (d, e) H
and S components in HSV, (f) gradient, and (g–l) histogram of gradient with 6 orientations (0o , 30o , 60o , 90o , 120o , 150°).

shows channel images of the bolt. Channel images in a and experimentally proven for objection detection. The
grayscale from Figures 3b to l are computed from the details of the gentle boost algorithm can be found in the
original RGB image (Figure 3a). The intensity of these following references (Friedman et al., 2000; Torralba
images varies depending on positions or color and this et al., 2004).
variation represents the unique feature of the object
patch. For example, Figures 3g–l show how different
edges of the bolt are highlighted depending on gradient
2.3 Object grouping
direction.
Using these channel images, features of each window Object grouping in this study is a process of matching
are computed by summing over a local rectangular two or more patches with the same object across the
region using Haar-like wavelets (Viola and Jones, images, and dividing them into groups of same object
2001). The integral image provides effective ways patches. If incorrect matching does not occur, matched
of calculating the local sum of channel images. For object patches are simply assigned as a same group.
simplicity, 1, 2, 3, and 4 rectangular Haar-like feature However, in reality, spurious matching does not allow
windows are used, which was proposed by the original such simple division. Moreover, object matching, espe-
work of Viola and Jones (2001). Features are computed cially for the applications in this study, is much more
from all training positive (object patches) and negative difficult than conventional matching problems because
(non-object patches) windows. Further modifications, all bolts in an image have very similar appearance and
not used in this study, are possible depending on are closely presented, causing failures in generating
complexity of object appearance and shape by increas- unique descriptors. To address these difficulties, robust
ing the number of possible rectangular or rotating matching and grouping algorithms are proposed by
window. integrating conventional matching algorithms and
Based on these features, a robust classifier is designed introducing a community detection technique for
to determine whether features at a certain window in grouping.
a test image indicate an object or not. In this study, a In general, object matching is accomplished by
boosting algorithm is implemented to produce a robust corresponding keypoints of object patches between
classifier. Boosting is a way of combining many weak images or selecting the closest object to an epipolar
classifiers to produce a strong classifier. By updating line after finding a fundamental matrix of a pair of
different weights of weak classifiers adaptively depend- images (Lowe, 2004; Snavely et al., 2008; Hartley and
ing on misclassification errors, the optimum strong Zisserman, 2003). However, in our application, a single
classifier which minimizes misclassification errors can use of these techniques produces large errors in match-
be obtained. There are several boosting algorithms ing. First, object patches having similar appearance in
introduced in the literature, but in this study, the gentle each image cannot be uniquely described by keypoints
boost algorithm, proposed by Friedman, is used because inside, causing wrong keypoint matching across images.
it is known as simple to implement, numerically robust, Second, epipolar constraint can help to remove the
 14678667, 2015, 10, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/mice.12141 by <Shibboleth>[email protected], Wiley Online Library on [27/02/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
764 Yeum & Dyke

worst of the non-corresponding object patches, but not 2.4 Crack damage detection
all of them. The center of the object patch does not
indicate the geometrical center of the object, and thus The proposed crack detection technique is applied to all
the epipolar line, which is computed from the center groups of object patches, which are found in the previ-
of the object patch, does not exactly pass through the ous step, to decide whether cracks exist or not. Before
corresponding object patch’s center. Moreover, due to introducing the technique, the type of crack referred to
the regional proximity of objects, more than one object in this study should be clearly defined. The crack visu-
is close to a certain epipolar line, making it difficult ally has a very sharp edge and an almost straight line,
to determine a corresponding object (Hartley and and initiates from bolt holes on steel plates because bolt
Zisserman, 2003). holes inherently have initial discontinuity in circumfer-
In this study, these two techniques are simply in- ence. A detectable crack in an image is one that can be
tegrated for better performance on matching objects. detected by human vision. Such “preliminary” informa-
Suppose that the object patch in the first image, called tion helps to detect the crack of interest by filtering out
target object patch, matches with one of the object non-crack edges.
patches in the second image. The object patch in the The first step in the proposed approach is to remove
second image is searched by ensuring that the distance the bolt areas from object patches. Each object patch
between its center and the epipolar line computed found in the previous step includes both the bolt and its
from the target object patch is within a set threshold. nearby area. The region of interest on each patch is the
If more than one object patches satisfy these criteria, area connected to the bolt, but not the bolt itself. The
keypoints inside of them are matched with those in benefits of removing the bolt area from the object patch
the target object patch. The object patch having the is that crack-like edges are not falsely detected from the
maximum number of keypoints matched is selected as bolt area and also a threshold boundary for true crack
the corresponding patch of the target object patch. The edges can be made, which will be mentioned later in this
integrated use of both keypoint matching and epipolar section. Detection of the bolt on object patches is car-
constraint highly improve matching performance. Cor- ried out by edge detection and binary morphology. The
respondence of object patches in every pair of images procedure is as follows: (1) A median filter is first ap-
are found using this integrated matching technique. plied to object patches to remove edges from real cracks
Despite the use of the proposed matching technique, or other textures, which are connected to the bolt. (2)
not all object patches are correctly matched with the Canny edge detector generates the edge image to detect
same object across images. The final step is grouping boundaries of the bolt. Then, a binary edge map is ob-
object patches based on the matching results between tained using a predetermined threshold (Canny, 1986).
object patches. This structure is almost identical to (3) Dilate operators using a disk structural element are
the community structure, which is widely used in data applied to the edge map to fill gaps between edges and
mining from large-scale data describing the topology of then, the convex hull of each connected entity is com-
network such as a social network. The object patches puted. Through this operation, several separated binary
(nodes) in a group (community) have dense matching entities are generated from a bolt or background. (4) A
(connection) internally but, some nodes are shared with true binary entity indicating the bolt is selected if it in-
other nodes in other communities due to incorrect con- cludes the center of the object patch. This criterion is
nections. From the standpoint of a network structure, reasonable because the object detector detects the bolt
the problem of grouping object patches is considered as to be presented at the center of the window. (5) Finally,
community detection. marginal pixels are added to the boundary of the de-
A very well-known community detection technique tected entity so that it includes full appearance of the
called modularity maximization is applied in this study. bolt. Example images are shown in Figures 4a and b.
Modularity is defined as a measure of the quality of Figure 4a is the detected object patch from Sections 2.2
particular division of a network into community. The and 2.3. Following the above procedure, the area of the
optimum community structure is obtained by iteratively bolt in the object patch is detected in Figure 4b. Subse-
amalgamating communities to maximize modularity in quent analysis is conducted on the area of the outside of
a network. The adjacent matrix A, which defines the the detected binary entity (object area).
connection between nodes, is the only input to this The second step is detecting crack-like edges. In this
technique. The matrix A is defined as Ai j if nodes i and study, the Hessian matrix-based vesselness measure-
j are connected, otherwise 0, where Ai j is the element ment technique, called Frangi filter, is used to detect
at ith row and jth column. This matrix can easily be crack-like edges. The approach is based on the fact
generated from the matching results. The technical that a crack on steel has a thin and bright line, similar
details are provided in Clauset et al. (2004). to the appearance of vessel in medical images. The
 14678667, 2015, 10, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/mice.12141 by <Shibboleth>[email protected], Wiley Online Library on [27/02/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Vision-based automated crack detection for bridge inspection 765

Fig. 4. Examples of bolt area and crack-like edge detection: (a) object patch including a bolt and its nearby area, (b) detection of
a binary entity (white area) indicating bolt, and (c) crack-like edge detected from the outside of the object area using the Frangi
filter.

Fig. 5. Examples of crack detection using Radon transformation: (a) one strong crack-like edge with the object boundary and (b)
Radon transformation of the strong crack-like edge and the range computed from the object boundary.

Hessian matrix-based edge detector does not produce crack’s appearance because the Frangi filter detects
double edges making good localization, and thus multi- several edges having similar appearance with cracks,
scale crack detection is possible (Frangi et al., 1999). as shown in Figure 4c. The third step is filtering out
The brief outline of the Frangi filter is first, a Hessian spurious non-crack edges using a region-based shape
matrix of the image is computed using a Gaussian characteristic. All connected components in the binary
derivative at multiple scales. Then, two eigenvalues edge image, obtained in the previous step, are iden-
(λ1 and λ2 ) of the Hessian matrix in each pixel of the tified using connected component labeling. Among
image are derived (|λ1 | > |λ2 |). An ideal crack edge on these connected components, the crack-like edges
image is λ2 ≈ 0 and |λ1 |  |λ2 |, and the sign of λ1 is are differentiated using their shape descriptors. For
negative because crack is a bright edge. The strength of example, fat and short edges are not real crack edges.
crack-like edges, V , is defined as The shape descriptor used in this study is eccentricity,
 which is defined as the ratio of major to minor axes of a
0 if λ1 > 0
V = R 2B
connected component and evaluate elongation of edges
S2
exp(− 2β 2 )(1 − exp(− 2c2 )) Otherwise (Yang et al., 2008). A threshold, which represents the
minimum eccentricity, is first set to filter out non-crack
 c are user-defined parameters, R B = λ2 /λ1 ,
where β and edges. Then, one strong edge line, which has the largest
and S = λ21 + λ22 . For example, if the edge is close to regional area, is detected, which is most likely a true
the ideal crack edge, R B goes to 0 and S becomes large, crack. Figure 5a shows the detected edge line from
and thus V is close to 1, which is the maximum of V. Figure 4c using these two criteria.
The final edge map can be obtained by thresholding As a final step, the detected crack-like edge, called
and removing edges connected to the border of the strong crack-like edge, is evaluated as to whether it
object patch. Figure 4c shows the result of crack- is the true crack or not. This decision is made based
like edge detection from the outside of the bolt area in on the assumption that crack is initiated from the bolt
the object patch (Figure 4a). The technical details about hole, and has an almost straight line as mentioned in the
the Frangi filter are provided in Frangi et al. (1999). previous section. Suppose that the axis perpendicular to
The remaining two steps are to detect real crack the detected edge goes through the center of the object.
edges from non-crack edges using prior knowledge of When the object (or object boundary) is projected
 14678667, 2015, 10, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/mice.12141 by <Shibboleth>[email protected], Wiley Online Library on [27/02/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
766 Yeum & Dyke

a Nikon D90 camera with 18–105 mm lens, and zoom

and flashing functions are not utilized. All 72 test images
having 4,288 × 2,848 resolution are sequentially taken
at roughly 23 m working distance, which is between
the camera and beam, and do not have much tilt angles
due to the issues in Section 2.2. For the object detec-
tion and grouping, downsized images with low resolu-
tion having 1,716 × 1,149 are used for fast computation.
However, once the areas of object patches are detected,
those are cropped from the full resolution images for
accurate crack detection.
Five randomly selected test images among the 72
images are used for training. A square of 68 object
Fig. 6. A test steel beam including 68 bolts. patches are cropped from these images. One hundred
forty-four negative patches, which are three times of
a number of positive patches, are randomly cropped
onto this axis, the range of the object projected on the from the non-object areas. Five hundred Haar-like
axis can be computed. If the detected edge is the true features in each object patch are randomly selected
crack, the line, which is drawn on the edge, will cross with respect to positions, sizes, and types. Thus, 5,500
the axis within this range. This concept is successfully features can be generated in each object patch, given by
implemented using the Radon transform. The Radon the product of 11 channel transforms and 500 Haar-like
transform in two dimensions is the integral projections features. A “strong” classifier is trained with 200 weak
of images along specified direction (MATLAB and classifiers, which are enough to ensure convergence of
Image Processing Toolbox, 2014; Deans, 1983). In this classification.
application, the maximum value of the Radon trans- For the sliding window technique, rough estimation
formation image indicates the direction and position of the size of the object patches in the images in ad-
of the detected edge line. Figure 5a shows the detected vance can reduce computation time and false-positive
crack-like edge and object boundary computed from detection. For example, suppose that the size of object
Figure 4a. Radon transformation of the strong crack- patches in image is unbounded, all scales of test images
like edge image is shown in Figure 5b. The dotted line are scanned, which is a very time-consuming process. In
is the range computed from projecting object boundary. this study, based on the physical bolt size and approxi-
The maximum value of the transformed image indicates mated working distance, the range of the bolt size on the
the edge’s direction and location on an angled axis, downsized images is found to be 64128 pixels. Conse-
respectively. The axis angle of the maximum point in quently, the final scale is set to from 1 to 2 with 21/5 scale
Figure 5b is around 140o and is perpendicular to the steps, causing a total of 6 scales. For object detection,
line of the edge. The true crack is determined if the test images are downsized with these scales so that the
maximum point is located inside of the range obtained different sizes of bolts in the images can be detected.
by projecting object boundary onto this axis. This All algorithms proposed in this study are imple-
criterion may not perfectly classify all true cracks from mented using MATLAB. VLFeat, which is an open
crack-like edges because non-crack edges also exist source library of computer vision algorithms, imple-
in the similar position of true cracks. However, it can ments the SIFT algorithm with MATLAB interfaces
successfully remove most non-crack edges. (Vedaldi and Fulkerson, 2010). Each image makes
pairs with the next 5 images in the set of sequential
images. The matching criterion of keypoint descriptors
proposed by VLFeat, called VL UBCMATCH, is used
3 EXPERIMENTAL VALIDATION
with a threshold of 2.5. If the number of matches is less
than 30, the fundamental matrix of this pair of images is
3.1 Description of the experiment
not computed and this pair of images is not considered
A large scale, rusted I-beam having 68 bolts, as shown in further during object matching. Using matched key-
Figure 6, is used for validating the proposed technique. points of an image pair, a fundamental matrix for this
As shown in introduction, a real fatigue crack is almost pair is computed using normalized 8 point algorithm
visually similar to a sharp scratch. Instead of producing and RANSAC. The function “estimateFundamen-
real fatigue cracks on the beam, two artificial scratches talMatrix” in the computer vision system toolbox in
are made with an awl at location (a) and location (b) MATLAB implements this algorithm (MATLAB and
in Figure 6. Test images of the beam are taken using Statistic Toolbox, 2012). If the number of remaining
 14678667, 2015, 10, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/mice.12141 by <Shibboleth>[email protected], Wiley Online Library on [27/02/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Vision-based automated crack detection for bridge inspection 767

Fig. 7. Examples of bolt detection results from test images: (a) true detection of bolts and (b) false-positive detection.

matches is less than 25, this pair is also not considered Table 1
for object matching. The threshold of the epipolar Results of object detection and grouping
constraint for matching is set to 181 pixels, which is
Object detection Object grouping
the diagonal distance of the maximum size of object
patches. For grouping object patches, if a group in- # of true object patches: # of matching: 2,922
cludes less than three object patches, this group and 1,326 # of object groups: 68
its object patches are removed and excluded from # of true-positive detection: # of object groups: 72 (with 4
a list of object patches. All object patches cropped 1,310 overlaps)
from original test images are resized as 240 × 240 # of true-negative detection: # of non-object groups: 5
pixel. As a rule of thumb, a threshold for Canny edge 16
detection is set to the high and low thresholds to # of false-positive detection:
products of 0.2 and 0.5 to the median value of the 91
grayscale object patch, respectively. The size of the
median filter for removing the crack edge for detecting
object areas is a 5 × 5 square. A disk structure element
having 8 pixel diameter is used to dilate images, and are removed before counting. The resulting object de-
10 pixels of the border margin are used. For the Frangi tector proposed in Section 2 achieves a 98.7% detection
filter, parameters β and c are set to 0.5 and the max- rate (1,310 object patches) and a 6.8% false-positive
imum Hessian norm, respectively, presented in the detection (91 non-object patches). The proposed object
original work and the threshold is 0.90 (Frangi et al., detection technique successfully attains a high rate
1999). Scales of the filter are up to 3 pixels with 0.2 of true detection and minimizes false-positive rate.
pixel step, which means the edges less than 6 pixels are Figure 7 shows samples of detected bolts and false-
more enhanced. For the shape descriptor, the minimum positive detection.
eccentricity is set to 8. Note that all parameters and Table 1 shows the results of object detection and
thresholds are not automatically determined. Users grouping. Based on the proposed matching and
should tune the parameters for different uses based on grouping techniques, 2,922 connections (matching)
ones that authors suggested in the training step and between nodes (objects) are found and 77 com-
by gaining experience. We chose values used based on munities (groups) are detected. All 68 bolts are
tests with an initial set of training images. However, successfully grouped and 5 non-object groups are
this simple process is conducted just once, and then the produced, which are visually similar to bolts. Here,
selected values are used for future visual inspections 5 non-object groups come from false-positive detection
without further tuning. in the object detection step. Non-object patches are
also classified as groups because they are consistently
detected across images. To avoid this problem, these
3.2 Experimental results
non-object patches are used for training as negative
A total of 1,326 bolts are shown in all test images. Bolts samples, not to be detected in future testing. Moreover,
having partial occlusion or distracted by foreign objects there are 4 communities overlapping, which means two
 14678667, 2015, 10, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/mice.12141 by <Shibboleth>[email protected], Wiley Online Library on [27/02/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
768 Yeum & Dyke

Fig. 8. Resulting groups of object patches at crack location (a) and location (b) in Figure 6.

Fig. 9. Examples of crack detection results: (a) true crack detection and (b) false-positive crack detection.

communities indicate the same bolt. These discrepan- Figure 9b is one of the false-positive detection results.
cies come from weak connections between sets of nodes The machine generated scratch next to the bolt, which
in a same group, and could be overcome by increasing is not induced by authors, is visually similar to a real
the number of pairs of each image for matching, but it crack. However, it is not a true crack and was originally
would be computationally expensive. A total of 1,147 present in the test structure. This is a limitation of the
object patches from 77 communities are grouped and proposed technique, but based on visual information
the remaining ones are removed due to a lack of nodes only, this scratch would be challenging to differentiate
in their allocated group. Figure 8 shows groups of object from a real crack, even by human inspectors.
patches having a crack associated with location (a) and
location (b) in Figure 6. Detected object patches prove
the necessity of images from different viewpoints. Only 4 CONCLUSIONS
a couple of patches show clear cracks that are visually
recognizable. In this study, a vision-based crack detection tech-
The proposed crack detection technique is applied nique is developed for automated inspection of large
to object patches in each group. There is a trade-off scale bridge structures only using images. Such images
between true and false-positive detection depending are collected from an aerial camera without advanced
on the threshold of the Frangi filter and Canny edge knowledge of the crack locations or special control of
detector because these two parameters determine camera positions or angles. The study focuses on pro-
crack-edges and the decision boundary, respectively. cessing these images to identify the presence of dam-
However, regardless of these variations, cracks are con- age on structure. The key idea is extracting images of
stantly detected from at least one of the object patches damage sensitive areas from different angles so as to
in the groups showing location (a) and location (b). Ex- increase detectability of damage and decrease false-
ample results are shown in Figure 9. Figures 9a and 5a positive errors. To achieve this goal, object detection
represent object patches and crack detection with ob- and grouping techniques, which are used in the area of
ject boundaries from location (a) and location (b). The computer vision, are implemented to extract images of
lines of true crack edges pass through the object areas. possible damage regions. Using these images, the
 14678667, 2015, 10, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/mice.12141 by <Shibboleth>[email protected], Wiley Online Library on [27/02/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Vision-based automated crack detection for bridge inspection 769

proposed damage detection technique can successfully Jahanshahi, M. R., Kelly, J. S., Masri, S. F. & Sukhatme, G. S.
detect cracks regardless of the small size or the possi- (2009), A survey and evaluation of promising approaches
bility of their not being clearly visible depending on the for automatic image-based defect detection of bridge struc-
tures, Structure and Infrastructure Engineering, 5(6), 455–
viewpoint of the images. The effectiveness of the pro- 86.
posed technique is successfully demonstrated using im- Jahanshahi, M. R., Masri, S. F. & Sukhatme, G. S. (2011),
ages of a large scale, rusty, steel beam with cracks us- Multi-image stitching and scene reconstruction for evalu-
ing a handheld camera instead of one mounted on a ating defect evolution in structures, Structural Health Mon-
UAV. In the future, the proposed technique will be vali- itoring, 10(6), 643–57, DOI: 10.1177/1475921710395809.
Jahanshahi, M. R., Sami, F. M., Curtis, W. P. & Gaurav, S.
dated using images of real civil structures collected from S. (2013), An innovative methodology for detection and
a UAV. quantification of cracks through incorporation of depth
perception, Machine Vision and Applications, 24(2), 227–
41.
ACKNOWLEDGMENTS Jiang, X. & Adeli, H. (2007), Pseudospectra, MUSIC, and
dynamic wavelet neural network for damage detection
The authors acknowledge support from National Sci- of highrise buildings, International Journal for Numerical
ence Foundation under Grant No. NSF-CNS-1035748. Methods in Engineering, 71(5), 606–29.
The authors would like to thank Dr. Robert J. Connor Lee, B. J., Shin, D. H., Seo, J. W., Jung, J. D. & Lee,
J. Y. (2011), Intelligent bridge inspection using remote
and Matt Hebdon at Purdue for providing the test controlled robot and image processing technique, in Pro-
structure and invaluable comments for this study. ceedings of International Association for Automation and
Robotics in Construction, Seoul, Korea, June 29–July 2,
1426–31.
REFERENCES Lowe, D. G. (2004), Distinctive image features from scale-
invariant keypoints, International Journal of Computer Vi-
Abdel-Qader, I., Abudayyeh, O. & Kelly, M. E. (2003), Anal- sion, 60(2), 91–110.
ysis of edge-detection techniques for crack identification in Malekian, V., Amirfattahi, R., Rezaeian, M., Aghaei, A. &
bridges, Journal of Computing in Civil Engineering, 17(4), Rahimi, P. (2012), Automatic detection and localization of
255–63. surface cracks in continuously cast hot steel slabs using digi-
Angel Aerial Survey (2014), Available at: http:// tal image analysis techniques, International Journal of ISSI,
angelaerialsurvey.com/, accessed May 14, 2014. 9(1), 30–40.
Canny, J. (1986), A computational approach to edge detec- MATLAB and Image Processing Toolbox (2014), The Math-
tion, IEEE Transactions on Pattern Analysis and Machine Works, Inc., Natick, MA.
Intelligence, 8(6), 679–98. MATLAB and Statistics Toolbox Release (2012), The Math-
Clauset, A., Newman, M. E. & Moore, C. (2004), Finding com- Works, Inc., Natick, MA.
munity structure in very large networks, Physical Review E, Miller, J. (2004), Robotic systems for inspection and surveil-
70(6), 066111. lance of civil structures. Doctoral dissertation, The Univer-
Dalal, N. & Triggs, B. (2005), Histograms of oriented gradi- sity of Vermont.
ents for human detection, in IEEE Computer Society Con- Moller, P. S. (2008), CALTRANS Bridge Inspection Aerial
ference on Computer Vision and Pattern Recognition, San Robot, Final Report, CA 08–0182.
Diego, CA, June 20-25, 2005, Vol. 1, pp. 886–93. O’Byrne, M., Schoefs, F., Ghosh, B. & Pakrashi, V. (2013),
Deans, S. R. (1983), The Radon Transform and Some of Its Texture analysis based damage detection of ageing infras-
Applications, John Wiley & Sons, New York. tructural elements, Computer-Aided Civil and Infrastruc-
Dollár, P., Tu, Z., Perona, P. & Belongie, S. (2009), Inte- ture Engineering, 28(3), 162–77.
gral channel features, in British Machine Vision Conference O’Byrne, M., Schoefs, F., Pakrashi, V. & Ghosh, B. (2014),
(BMVC), London, England, September 5-8, 2009. Regionally enhanced multi-phase segmentation technique
Felzenszwalb, P. F., Girshick, R. B., McAllester, D. & for damaged surfaces, Computer-Aided Civil and Infras-
Ramanan, D. (2010), Object detection with discriminatively tructure Engineering, 29(9), 644–58.
trained part-based models, IEEE Transactions on Pattern Phares, B. M., Rolander, D. D., Graybeal, B. A. & Washer,
Analysis and Machine Intelligence, 32(9), 1627–45. G. A. (2001), Reliability of visual bridge inspection, Public
Frangi, A. F., Niessen, W. J., Hoogeveen, R. M., Van Walsum, Roads, 64(5).
T. & Viergever, M. A. (1999), Model-based quantitation of Schanda, J. (2007), CIE colorimetry, in Colorimetry: Under-
3-D magnetic resonance angiographic images, IEEE Trans- standing the CIE System, John Wiley & Sons Inc., Hoboken,
actions on Medical Imaging, 18(10), 946–56. NJ.
Friedman, J., Hastie, T. & Tibshirani, R. (2000), Additive Shen, H. K., Po-Han, C. & Luh-Maan, C. (2013), Automated
logistic regression: a statistical view of boosting (with steel bridge coating rust defect recognition method based
discussion and a rejoinder by the authors), The Annals of on color and texture feature, Automation in Construction,
Statistics, 28(2), 337–407. 31, 338–56.
Hartley, R. & Zisserman, A. (2003), Multiple View Geometry Snavely, N., Seitz, S. M. & Szeliski, R. (2008), Modeling the
in Computer Vision, 2nd edn., Cambridge University Press, world from internet photo collections, International Journal
New York. of Computer Vision, 80(2), 189–210.
Indiana Department of Transportation (2013), Bridge Inspec- Torok, M. M., Golparvar-Fard, M. & Kochersberger, K. B.
tion Manual, Indianapolis, IN. (2013), Image-based automated 3D crack detection for
 14678667, 2015, 10, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/mice.12141 by <Shibboleth>[email protected], Wiley Online Library on [27/02/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
770 Yeum & Dyke

post-disaster building assessment, Journal of Computing in Viola, P. & Jones, M. (2001), Rapid object detection using a
Civil Engineering, 28(5), A4014004. boosted cascade of simple features, in Proceedings of the
Torralba, A., Murphy, K. P. & Freeman, W. T. (2004), Sharing 2001 IEEE Computer Society Conference on Computer Vi-
features: efficient boosting procedures for multiclass object sion and Pattern Recognition, Kauai, HI, 2001, Vol. 1, 511–
detection, in Proceedings of the 2004 IEEE Computer So- 18.
ciety Conference on Computer Vision and Pattern Recogni- Yang, M., Kpalma, K. & Ronsin, J. (2008), A sur-
tion, Washington DC, June 27–July 2 2004, Vol. 2, II-762– vey of shape feature extraction techniques, in P.-Y.
69. Yin (ed.), Pattern Recognition Techniques, Technology
US Aerial Video (2014), Available at: http://www. and Applications, InTech, pp. 43–90, DOI: 10.5772/
usaerialvideo.com/, accessed May 14, 2014. 6237.
Vedaldi, A. & Fulkerson, B. (2010), VLFeat: an open and Zhu, Z., German, S. & Brilakis, I. (2011), Visual retrieval of
portable library of computer vision algorithms, in Proceed- concrete crack properties for automated post-earthquake
ings of the International Conference on Multimedia, New structural safety evaluation, Automation in Construction,
York, NY, October, 1469–72. 20(7), 874–83.