ColorCrack: Identifying Cracks in Glass

James Max Kanter

Massachusetts Institute of Technology
77 Massachusetts Ave
Cambridge, MA 02139
[email protected]

Figure 1: ColorCrack automatically identifies cracks in arbitrary angled photos of cracked screens. It does this in four
main steps : (1) detect screen using HOG features (2) Transform screen by applying homophony (3) Find cracks using edge
detector (4) Process result to connect cracks and color regions.

Abstract 1. Introduction
Machine vision is a powerful tool to automate a tasks
Automatically identifying cracks in glass is an applica- currently performed by humans. However, what is instinc-
tion of machine vision that has not been studied much be- tive for humans is not necessarily for a computer. One such
fore. It poses the unique opportunity to incorporate the task is identifying cracks in glass.
physics of glass and how it cracks into a detection algo- Objects that have glass and have the potential to be
rithm. cracked are all around us. While it is simple to see cracks, it
ColorCrack is an automated method of finding cracks in is harder to quantitatively characterize the cracks that form
glass smartphone screens using insights from previous work in glass.
in object and edge detection.The key to ColorCrack is the Studying how glass break is especially relevant to sci-
ability to automatically detect cracked screens from nearly entists and engineers who study, design, and develop glass
any angle. This functionality allows the system to take ad- objects. For instance, in forensic science,researchers seeks
vantage of the fact that cracks are more visible at certain to model how cracks form. In studies like ”Star-Shaped
angles to improve the detection accuracy. Crack Pattern of Broken Windows”, researchers analyse
photographs of cracks by hand to develop physical mod-
The algorithm was developed and evaluated using a per-
els that infer the point of origin of a projectile[6]. In other
sonally collected dataset. Initial results show that Color-
fields like material science, scientists could benefit from
Crack can typically find the screen correctly or with little er-
large digitalized datasets on cracked glass. For example,
ror, as well as identify the main crack structures, efficiently
imagine if the engineering team designing the screen on the
enough for most applications.
next iPhone incorporated real world data on cracks in their
In the future, the ColorCrack algorithm could be ex- design.
tended to other glass types for applications and has uses The problem poses challenges for machine vision in at
in fields such as material and forensic science. least a few regards. First, the 3d nature of cracks means
that they cannot be viewed from all angles. Second, the re-

1
flective and transparent nature of glass differentiates it from that is in between T1 and T2 is classified as an weak edge if
a lot of other work in vision. And finally, glass is used in it connects to a pixel above the high threshold.
varying manners the differ in both shape and scale which While many methods since the Canny detector have used
present difficulties for general solutions. a gradient approach, there are other techniques to solve the
In this paper, I investigate two approaches to solving this problem using machine learning. Dollár and Zitnick use
problem and present the results. the fact that edges contain structures in the form of straight
lines and T-junctions to train a random forest [3]. A random
2. Previous work forest is an ensemble of decisions trees where a decision
ft (x) classifies an image patch x ∈ X using a binary split
While I found no previous academic work specifically function
on automatic crack detection in glass, the problem relates h(x, θj ) ∈ 0, 1 (1)
to several efforts in the fields of object detection and edge
detection. These trees are combined using an ensemble method such
as majority voting to classify the sample. Dollár and Zitnick
2.1. Object Detection and HOG Features apply this to images by predicting a 16 x 16 segmentation
A Histogram of Oriented Gradients (HOG) is a robust mask from a larger 32 x 32 input patch. To improve results
method for generating image features that has been shown they run this prediction at 3 different resolutions of the im-
to work well for object detection [2]. It is based on the idea age and average the result. Finally, they sharpen the image
that an image can be characterized well by the distribution by iteratively resigning pixels in the segmentation mask and
of local intensity gradients or edge directions. averaging the results. The technique computes results an or-
der magnitude faster than other methods while also achiev-
HOG is calculated for a given image by dividing it into
ing state of the art accuracy. This results in frame rates that
small regions called cells that can be either rectangular or
are high enough for of real-time edge detection.
radial. Each of these cells is represented as a histogram of
gradient directions of the pixels in that cell. In their paper,
Dalal and Triggs show that overlapping these cells can im- 3. Crack Detection Problem Set-up
prove performance[2]. For this project, I focus on cracked smart screens be-
To account for illumination and shadowing in a cell, the cause that is a dataset I could easily generate myself. How-
local regions are contrast normalized before using the HOG ever, through the process, ColorCrack was designed with
features. A common way to do this is to assemble nearby thoughts towards the general case.
cells into ”blocks” and then use the total energy of the block The problem of identifying cracks in glass in pho-
to normalize each cell. tographs can be broken down into two parts.
HOG features are then used to train a machine learning
model. A common approach is to train a support vector 1. Finding the glass in a photograph
machine (SVM), which is a binary classifier that tries to
optimizes a hyperplane between training points. Dalal and 2. Finding the clacks in the glass
Triggs used this approach to get near perfect separation on
In the first step, the input is a photo of a non-occluded
the MIT pedestrian database[2].
screen that can be at any angle. The algorithm used in this
2.2. Edge detection step needs to identify the location of the screen and trans-
form the photo so the plane of the screen is perpendicular to
Cracks in glass can be thought of edges in an image. the camera.
Edge detection is at core of many machine vision tasks In the second step, the input is the corrected screen and
and has a lot of research behind. Early work by John it must output a binary image with the locations of cracks.
Canny has been shown to sufficient for most edge detec- Therefore, ColorCrack considers a pixel of glass to either
tion tasks[4]. The Canny detector is based on the gradient be cracked or not cracked and nothing in between.
of image pixel intensities calculated using the derivative of
a Gaussian filter[1]. First, the algorithm smooths the image 4. Step 1: Screen Extractions from images
using a Gaussian filter to reduce noise. In the nonmaxium
suppression step, the algorithms takes the gradient in mul- The first stage of the problem is extracting the screen
tiple directions and scans the image to only keep pixels that from an angled photograph of a phone. ColorCrack does not
are local maximum in some direction. Finally, the Canny limit the input images to being taken in a certain way to keep
detector uses a high threshold, T1 , and low threshold T2 , to the system flexible. Additionally, most cracks photograph
classify pixels. Pixels above T1 are considered strong edges, better when taken at an angle. This means that allowing
while pixels below T2 are labelled as non-edges. Any pixel angled photos often produces better results.

2
 To extract the screen from an angled image, I explored is removing the checkerboard pattern. While the system
ways to identify the 4 corners in the image. Once the four knew the source checkerboard pattern that was displayed
corners are identified, then ColorCrack calculates the ho- on the screen, the colors are slightly differently in the ac-
mography that transforms the input image to one where the tual photo. To account for this, ColorCrack measures the
plane of screen is perpendicular to the camera. the color of the squares in the captured photo and adjusts
I experimented with two different approaches for screen the source checkerboard photo. With the modified source
extraction. checkerboard, ColorCrack subtracts that photo from the ac-
tual photo and takes the absolute value of the result.
Detecting checkerboard pattern on screen The result of this can be seen in Figure 2. As you can
One way to detect the screen is to display a known image see, this preserved the cracks, but left artifacts where the
that is easy to detect. Therefore, the first approach I took cells change color or the checkerboard was misaligned.
was to display a checkerboard pattern on the screen before While it may be possible to remove these artifacts, I
taking the photo. perceived it as too difficult and took another approach
The disadvantage of this approach is that it requires to identifying the screen without having to use a specific
downloading a specific image to the phone before pho- image.
tographing it. Even more, this method may be impractical
when extended to other glass scenarios where printing large Screen detection using HOG features
checkerboards won’t work. Even though this approach ul- Next, I redefined identifying the screen on a phone as the
timately did not work, it was important for providing ev- task of identifying each of the four corners. ColorCrack
idence that it is possible to capture the crack information implements this by training a multi-class SVM on the HOG
even after applying an homography. features of a training set of images.
Matlab provides the function
dectectCheckerboard that can find a checker- Making training set
board in a photo. This function does not return the full
checkerboard, but from the information it does return I assembled the training set manually using a custom
ColorCrack computes the average square size. Using the tool. The tool displayed a photo and created 100px by
average square size, the system extrapolates the location 100px patches around the points I clicked. Using this tool I
of the corners of the actual screen. Once these corners are assembled a training set of 12 examples of each type of cor-
identified ColorCrack calculates the homography and uses ner and 200 examples of non-screen corner objects. Figure
imtransform to get the warped image. The results of 3 shows what these patches look like for each of the 4 corner
extracting the checkerboard pattern can be seen in Figure 2. types. In the non-screen corner objects I included examples
of pictures of screens that did not contain corners as well
as examples of corners that were not screens. These two
changes improved performance on at least a few examples
in my data set.

Figure 3: An example of four corner examples extracted

from an image for training.

Figure 2: Left: The result of extracting screen using a

checkboard pattern displayed an crack phone. Right: The
result of trying to remove the checkboard pattern from the
image. This approach did not work as well as hoped, so in
the end ColorCrack used a different method.

The final step to prepare the image for crack detection Before training the 5 one vs all SVM’s on these patches

3
I had to decide on the cell size to use to computer my HOG Training SVM’s and Corner Detection
features. I determined this by trying several different cell
sizes as show in Figure 4. The visualization shows that a With the features determined, ColorCrack trains several
cell size of 32x32 does not seem to encode enough shape SVM’s to detect each type of corner – upper left, upper
information. On the other hand, a cell size of 8x8 encodes right, lower right, and lower left – as well as an SVM for
more information than is probably necessary. This means non corner objects. They were training in a one vs all fash-
that the dimensionality of the training examples would ion.
be very high, increasing time to train. The cell size of To find a corner, ColorCrack uses a sliding window of
16x16 seems to be a compromise that limits the size of the 100px by 100px that skips 10 pixels at a time to assemble a
resulting vector while encoding enough information. list of candidate coordinates. The candidate coordinate that
has the highest score for each corner type gets labelled as
that corner. While the corner detection generally worked
well, it did not always perform as well as necessary for
step 2. In cases like the one in Figure 6, I manually picked
out the corners in order to evaluate the performance of just
crack identification.

Figure 4: Visualization of the HOG features for for 3 differ-

ent cell sizes. ColorCrack uses a cell size of 16x16 to bal-
ance capturing shape information and limiting feature size.
Left to right cell sizes: 8x8, 16x16, 32x32.

Using image gradients for training

In the course of training, I discovered that taking the

gradient of the image before extracting HOG features
improved performance. This might be because the sharp
change in color between glass and the border around the
glass is a defining feature of glass that can be exploited
for better performance. Figure 5 shows the result of this
addition to the the processing pipeline. Figure 6: An example of when ColorCrack did not properly
identify the location corners. This error may be too much
depending on the application.

5. Step 2: Crack identification

Finding the cracks in the transformed image happens
in 4 steps: convert to grayscale and apply Gaussian blur,
run edge detection, filter out noise, and dilate then erode
edges. The results at each step of of this process are shown
in figure 7. As mentioned above, to access the performance
only of this step, I manually selected corners when step 1
did not not perform well enough.

Figure 5: The result of using the image gradient to calculate Convert to grayscale and apply Gaussian blur
HOG features. The classification of the upper right corner This is a preprocessing step. Color data is not used by
becomes noticeably more accurate. ColorCrack to finding cracks, so the input image can be

4
Figure 7: The results of applying ColorCrack’s crack detection algorithm to 4 sample photos. From left to right: input image,
result of applying Gaussian blur and Canny edge detector, result of removing regions that are too small to be cracks, result of
dilation, result of erosion, result of coloring in regions. Small tweaks were made to the parameters of each image to achieve
optimal results.

converted to grayscale. After that, ColorCracks applies falsely identified as a crack.

a Gaussian blur to the image. Cracks are very defined
parts of the input image, so blurring the photos preserves
the crack information while removing noise that might be Run edge detection
ColorCrack uses the Canny edge detector to locate cracks.

5
Cracks continue to form until they hit another crack [5]. of photos. This is largely because of the efficient imple-
This creates the branching pattern that the weak and strong mentation of all steps of the process in Matlab.
edge threshold the canny detector detects well. Thus, the
Canny edge detector is a good choice because the physics What did not worked and thoughts on how to improve
of how cracks form is similar to the mechanics of the Canny When ColorCrack incorrectly identifies the corner locations
edge detector. it heavily affects the homography calculation. A way to im-
However, it is important to set the threshold for strong prove this would be to have highly ranked candidate points
and weak edges in the Canny edge detector. ColorCrack ”vote” on where the corner should be. Another solution
modifies the default values of Matlab’s Canny edge detector might be to implement logic that selected the highest rank-
implementation based the insight that there will be at least ing four points that fulfilled the constraints of how they
on crack in the image that is very easy to recognize. This could be arrange (e.g an upper corner cannot be below a
means that the threshold for strong edges is set higher than lower corner).
the default. Another issue is that even when taking a photo at an an-
gle, cracks sometime do not show up clearly. In this case, it
Filter out noise is impossible to identify every crack. A possible solution to
After running the edge detector, there are sometimes this could be to compute image correspondences between
”cracks” that are just noise in the image. To filter this multiple image a different angles and combine the results.
out, ColorCrack uses the observation that cracks are Finally, the next step is to apply ColorCrack to other uses
typically large continuous regions. The Matlab function of glass. In particular, it would be useful to explore how
bwareaopen is used to remove regions that contain too much tweaking is necessary on a dataset of crack windows.
few pixels to be cracks.
References
Dilate edges then erode edges
[1] J. Canny. A computational approach to edge detection. Pat-
As mentioned above, cracks form until they hit another tern Analysis and Machine Intelligence, IEEE Transactions
crack. To fulfil this constraint, ColorCrack tries to connect on, PAMI-8(6):679–698, Nov 1986.
the cracks the Canny detector finds by dilating the pixels of [2] N. Dalal and B. Triggs. Histograms of oriented gradients for
the crack using imdilate and a disk structuring element. human detection. In Computer Vision and Pattern Recogni-
After applying the dilation to connect cracks, the Matlab tion, 2005. CVPR 2005. IEEE Computer Society Conference
function imerode with a smaller disk structuring element on, volume 1, pages 886–893. IEEE, 2005.
is applied to shrink the detected cracks, while leaving the [3] P. Dollár and C. L. Zitnick. Fast edge detection using struc-
new connections in tact. As an added benefit, the dilation tured forests. CoRR, abs/1406.5549, 2014.
and erosion has the effect of filling in the area between the [4] B. Green. Canny edge detection tutorial, 2002.
outsides edges of the crack, which is the desired behavior. [5] B. of Criminal Apprehension. Glass, 2014.
[6] N. Vandenberghe, R. Vermorel, and E. Villermaux. Star-
6. Evaluation shaped crack pattern of broken windows. Phys. Rev. Lett.,
110:174302, Apr 2013.
ColorCrack was evaluated using 8 different photos of
cracked screens. Below are the highlights of what worked
and thoughts on how to improve what did not work.

What worked
ColorCrack generally correctly found corners or found them
with small errors. This worked on a relatively small dataset
of just 248 images, which speaks to the robustness of HOG
features and SVM’s.
ColorCrack always found the most prominent cracks in
a photo and often found the weaker connecting cracks. This
is likely a result of the fit between the crack identification
problem and the Canny Edge detector. It struggled on the
higher resolution cracks and very weak cracks, but requiring
accuracy in these regards depends on the exact application.
ColorCrack is performant enough to be used on large
sets of data. In a reasonable amount of time (hours), it
could be applied to high speed photography or thousands