See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/339339931

Deep Machine Learning Approach to Develop a New Asphalt Pavement

Condition Index

Article in Construction and Building Materials · February 2020

DOI: 10.1016/j.conbuildmat.2020.118513

CITATIONS READS

53 973

3 authors:

Hamed Majidifard Yaw Adu-Gyamfi

University of Missouri University of Missouri
25 PUBLICATIONS 254 CITATIONS 45 PUBLICATIONS 365 CITATIONS

SEE PROFILE SEE PROFILE

William Buttlar
University of Missouri
217 PUBLICATIONS 6,203 CITATIONS

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Development of a Performance-related Asphalt Mix Design Specification for Tollway, Illinois View project

Performance Characteristics of Modern Recycled Asphalt Mixes in Missouri,Including Ground Tire Rubber, Recycled Roofing Shingles, and Rejuvenators View project

All content following this page was uploaded by Hamed Majidifard on 18 February 2020.

The user has requested enhancement of the downloaded file.

 1 Deep Machine Learning Approach to Develop a
2 New Asphalt Pavement Condition Index
3
4
5 Hamed Majidifarda, Yaw Adu-Gyamfib, William G. Buttlarc
6
a
7 Ph.D. Candidate, University of Missouri, Columbia, MO-65211
b
8 Assistant Professor, University of Missouri, Columbia, MO-65211
c
9 Professor and Glen Barton Chair in Flexible Pavements, University of Missouri, Columbia,
10 MO-65211
11
12
13 ABSTRACT:
14 Pavement condition assessment provides information to make more cost-effective and
15 consistent decisions regarding management of pavement network. Generally, pavement
16 distress inspections are performed using sophisticated data collection vehicles and/or foot-on-
17 ground surveys. In either approach, the process of distress detection is human-dependent,
18 expensive, inefficient, and/or unsafe. Automated pavement distress detection using road
19 images is still a challenging issue among pavement researchers and computer-vision
20 community. In recent years, developments in deep learning has enabled researchers to develop
21 robust tools for analyzing pavement images at unprecedented accuracies. Nevertheless, deep
22 learning models necessitate a big ground truth dataset, which is often not readily accessible for
23 pavement field. In this study, a labeled pavement dataset is presented as the first step towards
24 a more robust, easy-to-deploy pavement condition assessment system. In total, 7237 google
25 street-view images were extracted, manually annotated for classification (nine categories of
26 distress classes) and segmentation. A pavement distress evaluation model is developed using
27 the YOLO deep learning framework for distress classification and U-net for segmentation and
28 distress densification. Classification accuracies were evaluated using F1-scores. The output of
29 the distress classification and segmentation models are used to develop a comprehensive
30 pavement condition tool which rates each pavement image according to the type and severity
31 of distress extracted. In the current study, weights for the rating process are also learned using
32 various machine-learning approaches. As a result, we are able to avoid over-dependence on
33 human judgement throughout the pavement condition evaluation process. The outcome of this
34 study could be conveniently employed to evaluate the pavement conditions during its service
35 life and help to make valid decisions for rehabilitation or reconstruction of the roads at the right
36 time.

37 KEYWORDS: Pavement monitoring; Pavement distresses detection; Deep learning; Google

38 API; Machine learning; Pavement condition prediction.

1
 1 1. Introduction

2 Pavement distress detection is a first key step in developing a robust pavement management
3 system. It offers a comprehensive assessment of pavement conditions, which in turn generates
4 information needed to make more cost-effective and consistent decisions associated with the
5 pavement network preservation. Generally, pavement distress inspection is performed using
6 sophisticated data collection vehicles and foot-on-ground surveys. In either approach, the
7 current process of distress detection is human-dependent, expensive, inefficient, and unsafe.
8 For example, the total price of an Aran was reported by the Ohio Department of Transportation
9 for US$1,179,000, with an annual operating expense of US$70,000 [1]. Fully automated
10 distress detection systems requiring no specialized data collection equipment have the potential
11 to lower distress survey costs, reliability and scalability [2]. The primary goal of this study is
12 to leverage recent advances in machine learning to develop a low-cost, robust pavement
13 condition assessment system, capable of detecting, classifying and quantifying the density of
14 pavement cracks in an automated fashion.
15 The most promising approaches for automated distress analysis leverage image processing
16 and computer vision algorithms to detect edges of different types and severities of crack in
17 pavement images. The primary advances in automated pavement crack detection techniques
18 are as follow: intensity-thresholding [3-6], match filtering [7], edge detection [8], seed-based
19 approach [9], multiscale methods like wavelet transforms and empirical mode decomposition
20 [10-14]; texture-analysis, and machine learning [15-18]. Also, Zou et al developed CrackTree
21 as an automatic procedure for crack detection [19]. There are some challenges related to these
22 approaches. The first challenge is that these techniques rely on image pixel manipulations,
23 which are very slow processes, and then it is not applicable for large scale deployment.
24 Secondly, these techniques only work precisely if the image configuration is static, and the
25 models don't work well if different camera configurations are used. Finally, there are many
26 heuristic rules related to the use of these models, which make it impractical to be implemented
27 extensively.
28 In order to overcome these challenges, computer vision algorithms which use machine
29 learning models have been proposed as an alternative to traditional image manipulation
30 techniques. In fact, recent progresses in deep learning has directed to substantial improvements
31 in our ability to analyze streams of videos and image at unprecedented accuracies. The models
32 are leading advances in areas like self-driving cars and smart cities [20], nanomaterials [21],
33 healthcare, agriculture, retailing, and finance.
34 Deployment of machine learning approaches for automated pavement distress detection is
35 not novel anymore, however application of deep learning is still attractive for pavement
36 researchers [22-25]. Deep architecture with many hidden layers like deep convolution neural
37 networks (DCNNs) provide frequent abstraction levels [26-29]. CrackNet software was
38 established by Zhang et al as a crack detection model using raw image patches via the CNN
39 [22]. Afterward, Zhang et al applied the Recurrent Neural Network (RNN) to produce
40 CrackNet-R, which is more accurate in detecting small cracks and removing noise than
41 CrackNet [30].It must be noted that none of those mentioned above studies proposed models
42 based on a comprehensive dataset covering all pavement distress types from sections with
43 different conditions. Also, classification and quantifying the density of the distress did not take
44 into account simultaneously. Furthermore, none of these studies did not provide a pavement
45 condition tool which can be used for evaluating pavement condition based on the proposed
46 detection models.
47 In the current study, we develop four prediction models to evaluate pavement condition
48 using the proposed pavement distress detection deep learning based models. It must be noted
49 that the variety, quality and quantity of data utilized for training models is the key factor for
50 their robustness, which is not always available. Comprehending the importance of labeled
51 datasets to develop a strong pavement condition detection model, Majidifard et al represented
52 the ‘Pavement Image Dataset,’ or (PID) [31]. The dataset contains 7,237 images extracted from
53 22 different pavement sections, including both US highways and interstate routes. A python
54 code were developed to extract images using Google Application Programming Interface (API)
55 in Google street-view [31]. Images were hand annotated by pavement experts carefully

2
 1 considering nine different distress classes. Afterward, the performance of dataset was assessed
2 using a famous deep learning framework named You Look Only Once (YOLO v2) [31].
3 However, the proposed model by YOLO did not quantify the density of the cracks. Therefore,
4 in our new study we developed a new U-Net based model to quantify the density of the
5 distresses. Finally, various pavement condition indices tried to be developed based on the
6 proposed crack detection models (Yolo-based and U-Net based). The following summarizes
7 the primary contributions of this study (Figure 1):
8 • First, we introduce a unique dataset annotated for simultaneous classification and
9 densification of pavement distresses. The data is extracted from google street view,
10 which provides us with a variety of camera views needed to improve the system’s
11 ability to recognize different types of cracks and estimate their severity.
12 • Second, we implemented a distress segmentation model capable of delineating the
13 boundaries of different types of cracks in challenging environments characterized by
14 severe shadowing, non-crack-like objects such as cars, trees, etc. We overcome this
15 challenge through intelligent model training techniques and architectural designs.
16 • Developing a new pavement condition index based on inputs from a crack
17 classification model (YOLO), a density model (U-Net) and a hybrid of machine
18 learning-based models.
19 • Conduct a comparative analysis of proposed pavement condition index with PASER
20 ratings along pavement profile.
21

22 Figure 1. Experimental research flowchart

23
24 The arrangement of the article is organized as follows. The relate works including pavement
25 condition indices and automated crack detection methods are reviewed in Section 2. The
26 methodology of the research introduces main road segmentation, U-Net based Model for

3
 1 distress quantification, and developing pavement condition indices which discussed in section
2 3 followed by model validation briefed in Section 4. Section 5 center around the conclusion of
3 the research which summaries the methodology and the implementation of the study.

4 2. Literature Review

5 2.1. Pavement condition index development approaches

6 There are various indices that are utilized to characterize pavement conditions, such as the
7 pavement condition index (PCI), present serviceability index (PSI), present serviceability
8 rating (PSR), pavement surface evaluation and rating (PASER) [32]. PCI is the quantitative-
9 based pavement condition index, while the others can be grouped into class of qualitative
10 pavement indices. The PCI, arguably the most prevalent index for pavement condition
11 evaluation, depends on inspection data and visual observation. U.S. Army Corps of Engineers
12 developed PCI for the management of pavement rehabilitation and maintenance system [33].
13 In the PCI calculation method, various distress types with different severities are incorporated
14 into a univalent PCI value. The length or area and the severity (low, medium, and high) of each
15 distress are taken into account to calculate the deduct values for the surveyed section (34-3;
16 35]. The PCI ranges from 100 to 0, in which 100 is brand new pavement, and 0 is the worst
17 condition possible. The University of Wisconsin-Madison Transportation Information Center
18 developed the PASER system, rates pavement conditions from 1-10 . It utilizes visual
19 inspection to assess pavement surface conditions [36]. The type of distress is not required to
20 calculate PASER, as a result, PASER ratings cannot be disaggregated by distress types. The
21 advantage of this method is that pavement sections may be rated promptly, possibly even by
22 "windshield survey." A principal drawback is that the PASER outputs cannot be used in
23 mechanistic-empirical transportation asset management programs because PASER ratings
24 cannot be classified into component distress data [36].
25 Newer approaches for developing pavement condition indices have been proposed by
26 several research teams as a result of the limitations traditional methods such as the PCI,
27 PASER, PSI, etc. Eldin and Senouci used neural-network based algorithm to predict condition
28 of pavement using data provided by the Oregon State Department of Transportation (ODOT)
29 [37]. Fathi et al used a hybrid machine learning (ML) method that combines random forest
30 (RF) and artificial neural network (ANN) to predict the alligator deterioration index (ADI)
31 [38]. Piryonesi and El-Diraby used historical distress data in the LTPP database to develop
32 decition trees-based algorithm to predict PCI of asphalt roads [39]. They developed a Python™
33 program to produce the PCI values from distress data according to the ASTM methodology.
34 For this purpose, all deduct value graphs, and correction curves were digitized and
35 mathematically designated [39]. Shahnazari et al., used ANN and GP algorithms to predict the
36 PCI. The models established based on large dataset of PCI values which was collected from
37 road network of Iran. The proposed models were accurate and they can be used instead of
38 Micropaver software to calculate PCI [40].
39 The limitation of the previous models is that all of them rely on manual inspection of
40 distresses. Inspectors should calculate the area, length, and severity of distresses, and the output
41 can be used as an input in these models to calculate the pavement condition rate. Therefore, a
42 fully automated distress detection model is demanding.
43 2.2. Automated Crack detection, segmentation and classification approaches
44 Image segmentation is the procedure of subdividing an image into several segments
45 (sets of pixels, also known as image objects). Image segmentation is classically performed to
46 detect boundaries such as lines and curves in images. In another word, image segmentation is
47 the practice of extracting the interest objects from the background. Object classification is a
48 technology associated with image processing and computer vision that copes with detecting
49 semantic objects of a specific class such as humans, buildings, or cars in videos and images.
50 Object classification and segmentation – both are part of machine learning based image
51 processing to train AI algorithms through computer vision, and both are important for object
52 recognition precisely in machine learning and AI development. The classification process is

4
 1 easier than segmentation; in the classification approaches, all objects in a single image are
2 grouped or categorized into a single class. While in segmentation each object of a single class
3 in an image is highlighted with different shades to make them recognizable to computer vision.
4 In crack segmentation, the severity of the distresses is detectable although there is no ability to
5 classify them into different groups. On the contrary, in the classification approaches the
6 distresses can be categorized into different groups and the severity cannot be measured.
7
8 The primary segmentation approaches are thresholding and edge detection. Thresholding-
9 based segmentation is broadly utilized in automated pavement distress systems [41]. Another
10 regular procedure in image processing is edge detection. The significant advantage of edge
11 detection is the fast reduction of image data to beneficial information. There are many useful
12 edge detectors that have been proposed over the past 30 years, such as LOG, Sobel, Roberts,
13 and Prewitt edge detectors [42]. The main problem associated with the most edge detection
14 algorithms is that these algorithms only characterize a spatial scale for edges detection.
15 Pavement images are acknowledged to be challenging to work within the process of pavement
16 distresses because of different details at various scales. In the past decade, wavelet-based edge
17 detection at multiple scales became popular in pavement image processing [43]. Shadows and
18 lighting effects in pavement images introduced new challenges in the automatic pavement
19 distress detection field. Region-based image thresholding has been implemented to resolve the
20 difficulties caused by shadow and illumination variations. A neighboring difference histogram
21 procedure was used by Li and Liu to crack image segmentation using a globally optimized
22 threshold [44]. However, histogram-based procedures, do not consider photometric and
23 geometric characteristics of the cracks in road pavement images [45]. Dynamic local
24 thresholding for non-overlapping image blocks developed by Oliveira and Correia [4].
25 Although segmentation procedure based on thresholding is beneficial in various image
26 segmentation tasks, it is still problematic for the automatic threshold selection. Image
27 morphological procedure is another primary tool that has often been utilized in the automated
28 pavement distress detection studies [46]. Naoki and Kenji [47] developed a procedure using
29 fundamental top-hat transform and morphological operations to extract structural information
30 from road pavement image, and subsequently cracks detection. Although morphological image
31 processing provides the benefit of extracting prominent geometrical structure related to the
32 cracks in road surface images, the performance is highly parameter dependent [3]. In practice,
33 it is suggested to operate morphological processing along with other image processing
34 techniques.
35 Object classification generally divided to machine learning-based and deep learning-based
36 methods. In Machine Learning approaches, first the features defined by using Histogram of
37 oriented gradients (HOG), Viola–Jones object detection framework, Scale-invariant feature
38 transform (SIFT) [48], then classification frameworks such as support vector machine (SVM)
39 implemented to do the classification. While, deep learning techniques have the capability to do
40 end-to-end object detection without precisely defining features, and are classically based
41 on convolutional neural networks (CNN) such as You Only Look Once (YOLO) [49], Single-
42 Shot Refinement Neural Network for Object Detection (RefineDet) [50], Region Proposals (R-
43 CNN [51], Fast R-CNN [52], Faster R-CNN [53], Single Shot MultiBox Detector (SSD) [54].
44 Critical distresses on road surface must be detected in order to propose a strong automated
45 pavement distress detection model. Pavement deterioration rate is a function of various features
46 such as climate, structural layering, traffic, and layer age. Strategies are developed by road
47 administrators to repair the road surface based on the type, extent, and severity of the distresses.
48 Former researches made improvement in the direction of this target but falling short in one area
49 or another. To clarify, CrackNet [22; 55] focused on detecting the presence of cracks but did
50 not specifically identify individual types. Zalama et al. [56] classified the distress types
51 horizontally and vertically, while Karaköse et al. [57] categorized distresses into three classes
52 – vertical, horizontal, and alligator. Finally, other studies resulted in the blurry road markings
53 detection [58], while others concentrated on cracks classification, including cracks which were
54 sealed [59]. Quality of data used for training machine learning models is the essential key for
55 their robustness. Previous studies introduced several benchmarked datasets (private and public)
56 for training of machine learning models [60]. In spite of this, none of the aforementioned

5
 1 studies employed an inclusive dataset covering all distress types which are annotated cautiously
2 by pavement experts.
3 Until recently, very few studies have been performed to address the issue of fine-grained
4 classification of pavement distress. The accuracy of distress classification is dependent on
5 factors such as camera view angle, segmentation accuracy, etc. The adoption of recent
6 advances in machine learning has dramatically improved the robustness of distress
7 classification methods. However, despite the great achievement of deep learning models,
8 existence of shadows and poor lighting and low contrast have made it challenging for pavement
9 researchers to develop an intact automated pavement distress detection model. Most of current
10 models only work accurate when the cracks are discernible, usually with uniform illumination
11 and salient gray-level features. Also, the camera angle view parameter is a challenging issue in
12 crack segmentation and classification. Most of the trained crack classification models only
13 work precisely if the same camera angle view images are used for prediction.
14 Furthermore, the former studies did not try to develop a model to classify and quantify
15 distress density simultaneously. In section 2.3, we review our previous research, which we
16 introduced a dataset with both top-down and wide-view images which are available in GitHub
17 repository [61].
18
19 2.3. Developing Crack Detection Model using Deep-Learning Framework (Majidifard et
20 al., 2020)
21 The current study is inspired by Majidifard et al’s recent work which developed a
22 comprehensive dataset for training deep learning algorithms for classifying different types of
23 pavement distress. This section highlights their work and its relevance to the current study. The
24 study first defined and annotated nine types of the most critical distresses which affect
25 pavement condition selected after reviewing various studies [62-70]. Next, a large database of
26 pavement images obtained from 22 different pavement sections in the United States was
27 collected via Google’s open street data API. All the distresses on the acquired images were
28 then annotated using the pre-defined distress classification rules as shown in Table 1. Figure 2
29 also offers instances of images annotated with nine different distress types. As shown in the
30 table 1, reflective, lane longitudinal, sealed longitudinal, and block cracks are among the
31 highest number of boundary boxes and images found in our dataset. Two different images were
32 extracted at each coordinate location: Wide-View images with a pitch angle of -70⸰ and birds-
33 eye-view at -90⸰. The wide view images were found to be useful for distress classification,
34 whereas the top-down view image resulted in more precise distress severity quantification.
35
36
37
38
39
40
41
42
43
44
45
46
47

6
1 Table 1. Distress types versus their corresponding distress ID [31].
Number of
Distress Type Distress ID Image Example boundary boxes for
each class

Reflective Crack D0 12428

Transvers Crack D1 5343

Block Crack D2 9709

Longitudinal Crack D3 2892

Alligator Crack D4 4838

Sealed Reflective Crack D5 5910

Lane Longitudinal Crack D6 12102

Sealed Longitudinal Crack D7 13610

pothole D8 637

7
 1 Figure 2. Annotated images (wide-view images) in the PID dataset [31].
2 A state-of-the-art, deep learning framework named YOLO v2 was implemented to
3 automatically detect and classify nine types of pavement distress.
4 Figure 3 shows the architecture of the YOLO model that was applied. The reader is directed
5 to the original paper for model development and testing details (Figure 3).

6 Figure 3. YOLO architecture [49]

7 Figure 4 shows some examples of prediction results on top-down images. To challenge the
8 robustness of the model, full sunshine images and images containing shadows (trees and cars)
9 were selected. Although the model was trained on wide-view images, the prediction results on
10 top-down images confirm that the developed model has the capability to detect distresses in
11 both the shadow-containing and full sunshine images, accurately.
12

8
 1 Figure 4. Samples of detecting pavement distresses from top-down images [31].
2 Precision, Recall, F1 scores and confusion matrices were the parameters utilized to assess
3 accuracy of the model.
4 Table 2 represents detection and classification accuracies of the proposed model for the nine
5 classes in our dataset. The precision, recall and F1 values for the model were 0.93, 0.77 and
6 0.84, respectively [31].
7
8 Table 2. Detection and classification results for nine distress types [31].
Crack
class D0 D1 D2 D3 D4 D5 D6 D7 D8 Average
name
Precision 0.93 0.9 0.93 0.91 0.91 0.93 0.93 0.94 0.96 0.93
Recall 0.76 0.83 0.79 0.83 0.74 0.83 0.79 0.57 0.78 0.77
F1 0.84 0.86 0.85 0.87 0.82 0.87 0.85 0.71 0.86 0.84
9
10 In this section, we reviewed approaches to develop pavement condition models and crack
11 detections methods. Also, we explained why we introduced a new pavement image dataset,
12 which was used to develop a deep-learning-based distress detection tool. In the following
13 sections we highlight the main contributions of this study: first, we introduce a robust pavement
14 distress segmentation and densification approach based on deep convolutional neural networks.
15 Second, we develop a pavement condition index based on inputs from distress class types and
16 density information. To the authors’ knowledge, none of the previous studies have been able
17 to rate the condition of pavements by taking into account the type and extent of distresses
18 surveyed. Detecting and quantifying of distresses are not enough to evaluate the pavement
19 conditions. As mentioned in the literature review section, PASER is one the pavement
20 condition index which helps to rate roads consistently. However, PASER index is a qualitative
21 and human dependent index, we noticed that there are some mis-ranking and inconsistency

9
 1 between ranking of the sections. Figure 5 shows example of inconsistency in PASER rating
2 among the investigated sections.
3

Mo94-PASER= 9 US24-PASER=7.2 I44-PASER=8

US54-PASER = 5.5 MO13-PASER = 7 MO7-PASER= 6

US24-PASER=7 US136-PASER=5 US24(1)-PASER=5

4 Figure 5. Example of inconsistency in PASER rating among the investigated sections
5 In order to develop a robust prediction tool, we need to have a reliable response. Then, the
6 PASER values were revised by pavement experts and various approaches were implemented
7 to fit a model with the eight features to the response value.

8 3. Methodology of Developing Pavement Condition Prediction Models

9 The methodology of developing the pavement condition prediction models consists of four
10 main steps: First, 71 pavement sections were selected in the state of Missouri, and the PASER
11 values were extracted from MoDOT virtual portal. The PASER values were checked and
12 corrected in case of a discrepancy by pavement experts. Afterward, an average of 83 images

10
 1 per section was extracted from the Google map at corresponding GPS coordinates using the
2 developed python software. Second, shoulders were removed from images by road
3 segmentation technique, and the revised images were run through the developed YOLO model
4 to classify the distresses. Third, a U-Net based model was developed for distress quantification.
5 Forth the YOLO and U-net models merged together to create the hybrid model, and finally,
6 three prediction PASER models were developed using YOLO and hybrid models by
7 implementing various machine learning techniques. Figure 10 shows the flowchart of the three
8 prediction models.

9 Figure 6. The primary PASER prediction models developed in this study.

10
11 3.1. Main Road Segmentation
12 The types of mixtures used for road shoulder construction usually vary significantly from those
13 used for the main road. As a result, the types of distress and the underlying factors influencing
14 them are also different. In the current study, distress information on road shoulders are not used
15 in calculating the pavement condition index. We annotated and developed a shallow network
16 for cropping out the main road for each pavement image. Distress detection and segmentation
17 and subsequently carried out on the cropped image. Figure 5 shows examples of main road
18 cropping.

19 Figure 7. Automatic cropping of the main-road from shoulder via CNN

20
21 3.2. Developing a U-Net based Model for Distress Quantification
22 In the current study, a pre-trained U-Net convolutional network was used, which was
23 originally developed for biomedical image segmentation [71]. Herein, it was used to quantify
24 the density of cracks in roads.
25 Khanal and Estrada implemented a Neural baseline that does image segmentation to retinal
26 vessel images. The dataset included 40 retinal images (20 for training and 20 for testing) where
27 blood vessel were annotated at the pixel level to mark the presence (1) or absence (0) of a blood

11
 1 vessel at each pixel (i, j) of the image (Figure 9). The authors used the U-net architecture to do
2 blood vessel segmentation (Figure 8). It is an architecture that is widely used for semantic
3 segmentation tasks, especially in the medical domain. The U-net Architecture is an encoder-
4 decoder with some skip connections between the encoder and the decoder. The ability to
5 consider a wider context when making a prediction for a pixel is the primary benefit of this
6 algorithm. This comes from the large number of channels utilized in the up-sampling process.
7 Figure 8 shows the network architecture. The left and right side of the structure represents
8 contracting and an expansive path, respectively. The contracting path reflects the typical
9 architecture of a convolutional network. It contains the duplicated application of two 3×3
10 convolutions (unpadded convolutions), each accompanied by a rectified linear unit (ReLU) and
11 a 2×2 max pooling operation with stride 2 for downsampling. At each downsampling step, the
12 number of feature channels is doubled. The expansive path consists of an upsampling of the
13 feature map followed by a 2×2 convolution that divides feature channels. This feature is
14 connected with the correspondingly cropped feature map from the contracting path, and two
15 3×3 convolutions, each followed by a ReLU. At the final layer, a 1×1 convolution is utilized
16 to map each 64×64 element feature vector to the desired class numbers. Overall, the network
17 contains 23 convolutional layers [72].

18 Figure 8. U-net Architecture [72].

19 After processing pavement images with the pretrained model we found it very accurate.
20 However, the problem with this model is that it detects some external items like cars and
21 shadows as distresses. In order to address the mentioned issue, shadows and cars were removed
22 from the U_Net images with hand and the model retrained again. Figure 10 shows the images
23 with shadows, cars and cracks. The images in the first, second and third columns represent
24 original input, pretrained model and retrained model output, respectively. As seen in the Figure
25 10, cracks were detected accurately in the re-trained model although shadows covered them in
26 some cases. Also, the shadows and cars were removed and the noise decreased significantly in
27 the new model.

a) b) a)
28 Figure 9 a. One example of the training data to developed the U-net model, b. hand-crafted
29 annotations that were used to retrain the model.

12
 a) b) c)
1 Figure 10. Examples of detecting shadows and cars as cracks a) raw image b) pre-trained
2 model output, and c) retrained model output.

13
 1 In order to check the accuracy of the model, 20 images were selected for testing the model
2 performance. The pixel difference between the ground truth and the predicted image were
3 calculate as MSE (mean square error). Figure 11 represents GT, predicted and error for one
4 image in test dataset. The final MSE calculated as 0.25 for the 20 images in the test dataset.
5

a) b) c)
6 Figure 11. Image of a) ground truth, b) predicted and c) error.
7 3.3. Developing a Hybrid Model of YOLO and U-Net
8 In order to develop a robust and comprehensive pavement condition model we needed to
9 consider both type and density of the cracks. As mentioned before, none of the YOLO and U-
10 net models could not provide the most accurate pavement condition model. Therefore, both of
11 these models require to be integrated together. To address this concern, images were processed
12 through the proposed YOLO and U-net models individually and the detected objects in the
13 proposed yolo model image were masked on the corresponding images processed by the U-net
14 model. Afterward the density of cracks were calculated for each type of distresses (Figure 12).
15 The ratio of white pixels for each detected distress to the total image size are considered as the
16 new features for our hybrid pavement condition model, which will be explained in detailed in
17 section 3.4.3 and 3.4.4.
Ratio of white
Distress pixels to image
size
Reflective
0/(584*565)*100
crack
Transverse
0/(584*565)*100
crack
620/(584*565)*10
Block crack
0
Longitudina
0/(584*565)*100
l crack
Alligator
0/(584*565)*100
crack
Sealed
transverse 0/(584*565)*100
crack
Sealed
251/(584*565)*10
longitudinal
0
crack

14
 Lane
521/(584*565)*10
longitudinal
0
crack
Pothole 0/(584*565)*100
1 Figure 12. Integrated Yolo and U-Net based model and calculation of ratio of white pixels
2
3 3.4. Developing Pavement Condition Prediction Models
4 3.4.1 Pavement Condition Prediction Model Development based on YOLO Model outputs
5 Using Genetic Expression Programming
6 In order to develop pavement condition prediction model, all the extracted images for each
7 section were run through the developed YOLO crack detection model. The average numbers
8 of detected distresses were calculated for each section. Table 3 exhibits the detailed variables
9 statistics used in this study. The data (71 pavement sections) are presented by frequency
10 histograms (see
11

a)

b)
12 Figure 13) to visualize the distribution of the samples. As observed from
13

a)

15
 b)
1 Figure 13, the distributions of the predictor variables are not uniform.
2 Table 3. Statistical parameters of the dependent and independent variables.
D(0) D(1) D(2) D(3) D(4) D(5) D(6) D(7) D(8) PASER
Mean 0.58 0.26 0.82 0.21 0.11 0.20 0.41 0.33 0.00 7.2
Median 0.43 0.13 0.33 0.18 0.04 0.01 0.35 0.09 0.00 7.0
Mode 0.02 0.00 0.00 0.00 0.00 0.00 0.01 0.00 0.00 7.0
Range 2.09 1.07 4.32 0.94 0.92 4.07 1.37 2.65 0.08 6.0
Maximum 2.09 1.07 4.32 0.94 0.92 4.07 1.37 2.65 0.08 10.0
Minimum 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 4.0
3
4
5

b)

a)
6 Figure 13. Distribution histograms of a) the features and b) output.
7
8 GEP is known as the most advanced generation of traditional genetic programming (GP) to
9 generate nonlinear prediction models automatically. These techniques have been extensively
10 deployed for tackling complicated engineering problems [73-80]. The traditional GP creates
11 computational models by imitating the biological evolution of living organisms. It provides a
12 tree-like form of a solution, which represents the closed-form solution of the optimization
13 problem [81; 82]. The primary objective of GP is to generate a population of programs that
14 correlate inputs with outputs for each data point. The initial random functions are then
15 calibrated to seize fitting functions that can accurately estimate the output via an administered
16 trial and error methodology.
17 The GP algorithm has some advantages over other machine learning (ML) techniques [83]
18 such as adaptive neuro-fuzzy inference systems and artificial neural networks. The first benefit

16
 1 is that GP is not a black box and outputs are in the form of semi-complex mathematical
2 solutions. The other advantage is that GP-based models lie on their inclination to obtain precise
3 relationships without considering former patterns of the existing relationships. On the other
4 hand, the primary advantage of GEP over the traditional GP method is that it can compile
5 several sub-programs to create a single complex program. Moreover, the GEP algorithm can
6 be implemented significantly faster than GP due to evolving the binary bit patterns. However,
7 more fundamental details about the ML, GP and GEP methods can be found in [81; 82; 84]. In
8 the current study, GEP is deployed for developing the rutting prediction models.
9 To avoid overfitting, the whole dataset (71 sections) was divided randomly into three
10 categories as training (70%), validation (15%), and testing (15%). The training and testing
11 subsets were used to calibrate and evaluate the models, respectively, while validation dataset
12 was used as an external output to check the model performance.
13 This study presents a new ML-based model which predicts the PASER from the following
14 variables:
15
16 𝑃𝐴𝑆𝐸𝑅 = 𝑓[𝑑(1), 𝑑(2), 𝑑(3), 𝑑(4), 𝑑(5), 𝑑(6), 𝑑(7), 𝑑(8), 𝑑(9)] [1]

17 where,
18 d(1)= Reflective Crack, d(2)= Transverse Crack, d(3)= Block Crack, d(4)= Longitudinal
19 Crack, d(5)= Alligator Crack, d(6) = Sealed Reflective Crack, d(7)= Lane Longitudinal
20 Crack, d(8)= Sealed Longitudinal Crack, d(9)= Pothole. All the variable including d(1) to
21 d(9) are the average number of distresses per each section.
22 The model was developed using PASER values obtained from 71 pavement sections in the
23 Midwest United States. Various runs were performed to delineate the optimized GEP
24 parameters. There are various principal setting GEP parameters such as general setting,
25 complexity increase, genetic operators, numerical constant, and fitness function. In general
26 part, the number of chromosomes changes the simulation run time. The higher number of
27 chromosomes, the longer running time. The head size represents the complexity of terms in the
28 developed model. Table 4 shows a set of parameters used during the GEP simulations.
29 Table 4. The optimal parameter setting for the GEP algorithm.
Parameter Settings

Chromosomes 30
Genes
General 6
Head size 12
Linking Addition
function
+, -, ×, /, √, 3√, Ln, Log, power,
Functions Function set exp
,sin, cos, tan,

Generations
Complexity without change 2000
increase Number of tries 3
Max. 5
complexity

17
 Genetic Mutation rate 0.00138, 0.044
operators Inversion rate 0.00546
Data type Floating-point
Numerical Lower bound -10
constants
Upper bound +10

2 The optimal GEP-based prediction model for PASER is as follows:

3 𝑌1 = 𝑎𝑟𝑐𝑡𝑎𝑛((𝑑(9)^(1/3) − ((𝑑(5)
4 − 𝑑(3))^2)))); [2]

5 𝑌2 = 𝑠𝑖𝑛(𝑠𝑖𝑛(𝑡𝑎𝑛ℎ(((𝑑(9) + (𝑚𝑖𝑛((𝑑(8) − 𝑑(1)), 𝑚𝑖𝑛(𝐺2𝐶4, 𝑑(6))) + (𝐺2𝐶7

6 + 𝐺2𝐶7))) − 𝑑(4)))));

7 𝑌3 = (𝑒𝑥𝑝(𝑚𝑖𝑛(𝑑(6), ((((𝑑(3) + 𝐺3𝐶8)^(1/3) + 𝑑(2))/𝑒𝑥𝑝(𝑑(6))) − (𝐺3𝐶6 − (1.0

8 − 𝑑(1)))))));

9 𝑌4 = 𝑚𝑖𝑛(𝑑(9), 𝑠𝑖𝑛((𝑚𝑖𝑛((𝑒𝑥𝑝(𝑑(5)) ∗ 𝑚𝑖𝑛(𝑑(5), 𝑑(1))), (𝐺4𝐶2 − 𝑑(2))) + 𝑑(6))));

10 𝑌5 = (1.0/(((((𝐺5𝐶3 + 𝐺5𝐶2) + (𝑑(3) + 𝑑(1))))^(1/3) ∗ ((𝑑(7) + 𝐺5𝐶9))^(1/3)

11 + 𝑒𝑥𝑝(𝑟𝑒𝑎𝑙𝑙𝑜𝑔(𝑚𝑎𝑥(𝑑(6), 𝐺5𝐶4))))));

12 𝑌6 = (1.0 − (𝑚𝑖𝑛(𝑑(2), 𝐺6𝐶6) − (((1.0/((𝐺6𝐶9 − 𝑑(4)))) ∗ ((𝑑(3) + 𝑑(1)) + 𝑑(3)))

13 − 𝑑(9))));

14 𝑌 = 𝑌1 + 𝑌2 + 𝑌3 + 𝑌4 + 𝑌5 + 𝑌6

15 Where:

16 G2C7 = -0.19, G2C4 = 0.51, G3C6 = 9.60, G3C8 = 10.07, G4C2 = 5.40, G5C9 = 2.57, G5C4
17 = 1.83, G5C3 = 6.68, G5C2 = -7.18, G6C6 = -6.74, G6C9 = -3.36
18 d(1)= Reflective Crack, d(2)= Transverse Crack, d(3)= Block Crack, d(4)= Longitudinal
19 Crack, d(5)= Alligator Crack, d(6) = Sealed Reflective Crack, d(7)= Lane Longitudinal
20 Crack, d(8)= Sealed Longitudinal Crack, d(9)= Pothole.
21 Figure 14 represents measured versus predicted PASER values for the entire data. Although
22 the dataset size was small, acceptable performance for the proposed model was achieved
23 (Figure 15).
24

18
 12.00
Measured value Predicted value

10.00

8.00
PASER

6.00

4.00

2.00

0.00
0 10 20 30 40 50 60 70 80

Data Record
1

2 Figure 14. Measured versus predicted value for the entire data.
3 In order to measure the performance of model, coefficient of determination (R2), root mean
4 squared error (RMSE) and mean absolute error (MAE) were used.
(∑𝑛 ̅̅̅ ̅ 2
𝑖=1(𝑂𝑖 −𝑂𝑖 ) (𝑡𝑖 −𝑡𝑖 ))
5 𝑅 2 = ∑𝑛 ̅̅̅ 2 𝑛 ̅ 2
[3]
1=1(𝑂𝑖 −𝑂𝑖 ) ∑𝑖=1(𝑡𝑖 −𝑡𝑖 )

∑𝑛
𝑖=1(𝑂𝑖 −𝑡𝑖 )
2
6 𝑅𝑀𝑆𝐸 = √ [4]
𝑛
∑𝑛
𝑖=1|𝑂𝑖 −𝑡𝑖 |
7 𝑀𝐴𝐸 = [5]
𝑛
8 Where,
9 𝑂𝑖 : Measured value
10 𝑡𝑖 : Predicted value
11 𝑂𝑖 : Average of measured values
12 𝑡𝑖 : Average of predicted values
13 n: Samples number
14

19
 11 10 10

10
9 9
Predicted PASER 9

Predicted PASER

Predicted PASER
8 8
8
7 7
7 R² = 0.94
RMSE=0.38 6 6
6 MAE=0.28 R² = 0.876
R² = 0.87 RMSE=0.55
5 RMSE=0.6 5 MAE=0.44
5
MAE=0.46
4 4 4
4 9 14 4 9 4 9 14
Measured PASER Measured PASER Measured PASER

a) b) c)
1 Figure 15. Measured against predicted PASER using the GEP model: (a) training data; (b)
2 testing data, and; (c) validation data.
3 A sensitivity analysis was carried out to explore the relative importance of variables in the
4 GEP model. The results are presented in Figure 16. The major observations were:
5  Block cracks had the highest influence on the pavement condition (PASER).
6  Lane longitudinal, sealed longitudinal cracks and potholes had the lowest influence on
7 PASER. The small effect of potholes on pavement condition may seem nonsense at
8 first glance. However, none of the sections in our dataset have potholes on their
9 surfaces. Therefore, the actual effect of potholes on pavement condition ignored in our
10 prediction model. The same explanation is valid for the alligator and transverse
11 cracking.

80.0
70.0
Relative importance (%)

67.9
60.0
50.0
40.0
30.0
19.4
20.0
10.0 7.0
0.5 1.3 3.0 0.7
0.1 0.1
0.0

12 Figure 16. Variable importance in the GEP model.

13

20
 1 3.4.2. Pavement Condition Prediction Model Development based on Unet Model outputs
2 In this part of study a new pavement condition index was developed based solely on the U-net
3 model output. All the 71 sections images were processed through the proposed U-net model.
4 For each processed image, the ratio of white pixels (crack-like features) to the total size of the
5 image was used to measure the crack density. To reduce the effect of spurious, non-crack-like
6 features, the output of the U-Net mode was thresholded. Two thresholds were tested: the first
7 threshold only selects pixels with intensity values ranging from 127 to 255, and the second one
8 is from 200 to 255 counts as white pixels. The ratio of the white pixels were calculated for all
9 sections and it is reported as the distress index to rank the sections.
10
11 Figure 17 shows the correlation of the distress index from U-net model with corresponding
12 PASER values. As shown in the figure, PASER has stronger correlation with the crack density
13 threshold of 127 than the one with 200 threshold.
14

U-Net ratio relation PASER vs U-Net density PASER vs U-Net density

1.00 12.0 12.0

R² = 0.8928 10.0 10.0 R² = 0.3945

0.80 R² = 0.506

8.0 8.0
Threshold = 200

0.60

PASER
PASER

6.0 6.0
0.40
4.0 4.0

0.20 2.0
2.0

0.00 0.0 0.0

0.00 1.00 2.00 3.00 4.00 0.00 2.00 4.00 0.00 0.50 1.00
Thershold = 127 Thershold = 127 Thershold = 200

a) b) c)
15
16 Figure 17. Correlation of a) distress density with different threshold, b) PASER versus
17 distress density with threshold=127, c) PASER versus distress density with threshold=127
18
19 This model can rank sections based on the density of distresses. However, it cannot
20 differentiate types of distresses and this may lead to a misleading conclusion. For example, this
21 model cannot discriminate sealed cracks from unsealed cracks; hence it could penalize the
22 condition of a section unfairly. Also sealed cracks are usually thicker than the unsealed ones,
23 the ration of white pixels will therefore be higher than images from unsealed crack sections.
24 Hence, the main problem with U-net model ranking is that this model just rely on detecting the
25 density of distresses not types of them. It has been well known that different distresses affect
26 pavement condition differently (

27 Figure 18).

21
 1

2 Figure 18. Example of problems associated with crack density U-Net based model
3
4 According to the limitation of the YOLO and U-net models, we developed a hybrid model
5 including both of these models. Therefore, the type and density of distresses will be considered
6 to develop a new pavement condition index.
7 3.4.3. Linear regression PASER prediction model based on hybrid model
8 In this part of study, linear regression method was used to develop a prediction model using
9 the explained variables. In order to make the equation similar to PCI approach, the deducted
10 values introduced as the response instead of the measured PASER values. Table 5 shows the
11 estimated coefficient for the predicted model. Also, P-value were calculated for each variable
12 and the variable number three which is block cracking has the lowest p-value and highest
13 importance in the model. Considering confidence level of 95%, the variable 1, 3, 4, and 8 are
14 significant. Figure 19 shows the fitted value versus the measured ones for training, testing and
15 validation dataset.
16
17 𝑃𝐴𝑆𝐸𝑅 𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑒𝑑 = 0.1 ∗ (100 − ∑𝑛𝑖=1(𝐵𝑖 𝑑𝑖 )) [2]

18 where, d(i) = ratio of white pixels to total image size

19 d(1)= Reflective Crack, d(2)= Transverse Crack, d(3)= Block Crack, d(4)= Longitudinal
20 Crack, d(5)= Alligator Crack, d(6) = Sealed Reflective Crack, d(7)= Lane Longitudinal
21 Crack, d(8)= Sealed Longitudinal Crack.
22 Table 5. Estimated coefficient for the fitted prediction model using linear regression from R
23 software
Variable Coefficients Std.Error t-value Pr(>|t|) Importance
d(1) 100.433 13.745 7.307 2.49E-09 ***
d(2) -40.603 34.906 -1.163 0.2505
d(3) 47.709 3.312 14.405 2.00E-16 ***
d(4) 70.774 16.16 4.38 6.42E-05 ***
d(5) 40.781 27.861 1.464 0.1498
d(6) 14.959 11.834 1.264 0.2123
d(7) 20.856 16.236 1.285 0.2051
d(8) 7.887 3.602 2.189 0.0335 *
24

22
 10 10 10

8 8 8

Predicted PASER
Predicted PASER
Predicted PASER

6 6 6

4 4 4
R² = 0.9781
R² = 0.923
2 R² = 0.8966 2 2 RMSE=4.76
RMSE = 7.25 RMSE=5.17 MAE=3.93
MAE = 5.58 MAE=4.61
0 0 0
0 2 4 6 8 10 0 2 4 6 8 10 0 2 4 6 8 10
Measured PASER Measured PASER Measured PASER

a) b) c)
1 Figure 19. Measured PASER versus predicted PASER using linear regression: (a) training
2 data; (b) testing data, and; (c) validation data.
3
4 3.4.4. Weight based PASER prediction model based on hybrid model
5 The PCI decision matrix affords particular guidelines for the repairs required based on road
6 classifications. Using PCI, helps to establish a pinpoint for preventive preservation that
7 diminishes road deteriorating before the point that it needs expensive rehabilitation [33].
8 In this model, different weights were assigned to the distresses, and deduct values were
9 calculated similar to the PCI method (Table 6 and equation 2). Different weights were
10 considered according to the level of importance of distresses in pavement condition. Afterward,
11 the deduct values were subtracted from 100.
12 Table 6. Distress weights to calculate the predicted PASER using weight method
Ratio of white pixels to total image
Variable Weight
size
V1 Reflective 0.4
V2 Transverse 0.4
V3 Block 0.4
V4 Longitudinal 0.4
V5 Alligator 0.4
V6 Sealed Reflective 0.1
V7 Lane Longitudinal 0.1
V8 Sealed Longitudinal 0.1
13
14 𝑃𝐴𝑆𝐸𝑅 = 0.1 × (100 − ∑𝑛𝑖=1(𝑤𝑖 𝑑𝑖 )) [6]

15 where,
16 d(1)= Reflective Crack, d(2)= Transverse Crack, d(3)= Block Crack, d(4)= Longitudinal
17 Crack, d(5)= Alligator Crack, d(6) = Sealed Reflective Crack, d(7)= Lane Longitudinal
18 Crack, d(8)= Sealed Longitudinal Crack, d(9)= Pothole
19 Figure 20 shows the measured PASER values versus the predicted PASER using the PCI
20 weighting method. The coefficient of determination (R2) for the training, testing and validation

23
 1 dataset are 0.87. 0.94, and 0.94. One of the most interesting benefit about the weight-based
2 method over other approaches is that the weights can be adjusted regarding to the case project.
3 For example, for airport pavement monitoring, where reflective cracks are more important than
4 longitudinal cracks. Then desired weights can be assigned to the distresses and a new PASER
5 index will be obtained based on new weights.

10 10 10

8 8 8

Predicted PASER
Predicted PASER

Predicted PASER
6 6 6

4 4 4

R² = 0.9382 R² = 0.9445
2 R² = 0.8738 2
2 RMSE = 8.73
RMSE = 12.2 RMSE = 10.45
MAE = 9.54 MAE = 7.96
MAE = 10.17
0 0 0
0 2 4 6 8 10 0 2 4 6 8 10 0 2 4 6 8 10
Measured PASER Measured PASER Measured PASER

a) b) c)
6 Figure 20. Measured PASER versus predicted PASER using Weight Method: (a) training, b)
7 testing and (c) Validation dataset.
8 4. Model Validation
9 To further evaluate the generalization capability of the PASER prediction models, the
10 developed models were deployed to predict the condition of six pavement sections with
11 different conditions (Figure 21). First, a new set ofpavement images was extracted for each
12 section from Google maps using our developed software. Second, the images were analyzed
13 with the developed crack detection and segmentation models (YOLO and U-Net) and the ratio
14 of white pixels for each detected distresses to the total image size were calculated. Finally, the
15 PASER values were predicted using weight based and linear regression models for all the
16 extracted images. The images were extracted by 15 m and we considered 5 meter as the view
17 of each image. The moving average of PASER values were calculated by a period of 20 images
18 and plotted in Figure 22a and b. Figure 22c shows the predicted PASER which developed
19 based on distress classification alone (YOLO model output). As seen in this figure, the
20 fluctuation of the PASER values from YOLO model is less than the ones from hybrid models.
21 This is also expected because the hybrid models are more susceptible and precise due to its
22 dependency on pixels counting. Also Figure 22d shows the corresponding measured PASER
23 values extracted from MoDOT’s portal. As seen, the predicted PASER values correlate well
24 with the measured values. The rankings are in order according to the measured PASER except
25 swapping MO11 and US24. Contrary to our developed pavement condition prediction models,
26 PASER rating is very straightforward rating without considering any distress quantification in
27 details. The approximate-based nature of the PASER procedure, makes it challenging to
28 correlate our machine learning-based pavement condition model with the measured PASER
29 values in some cases. Also the PASER values from hybrid models are fluctuating more than
30 the corresponding measured PASER values due to variety of detected distresses image by
31 image. While measured PASER values are usually constant along 2000 meter which is
32 unrealistic (Figure 22).
33

24
 I70 US50_Sedelia MO11

US24_lexington US71 MO52_Stover

1 Figure 21. Pavement condition, a) I70 (good condition), b) US50 , c) MO11, d) US24,
2 e)US71 and f) MO52 (dense-block cracks).
3

Hybrid model-weight based PASER Hybrid model-linear regression PASER

10.0 10.0

8.0 8.0
Predicted PASER
Predicted PASER

6.0 6.0

4.0 4.0

2.0 2.0

0.0 0.0
0 500 1000 1500 2000 2500 0 500 1000 1500 2000 2500
Profile (m) Profile (m)

a) b)

25
 YOLO model-GEP based PASER Measured PASER

10 10

8 8
Predicted PASER

Measured PASER
6 6

4 4

2 2

0 0
0 500 1000 1500 2000 2500 0 500 1000 1500 2000 2500
Profile (m) Profile (m)

c) d)

1 Figure 22. Changing of Predicted PASER using different models a) Hybrid model-weight
2 based, b) Hybrid model-linear regression, c) YOLO model-GEP based and d) Measured
3 PASER, for six sections alongside pavement profile.

4 5. Conclusions

5 In this study, a deep machine learning approach was implemented to predict pavement
6 condition of asphalt-surfaced roadways. Models were trained using a comprehensive road
7 condition dataset consisting of 7,237 images extracted from Google Street-view. The images
8 were annotated with bounding boxes featuring nine different pavement distresses. A state-of-
9 the-art deep learning frameworks were implemented to detect, classify and segment nine types
10 of pavement distress automatically. A pre-trained U-Net based model was used to calculate the
11 density of the distresses. The pre-trained model was fine-tuned by using manually annotated
12 road images acquired in a variety of environmental conditions. The model can accurately
13 differentiate non crack-like features including shadows, cars from pavement distresses. In order
14 to develop a model to classify and quantify severity of the distresses, YOLO and U-net model
15 were integrated together as a hybrid model.
16 Various pavement condition prediction indices were developed based on the detected
17 distresses by YOLO, and hybrid model. Approximately 82 images per section was extracted
18 from 71 different road sections located in the state of Missouri. The distresses detected for all
19 the images using the developed deep-learning crack detection model, and the average number
20 of distresses per image were calculated for all the 71 sections. Afterward, the measured PASER
21 values were extracted from MoDOT portal for all the corresponding road sections. A GEP-
22 based model was developed to predict PASER using the average number of distresses per
23 image as an input. Also, two more PASER prediction models were developed based on the
24 hybrid model outputs. Linear regression, and weight-based prediction models were developed,
25 and the performance of them were verified by high values of coefficients of determination (R2)

26
 1 for all the training, validation, and testing dataset. The results of the sensitivity analysis indicate
2 that block cracking (an extensive distress in the Midway in modern times) is the most effective
3 parameter in explaining the variations in PASER as compared with the other predictor
4 variables. This may be due to the fact that block cracking is a clear indicator of advanced
5 asphalt age and age hardening, both of which are in turn correlated to many other pavement
6 distresses.
7 For further model validation, six sections with various surface conditions were selected from
8 Missouri roads. The predicted PASER values correlated well with the measured values.
9 However, there was a swap in ranking between two of the cases. Contrary to our proposed
10 pavement condition prediction models, traditional PASER rating is a qualitative-based rating
11 without considering any distress quantity in detail. The approximate and human-based nature
12 of the PASER procedure makes it challenging to correlate our machine learning-based
13 pavement condition model with the measured PASER values in some cases. Also, the predicted
14 PASER values from our models are fluctuating more than the corresponding measured PASER
15 values due to the variety of detected distresses image by image in each section, while PASER
16 values are usually constant over the whole section which is unrealistic.
17 Finally, the proposed models offer some advantages over traditional pavement monitoring
18 (expensive cost of ARAN vehicles and laser equipment), and as compared to previous deep
19 learning-based models. First, this tool excluded the dependency of PASER to human judgment
20 and made it more accurate. Also this study is the pioneer in concerning about developing a
21 prediction pavement condition index after developing a model to detect the distresses. Second,
22 the models were trained using Google street-view images, which are free and available for
23 virtually all roads in the US and abroad. Third, the models were developed based on
24 comprehensive pavement image dataset which was annotated considering wide variety of
25 common pavement distress types by pavement experts. Finally, the developed models are
26 robust and flexible, cost-effective, and able to predict distress from different camera views
27 towards convenient. The fact that these first-generation models appear to have an acceptable
28 average prediction error suggests that it may be very useful for DOTs and road agencies, as a
29 means to evaluate road sections conditions. This tool could be conveniently employed to
30 evaluate the pavement conditions during its service life and help to make valid decisions for
31 rehabilitation or reconstruction of the roads at the right time.

32 Future Studies

33 A current limitation in the PASER prediction model reported herein is that it was calibrated
34 using a relatively limited set of PASER results (71 sections). The model performance will be
35 enhanced by updating the model, allowing to continue to learn and ultimately culminate into a
36 well-built and broadly predictive tool, as more data becomes available. It is hoped that
37 collaborations with other research groups and owner-agencies will lead to a vastly larger
38 database, and an even more highly predictive model. It is recommended that a major, national
39 study is justified by the current results, and represents a next logical step forward.
40 Also the same idea can be implemented on 3D images and other distresses like rutting can
41 be taken into account in the final pavement condition prediction model. Furthermore, distresses
42 characteristics can be investigated precisely by introducing 3D images.
43 In order to facilitate the deployment of the proposed model, an integrated software will be
44 developed to incorporate all the steps, including extracting images, analyzing images,
45 predicting pavement condition together.

46 Acknowledgments

47 The authors would like to thank Missouri Department of Transportation for providing access
48 to MoDOT virtual portal. The findings and conclusions reported herein are those of the authors,
49 and not necessarily those of the MoDOT.

27
 1 Conflict of interest

2 The authors do not have any conflict of interest with other entities or researchers.

3 6. References

4 1. Vavrik, W., Evans, L., Sargand, S., & Stefanski, J. (2013). PCR evaluation: considering transition from
5 manual to semi-automated pavement distress collection and analysis.
6 2. Koutsopoulos, H. N., & Downey, A. B. (1993). Primitive-based classification of pavement cracking
7 images. Journal of Transportation Engineering, 119(3), 402-418.
8 3. Chambon, S., & Moliard, J. M. (2011). Automatic road pavement assessment with image processing:
9 review and comparison. International Journal of Geophysics, 2011.
10 4. Oliveira, H., & Correia, P. L. (2009, August). Automatic road crack segmentation using entropy and
11 image dynamic thresholding. In 2009 17th European Signal Processing Conference (pp. 622-626). IEEE.
12 5. Tsai, Y. C., Kaul, V., & Mersereau, R. M. (2009). Critical assessment of pavement distress segmentation
13 methods. Journal of transportation engineering, 136(1), 11-19.
14 6. Zhang, D., Li, Q., Chen, Y., Cao, M., He, L., & Zhang, B. (2017). An efficient and reliable coarse-to-
15 fine approach for asphalt pavement crack detection. Image and Vision Computing, 57, 130-146.
16 7. Zhang, A., Li, Q., Wang, K. C., & Qiu, S. (2013). Matched filtering algorithm for pavement cracking
17 detection. Transportation Research Record, 2367(1), 30-42.
18 8. Ayenu-Prah, A., & Attoh-Okine, N. (2008). Evaluating pavement cracks with bidimensional empirical
19 mode decomposition. EURASIP Journal on Advances in Signal Processing, 2008(1), 861701.
20 9. Zhou, Y., Wang, F., Meghanathan, N., & Huang, Y. (2016). Seed-Based Approach for Automated Crack
21 Detection from Pavement Images. Transportation Research Record, 2589(1), 162-171.
22 10. Zhou, J., Huang, P., & Chiang, F. P. (2005). Wavelet-based pavement distress classification.
23 Transportation research record, 1940(1), 89-98.
24 11. Subirats, P., Dumoulin, J., Legeay, V., & Barba, D. (2006, October). Automation of pavement surface
25 crack detection using the continuous wavelet transform. In 2006 International Conference on Image
26 Processing (pp. 3037-3040).
27 12. Wang, K. C., Li, Q., & Gong, W. (2007). Wavelet-based pavement distress image edge detection with a
28 trous algorithm. Transportation Research Record, 2024(1), 73-81.
29 13. Ying, L., & Salari, E. (2010). Beamlet transform‐based technique for pavement crack detection and
30 classification. Computer‐Aided Civil and Infrastructure Engineering, 25(8), 572-580.
31 14. Adu-Gyamfi, Y. O., Okine, N. A., Garateguy, G., Carrillo, R., & Arce, G. R. (2011). Multiresolution
32 information mining for pavement crack image analysis. Journal of Computing in Civil Engineering,
33 26(6), 741-749.
34 15. Koch, C., Georgieva, K., Kasireddy, V., Akinci, B., & Fieguth, P. (2015). A review on computer vision
35 based defect detection and condition assessment of concrete and asphalt civil infrastructure. Advanced
36 Engineering Informatics, 29(2), 196-210.
37 16. Oliveira, H., & Correia, P. L. (2012). Automatic road crack detection and characterization. IEEE
38 Transactions on Intelligent Transportation Systems, 14(1), 155-168.
39 17. Fujita, Y., Shimada, K., Ichihara, M., & Hamamoto, Y. (2017). A method based on machine learning
40 using hand-crafted features for crack detection from asphalt pavement surface images. In Thirteenth
41 International Conference on Quality Control by Artificial Vision 2017 (Vol. 10338, p. 103380I).
42 International Society for Optics and Photonics.
43 18. Hizukuri, A., & Nagata, T. (2017, May). Development of a classification method for a crack on a
44 pavement surface images using machine learning. In Thirteenth International Conference on Quality
45 Control by Artificial Vision 2017 (Vol. 10338, p. 103380M). International Society for Optics and
46 Photonics.
47 19. Zou, Q., Cao, Y., Li, Q., Mao, Q., & Wang, S. (2012). CrackTree: Automatic crack detection from
48 pavement images. Pattern Recognition Letters, 33(3), 227-238.
49 20. Huval, B., Wang, T., Tandon, S., Kiske, J., Song, W., Pazhayampallil, J., ... & Mujica, F. (2015). An
50 empirical evaluation of deep learning on highway driving. arXiv preprint arXiv:1504.01716.
51 21. Hajilounezhad, T., Oraibi, Z. A., Surya, R., Bunyak, F., Maschmann, M. R., Calyam, P., & Palaniappan,
52 K. (2019). Exploration of Carbon Nanotube Forest Synthesis-Structure Relationships Using Physics-
53 Based Simulation and Machine Learning.

28
 1 22. Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road crack detection using deep convolutional
2 neural network. In 2016 IEEE international conference on image processing (ICIP) (pp. 3708-3712).
3 23. Cha, Y. J., Choi, W., & Büyüköztürk, O. (2017). Deep learning‐based crack damage detection using
4 convolutional neural networks. Computer‐Aided Civil and Infrastructure Engineering, 32(5), 361-378.
5 24. Maeda, H., Sekimoto, Y., Seto, T., Kashiyama, T., & Omata, H. (2018). Road damage detection using
6 deep neural networks with images captured through a smartphone. arXiv preprint arXiv:1801.09454.
7 25. Fan, Z., Wu, Y., Lu, J., & Li, W. (2018). Automatic pavement crack detection based on structured
8 prediction with the convolutional neural network. arXiv preprint arXiv:1802.02208.
9 26. Mandal, V., Uong, L., & Adu-Gyamfi, Y. (2018). Automated Road Crack Detection Using Deep
10 Convolutional Neural Networks. In 2018 IEEE International Conference on Big Data (Big Data) (pp.
11 5212-5215).
12 27. Xie, D., Zhang, L., & Bai, L. (2017). Deep learning in visual computing and signal processing. Applied
13 Computational Intelligence and Soft Computing, 2017.
14 28. Agrawal, A., & Choudhary, A. (2016). Perspective: Materials informatics and big data: Realization of
15 the “fourth paradigm” of science in materials science. Apl Materials, 4(5), 053208
16 29. Liu, W., Wang, Z., Liu, X., Zeng, N., Liu, Y., & Alsaadi, F. E. (2017). A survey of deep neural network
17 architectures and their applications. Neurocomputing, 234, 11-26.
18 30. Zhang, A., Wang, K. C., Fei, Y., Liu, Y., Chen, C., Yang, G., ... & Qiu, S. (2019). Automated pixel‐level
19 pavement crack detection on 3D asphalt surfaces with a recurrent neural network. Computer‐Aided Civil
20 and Infrastructure Engineering, 34(3), 213-229.
21 31. Majidifard, H., Jin, P., Adu-Gyamfi, Y., and W.G. Buttlar. (2020). PID: A New Benchmark Dataset to
22 Classify and Densify Pavement Distresses, Journal of Transportation Research Record.
23 32. Huang, Y. H. (2004). Pavement Analysis and Design (2nd Edition), Book.
24 33. Shahin, M. Y. (2005). Pavement management for airports, roads, and parking lots (Vol. 501). New
25 York: Springer.
26 34. ASTM D5340-12 2018., Standard Test Method for Airport Pavement Condition Index Surveys
27 35. ASTM D6433 2018., Standard Practice for Roads and Parking Lots Pavement Condition Index Surveys.
28 36. Walker, D., Entine, L., & Kummer, S. (2002). Pavement surface evaluation and rating. Asphalt PASER
29 manual. Transportation Information Center, University of Wisconsin, Madison, WI.
30 37. Eldin, N. N., & Senouci, A. B. (1995). A pavement condition‐rating model using backpropagation neural
31 networks. Computer‐Aided Civil and Infrastructure Engineering, 10(6), 433-441.
32 38. Fathi, A., Mazari, M., Saghafi, M., Hosseini, A., & Kumar, S. (2019). Parametric Study of Pavement
33 Deterioration Using Machine Learning Algorithms. Airfield and Highway Pavements, 31-41.
34 39. Piryonesi, S. M., & El-Diraby, T. E. (2019). Data Analytics in Asset Management: Cost-Effective
35 Prediction of the Pavement Condition Index. Journal of Infrastructure Systems, 26(1), 04019036.
36 40. Shahnazari, H., Tutunchian, M. A., Mashayekhi, M., & Amini, A. A. (2012). Application of soft
37 computing for prediction of pavement condition index. Journal of Transportation Engineering, 138(12),
38 1495-1506.
39 41. Koutsopoulos, H. N., Sanhouri, I. E., & Downey, A. B. (1993). Analysis of segmentation algorithms for
40 pavement distress images. Journal of transportation engineering, 119(6), 868-888.
41 42. Davis, E. R. (2005). Machine vision: Theory, algorithms, practicalities, Morgan Hanfmann Publishers,
42 Calif.
43 43. Cuhadar, A., Shalaby, K., & Tasdoken, S. (2002, May). Automatic segmentation of pavement condition
44 data using wavelet transform. In IEEE CCECE2002. Canadian Conference on Electrical and Computer
45 Engineering. Conference Proceedings (Cat. No. 02CH37373) (Vol. 2, pp. 1009-1014). IEEE.
46 44. Li, Q., & Liu, X. (2008, May). Novel approach to pavement image segmentation based on neighboring
47 difference histogram method. In 2008 Congress on Image and Signal Processing (Vol. 2, pp. 792-796).
48 IEEE.
49 45. Jitprasithsiri, S., Lee, H., Sorcic, R. G., & Johnston, R. (1996). Development of digital image-processing
50 algorithm to compute unified crack index for Salt Lake City. Transportation Research Record, 1526(1),
51 142-148.
52 46. Bai, X., Zhou, F., & Xue, B. (2012). Multiple linear feature detection based on multiple-structuring-
53 element center-surround top-hat transform. Applied optics, 51(21), 5201-5211.
54 47. Naoki, T., and U. Kenji. (1998). A crack detection method in road surface images using morphology,
55 LAPR Workshop on Machine Vision (MVA98), Makuhari, Chiba, Japan. pp. 154–157.

29
 1 48. Dalal, Navneet (2005). "Histograms of oriented gradients for human detection" (PDF). Computer Vision
2 and Pattern Recognition.
3 49. Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016). You only look once: Unified, real-time
4 object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp.
5 779-788).
6 50. Zhang, S., Wen, L., Bian, X., Lei, Z., & Li, S. Z. (2018). Single-shot refinement neural network for object
7 detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp.
8 4203-4212).
9 51. Ross, G., (2014). "Rich feature hierarchies for accurate object detection and semantic segmentation"
10 (PDF). Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE: 580–
11 587. arXiv:1311.2524. doi:10.1109/CVPR.2014.81. ISBN 978-1-4799-5118-5.
12 52. Girshick, R. (2015). Fast r-cnn. In Proceedings of the IEEE international conference on computer vision
13 (pp. 1440-1448).
14 53. Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster r-cnn: Towards real-time object detection with
15 region proposal networks. In Advances in neural information processing systems (pp. 91-99).
16 54. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C. Y., & Berg, A. C. (2016). Ssd: Single
17 shot multibox detector. In European conference on computer vision (pp. 21-37). Springer, Cham.
18 55. Zhang, A., Wang, K. C., Li, B., Yang, E., Dai, X., Peng, Y., ... & Chen, C. (2016). Automated pixel‐
19 level pavement crack detection on 3D asphalt surfaces using a deep‐learning network. Computer‐Aided
20 Civil and Infrastructure Engineering, 32(10), 805-819.
21 56. Zalama, E., Gómez‐García‐Bermejo, J., Medina, R., & Llamas, J. (2014). Road crack detection using
22 visual features extracted by Gabor filters. Computer‐Aided Civil and Infrastructure Engineering, 29(5),
23 342-358.
24 57. Karaköse, M., Akarsu, B., Parlak, K. S., Sarimaden, A., & Erhan, A. (2016). A fast and adaptive road
25 defect detection approach using computer vision with real time implementation. International Journal of
26 Applied Mathematics, Electronics and Computers, 4(Special Issue-1), 290-295.
27 58. Kawano, M., Mikami, K., Yokoyama, S., Yonezawa, T., & Nakazawa, J. (2017, December). Road
28 marking blur detection with drive recorder. In 2017 IEEE International Conference on Big Data (Big
29 Data) (pp. 4092-4097).
30 59. Zhang, K., H. D. Cheng, and B. Zhang. (2018). Unified Approach to Pavement Crack and Sealed Crack
31 Detection Using Preclassification Based on Transfer Learning. Journal of Computing in Civil
32 Engineering. 2018b. 32: https://doi.org/10.1061/(ASCE)CP.1943-5487.0000736.
33 60. Gopalakrishnan, K. (2018). Deep Learning in data-driven pavement image analysis and automated
34 distress detection: A review. Data, 3(3), 28.
35 61. Majidifard., H. (2020). PID - Pavement Image Dataset. Available: https://github.com/hmhtb/PID-
36 Pavement-Image-Dataset.git, [Accessed: 05-Jan-2020].
37 62. McGhee, K. (2002). Development and implementation of pavement condition indices for the Virginia
38 Department of Transportation. Phase I Flexible Pavements.
39 63. Buttlar, W. G., Meister, J., Jahangiri, B., Majidifard, H., & Rath, P. (2019). Performance
40 Characteristics of Modern Recycled Asphalt Mixes in Missouri, Including Ground Tire Rubber,
41 Recycled Roofing Shingles, and Rejuvenators (No. cmr 19-002).
42 64. Buttlar, W., Rath, P., Majidifard, H., Dave, E. V., & Wang, H. (2018). Relating DC (T) Fracture Energy
43 to Field Cracking Observations and Recommended Specification Thresholds for Performance-
44 Engineered Mix Design. Asphalt Mixtures, 51.
45 65. Jahangiri, B., Majidifard, H., Meister, J., Buttlar, W. (2019). Performance Evaluation of Asphalt
46 Mixtures with Reclaimed Asphalt Pavement and Recycled Asphalt Shingles in Missouri. Journal of
47 Transportation Research Record (TRR). DOI: 10.1177/0361198119825638.
48 66. Wang, Y. D., Ghanbari, A., Underwood, B. S., & Kim, Y. R. (2019). Development of a Performance-
49 Volumetric Relationship for Asphalt Mixtures. Transportation Research Record, 0361198119845364.
50 67. Ghanbari, A., Underwood, B. S., & Kim, Y. R. Development of Rutting Index Parameter Based on
51 Stress Sweep Rutting Test and Permanent Deformation Shift Model. International Journal of
52 Pavement Engineering, In Press.
53 68. Morovatdar, A., Ashtiani, S. R., Licon, C., Tirado, C., Mahmoud, E. (2020). A Novel Framework for
54 the Quantification of Pavement Damages in the Overload Corridors. 99th TRB Annual Meeting,
55 Transportation Research Record (TRR): Journal of the Transportation Research Board.

30
 1 69. Mansourkhaki, A., Ameri, M., Habibpour, M., & Underwood, B. S. (2020). Relations between
2 colloidal indices and low-temperature properties of reclaimed binder modified with softer binder, oil-
3 rejuvenator and polybutadiene rubber. Construction and Building Materials, 239, 117800.
4 70. Mansourkhaki, A., Ameri, M., Habibpour, M., & Shane Underwood, B. (2020). Chemical Composition
5 and Rheological Characteristics of Binders Containing RAP and Rejuvenator. Journal of Materials in
6 Civil Engineering, 32(4), 04020026.
7 71. Khanal, A., & Estrada, R. (2019). Dynamic Deep Networks for Retinal Vessel Segmentation. arXiv
8 preprint arXiv:1903.07803.
9 72. Ronneberger, O., Fischer, P., & Brox, T. (2015). U-net: Convolutional networks for biomedical image
10 segmentation. In International Conference on Medical image computing and computer-assisted
11 intervention (pp. 234-241).
12 73. Alavi, A. H., Gandomi, A. H., Nejad, H. C., Mollahasani, A., & Rashed, A. (2013). Design equations for
13 prediction of pressuremeter soil deformation moduli utilizing expression programming systems. Neural
14 Computing and Applications, 23(6), 1771-1786.
15 74. Saghafi, M., Asgharzadeh, S. M., Fathi, A., & Hosseini, A. (2019). Image processing method to estimate
16 the wearing condition of slurry seal mixtures. In Transportation & Development Institute (T&DI),
17 International Airfield and Highway Pavements Conference (Vol. 2019).
18 75. Rashidi, M., Saghafi, M., & Takhtfiroozeh, H. (2018). Genetic programming model for estimation of
19 settlement in earth dams. International Journal of Geotechnical Engineering, 1-10.
20 76. Majidifard, H., Jahangiri, B., Buttlar, W. G., & Alavi, A. H. (2019). New machine learning-based
21 prediction models for fracture energy of asphalt mixtures. Measurement, 135, 438-451.
22 77. Mirzahosseini, M., Jiao, P., Barri, K., Riding, K. A., & Alavi, A. H. (2019). New machine learning
23 prediction models for compressive strength of concrete modified with glass cullet. Engineering
24 Computations.
25 78. Jiao, P., Roy, M., Barri, K., Zhu, R., Ray, I., & Alavi, A. H. (2019). High-performance fiber reinforced
26 concrete as a repairing material to normal concrete structures: Experiments, numerical simulations and
27 a machine learning-based prediction model. Construction and Building Materials, 223, 1167-1181.
28 79. Bolandi, H., Banzhaf, W., Lajnef, N., Barri, K., & Alavi, A. H. (2019, July). Bond strength prediction
29 of FRP-bar reinforced concrete: a multi-gene genetic programming approach. In Proceedings of the
30 Genetic and Evolutionary Computation Conference Companion (pp. 364-364).
31 80. Bolandi, H., Banzhaf, W., Lajnef, N., Barri, K., & Alavi, A. H. (2019). An Intelligent Model for the
32 Prediction of Bond Strength of FRP Bars in Concrete: A Soft Computing Approach. Technologies,
33 7(2), 42.
34 81. Ferreira, C. (2001). Gene Expression Programming: A New Adaptive Algorithm for Solving
35 Problems. Complex Systems, 13(2), 87-129.
36 82. Koza, J. R., & Koza, J. R. (1992). Genetic programming: on the programming of computers by means
37 of natural selection (Vol. 1). MIT press.
38 83. Mitchell, T. M. (1997). Does machine learning really work?. AI magazine, 18(3), 11-11.
39 84. Alavi, A. H., & Gandomi, A. H. (2011). A robust data mining approach for formulation of geotechnical
40 engineering systems. Engineering Computations.

41

31

View publication stats