Received 15 September 2022, accepted 20 October 2022, date of publication 26 October 2022, date of current version 3 November 2022.

Digital Object Identifier 10.1109/ACCESS.2022.3217227

A Survey of Defect Detection Applications Based

on Generative Adversarial Networks
XIANGJIE HE 1 , ZHENGWEI CHANG2 , LINGHAO ZHANG2 , HOUDONG XU3 , HONGBO CHEN4 ,
AND ZHONGQIANG LUO 1,5 , (Member, IEEE)
1 School of Automation and Information Engineering, Sichuan University of Science and Engineering, Yibin 644000, China
2 StateGrid Sichuan Electric Power Research Institute, Chengdu 610095, China
3 StateGrid Sichuan Electric Power Company, Chengdu 610095, China
4 Sichuan Shuneng Electric Power Company Ltd., Chengdu 610000, China
5 Artificial Intelligence Key Laboratory of Sichuan Province, Sichuan University of Science and Engineering, Yibin 644000, China

Corresponding author: Zhongqiang Luo ([email protected])

This work was supported in part by the National Natural Science Foundation of China under Grant 61801319, in part by the Sichuan
Science and Technology Program under Grant 2020JDJQ0061 and Grant 2021YFG0099, in part by the Sichuan University of Science and
Engineering Talent Introduction Project 2020RC33, in part by the Innovation Fund of Chinese Universities under Grant 2020HYA04001,
in part by the Artificial Intelligence Key Laboratory of Sichuan Province Project under Grant 2021RZJ03, and in part by the 2022 Graduate
Innovation Fund of Sichuan University of Science and Engineering under Grant Y2022131.

ABSTRACT With the development of science and technology and the progress of the times, automation and
intelligence have been popularized in manufacturing in all walks of life. With the progress of productivity,
product defect detection has become an indispensable part. However, in practical scenarios, the application
of supervised deep learning algorithms in the field of defect detection is limited due to the difficulty
and unpredictability of obtaining defect samples. In recent years, semi-supervised and unsupervised deep
learning algorithms have attracted more and more attention in various defect detection tasks. Generative
adversarial networks (GAN), as an unsupervised learning algorithm, has been widely used in defect detection
tasks in various fields due to its powerful generation ability. In order to provide some inspiration for the
researchers who intend to use GAN for defect detection research. In this paper, the theoretical basis, technical
development and practical application of GAN based defect detection are reviewed. This paper also discusses
the current outstanding problems of GAN and GAN-based defect detection, and makes a detailed prediction
and analysis of the possible future research directions. This paper summarizes the relevant literature on the
research progress and application status of GAN based defect detection, which provides certain technical
information for researchers who are interested in researching GAN and hope to apply it to defect detection
tasks.

INDEX TERMS Deep learning, generating adversarial networks, defect detection, adversarial learning.

I. INTRODUCTION technologies in industrial imaging, computer vision, deep

Defect detection [1], which aims to find the appearance learning and other fields, vision-based defect detection tech-
defects of various industrial products, agricultural products nology has made great progress and become an effective solu-
and construction roads, is one of the important technologies tion for product appearance inspection, which has aroused
to ensure product quality and maintain production stability. strong attention from academia and industry. Defect detection
Previous defect detection requires manual screening, which is not only can be used to detect all kinds of industrial prod-
costly and inefficient and difficult to cover large-scale quality ucts (such as metal, textile, semiconductor, etc.), agricultural
inspection needs. In recent years, with the emergence of new products (such as litchi, tomatoes, etc.) and building roads
(such as concrete, road pavement, etc.) if there is a defect
The associate editor coordinating the review of this manuscript and (various defect sample is shown in Fig. 1), and has good
approving it for publication was Bing Li . precision and efficiency, and can also provide a simple and

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
VOLUME 10, 2022 113493
 X. He et al.: Survey of Defect Detection Applications Based on Generative Adversarial Networks

FIGURE 1. Examples of surface images.

safe operation environment. Therefore, defect detection has learning methods such as convolutional neural network
become one of the important basic research and technologies (CNN) [3], Deep Belief Network (DBN) [4], Recurrent Neu-
in the fields of intelligent manufacturing, product storage and ral Network (RNN) [5], Autoencoder (AE) [6] and Generative
transportation management, and is widely used in scenarios Adversarial Network (GAN) [7] to various defect detection
such as unmanned inspection, intelligent inspection, produc- tasks and achieved good performance. With the advent of the
tion control and anomaly traceability. Vision-based defect information age, data volume and complexity show an expo-
detection not only has very important research value, but also nential growth trend. More and more deep learning methods
has broad application prospects. However, compared with the and their variants have been proposed and applied to various
general object detection task, the defect detection task is faced defect detection tasks. In actual scenarios, it is often unrealis-
with many difficulties, such as lack of defect samples, low tic to collect enough defect samples for deep defect detection.
visibility of defects, irregular shape, unknown type, etc. This Usually, there are only a large number of normal samples and
makes it difficult for many existing methods to meet the task a small number (or even no) defect sample. In this case, the
requirements of high precision and high speed at the same class imbalance problem [8] will be very serious, and even
time. Therefore, there are still a large number of problems to directly lead to the failure of defect detection task. Therefore,
be solved on the way to realize practical application [2]. the data imbalance problem is the biggest obstacle to the
Because the defects on the surface of the object can be practical application of deep defect detection method.
regarded as an ‘‘anomaly’’, some defect detection methods Due to the ability of distribution fitting, generative model
adopt the idea of anomaly detection. However, the defini- has become one of the best methods for defect detection
tion of anomaly detection is different from defect detection. [9]. Among them, Variational Auto-Encoder (VAE) [10] and
Specifically, the concept of anomaly detection is more exten- GAN (including their variants) are the most representative
sive and abstract. Image-based anomaly detection mainly ones, and GAN and its variants are one of the most popular
focuses on whether the input image is an anomaly instance, deep learning methods in recent years. The generative ability
while surface defect detection focuses more on the detection of GAN is excellent, and the generated image instances can
task at pixel level. At the pixel level, anomalies differ more even be indistinguishable from the real images. Fig. 2 shows
subtly from normal patterns and are much more difficult to the excellent defect generation capability of GAN, where the
detect. Therefore, the direct use of anomaly detection meth- generated image examples are from literature [11]. As shown
ods is difficult to meet the task requirements of surface defect in Fig. 2, more realistic defect samples can be generated
detection [2]. by training with a dataset containing only normal samples.
Since deep learning methods have been applied to com- The excellent generation ability of GAN alleviates the prob-
puter vision tasks, researchers have widely applied deep lem of insufficient defect samples to a certain extent [12].

113494 VOLUME 10, 2022

X. He et al.: Survey of Defect Detection Applications Based on Generative Adversarial Networks

FIGURE 2. An example of image generation based on GAN.

FIGURE 3. Papers on GAN and its application in defect detection were published from 2014.01 to 2022.07.

In addition, GAN can also learn the feature representation of we review in detail the theoretical development and evolution
the latent space by reconstructing (repairing) defect samples. of GAN, the research progress, development process and
And by comparing with normal samples, GAN can accurately implementation of GAN-based defect detection methods in
detect whether there are defects in the graph and locate the specific applications. In addition, this paper summarizes and
location of defects. At present, most of the theories and meth- prospects the latest progress, challenges and future research
ods based on GAN are suitable for defect detection. There- directions of GAN based defect detection, which is of great
fore, GAN has become a very hot research topic in recent significance.
years. At present, many defect detection methods based on This paper can be divided into the following parts:
GAN have been widely used in production and manufac- Section 2 defines the defect detection problem and briefly
turing, construction, road and agricultural product quality introduces the defect detection process. Section 3 summarizes
inspection and other fields. Fig. 3 shows the publication of the defect detection algorithm based on GAN, including the
GAN-related papers and their application in defect detection principle of GAN, various variants and their development
from January 2014 to July 2022, where (a) cites the statistical history in defect detection applications. Section 4 summarizes
data in literature [9]. the application of GAN-based defect detection in various
Compared with many current literatures devoted to sum- industries. Section 5 analyzes the limitations and existing
marizing defect detection technology [13], [14], [15], [16], problems of current GAN and GAN-based defect detection
this paper makes the following contributions: technology, and considers and looks forward to the next
This paper focuses on the research progress and applica- development direction of GAN and GAN-based defect detec-
tion status of GAN in defect detection. At present, many tion based on the actual situation. Section 6 summarizes the
studies are devoted to summarizing the research progress thesis.
and application status of defect detection, and comprehen-
sively and profoundly summarizing the research progress and II. OVERVIEW OF DEFECT DETECTION
application status of deep learning-based defect detection. A. THE DEVELOPMENT OF DEFECT DETECTION
However, only a brief introduction is made to the research TECHNOLOGY
progress and application status of GAN in defect detection, The development milestones of defect detection methods are
without a comprehensive summary. Therefore, in this review, shown in Fig. 4, where the timeline indicates that this class

VOLUME 10, 2022 113495

 X. He et al.: Survey of Defect Detection Applications Based on Generative Adversarial Networks

FIGURE 4. Development milestone of defect detection technology.

TABLE 1. Comparison of defect detection between traditional methods and deep learning methods.

of methods was first applied to defect detection. In Fig. 4, difference in the perception of the defect pattern leads to two
taking 2012 as the dividing line, existing defect detection very different methods of detection. And take the insulation
methods are divided into traditional methods and deep learn- board surface defect detection as an example, as shown in
ing methods. The research on defect detection based on tra- Fig. 5. The first method is a defect detection method based
ditional methods began in the 1980s, with abundant research on supervised learning [33], which uses defect images with
results, which could be further divided into statistical method, labels (including categories, rectangular boxes, pixel by pixel,
filtering method and model method. And since 2013, with etc.) to be input into the network for training. In this case,
the successful application of deep learning technology repre- ‘‘defect’’ means a marked area or image. Therefore, this
sented by convolutional neural network in many computer- method pays more attention to defect features, for example,
vision tasks, people have tried to apply it to defect detection in the training phase, regions containing continuous crack
and achieved rich results. In 2021, Su et al. [17] reviewed the ranges or images are labeled as ‘‘scratch’’ defects for network
surface defect detection based on visual perception, and made training. And in the test phase, a ‘‘scratch’’ defect is consid-
a profound summary of the application of traditional methods ered to have occurred when a characteristic continuous crack
and deep learning methods in defect detection. Table 1 shows is detected in the insulating partition image. The second is the
the comparison of the advantages and disadvantages of tra- defect detection method based on unsupervised learning [34],
ditional methods and deep learning methods on defect detec- which usually only needs to input normal non-defect samples
tion. At present, research on defect detection methods based into the network for training, also known as one-class learn-
on deep learning methods is very active, and various inno- ing. This method pays more attention to the features without
vative methods are constantly emerging. According to the defects (i.e., normal samples). When features that have not
different defect information obtained, they can be divided into appeared before (defect features) are found in the process of
whole image classification, object detection and pixel-by- defect detection, defects are considered to be detected. In this
pixel segmentation methods. And in the following sections, case, ‘‘defect’’ means Anomaly, so this method is also called
these methods are elaborated. methods are very active, and Anomaly detection.
various innovative methods are constantly emerging. Accord-
ing to the different defect information obtained, they can 2) DEFINITION OF DEFECT DETECTION
be divided into whole image classification, object detection Compared with the explicit tasks of classification, detection,
and pixel-by-pixel segmentation methods. in the following and segmentation in computer vision, the requirements for
sections, these methods are elaborated. defect detection are very general. In fact, its requirements
B. DEFINITION OF DEFECT DETECTION PROBLEM can be divided into three different levels, namely ‘‘what is the
defect’’, ‘‘where is the defect’’, and ‘‘how big is the defect’’.
1) DEFINITION OF DEFECT
In computer vision tasks, defects tend to be notions of human (1) Stage 1: ‘‘What is the defect’’, corresponding to
experience rather than a purely mathematical definition. The the target classification/recognition task in computer
113496 VOLUME 10, 2022
X. He et al.: Survey of Defect Detection Applications Based on Generative Adversarial Networks

FIGURE 5. Example definition of a defect detection problem.

vision [35]. And as shown in Fig. 5, four defect cat- process of ‘‘defect classification’’ in stage 1, and the ‘‘defect
egories are classified: scratch, pothole, ablation and segmentation’’ in stage 3 can also complete the ‘‘defect loca-
voltage breakdown. The task at this stage is called tion’’ in stage 2. And Phase 1, Defect Classification, also
‘‘defect classification or recognition’’. Category labels achieves Phase 2 and phase 3 goals in a number of ways.
include image labels and defect type labels, indicating Therefore, in the following, it is still referred to as defect
whether the current image has defects and the category detection according to the traditional industrial habits, and
of defects respectively. it is only distinguished for different network structures and
(2) Stage 2: ‘‘Where is the defect’’ corresponds to the tar- target functions.
get detection and localization task in computer vision
[36], and the defect localization in this stage is detec- 3) DEFECT DETECTION SYSTEM
tion in a strict sense. It not only gives the types of The basic structure of the surface defect detection system
defects in the image, but also gives the specific loca- based on deep learning includes three main modules, which
tion of the defects in the original image. As shown in successively complete the functions of image acquisition,
Fig. 5, scratches, pits, ablation and voltage breakdown image processing and image feedback [17]. In the image
defects are respectively marked with external rectangu- acquisition module, lighting and imaging system should be
lar boxes. constructed according to the surface properties and defect
(3) Stage 3: ‘‘How big is the defect’’, corresponding to the features of the object to be detected. And through the coop-
segmentation task in computer vision [37]. As shown erative configuration and operation of mechanical device,
in the defect segmentation area in Fig. 5, the defect is light source and camera, the surface image of the object to
segmented from the background pixel by pixel, and a be examined with obvious defect characteristics is obtained.
series of information such as the length, width, area and In the image processing module, the image processing algo-
location of the defect can be further obtained, which rithm is used to detect the defect target in the image and iden-
can assist the product to carry out a higher level of tify the defect type. Finally, in the image feedback module,
quality assessment, such as the judgment of pros and the current sample is judged to be qualified according to the
cons of the product. detection standard, and the judgment result is transmitted to
Although the functional requirements and goals of the the actuator. At the same time, the defect type, location, shape
three phases of defect detection are different, the three phases and size of the image can also be visualized and displayed,
actually contain each other and can be inter-transformed. and the image and defect information can be stored for
For example, the ‘‘defect location’’ in stage 2 includes the subsequent query and statistics. Deep learning-based defect

VOLUME 10, 2022 113497

 X. He et al.: Survey of Defect Detection Applications Based on Generative Adversarial Networks

FIGURE 6. Structure illustration of the Generative Adversarial Network (GAN).

models are iteratively optimized by continuous confrontation,

that is, the optimization problem of GAN is a binary minimax
adversarial problem, so that the data distribution of pseudo-
samples generated by generator G is as close as possible to
the data distribution of real samples. The final goal of GAN
network optimization is as follows: generator G needs to be
able to ‘‘fool’’ the discriminator, that is, generate spurious
samples that make it difficult for the discriminator to dis-
tinguish between true and false; The discriminator D needs
to distinguish real samples from generated samples as far
as possible from the input data. The discriminator outputs
‘‘1’’ and ‘‘0’’ for real and generated samples, respectively.
However, when the output probability of the discriminator is
basically ‘‘0.5’’ each time (that is, the discriminator can no
longer distinguish between true and false samples), it means
that the model has reached the Nash equilibrium, that is, the
optimal state, which is the adversarial thought of GAN. The
training process of GAN can be divided into three stages, one
is to fix the discriminator D and train the generator G, the
FIGURE 7. Training algorithm for GAN. other is to fix the generator G and train the discriminator D,
and then cycle the first and second stages to continuously
iterate the training. Fig. 7 shows the training algorithm for
detection systems have been widely used in various detection
GAN.
tasks.
The original objective function of the Generative adversar-
ial network (GAN), as shown in Equation (1).
III. OVERVIEW OF GAN-BASED DEFECT DETECTION
A. OVERVIEW OF GAN PRINCIPLE min max V (D, G) = Ex∼pr (x) [log(D(x))]
G D
Goodfellow et al. [38] proposed generative adversarial net-
+ Ez∼pz (z) [log(D(G(z)))] (1)
works in 2014, whose core idea is two-player zero-sum game
theory. The generator and the discriminator use the mutual where V represents the output value of the loss function. G
game strategy to continuously iterate to improve the effect. and D are generators and discriminators for GAN, respec-
And the final effect you want to achieve is that the discrimi- tively. Pr (x) is the real data distribution, Pz (z) is the distribu-
nator can’t tell the difference between the real sample and the tion of generated data, and E is the average.
generated pseudo-sample. The network structure of GAN is The training optimization loss function of the generator G
shown in Fig. 6. is shown in Equation (2).
Generative adversarial network (GAN) consists of two
important parts, namely, generator G and discriminator D. min V (D, G) = Ez∼pz (z) [log(1 − D(G(z)))] (2)
G
The generator G can generate pseudo samples whose sim-
ilarity approximates the real samples by learning the fea- The training optimization loss function of the discriminator
ture distribution of the real sample data and random noise D is shown in Equation (3).
(or other data); And the discriminator D is used to distin-
max V (D, G) = Ex∼Pr (x) [log(D(x))]
guish between the real samples obtained from the data and D
the pseudo-samples generated by the generator G. The two + Ez∼Pz (z) [log(1 − D(G(z)))]Pz (z) (3)

113498 VOLUME 10, 2022

X. He et al.: Survey of Defect Detection Applications Based on Generative Adversarial Networks

When the quality of the generated samples is the best, the DCGAN to improve its own network and methods, which
data distribution of the generated samples should be consis- improved the accuracy and robustness of defect detection.
tent with that of the real samples. Therefore, theoretically WGAN [46]. Arjovsky et al. proposed a new algorithm
speaking, a trained GAN network should be able to fit the data called WGAN. The original GAN performs model train-
distribution of any sample and determine whether a given new ing by minimizing the JS divergence [62] between the real
sample is in the distribution [9]. distribution and the generated distribution, but this differ-
ence cannot reach the optimum. However, by minimizing
B. COMMON VARIANTS Wasserstein distance and satisfying Lipschitz continuity [62],
In practice, how to find the Nash equilibrium of GAN network WGAN theoretically solves the training instability and mode
is a challenge and difficulty. Because GAN faces two major collapse problems of GAN and ensures the diversity of gen-
problems. First, the model is difficult to train and unstable. erated samples. And WGAN does not even need to elabo-
In the actual training, it is easy to have the discriminator rate the network architecture, the simplest multi-layer fully
D converge and the generator G diverge, and it is difficult connected network can do. The loss function of WGAN
to have good synchronization between the two networks. is shown in Equation (5). Literature [63], [64], [65] has
Therefore, the training of D and G needs to be carefully introduced WGAN to improve its own network and method,
designed. Second, GAN has the phenomenon of mode col- which improves the accuracy and robustness of defect
lapse in the learning process [38]. When the generator G detection.
learns a parameter setting, it can generate samples that are
min max V (D, G) = Ez∼pz (z) [log(D(G(z)))]
very realistic to the discriminator D, and can easily ‘‘fool’’ G D
the discriminator D; Therefore, generator G will generate the − Ex∼pr (x) [log(D(x))]
same pseudo-samples again and again, and eventually always + λgpEx̂∼p(x̂) ||∇x̂ D x̂ ||2 − 1 (5)


generate the same sample points, and the pattern is missing,
so it cannot continue to learn. In order to make GAN networks where x̂ is uniformly sampled in a straight line between a
suitable for a wide variety of tasks and solve the problems pair of real samples and generated samples, and λgp is a
there faces, researchers have proposed various variants based hyperparameter.
on GAN. Fig. 8 shows the various GAN variant networks that Cycle-GAN [49]. Zhu et al. proposed a method to learn
have emerged in recent years. The GAN variants that are more to transform an image from the source domain to the target
commonly used for defect detection are listed below. domain Y without pairwise examples. Cycle-GAN makes the
CGAN [39]. Conditional Generative Adversarial Nets inverse mapping from Y to X hold by introducing a cyclic
(CGAN) proposed by Mirza et al. is an extension of the consistency loss, namely F (G (X )) = X . The loss function
original GAN. And both generator G and discriminator D change of Cycle-GAN is shown in Equation (6). This method
condition on additional information y, which can be any has excellent performance in image style transfer, object
kind of additional information, such as category information deformation and photo enhancement. Moreover, Cycle-GAN
or data from other models. Since labels, vectors and even has excellent performance in data enhancement. And litera-
images can be used as conditional data, CGAN can effectively ture [66], [67], [68] all introduced Cycle-GAN to improve its
control the sample semantics. CGAN is regarded as a two- own network and methods, which improved the accuracy and
player minimax game with conditional probability, and the robustness of defect detection.
objective function is defined as Equation (4). Literature [56],
min max V (G, DY , X , Y ) = Ey∼pr (y) [log(DY (y))]
[57], [58] has introduced CGAN to improve its own network G D
and method, which improves the accuracy and robustness of + Ex∼pr (x) [log(1 − DY (G(x)))] (6)
defect detection.
With the continuous improvement and deepening of deep
min max V (D, G) = Ex∼Pr (x) log (D (x|y))
  learning theory, there is still a lot of research space for GAN
G D variants in the future. Table 2 lists the optimization methods,
+ Ez∼P(z) log (1 − D (G (z|y))) (4) advantages and disadvantages of some GAN variants, as well
 
as the suitable application scenarios.
DCGAN [40]. The DCGAN proposed by Radford et al.
provides a convolutional GAN architecture. By using decon- C. DEVELOPMENT OF DEFECT DETECTION BASED ON GAN
volution to replace the pooling layer in generator G and stride With the continuous deepening and improvement of GAN
convolutions to replace the pooling layer in discriminator theory research, various variant network models emerge in
D; And the batch-norm is used in generative model and endlessly. However, the defect detection based on tradi-
discriminant model, which changes the activation function tional deep learning methods is limited by the difficulties
and makes GAN structure more stable. DCGAN verifies that of data collection and expensive labor costs. Inspired by the
discriminators can be used for feature extraction in super- remarkable achievements of GAN-based anomaly detection,
vised learning tasks and generators can be used for semantic researchers gradually apply GAN and its variants to various
vector computation. Literature [59], [60], [61] all introduced defect detection tasks.

VOLUME 10, 2022 113499

 X. He et al.: Survey of Defect Detection Applications Based on Generative Adversarial Networks

FIGURE 8. Examples of GAN variants.

Defects detection based on GAN mainly has the following supervised discriminant learning for fabric defect detection.
three ideas: And Cheng et al. [72] proposed a generative adversarial net-
work with multi-head fusion strategy – IRT-GAN, which was
(1) GAN network can learn feature distribution by repair-
used to automatically detect defects in composite materials
ing defect images, and then determine whether the
using infrared thermal imaging technology. Based on the
sample has defects and locate the location of the defects
above content, there is still a lot of room for progress in
by comparing with the input samples.
defect detection of GAN, both its theoretical research and its
(2) Due to the scarcity of defect samples, GAN is used
application improvement will continue to move forward.
to generate defect samples and enhance the data set,
so that the defect detection task can continue or even
achieve better detection accuracy. IV. OVERVIEW OF DEFECT DETECTION APPLICATIONS
(3) Based on the network structure of GAN, by replacing BASED ON GAN
the generator and discriminator with other networks, Defect detection is a key problem and basic requirement in
the generator directly detects defect samples, takes the computer vision, but it is challenging to detect small and
detection results as the output of the generator, and then complex defects. And the defect detection is now widely used
inputs the output of the generator to the discriminator, in industry, agriculture, construction and road, etc., and is
which improves and enhances it. Finally, the detection playing an increasingly important role in quality detection in
result of the discriminator is regarded as the result of various fields. Today, GAN and their variants are trained to be
defect detection. widely used to detect a wide variety of defects, and as shown
In 2018, Zhao et al. [69] first proposed to apply GAN to in Fig. 10, using these methods achieves better results than
defect detection. The basic detection concept is to establish a previous methods. At present, there are two main ideas for
reconstruction network. If there are defect areas in the sample, GAN based defect detection. Firstly, GAN skillfully uses the
these defect areas can be repaired, and then the input sample is concept of game to train the generator and discriminator to
compared with the recovered sample to indicate the accurate detect the defect region successfully, and it does not need to
defect area. And based on this concept, Donahue et al. [70] know the real distribution of normal data. Second, GAN is
first applied AnoGAN [50] to defect detection of electronic used as a powerful means of data enhancement to expand the
components in 2019. In the training, only normal samples defect dataset. These advantages make GAN widely used in
are used to learn the feature distribution in the latent space defect detection tasks. In the future, GAN will still be one of
unsupervised. The generator outputs the reconstructed image the hot spots in the field of defect detection.
through forward propagation and compares it with the orig- In this section, we review GAN implementations for vari-
inal image to detect the defect region. With the improve- ous defect detection tasks. In particular, it summarizes the key
ment of theoretical work, more and more researchers have issues solved by GAN in these fields, and expounds the poten-
applied GAN and its variants to various defect detection tasks. tial research directions of GAN, as well as the new challenges
Fig. 9 shows the development of GAN based defect detection to be solved in the future. In recent years, deep learning tech-
theory and application. For example, Liu et al. [71] designed nology has become the most successful application in indus-
a GAN-based single-class classifier for steel plate surface trial defect detection. The detector is obtained by training a
defect classification. Liu et al. [7] proposed a GAN-based large number of samples. The detector can perform automatic

113500 VOLUME 10, 2022

X. He et al.: Survey of Defect Detection Applications Based on Generative Adversarial Networks

TABLE 2. List of advantages and disadvantages of some GAN variants.

VOLUME 10, 2022 113501

 X. He et al.: Survey of Defect Detection Applications Based on Generative Adversarial Networks

FIGURE 9. The development of GAN defect detection.

(3) In actual industrial production, how to realize real-

time and efficient detection is a difficult problem to be
solved urgently.
In actual scenarios, it is difficult to collect enough defect
samples, and the defect features extracted from insufficient
and unbalanced data by deep learning methods are not accu-
rate enough and even lead to the failure of the defect detection
task. And the GAN can not only effectively alleviate the
problem of lack of data by generating defective samples,
but also detect defects by comparing the differences between
input samples and reconstructed samples after learning to
characterize normal samples. Therefore, GAN is widely used
in various defect detection tasks. The following summarizes
the application of GAN in defect detection in various fields,
including quality detection of industrial products, textiles,
construction roads and agricultural products.

A. OVERVIEW OF THE APPLICATION OF GAN BASED

DEFECT DETECTION IN INDUSTRIAL PRODUCTS
In the process of industrial production, product defects are
FIGURE 10. Application of defect detection based on GAN in various
unavoidable due to improper manual operation, production
industries. technology problems, storage and transportation problems
and other reasons. And the defect will affect the appearance
detection tasks such as product defects, abnormal operation and performance of the product and bring economic loss to
of equipment, and fault diagnosis. At present, although deep the production enterprise. Strict control of the qualified rate
learning techniques have been successfully applied to many of products can effectively reduce the company’s production
industrial detection tasks and can achieve the most advanced costs. In the visual defect detection of industrial products,
performance, the defect detection task is still challenging due based on the assumption that the potential features of normal
to the need for a large number of training samples and high- samples and defect samples are different, GAN is used to
quality samples. The latest review of defect detection [13], obtain the feature distribution of normal samples in the poten-
[16], [89] points out that the three key issues of the current tial space, and the trained model is used to detect defects.
deep learning defect detection task can be summarized as Generally, normal samples are only used for training GAN,
follows: and the combination of normal samples and defective samples
(1) In the case of few or no actual defects, it is difficult is used for testing and verification.
for the detection model to correctly learn the feature In 2018, Zhao et al. [69] first proposed the combination
representation, which is very difficult for the defect of GAN and autoencoder for defect detection of industrial
detection task based on deep learning methods. products. They proposed a defect detection framework based
(2) There are many types of defects, the model can’t auto- only on normal sample training, reconstructed defect images
matically identify the types of defects, and the labeling by combining GAN and AE, compared input samples with
of new defects requires workers with expert knowledge recovered samples, and accurately detected defect areas.
and a lot of time. In addition, this method does not require defect samples

113502 VOLUME 10, 2022

X. He et al.: Survey of Defect Detection Applications Based on Generative Adversarial Networks

and manual labeling, and has high detection accuracy on in the industrial field and has made the most achievements.
DAGM2007 dataset [90]. Wang et al. [91] used the Otsu However, both in principle and application, there is still a
algorithm to determine the threshold of the residual image large room for progress in the future.
to repair the defect image, and then obtained the defect area
by comparing the difference between the input image and the B. OVERVIEW OF APPLICATION OF TEXTILE DEFECT
repaired image. Experimental results on engine cylinder head DETECTION BASED ON GAN
datasets show that the proposed method is effective in image In the process of manufacturing, transportation, storage and
restoration and defect detection. use, due to improper human operation or machine failure,
Due to the shortage of defect images and high labeling cost textiles are likely to produce a variety of defects (such as
in actual production lines, it is difficult to obtain sufficient scratches, pits, ablations and defects). With the continuous
diversity and quantity of defect data sets. Niu et al. [80] pro- development of deep learning theory and technology, more
posed a new generation method called Surface Defect Gen- and more researchers have applied GAN to textile defect
erative Adversarial Network (SDGAN). The defect images detection, and achieved relatively outstanding results. The
generated by this method have better image quality and diver- following is a review of recent studies on GAN applied to
sity, and show excellent performance in detecting the surface textile defect detection.
defects of the cylinder block of the commutator. Moreover, As early as 2018, Komoto et al. [94] proposed to use GAN
the defect classification trained on SDGAN-enhanced images for textile surface defect detection. They proposed Denoising
is robust to uneven and poor lighting conditions. SDGAN autoencoder Generation Adversarial Network (DAE-GAN).
has two generators. One generator generates defective sam- By introducing adversarial learning framework into DAE,
ples from normal samples, and the other generator restores defective images can be restored to clearer defect-free
defective samples to normal samples. In addition, four dis- images. However, there is no significant improvement in
criminators are used to distinguish real samples from gener- the accuracy of defect detection, so there is a large space
ated samples. Lian et al. [78] combined GAN and CNN to for optimization. In order to improve the problem of low
automatically identify small defects in images. The network detection accuracy and difficult generalization when fabric
uses the changes in the image as regularization terms to gen- texture and defects are complex, Liu et al. [7] proposed
erate defect-free images and corresponding enlarged defect a fabric defect detection framework based on generative
images. This method expands the limited data set of defect adversarial network, which can learn existing defect samples
detection training samples and achieves a defect detection and adapt to different fabric textures. Firstly, a conditional
accuracy of up to 99.2%. GAN is trained to generate a reasonable defect patch, and
In addition to the defect detection examples discussed then it is fused to a specific location. The network can
above, Liu et al. [71] proposed a single-class classification update the existing fabric defect dataset and better detect the
method for strip surface defect detection based on GAN, defects under different conditions. The final average detec-
and proposed a loss function to improve convergence speed tion accuracy reached 94.8%, the recall rate reached 97.6%,
and stability. Single-class classifier (OCC) can detect defects and the F-value reached 96.2%. Hu et al. [77] proposed a
of different sizes, shapes and types only by training normal novel unsupervised automatic fabric defect detection method
samples, and the average detection accuracy of the strip based on deep convolution Generative adversarial network
defect dataset reaches 94%. Zhang et al. [82] proposed a new (DCGAN). In this method, by introducing a new encoder
method based on the idea of GAN, which is a semi-supervised component, the model will restore the defect image to a
generative adversarial network (SSGAN) composed of a dual normal image, and then highlight the potential defect area
attention mechanism segmentation network and a full convo- through the residual error between the original image and
lution discriminator (FCD) network. To obtain more accurate the reconstructed image. Then the defect detection accuracy
segmentation results at pixel level. The segmentation network is improved by fusing the residual map with the likelihood
based on dual attention mechanism can segment defects from map of the original image. Finally, this method can achieve
labeled and unlabeled images, while FCD uses adversarial 93.45% defect detection accuracy. To solve the problem of
and cross-entropy loss functions to generate confidence maps image distortion in textile defect detection, Li et al. [76]
of unlabeled images in a semi-supervised learning manner. proposed SRGAN, a super-resolution image reconstruction
This method can achieve 81.8% defect segmentation accu- technique based on generative adversarial network, which
racy on the Severstal steel plate defect dataset [92] with only can reconstruct the obtained low-pixel image into high-pixel
1/8 marks, and it is robust and flexible in various scenarios. image. The generation network is responsible for generating
In the latest research, GAN has also achieved excellent per- high-resolution images, and the discrimination network is
formance in glass fiber material defect detection [72], steel responsible for identifying the authenticity of images. The
plate surface defect detection [88], and PCB board defect network is continuously optimized through generation loss
detection [86, 93]. Table 3 shows the research application and discrimination loss, and the generation of high-quality
and performance of GAN-based defect detection in the field images is guided. Experimental results show that SRGAN can
of industrial production. Among all kinds of applications obtain clearer images, reconstruct richer textures, more high-
based on GAN defect detection, it is the most widely used frequency details, and identify defects more easily, which

VOLUME 10, 2022 113503

 X. He et al.: Survey of Defect Detection Applications Based on Generative Adversarial Networks

TABLE 3. Research on defect detection application of GAN in industrial production field.

is very important in fabric defect detection. Zhang et al. 99.57%, respectively, with only a small number of samples.
[85] proposed an improved deep convolutional generative The literature [97, 98, 99, 100, 101] all proposed fabric defect
adversarial network, which introduced an autoencoder with detection methods based on GAN, and achieved relatively
an MLP layer in the generator module. By adding MLP layers excellent detection performance. Table 4 shows the research
to extract low-rank fabric image features, the model has a application and performance of GAN based defect detection
stronger ability to capture fabric texture features. Through the in the field of textile quality inspection.
reconstruction of the defect image, the reconstructed image is
compared with the defect image to segment the defect area. C. OVERVIEW OF GAN-BASED CONSTRUCTION ROAD
Compared with previous studies, this method can achieve DEFECT DETECTION APPLICATIONS
better defect segmentation effect. GAN is also gradually applied to defect detection in the
In the latest study, Li et al. [87] proposed a lightweight seg- construction industry, such as concrete surface, highway
mentation system for the detection of weak and small defects pavement, bridge surface, etc. Zhang et al. [11] proposed a
in fabrics. Firstly, the image repair mechanism based on gen- Defect synthesis network Defect-GAN, which can generate
erative adversarial network model is used to repair the defec- real defects in various image backgrounds with different
tive sample images. Then, the difference between the defect textures and appearance by introducing a strategy based on
sample and the fix sample is obtained. Finally, the defect layer composition. It can also simulate the random variation
region is segmented. The experimental results show that of defects and flexibly control the location and category of
the joint intersection of three different datasets is 77.84%, defects in the image background. The experimental results on
77.85% and 73.6%, respectively, and the proposed model is CODEBRIM dataset [102] show that defect-GAN has better
superior to the traditional semantic segmentation model. Wen performance than previous methods in Defect generation.
et al. [95] proposed a new cyclic consistency Adversarial net- The generated data sets are used in defect detection to achieve
work with attention mechanism (ATTECGAN). First, defect higher accuracy. Mei et al. [81] proposed a new road crack
samples were synthesized using ATTECGAN to expand the detection method, Conn-Crack, which combined conditional
sample size. Secondly, by discovering the discriminative part Wasserstein to generate adversarial network and connected
of the samples and enlarging the differences between the graph, used 121 layers of densely connected neural network
samples, the attention mechanism is used to enhance the with deconvolution layer as generator for multi-level feature
feature. ATTECGAN has been tested on KolektorSDD [96] fusion, and used 5-layer fully convolutional network as dis-
and DAGM2007[90] datasets, and its accuracy is 98.53% and criminator. The method was tested on CFD dataset [103] and

113504 VOLUME 10, 2022

X. He et al.: Survey of Defect Detection Applications Based on Generative Adversarial Networks

TABLE 4. Research on the application of GAN in textile defect detection.

EdmCrack600 dataset. The results show that compared with generative adversarial network. This method uses variational
other existing methods, the proposed method achieves the autoencoder (VAE) to encode real crack images. The latent
most advanced performance in terms of accuracy (96.79%), variable values obtained from VAE are provided as input to
recall (87.75%) and F1 score (91.96%). To address the prob- the DCGAN model generator, and the model hyperparame-
lem of data shortage and imbalance in building structural ters are optimized. Then, the Adaptive moment estimation
defects, Shin et al. [104] developed a data augmentation (Adam) optimizer is used to reconstruct the optimization
method using Generative adversarial networks (GAN). In the model, so as to improve the convergence speed and gener-
model experimentally applied with GAN-based data aug- alization ability of the model.
mentation, the average performance is improved by about In the latest study, Xu et al. [108] proposed a method
0.16 compared to the model trained with small datasets. to detect pavement cracks under small samples. Firstly, the
In order to solve the problem that it is difficult for CGAN image generated by GAN model is used to expand the origi-
to detect the shape of objects in the detection of road sur- nal small sample dataset, and convolutional neural network
face defects, Kyslytsyna et al. [105] proposed an improved (CNN) model is constructed at the same time. Then, the
CGAN with attention gate (ICGAN) method to detect road transfer learning method is used to train and test the data
surface defects. ICGAN first removes any information in the sets before and after the extension, respectively to verify the
image other than the road, then identifies flaws and adds validity of the extended data. It is proved that, compared
two attention gates to the U-Net architecture to improve the with the unexpanded dataset, the CNN model trained after
segmentation capability of the generator in Pix2Pix. Exper- the expansion improves the detection accuracy of the test
imental results on Unsupervised Llamas dataset [106] show set from 80.75% to 91.61%. Ali et al. [109] proposed a
that the ICGAN method has better performance than other new sensor technology that can detect road damage using a
state-of-the-art methods. In order to solve the problem of deep learning-based image processing algorithm. This tech-
small sample size in intelligent road detection, Pei et al. nique includes a super-resolution semi-supervised learning
[107] proposed a virtual image set generation method for method based on generative adversarial networks. The for-
asphalt pavement cracks based on improved deep convolution mer improves the quality of the road image and makes

VOLUME 10, 2022 113505

 X. He et al.: Survey of Defect Detection Applications Based on Generative Adversarial Networks

the damaged area clearly visible. The latter uses 5327 road and the patches can be automatically labeled, which greatly
images and 1327 tag images to enhance the detection perfor- improves the performance of YOLOv4 defect detection.
mance. The two methods are applied to four lightweight seg- Table 6 shows the research application and performance of
mentation neural networks. For 400 road images, the average GAN in fruit crop defect detection.
recognition rate reached 81.540% and 79.228%, respectively.
Literature [110], [111], [112], [113], [114] has also made V. ISSUE DISCUSSION AND FUTURE OUTLOOK
improvements and upgrades for GAN used in construction GAN network has made remarkable achievements in the field
defect detection. Table 5 shows the research application and of computer vision due to its powerful generating ability and
performance of GAN-based defect detection in the field of ‘‘coincidence’’ with the era of big data. Although GAN has
construction road quality inspection. been widely used in defect detection, there are still many
challenges, and it still has broad application prospects in the
D. OVERVIEW OF GAN-BASED APPLICATION IN DEFECT future. This section summarizes the challenges and possible
DETECTION OF AGRICULTURAL PRODUCTS future directions of GAN-based defect detection.
In recent years, researchers have also used GAN for surface The old problems [125] in the application of GAN network,
defect detection of agricultural products, so as to control such as mode collapse, gradient disappearance and training
the quality of agricultural products and improve economic instability, have not all been solved. Although various vari-
benefits. In 2019, Tian et al. [117] used the Cyclic Con- ants have emerged, each of these networks basically focuses
sistency Adversarial Network (Cycle-GAN) deep learning on improving only one or two problems, leaving the remain-
model to achieve data enhancement in the detection of Apple ing problems unimproved and possibly even more serious.
anthracnose lesions, and achieved higher detection accu- For example, CGAN [39] added constraints to the original
racy compared with the dataset without data enhancement. GAN model, making GAN generation direction controllable,
Abbas et al. [118] proposed a deep learning-based method for but the training of the model was still unstable. WGAN [46]
tomato disease detection, which uses conditional generation used Wasserstein distance to break through the traditional
adversarial network (CGAN) to generate synthetic images defects of GAN and make the model training more stable, but
of tomato plant leaves. Then, transfer learning was used also prone to the phenomenon of gradient dispersion. LSGAN
to train the DenseNet121 model on both synthetic and real [55] improved the loss function to the least square loss, which
images to classify tomato leaf images into ten categories of improved the quality of the generated samples, but the model
diseases. The proposed model has been extensively trained was still prone to the problems of gradient disappearance and
and tested on the publicly available Plant Village dataset gradient explosion. By sorting out various problems existing
[119]. The method divided tomato leaf images into 5 cat- in GAN itself and summarizing the defect detection methods
egories, 7 categories and 10 categories, with the accuracy based on GAN, future research on generative adversarial
of 99.51%, 98.65% and 97.11%, respectively. The proposed network and its potential breakthrough in the field of defect
method is shown to be superior to the existing methods. detection should mainly focus on the following aspects.
Wang et al. [120] adopted a transformer-based generative
A. GAN ITSELF
adversarial network (GAN) as a data enhancement means,
which can effectively enhance the original training set with 1) THEORETICAL EXPLORATION
more diverse samples to rebalance the three categories. The main purpose of theoretical research is to solve the
Experimental results show that the enhanced data sets get defects of GAN model, but the existing methods are mainly
higher detection accuracy. In order to alleviate the problem to adjust the training parameters and modify the training
of data scarcity, Brid et al. [121] adopted conditional GAN process, and the theoretical exploration of the defects of
to synthesize images to enhance the dataset (Lemons Quality GAN is not deep enough. Therefore, researchers should pay
Control Dataset [122]), and finally achieved 88.75% defect attention to the structural design of the basic algorithm and
classification accuracy. Even if the model is compressed to the design of application-oriented loss function to make the-
half the original size, the conditional GAN enhanced classi- oretical breakthroughs. For example, we can pay attention to
fication network can maintain the classification accuracy of the variation of traditional structure, such as CGAN network,
81.16%. Guo et al. [123] adopted a data expansion method and combine the advantages of existing algorithms to improve
combining deep convolution generative adversarial network the model architecture, and design general and reasonable
and rigid transformation (RT) to improve the data richness of constraints, so as to ensure that under the condition of model
defective dates and effectively solve the imbalance problem stability, we can pay attention to the loss function design with
among different types of date data. The defect detection accu- good quality and diversity of image generation. The theo-
racy after data enhancement is up to 99.2%. Chen et al. [124] retical exploration of GAN is one of the main development
proposed an automatic defect detection method based on directions in the future.
YOLOv4. Cycle-GAN in this method contributes the most to
the model training strategy. The pseudo-defects generated by 2) INTERNAL MECHANISM TRANSPARENCY
Cycle-GAN enrich the types of defects, and the patches can Compared with machine learning, the model complexity of
conform to the texture after pasting to the original position, deep learning increases by orders of magnitude, and the

113506 VOLUME 10, 2022

X. He et al.: Survey of Defect Detection Applications Based on Generative Adversarial Networks

TABLE 5. Research on defect detection application of GAN in construction road industry.

training and computation process is ‘‘hidden’’ and untrace- B. DEFECT DETECTION DIRECTION
able, making it particularly important to study the inner 1) NETWORK MODEL REPLACEMENT
working mechanism of models. Using appropriate tools to In the improvement of GAN model, the use of other network
realize the transparent research of the working mechanism replacement generators and discriminators should be consid-
of the information flow inside the model, we can find the ered in the future. In other words, based on the structure of
problems affecting the stability and training process of the GAN network, the idea of GAN network is still followed, and
model from the root, and then analyze and solve them to break various networks are used as generators and discriminators.
through the performance bottleneck of the model. Using For example, SSGAN [82] replaced the generator of tradi-
appropriate tools to realize the transparent research of the tional GAN with Semantic Segmentation Network (SegNet)
working mechanism of the information flow inside the model, and discriminator with full convolution discriminant network
we can find the problems affecting the stability and training (FCD), which directly avoided the problems existing in GAN
process of the model from the root, and then analyze and model and achieved good performance in strip defect detec-
solve the problems, so as to break through the performance tion. This is also a method that can directly avoid the prob-
bottleneck of the model. In particular, it is urgent to solve lems of pattern collapse, gradient disappearance and training
the representation problem of how GAN model generates instability in GAN, which is worth further study in the future.
images and the visualization problem of global convergence
of generator and discriminator. In addition, the controllability
problem of the generated network has not been completely 2) INTRODUCING ATTENTION MECHANISM
solved, and only the experimental effect of specific scenes has Attention mechanisms [126] are derived from the study
been achieved, but the universality of different scenes of the of human vision. In cognitive science, due to information
control effect has not been achieved. Transparency research processing bottlenecks, humans selectively focus on a por-
on the internal mechanism of GAN is also an important tion of all information while ignoring other visible informa-
development direction. tion. Attention mechanism in neural networks is a resource

VOLUME 10, 2022 113507

 X. He et al.: Survey of Defect Detection Applications Based on Generative Adversarial Networks

TABLE 6. Application of GAN in defect detection of agricultural products.

allocation scheme to allocate computing resources to more This paper also introduces the research progress and appli-
important tasks and solve the problem of information over- cation status of GAN defect detection methods in various
load when computing capacity is limited. In neural network fields in detail. In view of the outstanding problems in the
learning, generally speaking, the more parameters of the development and application of GAN and GAN-based defect
model, the stronger the expression ability of the model, and detection technology, the possible research directions and
the more information stored in the model, but this will lead to improvement ideas in the future are put forward. We hope
the problem of information overload. By introducing atten- that this review will be helpful to researchers working on
tion mechanism, the problem of information overload can GAN, especially those who are interested in using GAN for
be solved and the efficiency and accuracy of task processing defect detection tasks. We believe that with the innovation of
can be improved by focusing on the information that is more various theories and the iterative development of technology,
critical to the current task, reducing the attention paid to the defect detection technology based on GAN will enter a
other information, and even filtering out irrelevant informa- new era of development.
tion. By introducing attention mechanism into GAN network,
especially in the generator part, the model pays more attention REFERENCES
to the defect feature part, which may improve the feature [1] P. Runeson, C. Andersson, T. Thelin, A. Andrews, and T. Berling, ‘‘What
learning and generation ability of generator. Therefore, the do we know about defect detection methods? [software testing],’’ IEEE
Softw., vol. 23, no. 3, pp. 82–90, May 2006, doi: 10.1109/MS.2006.89.
introduction of attention mechanism in GAN is one of the [2] L. D. Luo, X. Y. Cai, H. Z. Yang, Y. Z. Zhang, Y. Zhou, and
directions that can be further studied in the future. X. Bai, ‘‘Survey on industrial defect detection with deep learning,’’
Scientia Sinica Informationis, vol. 52, pp. 1002–10039, Jun. 2022,
doi: 10.1360/SSI-2021-0336.
VI. CONCLUSION [3] J.-X. Gu, Z.-H. Wang, J. Kuen, L.-Y. Ma, A. Shahroudy, B. Shuai, T. Liu,
X.-X. Wang, G. Wang, J.-F. Cai, and T.-H. Chen, ‘‘Recent advances in
This paper reviews the research progress, development his- convolutional neural networks,’’ Pattern Recognit., vol. 77, pp. 354–377,
tory and application status of GAN based defect detection. Feb. 2018, doi: 10.1016/j.patcog.2017.10.013.
Through the thinking and definition of defect and defect [4] D. Ye, G. S. Hong, Y. Zhang, K. Zhu, and J. Y. H. Fuh, ‘‘Defect detection
in selective laser melting technology by acoustic signals with deep belief
detection, this paper discusses the implementation scheme networks,’’ Int. J. Adv. Manuf. Technol., vol. 96, nos. 5–8, pp. 2791–2801,
of current defect detection system, summarizes the princi- May 2018, doi: 10.1007/s00170-018-1728-0.
ple of GAN, and briefly introduces and compares various [5] Q. Xu, Q. Zhao, G. Yu, L. Wang, and T. Shen, ‘‘Rail defect
detection method based on recurrent neural network,’’ in Proc.
variants of GAN, and expounds the theoretical development 39th Chin. Control Conf. (CCC), Jul. 2020, pp. 6486–6490, doi:
and application status of defect detection based on GAN. 10.23919/CCC50068.2020.9188823.

113508 VOLUME 10, 2022

X. He et al.: Survey of Defect Detection Applications Based on Generative Adversarial Networks

[6] W. Liu, Z. Liu, H. Wang, and Z. Han, ‘‘An automated defect detection [28] X. Xie and M. Mirmehdi, ‘‘TEXEMS: Texture exemplars for defect
approach for catenary rod-insulator textured surfaces using unsupervised detection on random textured surfaces,’’ IEEE Trans. Pattern Anal.
learning,’’ IEEE Trans. Instrum. Meas., vol. 69, no. 10, pp. 8411–8423, Mach. Intell., vol. 29, no. 8, pp. 1454–1464, Aug. 2007, doi:
Oct. 2020, doi: 10.1109/TIM.2020.2987503. 10.1109/TPAMI.2007.1038.
[7] J. Liu, C. Wang, H. Su, B. Du, and D. Tao, ‘‘Multistage GAN for fabric [29] X. Xie, ‘‘A review of recent advances in surface defect detec-
defect detection,’’ IEEE Trans. Image Process., vol. 29, pp. 3388–3400, tion using texture analysis techniques,’’ Electron. Lett. Comput. Vis.
2020, doi: 10.1109/TIP.2019.2959741. Image Anal., vol. 7, no. 3, pp. 1–22, 2008. [Online]. Available:
[8] K. Oksuz, B. C. Cam, S. Kalkan, and E. Akbas, ‘‘Imbalance prob- https://raco.cat/index.php/ELCVIA/article/view/150223.
lems in object detection: A review,’’ IEEE Trans. Pattern Anal. [30] X. Chen, J. Chen, X. Han, C. Zhao, D. Zhang, K. Zhu, and
Mach. Intell., vol. 43, no. 10, pp. 3388–3415, Oct. 2021, doi: Y. Su, ‘‘A light-weighted CNN model for wafer structural defect
10.1109/TPAMI.2020.2981890. detection,’’ IEEE Access, vol. 8, pp. 24006–24018, 2020, doi:
[9] X. Xia, X. Pan, N. Li, X. He, L. Ma, X. Zhang, and N. Ding, ‘‘GAN-based
10.1109/ACCESS.2020.2970461.
anomaly detection: A review,’’ Neurocomputing, vol. 493, pp. 497–535,
Jul. 2022, doi: 10.1016/j.neucom.2021.12.093. [31] C. Hua, S. Chen, G. Xu, and Y. Chen, ‘‘Defect detection method
[10] D. P. Kingma and M. Welling, ‘‘Auto-encoding variational Bayes,’’ in of carbon fiber sucker rod based on multi-sensor information fusion
Proc. Int. Conf. Learn. Represent., Dec. 2013, pp. 1–14. and DBN model,’’ Sensors, vol. 22, no. 14, p. 5189, Jul. 2022, doi:
[11] G. Zhang, K. Cui, T.-Y. Hung, and S. Lu, ‘‘Defect-GAN: high-fidelity 10.3390/s22145189.
defect synthesis for automated defect inspection,’’ in Proc. IEEE Winter [32] X. Zhang, Y. Cui, Y. Wang, M. Sun, and H. Hu, ‘‘An improved AE detec-
Conf. Appl. Comput. Vis. (WACV), Jan. 2021, pp. 2524–2534. tion method of rail defect based on multi-level ANC with VSS-LMS,’’
[12] K. Liu, Y. Li, J. Yang, Y. Liu, and Y. Yao, ‘‘Generative principal com- Mech. Syst. Signal Process., vol. 99, pp. 420–433, Jan. 2018, doi:
ponent thermography for enhanced defect detection and analysis,’’ IEEE 10.1016/j.ymssp.2017.06.029.
Trans. Instrum. Meas., vol. 69, no. 10, pp. 8261–8269, Oct. 2020, doi: [33] J. Božič, D. Tabernik, and D. Skočaj, ‘‘Mixed supervision for
10.1109/TIM.2020.2992873. surface-defect detection: From weakly to fully supervised learn-
[13] A.-A. Tulbure, A.-A. Tulbure, and E.-H. Dulf, ‘‘A review on mod- ing,’’ Comput. Ind., vol. 129, Aug. 2021, Art. no. 103459, doi:
ern defect detection models using DCNNs—Deep convolutional neu- 10.1016/j.compind.2021.103459.
ral networks,’’ J. Adv. Res., vol. 35, pp. 33–48, Jan. 2022, doi: [34] S. Mei, H. Yang, and Z. Yin, ‘‘An unsupervised-learning-based
10.1016/j.jare.2021.03.015. approach for automated defect inspection on textured surfaces,’’ IEEE
[14] Q. Luo, X. Fang, L. Liu, C. Yang, and Y. Sun, ‘‘Automated visual defect Trans. Instrum. Meas., vol. 67, no. 8, pp. 1266–1277, Jun. 2018, doi:
detection for flat steel surface: A survey,’’ IEEE Trans. Instrum. Meas., 10.1109/TIM.2018.2795178.
vol. 69, no. 3, pp. 626–644, Mar. 2020, doi: 10.1109/TIM.2019.2963555.
[35] T. Czimmermann, G. Ciuti, M. Milazzo, M. Chiurazzi, S. Roccella,
[15] D. Martin, S. Heinzel, J. K. von Bischhoffshausen, and N. Kühl, ‘‘Deep
C. M. Oddo, and P. Dario, ‘‘Visual-based defect detection and classifica-
learning strategies for industrial surface defect detection systems,’’ in
tion approaches for industrial applications—A Survey,’’ Sensors, vol. 20,
Proc. 55th Annu. Hawaii Int. Conf. Syst. Sci. (HICSS), 2021, pp. 1–11.
[16] Q. Li, Z. Luo, H. Chen, and C. Li, ‘‘An overview of deeply optimized no. 5, p. 1459, Mar. 2020, doi: 10.3390/s20051459.
convolutional neural networks and research in surface defect classifica- [36] J. Iqbal, S. M. Pasha, K. Baizid, A. A. Khan, and J. Iqbal, ‘‘Computer
tion of workpieces,’’ IEEE Access, vol. 10, pp. 26443–26462, 2022, doi: vision inspired real-time autonomous moving target detection, tracking
10.1109/ACCESS.2022.3157293. and locking,’’ Life Sci. J., vol. 10, no. 4, pp. 3338–3345, 2013.
[17] H. Su, B. J. Zhang, H. B. Zhang, and W. Zou, ‘‘Review of [37] A. Garcia-Garcia, S. Orts-Escolano, S. Oprea, S. Oprea,
research on the inspection of surface defect based on visual percep- V. Villena-Martinez, and J. Garcia-Rodriguez, ‘‘A review on deep
tion,’’ Comput. Integr. Manuf. Syst., Apr. 2021, [Online]. Available: learning techniques applied to semantic segmentation,’’ in Proc. CVPR,
https://kns.cnki.net/kcms/detail/11.5946.TP.20210428.1807.024.html Apr. 2017, pp. 1–23.
[18] J. L. Raheja, S. Kumar, and A. Chaudhary, ‘‘Fabric defect detection based [38] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley,
on GLCM and Gabor filter: A comparison,’’ Optik, vol. 124, no. 23, S. Ozair, A. Courville, and Y. Bengio, ‘‘Generative adversarial nets,’’ in
pp. 6469–6474, Dec. 2013, doi: 10.1016/j.ijleo.2013.05.004. Proc. Adv. Neural Inf. Process. Syst. (NIPS), 2014, pp. 1–47.
[19] F. Arnia and K. Munadi, ‘‘Real time textile defect detection using [39] M. Mirza and S. Osindero, ‘‘Conditional generative adversarial nets,’’
GLCM in DCT-based compressed images,’’ in Proc. 6th Int. Conf. Comput. Sci., vol. 2, pp. 2672–2680, Nov. 2014.
Modeling, Simul., Appl. Optim. (ICMSAO), May 2015, pp. 1–6,
[40] A. Radford, L. Metz, and S. Chintala, ‘‘Unsupervised represen-
doi: 10.1109/ICMSAO.2015.7152203.
tation learning with deep convolutional generative adversarial net-
[20] X. Liu, F. Xue, and L. Teng, ‘‘Surface defect detection based on gradi-
works,’’ in Proc. Int. Conf. Learn. Represent. (ICLR), 2016, doi:
ent LBP,’’ in Proc. IEEE 3rd Int. Conf. Image, Vis. Comput. (ICIVC),
10.48550/arXiv.1511.06434.
Jun. 2018, pp. 133–137, doi: 10.1109/ICIVC.2018.8492798.
[21] O. Kaynar, Y. E. Isik, Y. Gormez, and F. Demirkoparan, ‘‘Fabric defect [41] M. Y. Liu and O. Tuzel, ‘‘Coupled generative adversarial networks,’’ in
detection with LBP-GLMC,’’ in Proc. Int. Artif. Intell. Data Process. Proc. Adv. Neural Inf. Process. Syst. (NIPS), 2016, pp. 1–9.
Symp. (IDAP), Sep. 2017, pp. 1–5, doi: 10.1109/IDAP.2017.8090188. [42] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel,
[22] A. Kumar and G. K. H. Pang, ‘‘Defect detection in textured ‘‘InfoGAN: Interpretable representation learning by information maxi-
materials using optimized filters,’’ in IEEE Trans. Syst., Man, mizing generative adversarial nets,’’ in Proc. Adv. Neural Inf. Process.
Cybern., B, Cybern., vol. 32, no. 5, pp. 553–570, Oct. 2002, doi: Syst. (NIPS), 2016, pp. 1–9.
10.1109/TSMCB.2002.1033176. [43] A. Odena, ‘‘Semi-supervised learning with generative adversarial
[23] A. Kumar, ‘‘Inspection of surface defects using optimal FIR filters,’’ networks,’’ in Proc. Int. Conf. Mach. Learn. (ICML), 2016, doi:
in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 10.48550/arXiv.1606.01583.
Apr. 2003, p. 241, doi: 10.1109/ICASSP.2003.1202339. [44] E. Dention, S. Gross, and R. Fergus, ‘‘Semi-supervised learning with
[24] D.-M. Tsai and C.-K. Huang, ‘‘Defect detection in electronic surfaces context-conditional generative adversarial networks,’’ in Proc. Int. Conf.
using template-based Fourier image reconstruction,’’ IEEE Trans. Com- Learn. Represent. (ICLR), 2017, doi: 10.48550/arXiv.1611.06430.
pon., Packag., Manuf. Technol., vol. 9, no. 1, pp. 163–172, Jan. 2019, doi:
[45] I. O. Tolstikhin, S. Gelly, O. Bousquet, C.-J. S.-Gabriel, and B. Schölkopf,
10.1109/TCPMT.2018.2873744.
‘‘AdaGAN: Boosting Generative Models,’’ in Proc. Adv. Neural Inf.
[25] B. Zorić, T. Matić, and Ž. Hocenski, ‘‘Classification of biscuit tiles for
Process. Syst. (NIPS), 2017, pp. 1–10.
defect detection using Fourier transform features,’’ ISA Trans., vol. 125,
pp. 400–414, Jun. 2022, doi: 10.1016/j.isatra.2021.06.025. [46] M. Arjovsky, S. Chintala, and L. Bottou, ‘‘Wasserstein generative
[26] X. Wang and M. Mirmehdi, ‘‘Archive film defect detection and adversarial networks,’’ in Proc. 34th Int. Conf. Mach. Learn., vol. 70,
removal: An automatic restoration framework,’’ IEEE Trans. Aug. 2017, pp. 214–223.
Image Process., vol. 21, no. 8, pp. 3757–3769, Aug. 2012, doi: [47] A. Odena, C. Olah, and J. Shlens, ‘‘Conditional image synthesis with aux-
10.1109/TIP.2012.2194505. iliary classifier GANs,’’ in Proc. 34th Int. Conf. Mach. Learn. (PMLR),
[27] H. Zhang, X. Jin, Q. M. J. Wu, Y. Wang, Z. He, and Y. Yang, ‘‘Automatic vol. 70, 2017, pp. 2642–2651.
visual detection system of railway surface defects with curvature filter and [48] R. D. Hjelm, A. P. Jacob, T. Che, A. Trichler, K. Cho, and Y. Bengio,
improved Gaussian mixture model,’’ IEEE Trans. Instrum. Meas., vol. 67, ‘‘Boundary-seeking generative adversarial networks,’’ in Proc. Int. Conf.
no. 7, pp. 1593–1608, Jul. 2018, doi: 10.1109/TIM.2018.2803830. Learn. Represent. (ICLR), 2018, doi: 10.48550/arXiv.1702.08431.

VOLUME 10, 2022 113509

 X. He et al.: Survey of Defect Detection Applications Based on Generative Adversarial Networks

[49] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, ‘‘Unpaired image-to-image [68] A. Khan, C.-H. Lee, P. Huang, and B. Clark, ‘‘Using cycle-GANS to
translation using cycle-consistent adversarial networks,’’ in Proc. IEEE generate realistic STEM images for defect identification,’’ Bull. Amer.
Int. Conf. Comput. Vis. (ICCV), Oct. 2017, pp. 2223–2232. Phys. Soc., vol. 67, Mar. 2022.
[50] T. Schlegl, P. Seeböck, S. M. Waldstein, U. Schmidt-Erfurth, and [69] Z. Zhao, B. Li, R. Dong, and P. Zhao, ‘‘A surface defect detection method
G. Langs, ‘‘Unsupervised anomaly detection with generative adver- based on positive samples,’’ in Trends in Artificial Intelligence (Lecture
sarial networks to guide marker discovery,’’ in Proc. Int. Conf. Notes in Computer Science), vol. 11013. Berlin, Germany: Springer,
Inf. Process. Med. Imag., vol. 10265, May 2017, pp. 146–157, Jul. 2018, pp. 473–481, doi: 10.1007/978-3-319-97310-4_54.
doi: 10.1007/978-3-319-59050-9_12. [70] E. A. Donahue, T.-T. Quach, K. Potter, C. Martinez, M. Smith, and
[51] D. Berthelot, T. Schumn, and L. Metz, ‘‘BEGAN: Boundary equilibrium C. D. Turner, ‘‘Deep learning for automated defect detection in
generative adversarial networks,’’ May 2017, arXiv:1703.10717, doi: high-reliability electronic parts,’’ Proc. SPIE, vol. 11139, pp. 30–40,
10.48550/arXiv.1703.10717. Sep. 2019, doi: 10.1117/12.2529584.
[52] J. Donahue, P. Krähenbühl, and T. Darrell, ‘‘Adversarial feature learning,’’ [71] K. Liu, A. Li, X. Wen, H. Chen, and P. Yang, ‘‘Steel sur-
in Proc. Int. Conf. Learn. Represent., Apr. 2017, pp. 1–18. face defect detection using GAN and one-class classifier,’’ in Proc.
25th Int. Conf. Autom. Comput. (ICAC), Sep. 2019, pp. 1–6, doi:
[53] Z. Yi, H. Zhang, P. Tan, and M. Gong, ‘‘DualGAN: Unsupervised dual
10.23919/IConAC.2019.8895110.
learning for image-to-image translation,’’ in Proc. Comput. Vis. Pattern
[72] L. Cheng, Z. Tong, S. Xie, and M. Kersemans, ‘‘IRT-GAN: A gen-
Recognit. (CVPR), Apr. 2017, pp. 2849–2857.
erative adversarial network with a multi-headed fusion strategy for
[54] H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. N. Metaxas, automated defect detection in composites using infrared thermogra-
‘‘StackGAN: Text to photo-realistic image synthesis with stacked genera- phy,’’ Composite Struct., vol. 290, Jun. 2022, Art. no. 115543, doi:
tive adversarial networks,’’ in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), 10.1016/j.compstruct.2022.115543.
Aug. 2017, pp. 5907–5915.
[73] J. Zhao, M. Mathieu, and Y. L. Cun, ‘‘Energy-based generative adver-
[55] X. Mao, Q. Li, H. Xie, R. YK Lau, Z. Wang, and S. P. Smolley, ‘‘Least sarial network,’’ in Proc. Int. Conf. Learn. Represent. (ICLR), 2017, doi:
squares generative adversarial networks,’’ in Proc. IEEE Int. Conf. Com- 10.48550/arXiv.1609.03126.
put. Vis. (ICCV), Apr. 2017, pp. 2794–2802. [74] H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena, ‘‘Self-attention
[56] R. Guo, H. Liu, G. Xie, and Y. Zhang, ‘‘Weld defect detection generative adversarial networks,’’ in Proc. 36th Int. Conf. Mach. Learn.,
from imbalanced radiographic images based on contrast enhance- vol. 97, May 2018, pp. 7354–7363.
ment conditional generative adversarial network and transfer learning,’’ [75] L. Haloui, J. S. Gupta, and V. Feuillard, ‘‘Anomaly detection
IEEE Sensors J., vol. 21, no. 9, pp. 10844–10853, May 2021, doi: with Wasserstein GAN,’’ Dec. 2018, arXiv:1812.02463, doi:
10.1109/JSEN.2021.3059860. 10.48550/arXiv.1812.02463.
[57] Z. Gao, G. Yang, E. Li, T. Shen, Z. Wang, Y. Tian, H. Wang, and Z. Liang, [76] J. Li, L. Wu, S. Wang, W. Wu, F. Song, and G. Zheng, ‘‘Super resolution
‘‘Insulator segmentation for power line inspection based on modified con- image reconstruction of textile based on SRGAN,’’ in Proc. IEEE Int.
ditional generative adversarial network,’’ J. Sensors, vol. 2019, pp. 1–8, Conf. Smart Internet Things (SmartIoT), Aug. 2019, pp. 436–439, doi:
Nov. 2019, doi: 10.1155/2019/4245329. 10.1109/SmartIoT.2019.00078.
[58] Y.-W. Lu, K.-L. Liu, and C.-Y. Hsu, ‘‘Conditional generative adver- [77] G. Hu, J. Huang, Q. Wang, J. Li, Z. Xu, and X. Huang, ‘‘Unsupervised
sarial network for defect classification with class imbalance,’’ in Proc. fabric defect detection based on a deep convolutional generative adver-
IEEE Int. Conf. Smart Manuf., Ind. Logistics Eng. (SMILE), Apr. 2019, sarial network,’’ Textile Res. J., vol. 90, nos. 3–4, pp. 247–270, Jul. 2019,
pp. 146–149, doi: 10.1109/SMILE45626.2019.8965320. doi: 10.1177/0040517519862880.
[59] J. Viola, Y. Chen, and J. Wang, ‘‘FaultFace: Deep convolutional [78] J. Lian, W. Jia, M. Zareapoor, Y. Zheng, R. Luo, D. K. Jain, and
generative adversarial network (DCGAN) based ball-bearing failure N. Kumar, ‘‘Deep-learning-based small surface defect detection via an
detection method,’’ Inf. Sci., vol. 542, pp. 195–211, Jan. 2021, doi: exaggerated local variation-based generative adversarial network,’’ IEEE
10.1016/j.ins.2020.06.060. Trans. Ind. Informat., vol. 16, no. 2, pp. 1343–1351, Feb. 2020, doi:
[60] H. Gao, Y. Zhang, W. Lv, J. Yin, T. Qasim, and D. Wang, ‘‘A deep 10.1109/TII.2019.2945403.
convolutional generative adversarial networks-based method for defect [79] Z. Mi, X. Jiang, T. Sun, and K. Xu, ‘‘GAN-generated image detection with
detection in small sample industrial parts images,’’ Appl. Sci., vol. 12, self-attention mechanism against GAN generator defect,’’ IEEE J. Sel.
no. 13, p. 6569, Jun. 2022, doi: 10.3390/app12136569. Topics Signal Process., vol. 14, no. 5, pp. 969–981, Aug. 2020, doi:
[61] Z. Geng, C. Shi, and Y. Han, ‘‘Intelligent small sample defect detection 10.1109/JSTSP.2020.2994523.
of water walls in power plants using novel deep learning integrating [80] S. Niu, B. Li, X. Wang, and H. Lin, ‘‘Defect image sample gen-
deep convolutional GAN,’’ IEEE Trans. Ind. Informat., early access, eration with GAN for improving defect recognition,’’ IEEE Trans.
Mar. 16, 2022, doi: 10.1109/TII.2022.3159817. Autom. Sci. Eng., vol. 17, no. 3, pp. 1611–1622, Jul. 2020, doi:
10.1109/TASE.2020.2967415.
[62] Y. LeCun, Y. Bengio, and G. Hinton, ‘‘Deep learning,’’ Nature, vol. 521,
[81] Q. Mei and M. Gül, ‘‘A cost effective solution for pavement
pp. 436–444, May 2015, doi: 10.1038/nature14539.
crack inspection using cameras and deep neural networks,’’ Con-
[63] M. Jalayer, R. Jalayer, A. Kaboli, C. Orsenigo, and C. Vercellis, ‘‘Auto- struct. Building Mater., vol. 256, Sep. 2020, Art. no. 119397, doi:
matic visual inspection of rare defects: A framework based on GP- 10.1016/j.conbuildmat.2020.119397.
WGAN and enhanced faster R-CNN,’’ in Proc. IEEE Int. Conf. Ind. 4.0,
[82] G. Zhang, Y. Pan, and L. Zhang, ‘‘Semi-supervised learning with GAN for
Artif. Intell., Commun. Technol. (IAICT), Jul. 2021, pp. 221–227, doi:
automatic defect detection from images,’’ Autom. Construction, vol. 128,
10.1109/IAICT52856.2021.9532584.
Aug. 2021, Art. no. 103764, doi: 10.1016/j.autcon.2021.103764.
[64] Y. Zhang, L. Lu, Y. Wang, Y. Ding, J. Yang, Z. Xu, B. Ma, and X. Lin, [83] S. Kong and D. Ramanan, ‘‘OpenGAN: Open-set recognition via open
‘‘An improved defect detection method of water walls using the WGAN,’’ data generation,’’ in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV),
J. Phys., Conf., vol. 1626, no. 1, Oct. 2020, Art. no. 012152. Oct. 2021, pp. 813–822.
[65] X. Le, J. Mei, H. Zhang, B. Zhou, and J. Xi, ‘‘A learning- [84] Y. Chen, Q. Gao, and X. Wang, ‘‘Inferential Wasserstein gen-
based approach for surface defect detection using small image erative adversarial networks,’’ Sep. 2021, arXiv:2109.05652, doi:
datasets,’’ Neurocomputing, vol. 408, pp. 112–120, Sep. 2020, doi: 10.48550/arXiv.2109.05652.
10.1016/j.neucom.2019.09.107. [85] Z. Zhang, X. Wan, L. Li, and J. Wang, ‘‘An improved DCGAN
[66] J. Liu, Z. Ma, Y. Qiu, X. Ni, B. Shi, and H. Liu, ‘‘Four discrimina- for fabric defect detection,’’ in Proc. IEEE 4th Int. Conf.
tor cycle-consistent adversarial network for improving railway defective Electron. Commun. Eng. (ICECE), Dec. 2021, pp. 72–76, doi:
fastener inspection,’’ IEEE Trans. Intell. Transp. Syst., vol. 23, no. 8, 10.1109/ICECE54449.2021.9674302.
pp. 10636–10645, Aug. 2022, doi: 10.1109/TITS.2021.3095167. [86] S. You, ‘‘PCB defect detection based on generative adversarial network,’’
[67] C.-F. Juang and W.-S. Chen, ‘‘Inspection of lead frame defects in Proc. 2nd Int. Conf. Consum. Electron. Comput. Eng. (ICCECE),
using deep CNN and cycle-consistent GAN-based defect augmenta- Jan. 2022, pp. 557–560, doi: 10.1109/ICCECE54139.2022.9712737.
tion,’’ in Generative Adversarial Learning: Architectures and Applica- [87] B. Li, Y. Zou, R. Zhu, W. Yao, J. Wang, and S. Wan, ‘‘Fabric defect
tions (Intelligent Systems Reference Library), vol. 217, R. Razavi-Far, segmentation system based on a lightweight GAN for industrial Internet
A. Ruiz-Garcia, V. Palade, and J. Schmidhuber, Eds. Berlin, Germany: of Things,’’ Wireless Commun. Mobile Comput., vol. 2022, pp. 1–17,
Springer, Feb. 2022, pp. 185–203, doi: 10.1007/978-3-030-91390-8_9. May 2022, doi: 10.1155/2022/9680519.

113510 VOLUME 10, 2022

X. He et al.: Survey of Defect Detection Applications Based on Generative Adversarial Networks

[88] H. Zhang, D. Pan, J. Liu, and Z. Jiang, ‘‘A novel MAS-GAN-based data [108] B. Xu and C. Liu, ‘‘Pavement crack detection algorithm based on
synthesis method for object surface defect detection,’’ Neurocomputing, generative adversarial network and convolutional neural network under
vol. 499, pp. 106–114, Aug. 2022, doi: 10.1016/j.neucom.2022.05.021. small samples,’’ Measurement, vol. 196, Jun. 2022, Art. no. 111219, doi:
[89] Z. Zeng, B. Liu, J. Fu, and H. Chao, ‘‘Reference-based defect detection 10.1016/j.measurement.2022.111219.
network,’’ IEEE Trans. Image Process., vol. 30, pp. 6637–6647, 2021, [109] R. Ali and Y.-J. Cha, ‘‘Attention-based generative adversarial
doi: 10.1109/TIP.2021.3096067. network with internal damage segmentation using thermography,’’
[90] M. Wieler and T. Hahn, ‘‘Weakly supervised learning for industrial optical Autom. Construction, vol. 141, Sep. 2022, Art. no. 104412, doi:
inspection,’’ in Proc. DAGM Symp., 2007, pp. 1–3. [Online]. Available: 10.1016/j.autcon.2022.104412.
https://hci.iwr.uni-heidelberg.de/node/3616 [110] S. Shim, J. Kim, S.-W. Lee, and G.-C. Cho, ‘‘Road damage detec-
[91] Q. Wang, R. Yang, C. Wu, and Y. Liu, ‘‘An effective defect detec- tion using super-resolution and semi-supervised learning with gener-
tion method based on improved generative adversarial networks (iGAN) ative adversarial network,’’ Autom. Construct., vol. 135, Mar. 2022,
for machined surfaces,’’ J. Manuf. Processes, vol. 65, pp. 373–381, Art. no. 104139, doi: 10.1016/j.autcon.2022.104139.
May 2021, doi: 10.1016/j.jmapro.2021.03.053. [111] X. Li, S. Yang, and H. Liu, ‘‘An unsupervised concrete crack
[92] Severstal. Accessed: Jul. 10, 2022. [Online]. Available: https://www. detection method based on nnU-Net,’’ in Proc. Int. Conf. Image,
kaggle.com/c/severstal-steel-defect-detection.which Vis. Intell. Syst. (ICIVIS), vol. 813, Mar. 2022, pp. 609–623,
[93] L. Zhu, D. Baolin, Z. Xiaomeng, F. Shaoliang, C. Zhen, Z. Junjie, and C. doi: 10.1007/978-981-16-6963-7_55.
Shumin, ‘‘Surface defect detection method based on improved semisuper- [112] F. Ni, Z. He, S. Jiang, W. Wang, and J. Zhang, ‘‘A generative adver-
vised multitask generative adversarial network,’’ Sci. Program., vol. 2022, sarial learning strategy for enhanced lightweight crack delineation net-
pp. 1–17, Jan. 2022, doi: 10.1155/2022/4481495. works,’’ Adv. Eng. Informat., vol. 52, Apr. 2022, Art. no. 101575, doi:
[94] K. Komoto, S. Nakatsuka, H. Aizawa, K. Kato, H. Kobayashi, and 10.1016/j.aei.2022.101575.
K. Banno, ‘‘A performance evaluation of defect detection by using [113] G. Chen, S. Teng, M. Lin, X. Yang, and X. Sun, ‘‘Crack detec-
denoising autoencoder generative adversarial networks,’’ in Proc. Int. tion based on generative adversarial networks and deep learning,’’
Workshop Adv. Image Technol. (IWAIT), Jan. 2018, pp. 1–4, doi: KSCE J. Civil Eng., vol. 26, no. 4, pp. 1803–1816, Jan. 2022,
10.1109/IWAIT.2018.8369766. doi: 10.1007/s12205-022-0518-2.
[95] L. Wen, Y. Wang, and X. Li, ‘‘A new cycle-consistent adversarial [114] C. Liu and B. Xu, ‘‘A night pavement crack detection method
networks with attention mechanism for surface defect classification based on image-to-image translation,’’ Comput.-Aided Civil Infras-
with small samples,’’ IEEE Trans. Ind. Informat., vol. 18, no. 12, truct. Eng., vol. 2022, pp. 1–17, May 2022, doi: 10.1111/mice.
pp. 8988–8998, Dec. 2022, doi: 10.1109/TII.2022.3168432. 12849.
[96] KolektorSDD. Accessed: Aug. 3, 2022. [Online]. Available: https://www. [115] A. Krizhevsky and G. Hinton. (Apr. 2009). Learning Multiple Lay-
vicos.si/resources/kolektorsdd/ ers of Features From Tiny Images. Department of Computer Sci-
[97] J. Silvestre-Blanes, T. Albero-Albero, I. Miralles, R. Pérez-Llorens, and ence. [Online]. Available: https://link.zhihu.com/?target=https%3A//w
J. Moreno, ‘‘A public fabric database for defect detection methods and ww.cs.toronto.edu/ kriz/cifar.html
results,’’ Autex Res. J., vol. 19, no. 4, pp. 363–374, Dec. 2019, doi: [116] M. Maguire, S. Dorafshan, and R. J. Thomas. (May 2018). SDNET2018:
10.2478/aut-2019-0035. A Concrete Crack Image Dataset for Machine Learning Applications.
[98] Y. Guo, L. Zhong, Y. Qiu, H. Wang, F. Gao, Z. Wen, and C. Zhan, Utah State University. [Online]. Available: https://orcid.org/0000-0002-
‘‘Using ISU-GAN for unsupervised small sample defect detection,’’ 7897-034
Sci. Rep., vol. 12, no. 1, pp. 1–13, Jul. 2022, doi: 10.1038/s41598-022- [117] Y. Tian, G. Yang, Z. Wang, E. Li, and Z. Liang, ‘‘Detection of apple
15855-7. lesions in orchards based on deep learning methods of CycleGAN
[99] Z. Liu, C. Gao, C. Li, N. Huang, Z, Guo, ‘‘Unsupervised fabric defect and YOLOV3-dense,’’ J. Sensors, vol. 2019, pp. 1–13, Apr. 2019, doi:
detection based on DCGAN with component-encoder,’’ in Pattern Recog- 10.1155/2019/7630926.
nition (Lecture Notes in Computer Science), vol. 13188, C. Wallraven, [118] A. Abbas, S. Jain, M. Gour, and S. Vankudothu, ‘‘Tomato plant dis-
Q. Liu, and H. Nagahara, Eds. Berlin, Germany: Springer, May 2022, ease detection using transfer learning with C-GAN synthetic images,’’
pp. 557–568, doi: 10.1007/978-3-031-02375-0_41. Comput. Electron. Agricult., vol. 187, Aug. 2021, Art. no. 106279, doi:
[100] W. Shi, W. Wang, L. Zhu, K. Wu, and J. Wu, ‘‘Clustering-based cycle 10.1016/j.compag.2021.106279.
GAN for fabric defect detection,’’ Nucl. Phys. B, Mar. 2022, doi: [119] D. Hughes and M. Salathé, ‘‘An open access repository of images on plant
10.2139/ssrn.4061500. health to enable the development of mobile disease diagnostics,’’ Comput.
[101] W. Li, J. Chen, J. Cao, C. Ma, J. Wang, X. Cui, and P. Chen, ‘‘EID-GAN: Soc., Apr. 2016, doi: 10.48550/arXiv.1511.08060.
Generative adversarial nets for extremely imbalanced data augmenta- [120] C. Wang and Z. Xiao, ‘‘Lychee surface defect detection
tion,’’ IEEE Trans. Ind. Informat., early access, Jun. 14, 2022, doi: based on deep convolutional neural networks with GAN-based
10.1109/TII.2022.3182781. data augmentation,’’ Agronomy, vol. 11, pp. 1–17, Jul. 2021,
[102] M. Mundt, S. Majumder, S. Murali, P. Panetsos, and V. Ramesh, ‘‘Meta- doi: 10.3390/agronomy11081500.
learning convolutional neural architectures for multi-target concrete [121] J. J. Bird, C. M. Barnes, L. J. Manso, A. Ekárt, and D. R. Faria, ‘‘Fruit
defect classification with the concrete defect bridge image dataset,’’ quality and defect image classification with conditional GAN data aug-
in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), mentation,’’ Scientia Horticulturae, vol. 293, Feb. 2022, Art. no. 110684,
Jun. 2019, pp. 11196–11205. doi: 10.1016/j.scienta.2021.110684.
[103] Y. Shi, L. Cui, Z. Qi, F. Meng, and Z. Chen, ‘‘Automatic road [122] M. Adamiak, ‘‘Lemons quality control dataset,’’ 2020.
crack detection using random structured forests,’’ IEEE Trans. Intell. Accessed: Jul. 20, 2022. [Online]. Available: https://github.com/
Transp. Syst., vol. 17, no. 12, pp. 3434–3445, Dec. 2016, doi: softwaremill/lemon-dataset, doi: 10.5281/zenodo.3965568.
10.1109/TITS.2016.2552248. [123] Z. Guo, H. Zheng, X. Xu, J. Ju, Z. Zheng, C. You, and Y. Gu, ‘‘Quality
[104] H. Shin, Y. Ahn, S. Tae, H. Gil, M. Song, and S. Lee, ‘‘Enhancement grading of jujubes using composite convolutional neural networks in
of multi-class structural defect recognition using generative adversarial combination with RGB color space segmentation and deep convolutional
network,’’ Sustainability, vol. 13, no. 22, p. 12682, Nov. 2021, doi: generative adversarial networks,’’ J. Food Process Eng., vol. 44, no. 2,
10.3390/su132212682. Feb. 2021, doi: 10.1111/jfpe.13620.
[105] A. Kyslytsyna, K. Xia, A. Kislitsyn, I. Abd El Kader, and Y. Wu, [124] S.-H. Chen, Y.-W. Lai, C.-L. Kuo, C.-Y. Lo, Y.-S. Lin, Y.-R. Lin,
‘‘Road surface crack detection method based on conditional generative C.-H. Kang, and C.-C. Tsai, ‘‘A surface defect detection system for
adversarial networks,’’ Sensors, vol. 21, no. 21, p. 7405, Nov. 2021, doi: golden diamond pineapple based on CycleGAN and YOLOv4,’’ J. King
10.3390/s21217405. Saud Univ. Comput. Inf. Sci., Jul. 2022, doi: 10.1016/j.jksuci.2022.
[106] K. Behrendt and R. Soussan, ‘‘Unsupervised labeled lane markers using 07.018.
maps,’’ in Proc. IEEE/CVF Int. Conf. Comput. Vis. Workshop (ICCVW), [125] Z. L. Wang and B. W. Zhang, ‘‘Survey of generative adversarial net-
Oct. 2019, pp. 832–839. work,’’ Chin. J. Netw. Inf. Secur., vol. 7, pp. 68–85, Aug. 2021, doi:
[107] L. Pei, Z. Sun, L. Xiao, W. Li, J. Sun, and H. Zhang, ‘‘Virtual generation 10.11959/j.issn.2096-109x.2021080.
of pavement crack images based on improved deep convolutional gener- [126] Z. Niu, G. Zhong, and H. Yu, ‘‘A review on the attention mechanism of
ative adversarial network,’’ Eng. Appl. Artif. Intell., vol. 104, Sep. 2021, deep learning,’’ Neurocomputing, vol. 452, pp. 48–62, Mar. 2021, doi:
Art. no. 104376, doi: 10.1016/j.engappai.2021.104376. 10.1016/j.neucom.2021.03.091.

VOLUME 10, 2022 113511

 X. He et al.: Survey of Defect Detection Applications Based on Generative Adversarial Networks

XIANGJIE HE received the B.S. degree in commu- HOUDONG XU received the M.S. degree in
nication engineering from the Sichuan University power system and automation from Tsinghua Uni-
of Science and Engineering, Yibin, China, in 2021, versity, in 2005. Currently, he is the Deputy Direc-
where he is currently pursuing the M.S. degree. tor of the Department of Digitalization, State Grid
His research interests include deep learning and Sichuan Electric Power Company. His research
computer vision. interests include big data analysis, 5G communi-
cation, and green low-carbon technology.

HONGBO CHEN received the M.S. degree

in electrical engineering and automation from
Wuhan University, Wuhan, China, in 2012. As the
main researcher, he participated in ‘‘Research
ZHENGWEI CHANG received the Ph.D. degree and application of non-electric quantity testing
in computer application technology from the Uni- technology in condition-based maintenance of
versity of Electronic Science and Technology of power transmission and transformation equip-
China (UESTC), in 2009. Since 2009, he has ment,’’ ‘‘Detection and evaluation of insulation
been with the State Grid Electric Power Research defects and pollution ultraviolet rays of high volt-
Institute, where he has been a Professor, since Jan- age power transmission and transformation equip-
uary 2019. His research interests include artificial ment,’’ ‘‘Pollution investigation, measurement, and analysis research along
intelligence, smart grid, and internet of energy. the Sichuan Section of Xiangjiaba–Shanghai and Jinping–Sunan ±800kV
UHV DC transmission lines,’’ and other key scientific and technological
projects of State Grid Corporation of China.

ZHONGQIANG LUO (Member, IEEE) received

the B.S. and M.S. degrees in communication engi-
neering and pattern recognition and intelligent sys-
tems from the Sichuan University of Science and
LINGHAO ZHANG received the Ph.D. degree in Engineering, Zigong, China, in 2009 and 2012,
computer software and theory from Nanjing Uni- respectively, and the Ph.D. degree in communica-
versity, in 2014. Currently, he is a Senior Engineer tion and information systems from the University
with the Energy Internet Technology Center, State of Electronic Science and Technology of China
Grid Sichuan Electric Power Research Institute. (UESTC), in 2016. Since 2017, he has been with
His research interests include electric power big the Sichuan University of Science and Engineer-
data analysis, artificial intelligence, and cyber- ing, where he is currently an Associate Professor. From December 2018 to
physical systems. In recent years, he has won many December 2019, he was a Visiting Scholar with the Department of Com-
awards in cyber security and big data competitions, puter Science and Electrical Engineering, University of Maryland, Baltimore
as a Principal Investigator or major research partic- County (UMBC). His research interests include information fusion, blind
ipant, he has worked on more than ten science and technology projects from source separation, signal processing for wireless communication systems,
U.S. National Science Foundation, China 863, State Grid Corporation and and intelligent signal processing.
Sichuan Provincial Government.

113512 VOLUME 10, 2022