2020 IEEE 5th Information Technology and Mechatronics Engineering Conference (ITOEC 2020)

Research on Defect Detection Method for Steel Metal

Surface based on Deep Learning
Xiaoyang Gai ,Peiran Ye,Jinglin Wang,Bingquan Wang
School of Information Engineering,Wuhan University of Technology
Wuhan, China
[email protected], [email protected],[email protected],[email protected]

Abstract—In the process of modern industry, the surface gives The formulation of standards brings inconvenience, so
defects of industrial products seriously affect the quality, safety, companies urgently need advanced technology and equipment
usability and aesthetics of products.Based on the method of deep for surface defect detection of industrial parts.
learning, this paper takes steel surface defects in industrial parts
as the breakthrough point, and mainly USES the convolutional With the rapid development of artificial intelligence, it has
neural network algorithm in deep learning to classify and detect become possible to use machine vision instead of manpower to
steel surface defects.Firstly, industrial cameras were used to solve such problems. Deep learning is a branch of machine
collect and pre-process the steel defect images to obtain relevant learning and one of the key breakthroughs and researches made
data sets. Secondly, VGG model was used to improve the in the field of machine learning in recent years. Convolutional
network features to improve the recognition of defects and neural networks, as an important deep learning algorithm, have
realize the classification and recognition of defects.Compared begun to be slowly applied to the field of image recognition.
with traditional methods, this method has higher accuracy and This paper is based on a convolutional neural network
efficiency. algorithm. Based on the VGG network model, by improving
the VGG network structure, the detection accuracy of steel
Keywords—surface defects;deep learning; convolutional parts is improved.
neural network; classification recognition
II. THE CONSTRUCTION OF THE DATE
I. INTRODUCTION
Industrial parts are widely used in a series of industries A. The data collection
such as aerospace, machinery, electronics, and automobiles, In this study, the object of study is steel industrial parts,
and are indispensable components in various industrially whose surface defects are not easy to distinguish with the
manufactured products.In the highly competitive modern naked eye. Therefore, it is necessary to use an industrial
industrial production, the quality of industrial parts is directly camera for image acquisition, and at the same time to
related to the final quality of the product.In mechanical and overcome the interference of external factors such as light
automatic processing engineering, changes in the tool's sources and set an appropriate focus And other parameters to
trajectory during processing and the material's own obtain pictures with high imaging quality, and also to ensure
characteristics, vibration, damage to the tool, and improper that the acquisition environment is clean and the acquisition
handling of the polishing process may cause appearance platform is stable, etc., to reduce the impact of external adverse
deformation, dents, and scratches on the surface of the factors on image acquisition.The CCD industrial camera used
machined part A series of flaws such as damage and in this study covers 300,000 to 14 million pixels, and uses a
unsatisfactory reflective characteristics. The surface flaws of more stable and versatile Gigabit Ethernet network for
industrial parts may not only make the appearance of the transmission. It has clear and stable image quality and supports
workpiece unsightly, but may also affect the performance of external trigger input control acquisition and signal output. The
the workpiece. industrial lens used with it has the characteristics of variable
With the continuous improvement of the processing magnification, large depth of field, no optical distortion, high
requirements of modern industrial products, the detection of definition and high contrast. The focal length method is
surface defects on machined parts has become an important adopted, and the measurement error is small, which is suitable
method and means of product quality inspection and control. for applications such as highly accurate detection, microscopic
The variety and quantity of industrial parts urgently require magnification, and size measurement.Through the use of
enterprises to realize industrialized production cameras and lenses, you can achieve image sample
automation.However, most companies currently use more magnification and high-quality collection. In addition, you
traditional manual visual inspection methods. This method has need to set up a collection platform, select a suitable light
a slow detection speed and cannot meet the high-speed source and camera stand, and solve the problems of external
manufacturing rhythm of industrial production. The sampling light interference and shooting stability, making the pictures
rate is low and the missed rate is high. , The workload is large, more Be clear and get high-quality data sets. The types of
the labor cost required is huge, and there are large errors. The defects in the collected steel sample set are divided into
test results are easily affected by the subjective factors of the rubbing, bottom leakage, bumping, and convex powder. Finally,
test staff. There is no consistent scientific guidance. Because the data samples are stored on the computer to complete the
there is no unified and reference evaluation mechanism, it also

978-1-7281-4323-1/20/$31.00 ©2020 IEEE 637

Authorized licensed use limited to: Auckland University of Technology. Downloaded on August 12,2020 at 16:43:06 UTC from IEEE Xplore. Restrictions apply.
data set establishment. The process of collecting the data is network parameters, making the constructed deep learning
shown in Fig.1. neural network better feature expression and classification
capabilities.

B. VGG network model

Fig. 1. The flow chart for collecting data. VGGNet is a convolutional neural network model first
proposed by Oxford University and Google Deep Mind. It uses
a small convolution kernel size instead of a large convolution
B. The enhancement of date
kernel size to increase the depth of the network, and repeatedly
Deep learning needs to learn based on a large amount of stacks multiple convolutional layers and maximum pooling
data. If the number of data is too small, overfitting will occur. layers. Reduce the input image size, maintain the translation
It is often difficult to obtain a sufficient amount of data in the invariance of the neural network, and use the pre-trained data
industrial production process, but the data set can be expanded of the specific layer to initialize the parameters. Compared with
by means of data enhancement. Therefore, the number of data the AlexNet network structure, it not only improves the
sets is greatly increased, thereby reducing the occurrence of recognition function of the discrimination function, but also
overfitting during the convolution training process, and the reduces unnecessary parameter.
network generalization ability is further enhanced.
The prominent feature of VGG is that it uses a small 3 × 3
At present, data enhancement mostly enhances the read-in convolutional layer, which increases the network depth and
images, such as performing flipping, panning, and other effectively improves the model's effect. Moreover, VGGNet
operations, so that the data set can grow non-linearly. has a good generalization ability for other data sets. The
Commonly used methods are rotation, which changes the VGGNet model studies the relationship between the depth of
orientation of objects in an image by angularly rotating the CNN and its performance. By repeatedly stacking a small
original image. You can also use the flip transform to reverse convolution kernel of 3 × 3 and a maximum pooling layer of 2
the image. Zooming is to enlarge or reduce the original image × 2, a CNN with a depth of 16-19 layers can be constructed.
proportionally. A translation transformation is a translation of
the position of an image. Image filtering can change the blur The basic algorithm model used in this study is the VGG16
degree of the image. Noise perturbation is to amplify the data model. VGG16 has 16 layers, 13 convolutional layers and 3
set by adding random perturbations to the image. While fully connected layers. After the first two convolutions with 64
performing data enhancement, you should also perform convolution kernels, one pooling is used, and the second time
category balance, that is, to enhance the different types of with 128 convolution kernels. After that, pooling is used, and
images to different degrees, so that the number of various types the three 512 convolution kernels are repeated twice, and then
of samples in the data set are similar. A type-balanced data set pooled, and finally three full connections are performed. The
is more conducive to model training. VGG16 network model is shown in Fig. 2.
Due to the limitation of the number of samples and
laboratory conditions, the number of sample sets obtained is
not very large. There are 300 images in total, of which 34 are
scratched images, 160 are missing images, 30 are concave
images, and 76 are convex images. However, this number of
data sets is far from sufficient for the training of deep
convolutional networks. Therefore, we use the image
enhancement method to enhance each type of image to expand
it to about 300. The main operations include image rotation,
translation, and miscutting. Finally, a defective sample set with
a sample size of about 1400 is established.

III. DETECTION ALGORITHM BASED ON CONVOLUTIONAL

NEURAL NETWORK

A. Convolutional neural network

Convolutional neural network is one of the most
representative algorithms for deep learning. It is a hierarchical
network structure containing convolution operations.
Convolutional neural networks have great advantages in image Fig. 2. VGG16 network model diagram.
processing because they share convolution kernels, are good at
processing high-dimensional data, and feature automatic
location. The so-called deep learning is actually to build a deep C. Improvement of network structure
neural network model containing multiple hidden layers, and Although VGG16 is a model with outstanding classification
with the support of the computer's powerful computing power, performance in convolutional neural networks, it was found in
it uses a large amount of training data to continuously optimize experiments that the network structure has a large loss in image

638

Authorized licensed use limited to: Auckland University of Technology. Downloaded on August 12,2020 at 16:43:06 UTC from IEEE Xplore. Restrictions apply.
feature extraction, which is insufficient, affecting the final c c c

target detection results, leading to the recognition of parts 

Zadd  （Xi  Yi） Ki   X  K  Y  K
i i i i (1)
targets The rate is not precise enough. With the continuous i 1 i 1 i 1

improvement of convolutional neural network models, deeper In the formula, X and Y are input channels, and the input
network models such as Inception and ResNet have appeared, and output channels in the algorithm are both 512. * is the
and deeper network layers have improved performance. convolution operation. In this module, since the 3 3
Therefore, combining the advantages of networks such as convolution layer is independent of the input and output, it is
Inception and ResNet, the feature extraction network is beneficial to the construction of the network. Therefore, the
improved, and the feature of the first three convolutional layers network module is designed by modifying the number of
in the neural network can be used to extract features well, and channels of the 3 3 convolution layer. The design module is
the image features can be extracted more effectively. The
shown in Fig.4.
improved VGG16 feature extraction network is shown in Fig.3.
The number of conv4 module parameters of VGG16
network and ResNet50 is 5898k, and each module of ResNet50
is about 983k parameters. Therefore, according to the
principles of building similar modules, the number of channels
of the 3 3 convolution layer is set to 172. In order to speed up
the convergence speed, adding a normalization layer before
each layer input is beneficial to correct the network output of
the previous layer so that its average value is 0 and then input
to the next layer. The improved feature extraction network can
take advantage of different networks, improve feature
utilization, and further enhance performance.
Fig. 3. The improved feature extraction network.

In the improved network structure, replace the Conv4

convolutional layer, borrow the idea of building a deep
network from ResNet50, and the inner product method of
Inception model split-transform-merge, and deepen the depth
and width of the network at the same time to get more Strong
expression skills.Increasing the depth and width of the network
is the most direct way to improve the performance of the neural
network. Since the Inception module is a network with a good
local topology, parallel convolution operations are performed
on the input images, and finally the different features obtained
are stitched together. Therefore, when designing the network
module, by combining the characteristics of ResNet and
Inception, the Incepton model is referenced in the residual
module to replace the convolutional layer in the residual
connection to form a new structure, so that nodes learn
between input and output. The difference mapping can avoid Fig. 4. The new design module.
the fitting of input and output features, eliminate gradient
dispersion and gradient explosion, and speed up the training of IV. EXPERIMENTAL RESULTS AND ANALYSIS
the network.
In the replacement module, in order to reduce the A. Performance evaluation index
calculation amount, the 5  5 convolution is replaced by two The performance evaluation index adopts average precision
3 3 convolutions. First, dimensionality reduction is performed AP.It is often used as a measurement standard, mainly
on the input by 1 1 convolution, then multiple including two values: precision and recall. Precision is
3 3 convolutions are used to transform, then channel merging precision, which means precision, and represents the
is performed in series along the channel dimensions, multi- proportion of samples belonging to this category in samples
scale detection is performed, and finally a 1 1 convolution is that are predicted to be positive in different categories. The
used to achieve The channels are the same, and the addition of formula is:
the residual module and the output linear vector is completed. TP TP
The formula is shown in formula (1).When changing the precision   (2)
TP  FP N
network module, it can be designed according to a similar
In the experiment, four types of samples, including scratch,
highly modular concept, so that the calculation complexity of
bottom leakage, concave impact and convex powder, were
the new module is similar to the conv4 module in VGG16, so
used as the detection targets to classify and predict the types of
that the network complexity can be maintained while
defects. TP(True Positive) indicated the number of defects
increasing the network depth and width.
correctly recognized. TN(False Negativa) represents the

639

Authorized licensed use limited to: Auckland University of Technology. Downloaded on August 12,2020 at 16:43:06 UTC from IEEE Xplore. Restrictions apply.
number of unrecognized defect samples. FP(FalsePositive) significantly improved compared to the basic network model,
error identifying flaws. The number of samples and recall which shows that the algorithm has a fairly obvious effect after
recall rate, on behalf of the correct number and test set to detect improvement, and further proves that the method is effective in
the target all the ratio of the number, the formula is: defect detection. It was also found in the experiment that due to
the large proportion of bottom leakage, the detection results are
TP (3)
recall  relatively close to this situation. Therefore, in future research, it
TP  FN is necessary to increase the number of defective data sets for
The formula of average accuracy is: training the network, while ensuring the data concentration The
1 N
(4) number of various types of defects is relatively average in
AP   p (r )dr   p (k )r (k ) order to increase the average accuracy and network detection
0
k 1
From the above, AP is an integral of precision and capabilities.
recall.That is, the product of precision and recall for each The total loss of the defect detection network (total_loss) is
threshold value is calculated respectively, and then the product composed of classification loss, regression loss in the regional
value under all thresholds is accumulated.By synthesizing recommendation network, and classification loss and
sample categories, the model can be evaluated by means of the positioning loss in the main network. It is necessary to
average precision rate mean MAP, which represents the mean visualize the loss curve on the training data set in time during
of the average accuracy of all sample categories in the model. the training process, which can guide the adjustment of various
The calculation formula is as follows: parameters in the defect detection model, including the number
N of iterations, learning rate, weight attenuation coefficient, and
1 (5)
MAP 
N
 AP
n 1
structure optimization.
The number of training cycles of the VGG improved
B. Test results network structure used in this project is 50. Each cycle will be
The experimental hardware platform operating system is trained on all training data sets, and the various types of loss
Win10 64-bit, CPU Intel Core (TM) i7-4790 3.60GHz, 8GB values of the network will be recorded every 50 steps. Fig. 5
memory. The entire experiment was developed based on the shows the total loss curve of the training process. As the
Tensorflow deep learning framework, the programming training progresses, the model training accuracy continues, the
language was Python, and GPU-accelerated training was loss function gradually decreases, and the loss value decreases
performed using a GTX1060 graphics card. greatly at the beginning of training, indicating that the learning
rate is appropriate and there is a gradient descent After the
The experiment uses the collected steel parts dataset as a training reaches a certain stage, the loss curve becomes stable,
training sample. The model first optimizes the algorithm's indicating that the trained model starts to converge.
weights. The momentum is set to 0.9, the weight attenuation
factor is 0.05, the batch size used in each step is set to 128, and
the initial learning rate is 0.01. When the number of batch
processing reaches 30,000, the learning rate is adjusted to
0.0001; when the number of batch processing reaches 50,000,
stop training to obtain the model, and save the trained model h5
file.
During the training of the basic model of defect detection, it
was found that the loss of the network did not change after
about 50 cycles of training, that is, the defect detection model
had converged. Therefore, in the subsequent improvement of
the network structure for training, the training period is set to
50.The training model of the VGG improved network structure
is transferred to the test data set, where the test set includes 400 Fig. 5. The large-scale network survivability chain.
defective samples (100 samples of four types of defective
samples). After testing the test set, the network evaluation
results obtained are shown in TABLE I. V. CONCLUSIONS
With the development of industrial automation, rapid and
TABLE I. TEST RESULTS accurate quality inspection of industrial products is of great
significance.With the development of deep learning, industrial
Algorithm model AP recall MAP detection algorithms have been greatly improved in both speed
VGG foundation 63% 67% 65% and accuracy compared with traditional algorithms.As an
model important product of industrial production, steel products will
The improved 74% 80% 77% have a variety of defects in the process of production and
network model processing.This paper is based on the deep learning detection
As can be seen from the table, the average accuracy rate of algorithm to identify defects, improve the detection efficiency,
the improved network structure model based on VGG is in industrial production has a certain degree of universality.

640

Authorized licensed use limited to: Auckland University of Technology. Downloaded on August 12,2020 at 16:43:06 UTC from IEEE Xplore. Restrictions apply.
 This paper mainly studied the VGG improved network ACKNOWLEDGMENT
based model of steel surface defect classification and detection, This work was supported by the National innovation and
through introducing convolution in deep learning neural entrepreneurship training program for college students.The
network algorithm solved the problem of the traditional image project number is S201910497052.
processing in the narrow as well as the traditional artificial
classification of strong subjectivity, low accuracy, the problem
of low efficiency of classification, and for introducing deep
learning other problems in the field of industrial provides
reference.In the experiment, we first used the industrial camera REFERENCES
to collect the steel images and set up the data set, and then [1] D. Qi, P. Zhang, L. Yu, “ Study on wood defect detection based on
enhanced the data to improve the recognition ability of the artificial neural network.. ” 2008 IEEE Conference on Cybernetics and
network and reduce the problem of overfitting.On the basis of Intelligent Systems(ICCI), Chengdu, China, pp. 21–24 Sept. 2008, IEEE,
pp. 951-956.
Tensorflow deep learning framework, the VGG basic model is
first built and then the network structure is improved, the [2] D. Dan, Y.Q. Zhang, “Research on Definition of Network
Survivability,” Journal of Computer Research and Development, vol.
number of network layers is deepened, and the ability to 43(Suppl.), pp. 525-529, 2006.
identify targets is improved.Through experiments, compared [3] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S.
with the original network, the improved feature extraction Ghemawat, G. Irving, M. Isard，“Tensorflow: a system for large-scale
network is more effective in feature extraction, and it is also machine learning ” 12th Symposium on Operating Systems Design and
found that the improved feature extraction network has higher Implementation (2016), pp. 265-283.
accuracy in target detection. [4] J. Dai, Y. Li, K. He, J. Sun，“R-fcn: object detection via region-based
fully convolutional networks ” Advances in Neural Information
In addition, in the defect detection process, due to the Processing Systems (2016), pp. 379-387.
uneven distribution of sample types and quantities, it is [5] H. Xu, L. Wang, S. Ni, “Application of Artificial Neural Network to
necessary to continue to improve the sample and algorithm Nondestructive Testing of Internal Wood Defects Based on the Intrinsic
model in the later work to make the training model more Frequencies,” 2010 International Conference on System Science,
Engineering Design and Manufacturing Informatization, Yichang, China,
reasonable and accurate, so as to further improve the accuracy 12-14 Nov. 2010, IEEE, pp. 207–210.
of the training results.In the experiment, the deepening of [6] L.H. Wang, W. Qi, et al.Pattern recognition and size determination of
network structure will lead to the problem of time wasting, internal wood defects based on wavelet neural networksComput.
long training time and slow response.In order to better apply to Electron. Agric., 69 (2009), pp. 142-148.
the industrial scene, in the future research, the network can be [7] N. Chen, X. Men, X. Han, et al. “Edge detection based on machine
changed to a lightweight network for training, to better meet vision applying to laminated wood edge cutting process.” 2018 13th
the needs of industrial detection.The improved detection IEEE Conference on Industrial Electronics and Applications (ICIEA),
Wuhan, China, 31 May-2 June 2018, IEEE, pp. 449–454.
algorithm means the improvement of detection technology,
[8] Huibin Sun,Jiduo Zhang,Rong Mo,Xianzhi Zhang. In-process tool
which has great practical significance to improve the level of condition forecasting based on a deep learning method[J]. Robotics and
industrial production. Computer-Integrated Manufacturing,2020,64.

641

Authorized licensed use limited to: Auckland University of Technology. Downloaded on August 12,2020 at 16:43:06 UTC from IEEE Xplore. Restrictions apply.