“Application of Convolutional Neural Network in Defect Detection of 3C Products”

CHAPTER 1
INTRODUCTION
1.1 Overview of project:
With the continuous improvement of science and technology, the intelligent, automated, and
unmanned manufacturing industry will be an inevitable trend in the future. As an important
form of metal materials, metal work pieces are widely used in daily life and industrial
production by virtue of their excellent mechanical and physical properties. In the production
process of the product, due to the influence of equipment and technology, different kinds of
defects often appear on the surface of the product, such as scratches, holes, and cracks in the
metal work piece.

The surface quality of metal work pieces not only affects the appearance and image of
the product, but may also affect the functional characteristics of the product and cause
significant losses to the enterprise. Therefore, it is very necessary to detect the surface defects
of the product, and it is particularly important to design a real-time and effective surface
defect detection method for metal work pieces. Conventional non-destructive inspection
methods for surface defects of metal work pieces include magnetic particle inspection,
penetrant inspection, infrared thermal imaging inspection, ultrasonic inspection, visual
inspection, etc. Machine vision inspection technology is constantly being used in inspections
in various fields. Machine vision inspection mainly uses high-resolution industrial cameras to
obtain images of specimens to be inspected, and uses digital image processing inspection
algorithms to complete the inspection of defects.

The visual inspection is a non-contact inspection, and it will not cause any damage to the
work piece to be inspected during inspection. At the same time, the visual inspection has a
high degree of automation, which can be realized for a long time and work continuously and
smoothly. In recent years, research on artificial intelligence technology has continued to
deepen, and machine learning and deep learning methods have been rapidly developed, and
have been gradually applied in various fields, providing a new method for solving the
detection problem of metal work piece surface defects.

Deep learning can directly learn two-dimensional images, reducing image pre-
processing, without manually extracting features, and can automatically learn more
appropriate features layer by layer, greatly reducing the impact of human factors. With the
development of
DEPT OF IS&E, AIT, CHIKKAMAGALUR PAGE 1
 “Application of Convolutional Neural Network in Defect Detection of 3C Products”

Machine vision technology, researchers use image processing to detect screen defects of 3C
products. Proposed an algorithm combining the line intercept as the threshold and particle
swarm optimization, which solved the problem of low detection accuracy of mobile phone
screen defects. Improved the median filter algorithm and utilized the image processing
algorithm to pre-process the defective image. In addition, they proposed an image difference
algorithm based on fast image matching for detection. Z hang et al posed an improved
differential image detection method for mobile phone screen defects to improve detection
efficiency and accuracy. However, the performance of pure image processing technology
cannot achieve satisfactory detection results.

Convolutional Neural Networks (CNN), a research hotspot in the field of image

detection and pattern recognition, has received more and more attention from scholars. Ma et
al. Designed the CNN based on the Google Net network, which greatly reduced the number
of parameters without affecting the prediction rate. Experimental results showed that the
defect detection rate of the designed CNN could reach 99.5%. Wei et al proposed a
multivariable CNN-based defect detection method in the production process of cover glass,
touch screen and mobile phone display under parallel light sources. Experiments verified that
the method had higher accuracy, better stability and faster speed. The CNN architecture
mainly includes weight parameter sharing and pooling operations which is more complicated.
However, CNN is easy to train, and the learned features are translation invariant.

The main content of this method is to review the surface defect detection methods with
different levels of CNN in 3C products such as glass display screens, PCBs and TFT-LCD
screens. Then, the detection methods and the problems of various methods are summarized
and discussed. First, we outline the types of defects in 3C products and the comparison of
different detection methods in the context of Industry 4.0. Subsequently, the improved CNN
structures in various scenarios are compared, and the technical limitations and detection
performance of different methods in practical applications are shown. This research
summarized and compared noteworthy research based on CNN in recent years to overcome
the challenges of surface defect detection in 3C products.

DEPT OF IS&E, AIT, CHIKKAMAGALUR PAGE 2

 “Application of Convolutional Neural Network in Defect Detection of 3C Products”

1.2 Overview of algorithm used:

Alexey Bochkovskiy who produced YOLOv4 in 2020. YOLOv5 is the next controversial
member of the YOLO family released in 2020 by the company Ultralytics just a few days after
YOLOv4. YOLO is an acronym that stands for You Only Look Once. We are
employing Version 5, which was launched by Ultralytics in June 2020 and is now the most
advanced object YOLO stands for You Look Only Once and it is one of the finest family of
object detection models with state-of-the-art performances . Its first model was released in 2016
by Joseph Redmon who went on to publish YOLOv2 (2017) and YOLOv3 (2018). In 2020
Joseph Redmon stepped out from the project citing ethical issues in the computer vision field
and his work was further improved by identification algorithm available. It is a novel
convolutional neural network (CNN) that detects objects in real-time with great accuracy. This
approach uses a single neural network to process the entire picture, then separates it into parts
and predicts bounding boxes and probabilities for each component. These bounding boxes are
weighted by the expected probability. The method “just looks once” at the image in the sense
that it makes predictions after only one forward propagation run through the neural network. It
then delivers detected items after non-max suppression (which ensures that the object detection
algorithm only identifies each object once).
1. Backbone: Model Backbone is mostly used to extract key features from an input image.
CSP (Cross Stage Partial Networks) are used as a backbone in YOLO v5 to extract rich in
useful characteristics from an input image.
2. Neck: The Model Neck is mostly used to create feature pyramids. Feature pyramids aid
models in generalizing successfully when it comes to object scaling. It aids in the identification
of the same object in various sizes and scales.
Feature pyramids are quite beneficial in assisting models to perform effectively on previously
unseen data. Other models, such as FPN, BiFPN, and PANet, use various sorts of feature
pyramid approaches. PANet is used as a neck in YOLO v5 to get feature pyramids
, objectless scores, and bounding boxes
3. Head: The model Head is mostly responsible for the final detection step. It uses anchor
boxes to construct final output vectors with class probabilities

DEPT OF IS&E, AIT, CHIKKAMAGALUR PAGE 3

 “Application of Convolutional Neural Network in Defect Detection of 3C Products”

CHAPTER 2
PROBLEM DEFINITION

2.1 Problem Statement:

The suitability of deep learning models will be investigated for steel surface, PCB, glass
display screens, including cracks, floaters and point defect detection. The main challenge of
this task is to establish a robust system with a limited number of samples In this method, we
will analyze the performance of convolutional neural network architecture and data
augmentation strategies in order to classify and compare their performance when solving the
main challenges.

2.2 Existing System:

Image classification is a branch of computer vision that uses a computer to acquire image data
and interpret them by mimicking human biological systems. This is a very important topic in
today's situation because every second a large number of image data are acquired and used for
various purposes around the world. One application of image classification is to detect defects
on the surfaces of industrial products. Quality inspection is usually the final stage in a
production line, and has been conducted mainly by human experts. This can be time consuming
and mistake-prone. In this study, we investigate the possibility of replacing, fully or partially,
human experts with a machine learner when the product defects are visible using Open CV
concepts.

2.3 Proposed System:

In this Proposed system, we investigate several methods based on deep learning especially
CNN (Convolutional Neural Network) based YOLO (You Only Look Once) algorithms which
have the highly efficient in the live time prediction based on the latest version of YOLOV5. The
first one is to use a deep learner directly to detect the existence of defects in a given product
surface image. The second one segments suspected parts first and then uses the deep learner to
classify the segmented parts. The third method employs an ensemble of deep learners. Results
show that the third method can provide the best results, and can be practically useful if we
introduce a proper rejection mechanism with the NMS and IOU techniques (intersection of
union).

DEPT OF IS&E, AIT, CHIKKAMAGALUR PAGE 4

 “Application of Convolutional Neural Network in Defect Detection of 3C Products”

CHAPTER 3
LITERATURE SURVEY

(1) ‘‘Defect detection of mobile phone screen based on improved difference image
method’
Using the CNN to defect detection of product. CNN is a feed forward neural network.
CNN consist of one or more convolutional layers and fully connected layers, as well as
associated weights and pooling layers. Literature is a very popular LeNet convolution
neural network structure. LeNet network structure can be used to detect defects in two
situations: one is to design a complex multi-layer CNN structure, use different network
structure to extra image content features, and complete end-to-end training to detect
defects in images; the other is to combine CNN with CRF model, train CNN with CRF
energy function as constraint or optimize network prediction results with CRF. And to
achieve the detection of product defects

(2) Peng, Y.; Liu, G.; Quan, Y.; Zeng, Q. The depth measurement of internal defect
based on laser speckle shearing interference. Opt. Laser Technol. 2017,92, 69–73
Using the CNN to defect detection of product. CNN is a feedforward neural network.
CNN consist of one or more convolutional layers and fully connected layers, as well as
associated weights and pooling layers. Literature is a very popular LeNet convolution
neural network structure. LeNet network structure can be used to detect defects in two
situations: one is to design a complex multi-layer CNN structure, use different network
structure to extra image content features, and complete end-to-end training to detect
defects in images; the other is to combine CNN with CRF model, train CNN with CRF
energy function as constraint or optimize network prediction results with CRF. And to
achieve the detection of product defects

(3) Xiao, X.; Yu, L.; Dong, Z.; Mbelek, R.; Xu, K.; Lei, C.; Zhong, W.; Lu, F.; Xing,
M. Adipose stem cell-laden injectable thermosensitive hydrogel reconstructing
depressed defects in rats: Filler and scaffold. J. Mater.Chem. B 2015,27, 5635–
5644.
DEPT OF IS&E, AIT, CHIKKAMAGALUR PAGE 5
 “Application of Convolutional Neural Network in Defect Detection of 3C Products”

The product defect-detection technology based on the neural network. Autoencoder

network mainly includes two stages: coding and decoding. In the coding stage, the input
signal is converted into a coding signal for feature extraction; in the decoding stage, the
feature information is converted into a reconstruction signal, and then the reconstruction
error is minimized by adjusting the weight and bias to realize the defect detection. The
difference between autoencoder networks and other machine learning algorithms is that
the learning goal of the autoencoder network is not for classification, but for feature
learning.

(4) Hui, K.; Jian, Y.; Chen, Z. Accurate and Efficient Inspection of Speckle and
Scratch Defects on Surfaces of Planar Products. IEEE Trans. Ind. Inform. 2017,4,
1855–1865.
The product defect-detection technology of deep residual neural network. The deep
residual network adds a residual module on the basis of the convolutional neural
network. The residual network is characterized by easy optimization and can improve
the accuracy by increasing the network depth. CNN, Generative Adversarial Networks,
etc. with the depth of the network increases, the extraction feature increases, but it is
easy to cause the activation function not to converge.

(5) Chen, J.; Li, C. Prediction and Control of Thermal Scratch Defect on Surface of
Strip in Tandem Cold Rolling. J. Iron Steel Res. Int. 2015,22, 106–114.
Full convolution neural network. The fully connected layer is a connection between any
two nodes between two adjacent layers. A fully connected neural network uses a fully
connected operation, so there will be more weight values, which also means that the
network will take up more memory and calculations. During the calculation of the fully
connected neural network, the feature map generated by the convolution layer is
mapped into a fixed-length feature vector

DEPT OF IS&E, AIT, CHIKKAMAGALUR PAGE 6

 “Application of Convolutional Neural Network in Defect Detection of 3C Products”

CHAPTER 4
SYSTEM SPECIFICATION

4.1 Hardware specification:

 Processor : Pentium IV 2.4 GHz.
 Hard Disk : 250 GB.
 Monitor : 15 VGA Color.
 RAM : 1 GB
 Mouse : Optical
 Keyboard : Multimedia

4.2 Software specification:

 Operating system : Windows XP/Windows 7
 Coding language : Python 3.7.4 IDE
 Software : Anaconda navigator
 Web-based IDE : Jupyter notebook

DEPT OF IS&E, AIT, CHIKKAMAGALUR PAGE 7

 “Application of Convolutional Neural Network in Defect Detection of 3C Products”

CHAPTER 5
METHODOLOGY

5.1 System design and flowchart:

DEPT OF IS&E, AIT, CHIKKAMAGALUR PAGE 8

 “Application of Convolutional Neural Network in Defect Detection of 3C Products”

Figure 1.1: Flowchart of system design

5.2 Explanation of each step in above flowchart:

This fig 1.1 explains, the complexity of the network structure is related to the complexity of the
recognition target. The basic characteristics of defect detection are shown for metal work

DEPT OF IS&E, AIT, CHIKKAMAGALUR PAGE 9

 “Application of Convolutional Neural Network in Defect Detection of 3C Products”

pieces. The convolutional neural network structure designed in this method. The first layer is
the input layer, and the input data is a 128*128 grayscale image. Next is the alternating
connection of the three convolutional layers and the pooling layer. The feature filters used in
the convolutional layers are all the same in size. The S3 layer outputs the final two-dimensional
feature map, and maps the two-dimensional feature map to a one-dimensional vector as the
input of the fully connected layer.
The SoftMax classifier classifies the output of the fully connected layer. It is equivalent to a
multi-class logistic regression. The result is the position corresponding to the largest component
output by the classifier. The training process of the network mainly involves the calculation of
forward propagation and back propagation of the network. Forward propagation reflects the
transfer of characteristic information, while back propagation reflects the correction of model
parameters by error information.
When the amount of data is insufficient or the network complexity is large, the training error of
the model will decrease and tend to zero, and the test error will first decrease and then increase.
The model can recognize the training samples very well, but the recognition effect on the test
samples is not very satisfactory, that is, over fitting occurs. Therefore, a regularization model
can be selected, and a regularization attenuation term or penalty term is added to the empirical
risk. The attenuation term will penalize too large parameter values to prevent over fitting.

5.3 Explanation of the Algorithm used:

DEPT OF IS&E, AIT, CHIKKAMAGALUR PAGE 10

 “Application of Convolutional Neural Network in Defect Detection of 3C Products”

Figure 1.2: Explanation of YOLOv5 Algorithm

The YOLO algorithm is a one-stage object detection algorithm characterized by the direct
regression of the location and category of an object after feature extraction. It has the
advantages of fast inference speed and high detection accuracy. The author of the YOLO
algorithm then proposed the YOLOv2 and YOLOv3 models successively, and their
performance showed continuous improvements. In 2020, Alexey Bochkovskiy proposed the
YOLOv4 algorithm, which modified the backbone network to CSPDarknet53, replaced the
activation function with Mish and added a PANet structure based on the FPN from YOLOv3.
The YOLOv5 model has been improved on the basis of YOLOv4, with the detection speed
being significantly improved and the model size greatly reduced, and it is more suitable for
engineering applications.

DEPT OF IS&E, AIT, CHIKKAMAGALUR PAGE 11

 “Application of Convolutional Neural Network in Defect Detection of 3C Products”

The YOLOv5 algorithm has four models: YOLOv5s, YOLOv5m, YOLOv5l and
YOLOv5x. The depth and width of these models increase in turn, and the feature extraction
ability also gradually improves. Considering that our algorithm was intended to be deployed
and applied in mobile terminals in the future, we selected the lightest YOLOv5s network as the
research object, improved it on the basis of our needs and established a model for detection of
surface defects in aero-engine components.
YOLOv5 is mainly composed of three parts: the backbone, neck and head. The backbone
consists of a series of convolutional neural networks used to extract image features, mainly
focus, C3 and SPP modules. The focus module slices the images and splices them into the
channel dimension to integrate the width and height information into the channel dimension,
which can effectively improve the speed of feature extraction. The C3 module is improved from
the structure of the cross-stage partial (CSP) connections, having one less convolution layer and
changing the activation function, and its main function is to extract features from images and
reduce the repetition of gradient information. The spatial pyramid pooling (SPP) module
respectively uses three pooling kernels of sizes 5, 9 and 13 to perform max-pooling operations
on the images. This module can increase the receptive field of the network and obtain features
of different scales. The neck is the feature fusion network of the model, where feature pyramid
networks (FPNs) and path aggregation network (PANet) are adopted. The structure of the FPNs
transmits semantic information from the top down, while the PANet additionally transmits
location information from the bottom up on the basis of the FPNs. The head is the prediction
network of the model and, through convolution operations, three groups of feature vectors
containing the categories prediction boxes, confidence and coordinate position are output,
which predict at scales of 80 × 80, 40 × 40 and 20 × 20, respectively.
The architecture of YOLOv5s is shown in Figure1.2, where k represents the convolution
kernel size, C3(T) represents the C3 module containing a residual structure, C3(F) does not
contain a residual structure, Upsample is the upsampling operation and CBS is the standard
convolution process consisting of Conv (the convolution layer), BN (batch normalization) and
SiLu (activation function).

CHAPTER 6
CONCLUSION

DEPT OF IS&E, AIT, CHIKKAMAGALUR PAGE 12

 “Application of Convolutional Neural Network in Defect Detection of 3C Products”

REFERENCES

DEPT OF IS&E, AIT, CHIKKAMAGALUR PAGE 13

 “Application of Convolutional Neural Network in Defect Detection of 3C Products”

[1] J. Zhang, Y. Li, C. Zuo, and M. Xing, ‘‘Defect detection of mobile phone screen based on
improved difference image method,’’ in Proc. Int. Conf. Intel. In format. Biomed. Sci.
(ICIIBMS), Nov. 2019, pp. 86–92.
[2] Peng, Y.; Liu, G.; Quan, Y.; Zeng, Q. The depth measurement of internal defect based on
laser speckle shearing interference. Opt. Laser Technol. 2017,92, 69–73

[3] Xiao, X.; Yu, L.; Dong, Z.; Mbelek, R.; Xu, K.; Lei, C.; Zhong, W.; Lu, F.; Xing, M.
Adipose stem cell-laden injectable thermosensitive hydrogel reconstructing depressed defects
in rats: Filler and scaffold. J. Mater.Chem. B 2015,27, 5635–5644.

[4] Hui, K.; Jian, Y.; Chen, Z. Accurate and Efficient Inspection of Speckle and Scratch
Defects on Surfaces of Planar Products. IEEE Trans. Ind. Inform. 2017,4, 1855–1865.

[5] Chen, J.; Li, C. Prediction and Control of Thermal Scratch Defect on Surface of Strip in
Tandem Cold Rolling. J. Iron Steel Res. Int. 2015,22, 106–114.
[6] B.Z. Chen, Z.H. Fang, Y. Xia, etc. Automatic detection of pore defects in thick steel pipe
welds based on X-ray images. Computer Applications, 2017, 37(03): 849-853.
[7] X.L. Qian, H.Q. Zhang, Y.X. Chen, etc. Research status and prospect of surface defect
detection for solar cells based on machine vision. Journal of Beijing University of
Technology, 2017, 43(01): 76-85.
[8] L.Z. Zhu, M.Y. Chen. A Research on Auto Glue Defect Detection Based on Machine
Learning. Journal of Shenyang Ligong University, 2018, 37(04): 18-22.
[9] L. Guo, L. Shu, G.C. Wu. Detection method of arc-extinguishing grid surface defects
based on machine learning. Mechanical Engineering and Automation, 2019(01): 4-7.
[10] C. Wang, Y.T. Liu, X.Y. Xu, et al. Research on magnetic tile defect detection based on
convolutional neural network. Journal of Dalian Nationalities University, 2019, 21(03): 220-
224.
[11]J .C. Hu, X.Y. Wang, H. Liu. Surface defect detection of continuous casting slab based on
deep learning. Journal of Shanghai University (Natural Science Edition), 2019, 25(04): 445-
452.

DEPT OF IS&E, AIT, CHIKKAMAGALUR PAGE 14