Journal of Manufacturing Processes 45 (2019) 208–216

Contents lists available at ScienceDirect

Journal of Manufacturing Processes

journal homepage: www.elsevier.com/locate/manpro

Weld image deep learning-based on-line defects detection using T

convolutional neural networks for Al alloy in robotic arc welding
Zhifen Zhanga, Guangrui Wena, , Shanben Chenb
⁎

a
School of Mechanical Engineering, Xi’an Jiao Tong University, Xi’an, 710049 China
b
School of Material Sciences and Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China

ARTICLE INFO ABSTRACT

Keywords: Accurate on-line weld defects detection is still challenging for robotic welding manufacturing due to the complexity
Deep learning of weld defects. This paper studied deep learning–based on-line defects detection for aluminum alloy in robotic arc
Defects detection welding using Convolutional Neural Networks (CNN) and weld images. Firstly, an image acquisition system was
Al alloy developed to simultaneously collect weld images, which can provide more information of the real-time weld images
Robotic arc welding
from different angles including top front, top back and back seam. Then, a new CNN classification model with 11
Convolutional neural networks
Weld images
layers based on weld image was designed to identify weld penetration defects. In order to improve the robustness and
Feature visualization generalization ability of the CNN model, weld images from different welding current and feeding speed were cap-
tured for the CNN model. Based on the actual industry challenges such as the instability of welding arc, the com-
plexity of the welding environment and the random changing of plate gap condition, two kinds of data augmentation
including noise adding and image rotation were used to boost the CNN dataset while parameters optimization was
carried out. Finally, non-zero pixel method was proposed to quantitatively evaluate and visualize the deep learning
features. Furthermore, their physical meaning were clearly explained. Instead of decreasing the interference from arc
light as in traditional way, the CNN model has taken full use of those arc lights by combining them in a various way
to form the complementary features. Test results shows that the CNN model has better performance than our pre-
vious work with the mean classification accuracy of 99.38%. This paper can provide some guidance for on-line
detection of manufacturing quality in metal additive manufacturing (AM) and laser welding.

1. Introduction defects during the robotic welding process is quite urgent in need.
Weld surface defects including under penetration and burning
Aluminum alloy is one of the key material for light weighing through can greatly weaken the strength of joints. Even if precautionary
manufacturing in various industry applications, such as aerospace, measures are well made before welding, the defects still cannot be ef-
automotive and so on. Arc welding manufacturing of aluminum alloy fectively controlled because of its high randomness and complex influ-
components, for instance gas tungsten arc welding (GTAW) has always ence factors, such as assembly form, welding structure, welding process
been the main technology for aerospace industry due to its high pre- parameters and heat accumulation effect. This kind of defects are not
cision and good stability. Therefore, guaranteeing the welding quality allowed in the welding products based on ISO10042 inspection stan-
of aluminum alloy component with zero-defect is of great importance. dards. So, using real-time monitoring and detection technology of
At present, more and more robotics are gradually replacing human welding seam quality might timely detecting seam defects, improving the
welder, which can greatly improve the production efficiency. However, stability of welding quality and manufacturing efficiency; moreover,
the down side is that there is no more human intelligence interference promoting the intelligent degree of robotic welding manufacturing.
to control or stop the weld defect during the dynamic welding process. Sensing technology is the key to achieve on-line defects detection.
Furthermore, due to the complexity of welding process and randomness By accurately sensing certain type of information, the welding robot
of process interference, weld defects are usually inevitable. Various might be “smart” enough to identify the seam quality of aluminum alloy
technologies of Non Destructive Testing (NDT) are applied post welding in real-time [1]. During the dynamic welding process, multiple types of
and performed by experienced workers, which are costly and low effi- information are releasing including welding audible sound, arc light,
cient. Therefore, achieving real-time monitoring and detecting of weld welding current and voltage. Arc audible sound information is reported

⁎
Corresponding author.
E-mail addresses: [email protected] (Z. Zhang), [email protected] (G. Wen), [email protected] (S. Chen).

https://doi.org/10.1016/j.jmapro.2019.06.023
Received 25 March 2019; Received in revised form 11 June 2019; Accepted 24 June 2019
1526-6125/ © 2019 The Society of Manufacturing Engineers. Published by Elsevier Ltd. All rights reserved.
Z. Zhang, et al. Journal of Manufacturing Processes 45 (2019) 208–216

to be related to seam penetration [2,3]. Zhang [3] found that the po- images [20] and so on. CNN has been applied to weld defects detection
larity exchange of AC welding power is the key source of arc sound both for on line [21,22] and off line [23]. Literature [21] combines CNNs
pressure and signal frequency energy. Seam defects of under penetra- with long short-term memory networks to extract more comprehensive
tion and burning through were identified using the proposed classifi- and fusion features while welding burning through and welding devia-
cation model. However, environment noise in welding workshop can be tion were detected for CO2 welding. S.A. Shevchik et.al [22] studied the
loud and uncertain, which normally requires complex algorithm and is on-line detection of porosity with different types and concentrations in
difficult for industry application. Arc spectrum emission is also reported selective laser melting of stainless steel using acoustic emission and
to be able to detect certain macro weld defects by means of non-contact spectral convolutional neural network.
spectrometer since it contains abundant information about emission However, to get the clear information of defects from real-time weld
intensity of various chemical elements. Mirapeixa [4] quantitatively images often encounter several challenges for CNN as follows:
researched the seam quality of Usibor blanks in the laser welding
process while the ratio of line-to-continuum for the Al I emission line at 1) Difficult to get a clear vision of the welding pool under the powerful
396.15 nm was extracted as the monitoring feature. Song et al. [5] exposure of arc light; hard to be characterized using simple features;
predicted the Al concentration during a laser additive manufacturing 2) Dynamic changing of arc light intensity under different welding
process by means of Al/Ti line-intensity ratio and support vector re- power; Arc drifting and unstable arc plasma because of constantly
gression. Zhang [6] proposed several spectrum bands of interest and searching for the minimum ionization energy path, leading to dy-
extracted their statistical features to detect surface oxidation defects. namic weld images with quite different features.
Furthermore, Huang et al. [7] utilized a k-medoids algorithm to select 3) Due to the high diversity of welding condition in the application of
spectral lines with more sensitivity to seam porosity. However, arc robotic rapid welding, the model generalization ability needs to be
optical spectrum is still researched on laboratory-level. More important, improved using technologies such as data augmentation, which has
it can be quite sensitive to the setup position and material component. not been reported yet as well as the visualization of CNN features.
Its reliability and generality still need to improve. Besides, its results are
highly dependent on equipment resolution of the spectrometer, which In this paper, on-line detection of weld defects for Al alloy in pulsed
leads to high cost and is unacceptable for industry application. GTAW was researched using convolutional neural networks and weld
Accurate detection of multiple seam defects, either surface one or inner image. Contributions can be included as follows:
one is still difficult due to the complex generation mechanism of defects
and their own limitations. In addition, those defects can happen in short 1) A weld image-based CNN classification model was proposed to
time and be probably unseen inside the weld pool and seam. Multisensory identify weld seam penetration defects in real-time for Al alloy in
fusion technology [8–11] might provide a possible solution for this pro- pulsed GTAW based on the three-way pool images, which can pro-
blem, but it will bring much more burden to the monitoring system with vide more information of the real-time weld images from different
all kinds of sensors, different information processing methods and complex angles including top front, top back and back seam.
algorithms. With all the high cost, it is not easy for industry application at 2) Data augmentation including noise addition and image rotation was
present. Welding pool vision and X ray imaging are more direct in terms of utilized to boost the CNN dataset and has improved the classification
evaluating the welding process and seam quality. Liu et al. [12] acquired accuracy about 3.88% based on the actual industry challenges ana-
the welding current and weld pool vision signal during the GTAW process lyzed in Fig.3 such as the instability of welding arc, the complexity of
to control the automated process using a data-driven method. Further- the welding environment and the random changing of plate gap
more, the fusion [13] between the machine algorithm and human in- condition. In order to improve the robustness and generalization
telligence was investigated for the next generation of intelligent welding ability of the CNN model, weld images from different welding current
robots. Liu and Zhang [14] studied to extract the human welder's response and feeding speed were captured for the CNN model.
against 3-D weld pool surface and transfer to the welding robots to per- 3) The deep learning image features of the CNN model were visualized
form automated welding tasks. He et al. [15] collected images of a seg- and analyzed using the proposed Non-zero Pixel (NOP) method.
ment gap and welding pool using a vision sensor that integrated the laser Their physical meaning were clearly explained. Instead of de-
structure light with a CCD image. Calta et al. [16] developed a synchrotron creasing the interference from arc light as in traditional way, the
X-ray imaging system to collect the dynamic information of a melt pool in CNN model has taken full use of those arc lights by combining them
a laser powder bed fusion process. The formation of keyhole pores was in a various way to form the complementary features.
revealed for Ti-6Al-4 V. X-ray imaging can collect more inner information
of the weldment, but higher resolution comes with higher radiation and 2. Theory of CNN
damage to workers. Its acquiring equipment can be very complex and still
on laboratory research stage. Convolutional neural networks [24,25] are a specialized kind of
Weld pool vision can acquire more macro level and rich information, neural network that use convolution in place of general matrix multi-
including wire filling, shape of weld pool and arc shape with high re- plication in some of their layers. Typical CNN consists convolutional
solution and low cost since CCD technology is more mature and has been layers, pool layers and full connection layers.
widely used in industry. To overcome these problems, traditional methods Convolutional layer introduces two important ideas: local receptive
includes two steps including feature extraction based on image processing fields and shared weights. Within the convolutional layer’s feature map,
and classification modelling, as shown in Fig. 1. Its real-time performance, each neuron is connected to previous layer’s maps through a learnable
stability and robustness of image system is still a huge challenge not to filter (convolutional kernel). Neurons within the same feature map
mention the complexity, efficiency and huge cost of algorithm. share the same filter which has a small receptive field, but extends
The convolutional neural network, as one of the well-known deep through the full corresponding previous layer’s map. Multiple maps
learning architectures, has powerful ability of feature learning owning to presenting different features are produced by using distinct filters.
multiple convolution and pooling layers. The greatest advantage of CNN Mathematically, the convolutional layer is defined as follows:
is that the features of each hidden layer are not designed manually, but
are learned from the input data automatically. With decades developing,
x lj = f x i(l 1)
kijl + blj
many various CNN models have been proposed and applied in diverse
i Mj (1)
applications, such as cracks detection in civil infrastructures [17], myo-
cardial infarction detection using ECG signals [18], fault classification of Where * represents the convolution operator; x lj represents the jth
machinery [19], detect cerebral micro bleeds from magnetic resonance feature map in the lth layer; Mj represents the input subset of feature

209
Z. Zhang, et al. Journal of Manufacturing Processes 45 (2019) 208–216

Fig. 1. Comparison of traditional method and proposed method in this paper.

a) Schematic of the image sensing system b) The photograph of visual sensor. c) Real-time three-way weld image.

maps that are used to calculate x lj kijl is the weight matrix of the kernel three directions including top center of the welding pool, front of the
connecting the ith feature map in the l-1th layer with the jth feature welding pool as well as back of the welding seam. The schematic dia-
map in the lth layer; blj is bias; f ( ) represents the activation function, gram of the image sensing system and the photograph of visual sensor
which enables the network to acquire a nonlinear expression of the were given in Fig.2-a and b. It includes a CCD camera (DH-SV1420FM),
input signal to enhance the representation ability. mirror system, filter system and “L” type aluminum alloy holder. The
In this paper, Leaky Rectified Linear Units (L-ReLU) activation USB digital transmission method was adopted to improve the trans-
function is used in convolution layers and full connection layers for its mission speed and anti-interference ability. Multiple mirror reflection
multiple advantages, such as making the weights more trainable, better was designed in order to capture the weld image from the three di-
gradient propagation, scale-invariant and avoiding neurons death. The rections. The adjustable secondary mirror is fixed at a 45 ° angle, which
formula of L-ReLU is described as follows: facilitates the adjustment of the reflected light path. The filter system
consists of proper filtering and dimming in order to decrease the in-
f (x ) = {0.01x,x, xx 0
0 (2) terference from arc light. In addition, considering the pulsed welding
current of the pulsed GTAW method, the three-way images were ac-
After convolution, pooling operations are used to replaces the quired within the period of pulse base current, which might have less
output of a certain location with a summary statistic of nearby. arc light. Its specific time was set to 36 ms after the peak current drops
x lj = pooling (x lj 1) + blj behind the edge. Its sampling rate is the same with pulse frequency, e.g.,
(3)
1 Hz, with the pixel size of 1392*1040. Fig.2-b displays the acquired
Where, pooling ( ) denotes the pooling operation, which is a form of real-time three-way weld image, from which the front edge of weld pool
non-linear down-sampling. Max pooling is the most common pooling and wire filler can be seen from the top front image, while the clear
functions. Pooling helps to make the features become approximately view of the weld pool and arc can be obtain from the top back image.
invariant to small translations of the input and improve statistical and Moreover, the back view of the weld seam gives the real-time in-
computational efficiency further. It can be related to a feature selection. formation about the forming quality.
Finally, the extracted features are input into the full connection layers, For precise defects detection based on weld images during the dy-
which is similar with multiplayer perception (MLP) network. All layers namic welding process of Al alloy, several challenges are encountered
are trained simultaneously using a version of the back-propagation and are demonstrated in Fig.3.
learning algorithm. During the training progress, a CNN with para-
meters, = {kij, wij, bj} , is trained and optimized by minimizing its loss 1) Due to the instability of welding arc, arc light is dynamically
function, where cross entropy is used and can be described: changing even the acquisition time is settled. Therefore, arc light
might be quite strong and useful information about weld quality and
H (p , q ) = p (x )log q (x )
(4) defects can be submerged by the strong light interference, as shown
x
in Fig.3-a.
Where, H is the cross entropy, which can determine how close the ac- 2) Due to the complexity of the welding environment, it is difficult to
tual output is to the expected output. p is the expected output of sample keep the CCD camera and vision acquisition system in a constant
x, q is the actual output of sample x. position. Therefore, the captured images might have different
characteristics as shown in Fig.3-b, even miss certain information as
2.1. Experiment setup shown in Fig.3-d.
3) Gap between the weld plate might randomly change due to the as-
The experimental system includes multiple signal acquisition system, semble error or plate transformation. Thus, it leads to arc light
welding system and process controlling and motion system. The experi- leaking from the gap and disturbs the real-time weld image, as
mental system diagram and welding parameters can be found in literature shown in Fig. 3-c.
[3]. Both arc sound pressure and voltage signal were synchronously ac- 4) Reflection of arc light is another huge problem when welding Al
quired with the sampling rate of 40 kHz by a data acquisition card. The alloy, which can be found in Fig.3-b–d. The reflection obviously
welding system includes the TIG welding power with OTC INVERTER increase the complexity of detection samples.
ELESON 500P-type, CM-271 type wire feeder, water cold tank, welding
torch and 99.99% pure Argon as the shielding gas. For the control and 3. Method and results
motion system, a host computer using VC++ multithreading application
was utilized to control the welding parameters such as feed speed, time and 3.1. Datasets prepare
so on. It can also drive work piece to move with multiple freedoms while the
torch and the designed multisensory system remained quiescent. In this section, datasets of weld images used for training and testing
For the weld image acquiring, a three-way image acquisition system the CNN classification model was carefully introduced. As listed in
was developed to simultaneously collect the vision information from Table 1, three types of weld seam quality including under penetration

210
Z. Zhang, et al. Journal of Manufacturing Processes 45 (2019) 208–216

Fig. 2. Weld image acquiring for Al alloy in GTAW.

a)Full penetration b) Changing of acquisition angle.
c) Arc light leaking from gap d) Under penetration.

defects, normal penetration and burning through defects were obtained of the welding, while presetting gap can generate the burning through
by designing the rectangle-shaped, dumbbell-shaped and ladder-shaped defects. Moreover, different weld parameters were used such as welding
weld plates, which can be seen in Fig. 4. Besides, considering the ob- current, feeding speed in order to cover more welding conditions, in-
vious heat accumulation effect of Al alloy thin plate, there is no pre- crease the diversity of the datasets and then improve the generation
heating of plates, which caused the under penetration at the beginning ability of classification model.

Fig. 3. Different weld images acquired for Al alloy in GTAW.

a) Rectangle-shaped plate b) dumbbell-shaped plate c) ladder-shaped plate.

211
Z. Zhang, et al. Journal of Manufacturing Processes 45 (2019) 208–216

Table 1
Data preparing of different seam penetration.
Shape of plate Welding current Feeding speed Weld seam penetration

Rectangle shape 240A, 260A 10 mm/s,7 mm/s Under penetration Normal penetration Burning through
Dumbbell-shaped 195A, 205A, 230A, 10 mm/s,7 mm/s
Ladder-shape

3.2. CNN based on weld image 3.3. Activation and optimization

In this section, a CNN defects classification model for Al alloy in GTAW In this paper, Rectified Linear Unit (ReLU) function is used as the
was developed based on CNN and weld images. Its framework was carefully activation function for the CNN model both in convolution layer and
designed based on the feature of weld image as displayed in Fig.5. full connection layer. It enables the network to acquire a nonlinear
expression of the input signal to enhance the representation ability and
3.2.1. Convolution and pooling make the learned features more dividable. ReLU makes the weights in
The original size of the weld image was 1392*1040 pixel, which the shallow layer more trainable when using back-propagation learning
needs to be resized into 100*100 before inputting into the CNN model, method to optimize the parameters. Here, Adam (Adaptive Moment
which can be seen in Fig. 6. The original image was resized by per- Estimation) algorithm is used for the optimization in this paper. Its can
forming interpolation to down-size the images. For down-sampling, the be described as below:
arithmetic mean was applied. The data in the three channels have been
mt = µ mt + (1 µ) gt ,
analyzed and shows little difference. Therefore, the original images 1

have been transferred into grey-scale and used the kernel size with 5*5 nt = nt 1 + (1 ) (gt ) 2,
other than 5*5*3. As listed in Table 2, the kernel size for the first mt
mˆ t = ,
convolution layer (L1-C1) is 5 × 5 with 1stides, which means it will 1 µt
convolve by skipping one pixel in x-direction and y-direction. The nt
nˆt = ,
second convolution layer (L3-C2) is the same with the first one, while 1 t

the third (L5-C3) and fourth(L7-C5) convolution layer have the kernel mˆ t
t =
size of 3 × 3 in 3 dimension with 1stides. Each convolution layer is nˆt + (6)
followed by a pooling layer with the size of 2 × 2 in 3 dimension with 2
strides as in L2-P1, L4-P2, L6-P3 and L8-P4. In this paper, max pooling mt and nt are the first order moment estimation and the second order
mˆt
performs the local max operation over the input features from the last moment estimation of the gradient; ˆ
is an dynamic regularizer to
nt +
layer, which can reduce the data size and obtain location-invariant the learning rate; μis the momentum factor, ηis a global learning rate,
features. That means the pooling layer can reduce the size of the weld εis a constant to make sure the denominator is not zero.
image, moreover, it can extract the invariance feature to shift, scale and The output layer will use Softmax to calculate the probability of the
distortion. Thus, it is proper for the complex welding environment and sample belonging to each class. All the layers are trained simulta-
randomly changing weld images. The number of filters for each con- neously using a version of the back-propagation learning algorithm.
volution layer are listed in Table 2, which increased with the layer During the training progress, a CNN with parameters,
number in order to extract deeper feature of the input weld image. = {kij, wij, bj} , is trained by minimizing the cross-entropy loss function.
Specially, the loss function is defined as follow:
3.2.2. Fully connected network
The fully connected network is similar with the neural network, e f yi
Li = log
which can defined as follow. j
e fj

x lj = f x il 1
Wijl + bjl e zj
fi (z ) =
i Wj (5) k
e zk (7)

Where, W ij represents the weight connecting the ith neuron in l-1th Where fi is the softmax function and its output changes from 0 to 1. To
layer with the jth neuron of in lth layer. Wj represents the number of minimize the loss function, those weight tensors are first assigned as
neurons in l-1layer. After the last layer of pooling, the original size of random numbers near to zero.
100*100 is down to 6*6*128. 3-D maps is rasterized to 1-D vectors and For the initialization of the training model, normal distribution was
connected with each units in the following full connection layers L9 –F1 applied to generate the random number for model parameters.
and L10 –F2 with the neural number of 1024 and 512. The last layer is a And then the forward propagation is executed. Cross entropy is used
logistic-regression layer containing three outputs corresponding to the to determine how close the actual output is to the desired output, which
three class of welding defects. refers to under penetration, normal penetration and burning through.

Fig. 4. Welding seams with different penetration defects of Al alloy in GTAW.

212
Z. Zhang, et al. Journal of Manufacturing Processes 45 (2019) 208–216

Fig. 5. Architecture of CNN model of weld defects classifica-

tion.

Table 3
Datasets after data augmentation.
Datasets Under Full penetration Burning Total Data
penetration through number augmentation

Total 1200 2148 1416 4764 Noise addition

Training 960 1718 1132 3810 Rotation
Testing 240 429 283 952

Fig. 6. Comparison between the datasets with and without data augmentation.
batch size of 10. In order to stop iterating of the model, iteration number
is decided when the changing curve of classification rate is stable.Fig.6
Table 2
displayed the curves of test accuracy along with the iteration times for
Parameters for the CNN model of weld defects classification.
datasets with and without data augmentation. It can be clearly seen from
Layer Kernel size Stride Operator Number or method Fig. 7-a and b that the model has shown higher accuracy. By calculating
its mean value, the average test accuracy without DA is 92.3% while it is
L1-C1 5×5 1 convolution Filters number 32, strides(1,1)
L2-P1 2×2 2 pooling Max 96.18% for data with DA, which means the average accuracy was im-
L3-C2 5×5 1 convolution Filters number:64 Padding:same proved by 3.88% by means of the data augmentation technology.
L4-P2 2×2 2 pooling Max
L5-C3 3×3 1 convolution Filters number:128 Padding:same
3.5.2. Resize of input image
L6-P3 2×2 2 pooling Max
L7-C4 3×3 1 convolution Filters number:128 Padding:same
In this section, the size of the input image was investigated by re-
L8-P4 2×2 2 pooling Max sizing to 100*100, 200*200 and 300*300 from original size of
L9 –F1 – – full connected 1024 1390*1040 in terms of the test accuracy and test time of the CNN model.
L10–F2 – – full connected 512 The learning rate was set to 0.0015, while 20 of iteration times and batch
– – – activation ReLu
size of 10 were used with maxpooling. For resizing image of 200*200,
– – – loss softmax_cross_entropy
Output – – – 012 the feature dimension was 100*100*32, 50*50*64, 25*25*128 and
12*12*128 for the four layers respectively. For resizing image of
300*300, the feature dimension was 150*150*32, 75*75*64, 37*37*128
3.4. Data augmentation and 18*18*128 for the four layers respectively. Their test accuracy and
testing time for each sample were compared and displayed in Fig.7.
In this section, two kinds of data augmentation were applied to Test time was also calculated under the computer environment of
boost the original dataset including noise addition and rotation. For salt Inter (R) Core (TM) i7-8700 [email protected] GHz and 32 G memory in 64bits
and pepper noise, the noise density is considered as the percentage of of Windows 10 while the programming environment is Python.
the image noise area, which is set to 0.05. Gaussian white noise of mean Fig. 7-a and b display the test accuracy and time for different size of
0 and variance 0.01 was added to weld images. For normal penetration input image, from which it can be seen that resizing to 100*100 shows
sample, salt-pepper noise was added to each weld image. For the under the best performance in terms of the test classification accuracy and
penetration and burning through defects, both two types of noise were time efficiency. In comparison, the size of the input image might have
added in order to boost the defect samples. Besides, the images were more influence on the test time efficiency than the classification accu-
randomly rotated from 15 degrees to 30 degrees. Furthermore, the data racy. Both factors should be weighted because for the industry of ro-
in the three channels of the weld image have been analyzed. The value botic welding, especially the rapid manufacturing one, on-line detec-
from the three channels shows little difference. Therefore, the original tion efficiency is quite important as well as the inspection accuracy.
images have been transferred into grey-scale. The state of weld pene-
tration was manually offline identified based on the state of back seam 3.5.3. Filters of convolution layer
as seen in Fig.4. There is no formation of back seam for under pene- In this section, the filters number of convolution layer was carefully
tration while the back seam is burned through for burning through, investigated. Four different structures and number of convolution filters
which both weaken the strength of weld seam. The final datasets for were designed and compared regarding the test accuracy and test time
CNN defects classification model was given in Table 3, in which the of the CNN model. Besides the number change of number of convolu-
whole data was divided into two parts with certain proportion before tion filters, other parameters are the same with that in Table 2.
training. Specifically, 80% of the total data was used for training of the Fig. 8-a shows the mean statistics of the test accuracy for the four
CNN model while the rest is for model testing. different number of filters in each convolution layers. It can be seen that
structure of 32-64-128-128 has shown the best performance with the
3.5. Optimization of the CNN model highest mean test accuracy of 94% and lower standard deviation. For the
bigger number of filters of 64-128-256-256, although the test accuracy
3.5.1. Data augmentation and std showed the similar value with that of 32-64-128-128, the feature
Based on the framework of CNN in last section, the data augmenta- number might be excessive and redundant feature can decrease the
tion of noise addition was investigated regarding its influence to classi- performance the model, especially the testing time and efficiency. Jud-
fication accuracy of the model. In addition, the learning rate was set to ging from Fig. 8-b, the testing time for 10 samples increased with the
0.001 with maxpooling, resized to 100*100, 30 of iteration times and number of filter, which means that the test efficiency is greatly affected

213
Z. Zhang, et al. Journal of Manufacturing Processes 45 (2019) 208–216

Fig. 7. Comparison of input image size for the CNN model.

a) Mean test accuracy b) Testing time.

by the filters number in convolution layer. In addition, proper filter convolution-pooling algorithm. Therefore, the key of NOP is to find the
number is important for the accurate classification in order to extract images with non-zero pixels. Fig. 9 gives the example of NOP for image
appropriate number of image features for the specific classification features extraction for the CNN model designed in this paper. The output
purpose. For comprehensive consideration, 32-64-128-128 contributes dimension from first convolution-pooling layer is 100*100*32, therefore,
the best performance of CNN in terms of test accuracy and testing time. when doing NOP its output data needs to be reshaped into 100*100 with
the number of 32 for 2D weld feature images. For the fourth convolution-
3.6. Visualization of deep learning features pooling layer, the number of 2D weld feature image will be 128, NOP can
determine how much of them have the effective image feature and map
Number In this section, the deep learning image features from each the extracted feature back to the original weld images.
convolution-pooling layer were analyzed and visualized using the
proposed Non-zero Pixel (NOP) method in order to reveal the extraction 4. Quantitative comparing based on NOP
pattern of the CNN model and explain the physical meaning of the deep
learning features. Different number of filters in convolution layer were Table 4 lists the numbers of NOP feature for different framework of
quantitatively compared and the extracted image features in each layer CNN model in each convolution-pooling layer, from which it can be
were visualized based on NOP. seen that the structure of 32-64-128-128 owns the highest number of
NOP feature comparing with other network structures. This might
3.7. NOP contribute the highest classification accuracy (Fig. 8a) of CNN model
due to more features have been deeply extracted. It also means that the
NOP was proposed to quantitatively evaluate the features extracted more the filters is, the deeper the CNN can dig, the more the effective
in each convolution –pooling layer and then visualize the images with feature is. Besides, the NOP features for 64-128-256-256 is much
learned features from the CNN classification model. Bellowing is the smaller even through its classification accuracy is close to the former
details of the NOP method. one, which limits its performance because of the limited extracted
The key of NOP method is to find the images with non-zero pixels from feature. The other thing is that there are only few images owning useful
all the output data in each layer. First, the original output data needs to be information based on the NOP method as seen in Table 4. We have tried
reshaped into 2-dimension image with the size of w*l. Then, search every to use less or more filters to change and compare the model structure as
data point from the 2D image to find out if there is any pixel whose value displayed in Fig. 8-a, from which the mean test accuracy and standard
is zero and count the number of images with non-zero pixels. Based on the deviation have been compared. One thing is clear that the performance
careful reviewing all the feature data, it is found that most of the feature did not improved when less or more filters are used. In future, it might
value is zero providing little information of the original image, which be worth to investigate other CNN model structures such as RCNN to
means that nothing has been learned from the image using the focus on the main area of the image to increase the efficiency of CNN.

Fig. 8. Comparison between different filters numbers of convolution layer for the CNN model.

214
Z. Zhang, et al. Journal of Manufacturing Processes 45 (2019) 208–216

wire filler have been blurred due to the further convolution. However,
the different degree of arc light has been extracted from the top weld
pool, wire filler and back seam. In the third layer, the extracted features
were quite similar with each other and cannot be clearly distinguished
in naked eyes.
Finally, more deep features have been extracted with different
characteristics, which were marked in red rectangle in the fourth layer.
First, several bright areas with different degree of brightness was ex-
tracted as deep features, which were originated from the arc light or its
refection. Second, those bright areas either solely appears or combines
with each other, which demonstrates the complementary features. For
instance, the bright area from top weld pool and back seam were
combined into one feature while the top weld pool and wire filler were
also combined to form the new feature. Therefore, by means of visua-
lization of the deep learning features, those image features can be well
explained in terms of its physical meaning and extraction pattern.
Fig. 9. Extraction of image features based on NOP for the CNN model.
5. Discussions and conclusions
Table 4
Numbers of NOP features for different framework of CNN and pooling method. In this paper, a new on-line detection of weld defects for Al alloy in
Convolution layer 8-16-32- 16-32-64- 32-64-128- 64-128-256-
pulsed GTAW was proposed based on convolutional neural networks
64 128 256 256 and weld images. The image acquisition system developed in this paper
can capture the images of weld pool from different angles and provide
1stlayer 5 3 14 7 more abundant information for accurate defects classification.
2nd layer 5 3 3 8
3rd layer 0 8 79 12
However, it contains multiple mirror reflection and might be difficult to
4th layer 0 30 35 55 use in practical industry application. In future, the image acquisition
system will be more simplified to improve its practicability.
Two kinds of data augmentation methods were applied to boost the
4.1. Visualization of DL features based on NOP original dataset including noise addition and image rotation and has
improved the accuracy about 3.88%. Moreover, several key parameters
Fig. 10 demonstrated the learned deep image features from the CNN including filters of convolution layer and size of input image were op-
model with the convolution framework of 16-32-64-128 using max- timized for the CNN model. Then, a new CNN classification model
pooling for a full penetration weld image. NOP was utilized to select based on weld image with 11 layers was designed to identify weld seam
these effective features with non-zero pixels. penetration defects.
After convolution-pooling processing in the first layer in Fig. 10, it Comparing with our previous work [3], audible arc sound was
can be seen that the image has been enforced by denosing and contrast utilized to identify the defects of under penetration and burning
improving, especially for the back view of weld seam, which has shown through based on the proposed classification model SVM-GSCV, by
the higher brightness. Then, the second layer of convolution reserved means of which the test accuracy was changing from 81.52% to 98.46%
only one effective image feature, in which the weld seam, arc light and and the average accuracy of classification is 87.16%. In this paper the

Fig. 10. Visualization of deep learning features in the CNN model based on NOP.

215
Z. Zhang, et al. Journal of Manufacturing Processes 45 (2019) 208–216

same batch of experiment dataset were researched for the same de- [5] Song L, Huang W, Han X, Mazumder J. Real-time composition monitoring using
tection purpose using the designed CNN model based on weld images, support vector regression of laser-induced plasma for laser additive manufacturing.
Ieee Trans Ind Electron 2017;64:633–42.
which has shown the better performance with the mean accuracy of [6] Zhang Z, Yu H, Lv N, Chen S, Zhang Z, Yu H, et al. Real-time defect detection in
99.38% with the std value of 0.018. pulsed GTAW of Al alloys through on-line spectroscopy. J Mater Process Technol
Finally, visualization of the deep learning features in each con- 2013;213:1146–56.
[7] Huang Y, Wu D, Lv N, Chen H, Chen S. Investigation of porosity in pulsed GTAW of
volution layer was carried out using the proposed NOP method to ex- aluminum alloys based on spectral and X-ray image analyses. J Mater Process
plain the physical meaning of the deep learning features. It was found Technol 2017;243:365–73.
that the deep learning features were mainly dependent on the bright- [8] Chen B, Wang J, Chen S. A study on application of multi-sensor information fusion
in pulsed GTAW. Ind Rob 2010;37:168–76.
ness of arc light due to the strong arc light emission. Instead of de- [9] Wu D, Huang Y, Chen H, He Y, Chen S. VPPAW penetration monitoring based on
creasing or eliminating the interference from arc light as in traditional fusion of visual and acoustic signals using t-SNE and DBN model. Mater Des
research way, the CNN model has taken full use of those arc lights by 2017;123:1–14.
[10] Zhang Z, Chen S. Real-time seam penetration identification in arc welding based on
combining them in a various way to form the complementary features,
fusion of sound, voltage and spectrum signals. J Intell Manuf
which provides us a new and innovative solution for real-time mon- 2017;28(January):207–18.
itoring information processing and defect detection. [11] You D, Gao X, Katayama S. Multisensor fusion system for monitoring high-power
In future, more macro weld defects should be included and in- disk laser welding using support vector machine. IEEE Trans Industr Inform
2014;10(May):1285–95.
vestigated to expand the datasets and improve the robustness of the CNN [12] Liu Y, Zhang Y. Iterative local ANFIS-Based human welder intelligence modeling
model. It is worth to mention that the proposed defects classification and control in pipe GTAW process: a data-driven approach. Ieee/asme Trans
model might be limited to surface defects and can not detect the inner Mechatron 2015;20(June):1079–88.
[13] Liu YK, Zhang YM. Fusing machine algorithm with welder intelligence for adaptive
defects like porosity, cracks due to the limitation of the vision sensing. welding robots. J Manuf Process 2017;27(June):18–25.
Although, other types of sensing technologies, such as arc optical spec- [14] Liu YK, Zhang YM. Supervised learning of human welder behaviors for intelligent
trum, thermal infrared temperature might provide better solution to robotic welding. Ieee Trans Autom Sci Eng 2017;14(July):1532–41.
[15] He Y, Chen Y, Xu Y, Huang Y, Chen S. Autonomous detection of weld seam profiles
achieve on-line detection of inner defects. In addition, more effective via a model of saliency-based visual attention for robotic arc welding. J Intell Robot
optimization method is needed for multiple parameters of the CNN Syst 2016;81(March):395–406.
model. Other advanced method including R-CNN or Fast R-CNN might be [16] Calta NP, Wang J, Kiss AM, Martin AA, Depond PJ, Guss GM, et al. An instrument
for in situ time-resolved X-ray imaging and diffraction of laser powder bed fusion
considering to selectively search regions of interests of the image. additive manufacturing processes. Rev Sci Instrum 2018;89:055101.
[17] Cha Y-J, Choi W, Buyukozturk O. Deep learning-based crack damage detection
Acknowledgement using convolutional neural networks. Comput Civ Infrastruct Eng
2017;32(May):361–78.
[18] Acharya UR, Fujita H, Oh SL, Hagiwara Y, Tan JH, Adam M. Application of deep
The work was supported by National Natural Science Foundation of convolutional neural network for automated detection of myocardial infarction
China, No.51605372, China postdoctoral science foundation funding, using ECG signals. Inf Sci (Ny) 2017;415:190–8.
No.2018T111052, No.2016M602805, National Natural Science [19] Jia F, Lei Y, Lu N, Xing S. Deep normalized convolutional neural network for im-
balanced fault classification of machinery and its understanding via visualization.
Foundation of China, No. 51775409, the Program for New Century Mech Syst Signal Process 2018;110:349–67.
Excellent Talents in University (NCET-13-0461). [20] Dou Q, Chen H, Yu L, Zhao L, Qin J, Wang D, et al. Automatic detection of cerebral
microbleeds from MR images via 3D convolutional neural networks. IEEE Trans
Med Imaging 2016;35:1182–95.
References [21] Liu T, Bao J, Wang J, Zhang Y. A hybrid CNN–LSTM algorithm for online defect
recognition of CO2 welding. Sensors 2018;18:4369.
[1] Chen SB, Wu J. Intelligentized methodology for arc welding dynamical processes: [22] Shevchik SA, Kenel C, Leinenbach C, Wasmer K. Acoustic emission for in situ
visual information acquiring, knowledge modeling and intelligent control vol. 29. quality monitoring in additive manufacturing using spectral convolutional neural
Springer Verlag; 2008. networks. Addit Manuf 2018;21(May):598–604.
[2] Lv N, Xu Y, Li S, Yu X, Chen S. Automated control of welding penetration based on [23] Zhenhua C, Guoliang H, Chao L, Guo C. Automatic recognition of weld defects in
audio sensing technology. J Mater Process Technol 2017;250:81–98. TOFD D-Scan images based on faster R-CNN. J Test Eval 2018;48:14. pp.-14 pp.,
[3] Zhang Z, Wen G, Chen S. Audible Sound-based Intelligent Evaluation for Aluminum 2018.
Alloy in Robotic Pulsed GTAW: mechanism, feature selection and defect detection. [24] Fukushima K. Neocognitron: a self-organizing neural network model for a me-
IEEE Trans Industr Inform 2017. chanism of pattern recognition unaffected by shift in position. Biol Cybern
[4] Mirapeix J, Vila E, Valdiande JJ, Riquelme A, Garcia M, Cobo A. Real-time detection 1980;36:193–202.
of the aluminium contribution during laser welding of Usibor1500 tailor-welded [25] Bengio Y. Learning deep architectures for AI. Found Trendsâ® Mach Learn
blanks. J Mater Process Technol 2016;235:106–13. 2009;2:1–127.

216