Measurement 185 (2021) 110032

Contents lists available at ScienceDirect

Measurement
journal homepage: www.elsevier.com/locate/measurement

An online visual measurement method for workpiece dimension based on

deep learning
Xuebing Li a, b, c, Yan Yang a, b, c, Yingxin Ye d, Songhua Ma e, Tianliang Hu a, b, c, *
a
School of Mechanical Engineering, Shandong University, Jinan 250061 China
b
Key Laboratory of High Efficiency and Clean Mechanical Manufacture at Shandong University, Ministry of Education, Jinan 250061 China
c
National Demonstration Center for Experimental Mechanical Engineering Education at Shandong University, Jinan 250061 China
d
School of Mechanical and Electronic Engineering, Shandong Jianzhu University, Jinan 250101 China
e
Shenzhen Research Institute of Shandong University, Shenzhen 51800 China

A R T I C L E I N F O A B S T R A C T

Keywords: Image preprocessing and edge detection are the two most critical steps when applying machine vision to measure
Dimension measurement workpiece dimension in industrial field. The surface of the workpiece is usually covered with interference re­
Interference regions detection gions, which makes edge detection difficult. Aiming at the problem, the Full Convolutional Network (FCN) model
Edge detection
is used to detect interference regions of workpiece images, and then interference regions are processed by the
Deep learning
directional texture repair method. To avoid the influence of workpiece surface texture after rough machining, a
method based on Holistically-nested Edge Detection (HED) model for rough edge detection of workpiece images
is proposed. The edge is obtained by HED, and then it is post-processed based on the non-maximum value
suppression and double threshold segmentation methods in the Canny operator to obtain a refined edge image. In
order to further improve the accuracy of the dimension measurement, sub-pixel level edge detection accuracy is
achieved by cubic spline interpolation. Finally, the method is validated by a shaft workpiece. The measuring
accuracy of the outer diameter of the workpiece can reach 0.02 mm, which can effectively meet the requirements
of fast semi-finishing inspection. This research provides a common workpiece dimension measurement method in
industrial field and methodological guidance for the application of deep learning in industrial inspection.

1. Introduction the profile information of workpiece surface, laser measurement needs

to scan the whole surface, which takes a long time. In contrast, the
In industrial production, quality control determines whether a surface information of workpiece can be collected by a camera imme­
product meets the delivery requirements, and thus plays an important diately. The advantages of high precision and high speed makes machine
role along the process of production. Industrial measurement is one of vision more suitable for online measurement than laser measurement.
the most important quality control methods, including online mea­ Traditional machine vision measurement usually detects the edge of
surement and offline measurement. Online measurement superiors to workpieces after various image preprocessing methods, and finally
the offline measurement since it has advantages of automation, high transforms the distance between edge pixels into physical distance [4,5]
reliability, high precision, and low human influence. Among all the or the surface roughness [6] of workpieces. [4] proposed an on-line
types of online measurement, contact measurement is widely used machine vision to measure the rubber extrusions, and the measure­
because of its stability and anti-interference ability. However, it takes a ment uncertainty was constrained within 0.1 mm. [7] proposed a ma­
long time and affects the processing cycle, and only a few specific points chine visual algorithm and successfully realized the detection of
can be measured at a time. With the rapid development of science and machine tool wear, and the error was controlled 6.7% compared to
technology, non-contact measurement attracts more and more attention traditional manual detection. [8] introduced an image-based cross-cor­
due to its wide measuring range, fast measuring speed and low cost. relation analysis method to detect the machine tool wear at a microscale
Commonly used non-contact measurement technologies include down to 0.1 mm. In addition, it has high requirements on the quality of
laser measurement [1], machine vision [2,3] and so on. In order to get original images and the detection environment, which is the key to

* Corresponding author.
E-mail address: [email protected] (T. Hu).

https://doi.org/10.1016/j.measurement.2021.110032
Received 9 January 2021; Received in revised form 29 July 2021; Accepted 10 August 2021
Available online 14 August 2021
0263-2241/© 2021 Elsevier Ltd. All rights reserved.
X. Li et al. Measurement 185 (2021) 110032

ensure the success of measurement. However, obtaining high quality 0.8055 (individual connected components) on the proposed challenging
images in industrial field is difficult due to the harsh environment. In dataset [25]. Huang et al. proposed an intelligent Steel Wire Rope(SWR)
order to use machine vision technology in industrial field, the following damage detection method based on CNN [26], which can automatically
two problems must be solved: extract discriminant features by training surface images of the SWR and
Firstly, there may be interference regions such as cutting fluid and achieves a F1 score of 0.99. Zhong et al. proposed a three-stage auto­
chips on the surface of workpiece. The existence of interference regions matic defect inspection system for highspeed railways based on CNN
will produce noise edges, especially the cutting fluid at the bottom of [27]. Zhang et al. proposed a vision-based fusion method for defect
workpiece affects the integrity of edges. detection of milling cutter spiral cutting edge, and the precision was
Secondly, it is difficult to get clear edge image of workpiece by using 98.86% [28]. These application cases show the feasibility of applying
operator detection or threshold segmentation directly due to the rich CNN to industrial measurement.
texture on the workpiece surface after roughing. The contributions of this paper are as follows:
Detection of interference regions is similar to defect detection, which
can be achieved by thresholding method. Otsu proposed an automatic (1) FCN deep leaning model is used to detect interference regions of
thresholding method to solve the unstable problem of fixed threshold workpiece images, and then interference regions are processed by
[9]. But this method will fail if the histogram is unimodal or close to the directional texture repair method.
unimodal. Ng revised the Otsu method for selecting optimal threshold (2) To avoid the influence of workpiece surface texture after rough
values for both unimodal and bimodal distributions [10]. However, it is machining, a method based on HED deep leaning model for rough
still difficult to detect interference regions in the complex processing edge detection of workpiece images is proposed.
environment. In addition, it is also difficult to detect the edge with (3) The measurement accuracy of the proposed method can reach
traditional methods. Traditional method has poor robustness and it is 0.02 mm, which can effectively meet the requirements of fast
also very difficult to adjust parameters [11]. The reason why the tradi­ semi-finishing inspection.
tional method fails in the industrial field is that it cannot distinguish the
features to be extracted correctly. Feature engineering plays a very Thus, an online measurement method that utilizes CNN is proposed
important role in traditional machine vision, and feature selection and in this paper to solve the problems of interference detection and edge
extraction are often the core of problem. It is necessary to explore a detection in industrial field. The rest of this paper is organized as fol­
general online vision measurement method of workpiece dimension, lows. Section 2 introduces the general idea of the proposed online
which can automatically select image features. measurement method. Section 3 designs the detection and processing
Compared with traditional methods, the Artificial Intelligence (AI) method for interference regions detection using FCN model. Section 4
does not need to perform any preprocessing operations on the input designs the edge detection method using HED model. Then, this method
image. It can directly learn the mapping relationship between the input has been implemented and verified through an experiment of shaft
and the target. Research shows that the characteristics of AI make it have detection in Section 5. Finally, conclusions are discussed in Section 6.
good performance in image processing [12]. Support Vector Machine
(SVM) [13], Neural Network(NN) [14], and other algorithms are grad­ 2. Overview
ually introduced into image processing to replace manual feature
extraction. It can be seen that applying CNN to industrial measurement is
SVM is mainly applied to classification problems, such as character feasible, while for specific problems in the industrial measurement, new
recognition, face recognition, pedestrian detection, text classification models are derived from CNN.
and so on. Therefore, it is not suitable for detecting interference regions In the field of semantic segmentation, the Full Convolutional
and extracting edge images. Though SVM performs well in the case of Network (FCN) proposed by Jonathan Long has brought about signifi­
small samples, its computational cost is high and the calculation speed is cant changes [29]. This network changes the architecture of CNN, re­
slow, and the performance of SVM is gradually surpassed by NN. alizes the mapping of image to image. Semantic information refers to the
The structure of the original NN is simple and difficult to solve the content of image, including image texture, color, shape and so on. Image
complex image problems. As a kind of NN, Convolutional Neural Net­ semantics segmentation refers to annotating different information in
works (CNN) has achieved better prediction results than traditional image from the perspective of pixels. The process of interference regions
methods or even other deep network models on many computer vision detection is to segment interference regions from the background of the
problems [15]. In the early stage of model implementation, the archi­ workpiece images accurately. FCN can achieve better end-to-end
tecture of CNN is too simple to solve complicated problems. Lecun et al. training and image semantics segmentation. It can input the original
designed LeNet-5 and trained the network model using an error image, classify each pixel, and output a labeled semantics segmentation
gradient-based algorithm [16,17]. It has achieved great success in image. the FCN model has the following characteristics: (1) It has no
handwritten character recognition and laid a solid foundation for the limitation on the size of the input image and does not need any pre­
future development of CNN. Since then, deeper and more complex CNN processing operations. (2) It classifies each pixel and has scale and
network models such as AlexNet [18], VGG [19], GoogleNet [20], and rotation invariance. (3) the model architecture has the ability to learn
ResNet [21] have been proposed, and the classification accuracy on multi-level features. In addition, there are many successful applications
ImageNet [22] has been continuously improved. The CNN has entered based on FCN. Zhang et al. proposed a concrete crack detection method
the stage of extensive research and application. In the past few years, it based on improved FCN [30], which distinguishes background from
has been widely used in object detection, semantics segmentation and crack at the pixel level effectively. Zhou et al. propose an approach of
other image processing fields [23]. In addition, a considerable portion of combining fully convolutional network and conditional random field for
deep leaning focuses on Vision Based Measurement (VBM) systems and image semantic segmentation [31], which achieves improved accuracy
CNN are the most widely used deep learning technique, especially in compared with several other methods on Pascal VOC 2012 dataset.
VBM [24]. According to the above successful applications and characteristics, a
After the above analysis, CNN is more suitable for solving above preliminary conclusion can be drawn that the model is suitable for
mentioned problems of interference detection and edge detection. In interference regions detection.
addition, there are many specific applications of CNN in industrial On this basis, the research of CNN in semantics segmentation is also
detection. Zhou et al. proposed a novel end-to-end attention-based fully flourishing [32–34]. Based on the idea of end-to-end of FCN, Xie et al.
convolutional neural network framework for automated defect inspec­ designed a Holistically-nested Edge Detection(HED) model [35]. The
tion on specular surface, achieves F1 score over 0.7513 (pixel level) and model overcomes the shortcomings of traditional edge detection

2
X. Li et al. Measurement 185 (2021) 110032

Fig. 1. Flow chart of online visual measurement method based on deep learning.

methods, such as difficulty in parameter adjustment and poor robustness relatively simple, and the use of fine-tuning, these data sets can meet the
to noise. The edge obtained by HED model is complete, and the preliminary needs through experiments. After labeling, the FCN model
robustness to noise is strong. The model is also suitable for edge that can be used to correctly detect interference regions, was trained
detection. with original images as input and corresponding labeled images as
However, FCN and HED cannot be directly used in the field of in­ output. Finally, the directional texture restoration method was used to
dustrial measurement because they are general algorithms, which need restore the interference regions for subsequent edge detection.
to be optimized and improved for specific purpose. In addition, some At the step 4, aiming at the problem of rich texture on the surface of
auxiliary operations are also required to fit the application scenarios. workpiece, a general edge detection method was explored in this paper.
Many factors should be taken into consideration such as network Similar to the FNC model training, HED model was trained with original
structure, layer number, learning rate, etc. Besides, it is necessary to images as input and the corresponding edge images as output. After
determine a suitable termination conditions of iteration for the ideal workpiece edge is detected using HED model, it is post-processed based
result in practice. on the non-maximum suppression and double threshold segmentation
Under the guidance of theory and practice, the original neural methods in the Canny operator to obtain a refined edge image. Finally,
network was improved and added some auxiliary operations in this in order to improve the accuracy of dimension measurement, the sub-
paper to improve the accuracy. Our solutions to measure workpiece pixel edge algorithm is used to precisely locate the edge.
dimension are summarized in Fig. 1.
Firstly, the workpiece images are captured and the distortion 3. Detection and processing method for interference regions of
correction is carried out. Then, the interference region is detected and workpiece images
processed before extracting edge image. Finally, the diameter of the
outer circle is obtained through interpolation. The step 3 and step 4 are In the process of machining, workpiece surface is covered with
the focus of this paper, their general processes are introduced next and irregular interference regions such as cutting fluid and chips. To
the implementation methods will be introduced in Section 3 and Section improve the quality of edge detection, interference regions must be
4 subsequently. detected and processed firstly.
At the step 3, aiming at the problem of interference regions on
workpiece surface in the process of machining, the detection of inter­
3.1. Detection of interference regions of workpiece images based on FCN
ference regions was studied. In the actual processing environment, 500
workpiece images were collected for training FCN model. In addition,
According to the analysis in the introduction, it is feasible to use FCN
because the detection of interference regions and edge detection are
network to solve the problem of interference regions detection.

3
X. Li et al. Measurement 185 (2021) 110032

Fig. 2. The design drawing of the shaft workpiece.

(a) the input image of the FCN model. (b) the label image of the FCN model

Fig. 3. The workpiece image dataset of the FCN model.

Therefore, a method of interference regions detection based on FCN Cutting fluid and chips exist in the interference regions of the images.
model was proposed. MIT image labeling tool LabelMe [36] is used to improve the efficiency
of labeling. After labeling, the original image will be converted into the
3.1.1. Architecture construction of FCN model corresponding labeled image. As shown in the Fig. 3, each closed curve
FCN has a skip architecture, which fuses the feature hierarchy represents an interference region, the red label is cutting fluid and the
through combining deep semantic information and shallow appearance green label is a chip.
information. The model consists of seven convolution layers and five
pooling layers. The pooling operation will reduce the feature map size to 3.1.3. Model training of FCN
half of the original size, and the convolution operation does not change Fine-tuning refers to loading the weights of pre-trained model into
the feature map size. After five convolution and pooling operations, the current model, and then using private dataset to retrain the model, so as
size of the output feature map is reduced to 1/32 of the original image. to avoid overfitting and improve training efficiency. Through analyzing
In order to restore the size of the original image, an up-sampled oper­ the logic of FCN, the model has been trained with the idea of fine-tuning.
ation at stride 32 is performed on the output feature map, and each pixel The whole training process of the model is shown in Fig. 4. The FCN-32s
is classified to obtain a network model FCN-32s. Similarly, combining model that initialized by the weights of VGG16 [19] trained on Image­
predictions from both the final layer and the pool4 layer, the FCN-16s Net [22] was trained firstly, then load the weights of the FCN-32s into
model can be obtained at stride 16; and combining all predictions the FCN-16s and continue training, and finally load the weights of the
from pool3, pool4 and final layer, the FCN-8s model can be obtained at FCN-16s into the FCN-8s and finish training. At each stage, the model is
stride 8. trained end-to-end and initialized with the weights of earlier model.
The whole process of training can be divided into 3 steps, which are
3.1.2. Construction of image dataset used for training FCN introduced as follows.
The images of a shaft during processing were taken by industrial Step 1: Initialize and train FCN-32s model.
cameras at different times in the same environment and then constituted The VGG16 model trained on the ImageNet [22] was used as the pre-
dataset. The design drawing of the shaft is shown in Fig. 2. There are 500 trained model. For FCN-32s construction, the original weights are kept
workpiece images in the dataset, and 450 images were selected unchanged and all fully connected layers are replaced with convolution
randomly as training set of FCN, and the remaining 50 images were used layers. Then, FCN-32s model can be trained based on the constructed
as test set of trained FCN model. dataset.

4
X. Li et al. Measurement 185 (2021) 110032

Fig. 4. Training process of the FCN model.

Table 1
Training parameters for FCN.
Loss function Back propagation parameter Learning strategy Learning rate of FCN-32s Learning rate of FCN-16s Learning rate of FCN-8s

Softmax function SGD optimization algorithm Fixed le− 10 le− 12 le− 14

accuracy
accuracy

accuracy
accuracy

Fig. 5. The evaluation indicators of FCN-32s, FCN-16s and FCN-8s.

5
X. Li et al. Measurement 185 (2021) 110032

Table 2 As shown in the Fig. 6, FCN-8s model also achieves the best perfor­
The evaluation indicators of FCN-32s, FCN-16s and FCN-8s with 5000 epochs. mance from the illustration intuitively. With the continuous forward
Models PA MPA MIoU FWIoU integration of network models, the detected interference regions are
more standardized and more accurate.
FCN-32s 0.9872 0.6038 0.5763 0.9761
FCN-16s 0.9921 0.7024 0.6531 0.9851
FCN-8s 0.9940 0.8685 0.8271 0.9884 3.2. Processing of interference regions of workpiece images

Interference regions will affect the edge detection of workpiece,

Step 2: Initialize and train FCN-16s model. especially the cutting fluid accumulate in the workpiece contour easily,
According to the architecture of FCN, it is not necessary to recon­ which destroys the contour integrity of the workpiece images.
struct the convolution layer, so the weights in FCN-32s can be directly Image inpainting is not the focus of this paper because the perfor­
updated to FCN-16s. Then, train FCN-16s with the constructed dataset. mances of existing image inpainting methods can meet the re­
Step 3: Initialize and train FCN-8s model. quirements. Telea et al. proposed an image inpainting technique based
Similar to Step 2, update the weights of FCN-16s to FCN-8s and train on the Fast Marching Method(FMM) [38], which produces nearly
FCN-8s with the constructed dataset. identical results to more complex, and usually slower known methods.
As shown in Table 1, these training parameters achieved the best Sapiro et al. proposed an approach based on the Navier-Stokes(NS)
results after many experiments. equations for fluid dynamics [39], which has the immediate advan­
tage of well-developed theoretical and numerical results. Wang et al.
3.1.4. Model assessment of FCN proposed a directional texture image inpainting algorithm, which can
After the model training is completed, the performance of the FCN repair directional texture image well [40].
model needs to be evaluated. The image semantic segmentation field As shown in the Fig. 7, for the kind of workpiece images with very
often uses these four evaluation indicators to measure the algorithm rich surface texture, the FMM and NS image inpainting algorithms have
[37]: Pixel Accuracy (PA), Mean Pixel Accuracy (MPA), Mean Inter­ not achieved good results after the experiment. The image inpainting
section over Union (MIoU), Frequency Weighted Intersection over algorithm proposed by Wang has a very good performance. Therefore, it
Union (FWIoU). As shown in the Fig. 5. was selected to process interference regions of workpiece images.
MPA is the standard measure indicator of image semantics segmen­
tation, but FWIoU is more convincing because the amount of cutting 4. Edge detection method for workpiece images
fluid is much larger than the amount of chips in the selected images.
After analysis, FCN-8s achieves the best performance when iteration Through the previous section, the irregular interference regions have
times is 50000, therefore it was chosen as the final model. In addition, been repaired. The repaired image eliminates the influence of interfer­
the batch size of model training is 45, so 10 iterations is an epoch. The ence regions and can be used for edge detection. After edge detection,
evaluation indicators of each model with 5000 epochs are shown in the diameter of the shaft can be fitted.
Table 2.

(a)original image. (b) result of FCN-32s model. (c) result of FCN-16s model. (d) result of FCN-8s model.

Fig. 6. Detection results of different FCN models.

(a) Original image (b) The image processed by the FMM (c) The image processed by the NS (d) The image processed by the algorithm

algorithm[38] algorithm[39] proposed by Wang [40]

Fig. 7. Processing results of different image inpainting algorithms.

6
X. Li et al. Measurement 185 (2021) 110032

Fig. 8. Rough edge detection method of workpiece images based on HED.

(a) the input image of the HED model. (b) the label image of the HED model.

Fig. 9. The workpiece image dataset of the HED model.

4.1. Rough edge detection of workpiece images based on HED

Table 3
Training parameters for improved HED.
HED model is a general edge detection model improved on the basis
of FCN and VGG model [35]. The edge obtained by HED model is Loss Back Learning Learning Learning Learning
function propagation strategy rate of rate of rate of
complete, and its robustness to noise is strong. This model has reached
parameter FCN-32s FCN-16s FCN-8s
the human level in the standard benchmark test.
However, there is still a problem of edge blurring when HED model is Weighted SGD Fixed le-10 le-12 le-14
cross optimization
used to extract the edge of the workpiece from images. To solve this entropy algorithm
problem, the following two improvement methods are proposed. Firstly, function
improve the training method of HED model. In the process of backward
propagation of the traditional training, the feature maps output at each
scale are involved in the calculation of the loss value. While in this Fig. 9, the model takes processed workpiece images as input, and out­
paper, only the fused images are used to calculate the loss values. Sec­ puts the hand-labeled edge of the corresponding workpiece images.
ondly, the edge images that output by the modified HED model are
further processed by the non-maximum suppression and double 4.1.2. Characteristics and training of improved HED
threshold segmentation algorithms of Canny operator. HED is a general edge detection model and owns a multi-scale and
The flow of rough edge detection is as follows (see Fig. 8). multi-level network architecture. It can achieve end-to-end training.
In the scene of original paper [35], images are usually complex with
4.1.1. Construction of image dataset used for training HED many edges to be detected. If only the loss value of the last layer is
After interference region processing, the repaired images constituted calculated for training, some edges will not be detected due to the lack of
the dataset, which are used for training HED model. 500 workpiece deep information. The global nesting architecture takes all levels of in­
images after processing interference regions are used as dataset. 450 formation into account to ensure the image quality. While for the scene
images were selected as training set of HED model at random, and the of industrial measurement, images are usually simple with a few regular
remaining 50 images were used as test set of HED model after training. edges of workpiece. And the global nesting architecture will blur the
For the application scenario of industrial measurement, it is necessary to edge because it takes too much deep information into account. In order
make the edge image of the workpiece as a class label. As shown in to make HED more suitable for industrial measurement, only the fused
images are used to calculate the loss values in the paper. Compared with

7
X. Li et al. Measurement 185 (2021) 110032

From the perspective of pixel gray, the edge of workpiece is the place
where the gray value of the image pixel changes the most. Non-
maximum suppression aims at suppressing the points where the
gradient of the pixels in the local region is not the maximum, so that
some non-edge points can be removed. The edges obtained by non-
maximum suppression method are further refined, but there are still
some false edges.

(2) Double threshold segmentation

Double threshold segmentation algorithm was used to further detect

and connect edges of the image. In this paper, the threshold obtained by
Otsu algorithm [9] is used as the high threshold T1 , and the 1/2 of T1 is
used as the low threshold T2 . The pixel type can be obtained by the
formula (1).
⎧
⎨ strong edge pixel t > T1
pixel = weak edge pixel T1 > t > T2 (1)
Fig. 10. The loss functions of HED models. (loss1 represents the loss function ⎩
suppressed edge pixel t < T2
value of the original HED model. loss2 represents the result after optimizing.)
All edges in a strong edge image are recognized as real edges. For the
the original HED model, the edge of the workpiece images detected by pixels in a weak edge image, they are retained if there are strong edge
the improved model is finer and the location is more accurate. pixels in its eight neighborhoods. In this way, the broken strong edges
As shown in Table 3, these training parameters achieved the best
results after many experiments.

4.1.3. Model assessment of improved HED

As shown in the Fig. 10, the value of loss function decreases and
converges gradually, indicating that the edge of workpiece images
generated by HED model approximates the edge of the labeled image
gradually.
According to Fig. 10, the loss value decreases before 9000 iteration
times and fluctuates steadily after 9000 iteration times. In other words,
the model with 900 epochs is chosen as the final detection model. As
shown in the Fig. 11, It can be seen that the edge of the workpiece image
detected by the improved HED model is more accurate and clearer.

4.1.4. Image post-processing

The edge detection model of workpiece images based on HED can
obtain complete edges and less interference contours. To further
improve the accuracy of the edge, the non-maximum suppression and
double threshold segmentation in Canny operator detection are adopted.

(1) Non-maximum suppression

Fig. 12. Edge detection result after post-processing.

(a) the result of the original HED model. (b) the result after optimizing.

Fig. 11. Edge detection results of workpiece images.

8
X. Li et al. Measurement 185 (2021) 110032

Table 4 5.1. Experimental design

X-direction sub-pixel coordinate.
Number of Rough edge Rough edge Sub-pixel In order to verify the effectiveness of the method, the results of co­
pixels ordinate abscissa abscissa ordinate measuring machine were selected as the benchmark and
1 50 130 129.067612 compared with the measurement results from experiment.
2 50 338 338.034850 Components of the image acquisition system were selected and
installed in the CNC lathe to build a hardware platform. Industrial
camera model is MV-CA050-10GM of HIKVISION, lens model is
Table 5 MF2520M-5MP of HIKVISION and light source is LED annular light.
Y-direction sub-pixel coordinate. In order to ensure the image quality, the camera system and the light
source system are required to be coaxial with the workpiece. Therefore,
Number of Rough edge Rough edge Sub-pixel
pixels abscissa ordinate ordinate the camera bracket was designed according to the tailstock structure,
which can be directly embedded in the tailstock of CNC machine tool.
1 100 81 80.060291
2 100 153 152.461962
The light source can be directly coaxially mounted on the lens. The
3 100 200 199.149723 relative position of the industrial camera, the workpiece and the light
4 100 278 277.51657 source are shown in Fig. 13.

can be connected. As shown in Fig. 12, the rough edge image of the 5.2. The implementation of measurement
workpiece is more accurate and clearer after post-processing.
The proposed methods are applied to online measurement of shaft.
4.2. Sub-pixel edge detection of workpiece images After the rough machining of workpieces, the vision system will collect
the image and give the outer circle dimension to determine whether the
In industrial measurement, it is necessary to precisely locate sub- workpiece dimension meets the requirements of the next semi-finishing.
pixel edges because edge localization at the pixel level cannot meet The workpiece images are collected in the CNC lathe processing of a
the accuracy requirements. In this paper, the cubic spline interpolation shaft.
[41] method is used to obtain sub-pixel edges, which has characters of Camera calibration is an indispensable part of online visual mea­
simple calculation and high accuracy. The edge pixels of the image after surement of workpiece dimension, which should be done before image
rouge edge detection were used to calculate sub-pixel edges. For an edge collecting. The measurement system can obtain internal parameters by
pixel, its subpixels in X-direction can be obtained by selecting some calibration and correct the distortion caused by manufacturing process
pixels in X-direction for cubic spline interpolation. It’s the same thing for to reduce the influence of image distortion on dimensional measurement
Y-direction. Finally, the sub-pixels of the image are calculated. accuracy. The internal parameters of the camera are obtained by
Some sub-pixel interpolation data were listed. The abscissa edge is OpenCV, and the distortion of the image is corrected. Then the physical
accurately positioned with the ordinate of 50, and the corresponding space is calibrated by high precision standard blocks to calculate the
data is shown in Table 4. While the ordinate edge is accurately posi­ pixel equivalent, which can be obtained from the following equation:
tioned with the abscissa of 100, and the corresponding data is shown in a
Table 5. k= (2)
r
So far, an accurate sub-pixel edge image of workpieces has been
obtained. Next, the obtained sub-pixel edge coordinates were fitted by where a is the actual dimension of the workpiece and r is the number of
the least square method to calculate the workpiece outer circle diameter. pixels. The unit of pixel equivalent k is mm/pixel. Its physical meaning is

5. Measurement implementation and verification Table 6

Calibration results of a standard block of 30 mm.
A shaft in the practical industrial production environment was Number of Physical Number of Pixel
measured as a case. Firstly, the experiment design is introduced, and measurements dimensions pixels equivalent
then the implementation method of measurement is explained. Finally, 1 30.02 882.26 0.034026
the measurement results of the shaft are analyzed. 2 30.00 882.10 0.034010
3 30.00 882.11 0.034009
4 30.01 882.20 0.034017
5 30.02 883.01 0.034000

Fig. 13. Hardware platform of the online visual measurement system.

9
X. Li et al. Measurement 185 (2021) 110032

(a) original image. (b) the repaired image. (c) the result of this paper. (d) the result of traditional method[42].

Fig. 14. The results of experimental image.

relative errors, and the results are listed in Table 7.

Table 7
The measurement results of the workpiece outer circle diameter. Δ = x− μ (4)
Number of The The result of The Absolute Relative
measurements result traditional result of error error Δ x− μ
δ= = (5)
of method [42] this μ μ
CMM paper

1 44.89 44.866 0.024 0.053% where x is an observed value, μ is the true value, Δ is the absolute error, δ
2 44.910 0.020 0.045% is the relative error.
3 – 44.868 0.022 0.049% As shown in the Table 7, the maximum absolute error of the diameter
4 44.867 0.023 0.051% of the shaft is 0.024 mm and the relative error is 0.053%. The result
5 44.912 0.022 0.049%
meets the requirement of semi-finishing.

the actual dimension of a pixel on the image. 6. Discussion and conclusion

A standard block of 30 mm was measured five times and the result of
measurements is shown in Table 6. The average value of multiple In this paper, an intelligent online visual measurement system based
measurements is 0.034012 mm/pixel, which will be used in subsequent on deep learning was developed to solve problems of interference re­
experiments under the same physical environment. gions processing and edge detection in industrial measurement.
After the calibration, the workpiece can be measured automatically. According to the above, the method in this paper has achieved good
Firstly, set internal calibration parameters of the camera and pixel results in each stage. Compared with FMM and NS methods, the method
equivalent. Then, the camera will acquire the workpiece images after selected in this paper can remove the interference regions effectively.
the CNC machine tool finished the current program. Finally, the work­ Compared with the traditional edge detection method, the improved
piece images are processed by a series of methods mentioned above, the HED method can effectively detect the edge of the workpiece and ach­
dimension of the workpiece is measured according to the pixel ieve the measurement accuracy of 0.02.
equivalent. Deep learning has been successfully applied to the field of industrial
measurement. In addition, it provides methodological guidance for the
application of deep learning in industrial measurement. Future work
5.3. Analysis of experimental results will continue to explore new solutions based on this method, expand the
scope of application, reduce errors and improve measurement accuracy.
As shown in the Fig. 14(a), there are two pieces of cutting fluids and The detection of the surface finish quality will also be studied in the
two pieces of chips on the workpiece. The dimension of the shaft is future.
measured by the methods presented in this paper. After the detection
and processing of interference regions on the workpiece surface, the CRediT authorship contribution statement
image is shown in Fig. 14(b). Both cutting fluids and chips on the
workpiece surface have been removed. The repaired image does not Xuebing Li: Formal analysis, Investigation, Writing – original draft.
leave obvious voids and the traces of restoration were slight. The image Yan Yang: Software, Investigation. Yingxin Ye: Supervision, Writing –
is detected by the HED model and post-processed to obtain a rough edge review & editing. Songhua Ma: Project administration. Tianliang Hu:
image as shown in Fig. 14(c). The edge image has a complete contour Conceptualization, Methodology, Writing – review & editing.
without noise edges such as textures. As shown in the Fig. 14(d), it is the
result of traditional edge detection methods [42]. It is difficult or
Declaration of Competing Interest
impossible to measure the outer circle of the workpiece.
Based on cubic spline function, the edges shown in Fig. 14(c) are
The authors declare that they have no known competing financial
interpolated. The obtained sub-pixel edge coordinates are fitted by the
interests or personal relationships that could have appeared to influence
least square method to obtain the equation of circle, which are as
the work reported in this paper.
follows:

(a − 801.897)2 +(b − 1133.7)2 = 659.5612 (3) Acknowledgement

where the unit is the number of pixels. This research is supported by Major Projects for Scientific and
The number of pixels of the workpiece outer circle diameter was Technological Innovation of Shandong Province (Grant
obtained by five repeated experiments and was converted into the actual no.2019JZZY010123-1); Innovative Development Project of Industrial
dimensions by the pixel equivalent. It is considered that the result of Internet, 2019(Construction of Innovative Application Experiencing
CMM is the true value and the result of this paper is the observed value. Center of Industrial Internet); the Science and Technology Innovation
The following two formulas (4)–(5) are used to calculate absolute and Program of Shenzhen Municipality (Grant no.

10
X. Li et al. Measurement 185 (2021) 110032

JCYJ20180301171337648). [22] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, F.F. Li, ImageNet: a Large-Scale
Hierarchical Image Database, 2009, pp. 248–255.
[23] R. Girshick, J. Donahue, T. Darrell, J. Malik, Rich feature hierarchies for accurate
References object detection and semantic segmentation, in: Proceedings of the IEEE Computer
Society Conference on Computer Vision and Pattern Recognition, 11/11 2013.
[1] D. Ding, Z. Zhao, X. Zhang, Y. Fu, J. Xu, Evaluation and compensation of laser- [24] M. Khanafer, S. Shirmohammadi, Applied AI in instrumentation and measurement:
based on-machine measurement for inclined and curved profiles, Measurement 151 the deep learning revolution, IEEE Instrum. Meas. Mag. 23 (2020) 10–17.
(2020). [25] Q. Zhou, R. Chen, B. Huang, W. Xu, J. Yu, DeepInspection: deep learning based
[2] S. Dworkin, T. Nye, Image processing for machine vision measurement of hot hierarchical network for specular surface inspection, Measurement 160 (2020).
formed parts, J. Mater. Process. Technol. 174 (2006) 1–6. [26] X. Huang, Z. Liu, X. Zhang, J. Kang, M. Zhang, Y. Guo, Surface damage detection
[3] A. Glowacz, Ventilation diagnosis of angle grinder using thermal imaging, Sensors for steel wire ropes using deep learning and computer vision techniques,
21 (8) (2021) 2853. Measurement 161 (2020).
[4] A. Pietrosanto, An on-line stereo-vision system for dimensional measurements of [27] J. Zhong, Z. Liu, Z. Han, Y. Han, W. Zhang, A CNN-based defect inspection method
rubber extrusions, Measurement 35 (3) (2004) 221–231. for catenary split pins in high-speed railway, in: IEEE Transactions on
[5] R.C. Crida, G. de Jager, An approach to rock size measurement based on a model of Instrumentation and Measurement, 10/06 2018.
the human visual system, Miner. Eng. 10 (10) (1997) 1085–1093. [28] T. Zhang, C. Zhang, Y. Wang, X. Zou, T. Hu, A vision-based fusion method for
[6] S. Ghodrati, M. Mohseni, S. Gorji Kandi, Application of image edge detection defect detection of milling cutter spiral cutting edge, Measurement 177 (2021)
methods for precise estimation of the standard surface roughness parameters: 109248.
Polypropylene/ethylene-propylene-diene-monomer blend as a case study, [29] J. Long, E. Shelhamer, T. Darrell, Fully convolutional networks for semantic
Measurement 138 (2019) 80–90. segmentation, Arxiv, vol. 79, 11/01 2014.
[7] J. Yu, X. Cheng, L. Lu, B. Wu, A machine vision method for measurement of [30] J. Zhang, C. Lu, J. Wang, L. Wang, X.-G. Yue, Concrete cracks detection based on
machining tool wear, Measurement (2021) 109683. FCN with dilated convolution, Appl. Sci. 9 (2019) 2686.
[8] K.M. Fong, X. Wang, S. Kamaruddin, M.-Z. Ismadi, Investigation on universal tool [31] H. Zhou, J. Zhang, J. Lei, S. Li, D. Tu, Image Semantic Segmentation based on FCN-
wear measurement technique using image-based cross-correlation analysis, CRF Model, 2016, pp. 9–14.
Measurement 169 (2021) 108489. [32] V. Badrinarayanan, A. Kendall, R. Cipolla, SegNet: a deep convolutional encoder-
[9] N. Otsu, A threshold selection method from gray-level histograms, IEEE Trans. Syst. decoder architecture for image segmentation, IEEE Trans. Pattern Anal. Mach.
Man Cybernet. 9 (1) (2007) 62–66. Intell. 39 (12) (2017) 2481–2495 (in English).
[10] H.F. Ng, Automatic thresholding for defect detection, Pattern Recogn. Lett. 27 [33] A.L. Yuille, DeepLab: semantic image segmentation with deep convolutional nets,
(2006) 1644–1649. atrous convolution, and fully connected CRFs, IEEE Trans. Pattern Anal. Mach.
[11] M. Sonka, V. Hlavac, R. Boyle, Image processing, Anal. Mach. Vision (1995). Intell. 40 (4) (2018) 834.
[12] Y. LeCun, Y. Bengio, G. Hinton, Deep learning, Nature 521 (7553) (2015) 436–444. [34] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, SSD: Single Shot MultiBox
[13] F. Meng, W. Lin, Z. Wang, Space edge detection based SVM algorithm, in: Artificial Detector, 12/07 2015.
Intelligence and Computational Intelligence, Berlin, Heidelberg, Springer Berlin [35] S. Xie, Z. Tu, Holistically-nested edge detection, Int. J. Comput. Vision 125 (2017)
Heidelberg, 2011, pp. 656–663. 1–16.
[14] S.H. Ong, Edge detection using a neural network, Pattern Recogn. 27 (12) (1994) [36] B.C. Russell, A. Torralba, K.P. Murphy, W.T. Freeman, LabelMe: a database and
1653–1662. web-based tool for image annotation, Int. J. Comput. Vision 77 (1–3) (2007)
[15] A. Kusiak, Convolutional and generative adversarial neural networks in 157–173 (in English).
manufacturing, Int. J. Prod. Res. 58 (5) (2019) 1594–1604. [37] A. Garcia-Garcia, S. Orts, S. Oprea, V. Villena Martinez, J. Rodríguez, A Review on
[16] B.L. Lecun, Gradient-based learning applied to document recognition, Proc. IEEE Deep Learning Techniques Applied to Semantic Segmentation, 04/22 2017.
86 (11) (1998) 2278–2324. [38] A. Telea, An image inpainting technique based on the fast marching method,
[17] Y. Lecun, L. Bottou, Y. Bengio, P. Haffner, Gradient-based learning applied to J. Graph. Tools 9 (2004) 01/01.
document recognition, Proc. IEEE 86 (1998) 2278–2324. [39] M. Bertalmio, A. Bertozzi, G. Sapiro, Navier-Stokes, Fluid Dynamics, and Image
[18] A. Krizhevsky, I. Sutskever, G. Hinton, ImageNet classification with deep and Video Inpainting, 2001, pp. I–355.
convolutional neural networks, Neural Inform. Process. Syst. 25 (2012). [40] J.T. Zhongqing, W.H. Lei, Algorithm of completion for strong computer
[19] K. Simonyan, A. Zisserman, Very Deep Convolutional Networks for Large-Scale engineering an applications, Comput. Eng. Appl. 42 (34) (2006) 29–32.
Image Recognition, arXiv 1409.1556, 09/04 2014. [41] S. McKinley, M. Levine, Cubic spline interpolation, Coll. Redw. 45 (1999) 02/28.
[20] C. Szegedy et al., Going Deeper with Convolutions, 09/16 2014. [42] J.F. Canny, A computation approach to edge detection, IEEE Trans. Pattern Anal.
[21] S. Jian, Deep Residual Learning for Image Recognition, in: IEEE Conference on Mach. Intell. 8 (1986) 769–798.
Computer Vision & Pattern Recognition, 2016.

11