See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/360881323

Enabling deep learning using synthetic data: A case study for the automotive
wiring harness manufacturing

Article in Procedia CIRP · January 2022

DOI: 10.1016/j.procir.2022.05.142

CITATIONS READS

13 530

3 authors, including:

Huong Giang Nguyen Jörg Franke

Friedrich-Alexander-University of Erlangen-Nürnberg Friedrich-Alexander-University of Erlangen-Nürnberg
25 PUBLICATIONS 155 CITATIONS 1,149 PUBLICATIONS 8,449 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Huong Giang Nguyen on 06 September 2022.

The user has requested enhancement of the downloaded file.

 Available online at www.sciencedirect.com
Available online at www.sciencedirect.com
ScienceDirect
ScienceDirect
Available online at www.sciencedirect.com
Available online
Procedia CIRP at www.sciencedirect.com
00 (2022) 000–000
Procedia CIRP 00 (2022) 000–000
ScienceDirect
ScienceDirect
www.elsevier.com/locate/procedia
www.elsevier.com/locate/procedia

ProcediaProcedia
CIRP 00CIRP
(2017)
107000–000
(2022) 1263–1268
www.elsevier.com/locate/procedia

55th CIRP Conference on Manufacturing Systems

55th CIRP Conference on Manufacturing Systems
Enabling deep learning using synthetic data:
Enabling deep learning using synthetic data:
A case study 28thforCIRP the automotive
Design Conference,wiring May 2018, harness manufacturing
Nantes, France
A case study for the automotive wiring harness manufacturing
A new methodology Huong to Giang Nguyenthe
analyze
a,
*, Resul
functional Habiboglu and
a
, Jörg Frankeaa architecture of
physical
Huong Giang Nguyen *, Resul Habiboglu , Jörg Franke
a, a

existing products for an assembly oriented

Fuerther Straße(FAU),
Friedrich-Alexander-Universität Erlangen-Nürnberg
a 246b, 90429 Nuernberg,
Institute
product
Germany
for Factory
family identification
Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), Institute for Factory Automation and Production Systems (FAPS),
a

Automation and Production Systems (FAPS),

Fuerther Straße 246b, 90429 Nuernberg, Germany
* Corresponding author. Tel.: +49 911 5302 9115. E-mail address: [email protected]
Paul Stief *, Jean-Yves Dantan, Alain Etienne, Ali Siadat
* Corresponding author. Tel.: +49 911 5302 9115. E-mail address: [email protected]
École Nationale Supérieure d’Arts et Métiers, Arts et Métiers ParisTech, LCFC EA 4495, 4 Rue Augustin Fresnel, Metz 57078, France
Abstract
*Abstract
Corresponding author. Tel.: +33 3 87 37 54 30; E-mail address: [email protected]
The wiring harness manufacturing follows a high-variant manufacturing philosophy with a high degree of manual labor to produce customized
The wiringTherefore,
products. harness manufacturing
the process step follows a high-variant
of optical inspection manufacturing philosophy
is crucial to monitor, with
assess, andaproactively
high degreesafeguard
of manualthe labor to produce
quality customized
of produced wiring
products.
harnesses.Therefore,
To automate the process
the optical step inspection,
of optical inspection
which is is crucial to
currently monitor,conducted
manually assess, andby proactively
operators,safeguard the quality
deep learning of produced
has become wiring
a powerful
Abstract
harnesses. To automate the
algorithm outperforming optical computer
traditional inspection,vision
whichapproaches.
is currentlyDeepmanually conductedautomated
learning-based by operators, deep
optical learningsystems
inspection has become a powerful
can robustly and
algorithm outperforming
reliably detect traditional components,
rigid and deformable computer vision suchapproaches.
as connectors, Deep learning-based
clips, single wires, automated optical inspection
and wire bundles. However, the systems can robustly
bottleneck and
for scalable
Inreliably
today’s
deep business
detect
learning rigid environment,
and deformable
solutions the components,
in the industrialtrend towards more
such
environment product
asand highvariety
connectors, and
clips,
deep customization
single
learning wires,
model and iswire
unbroken. is Due
bundles.
performance to this the
However,
the database development,
bottleneck
for the needand
of
for scalable
model training
agile
deepand reconfigurable
learning
optimization. To addressproduction
solutions in the
this systems
industrial
research emerged
proposetoa and
gap, environment
we copehigh
deep with deep
various products
learning
learning-based and performance
model
data processingproduct families.
pipeline Todatabase
design
is automated
for the and
for optimize
optical production
model training
inspection and
of wiring
systems
harnessesas using
optimization.wellToas toand
real choose
address this the
syntheticallyoptimal
research gap,product matches,
we propose
generated point a deep
clouds. product analysis
learning-based
This paper methods
data the
outlines are needed.
processing
process of realIndeed,
pipeline most ofdata
forsynthetic
and automated the known
optical methods
inspection
generation ofaim to
wiring
and evaluates
analyze
harnesses
the a product
potentialusing or one
real andproduct
of synthetic data tofamily
synthetically on thedata
enrichgenerated
real physical
point level.training.
clouds.
for model Different
This paperproduct
The families,
outlines
data however,
the process
processing may
of real
pipeline differ
and largely
synthetic
is implemented inexperimental
data
and terms of theand
generation number and
evaluates
findings are
nature of components.
the potential
generated of synthetic
to deduct This
important fact
data to impedes
enrich real
parameters an efficient
fordata comparison
for model
synthetic data training. and
generationThechoice
anddata oflearning
appropriate
processing
deep product
pipeline
model family
whichcombinations
is implemented
training modelfor
and experimental
impact the findings
production
performance. are
system.
generatedA new methodology
to deduct important is parameters
proposed tofor analyze existing
synthetic products inand
data generation view of their
deep functional
learning and physical
model training whicharchitecture.
impact model Theperformance.
aim is to cluster
these products
© 2022
2022 in new assembly
The Authors.
Authors. Publishedoriented
by Elsevierproduct families
ELSEVIER for the optimization of existing assembly lines and the creation of future reconfigurable
© The Published by B.V.B.V.
assembly
This
© is
2022 ansystems.
Theopen Based
access
Authors. on Datum
article
Published under
by Flow
the Chain,
CC
ELSEVIER the physical
BY-NC-ND
B.V. structure
license of the products is analyzed. Functional subassemblies are identified, and
(https://creativecommons.org/licenses/by-nc-nd/4.0)
This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0)
aPeer-review
functional
This
Peer-review analysis
under responsibility of the International Programme committeephysical
is an open- accessis
Peer-review performed.
article
underunder Moreover,
the CC
responsibility a hybrid
BY-NC-ND
of the functional
license
International and architecture graph (HyFPAG)
(https://creativecommons.org/licenses/by-nc-nd/4.0)
Programme committee
of the 55th of the
CIRP Conference 55th CIRP
on is the output
Conference
Manufacturing which depicts the
on Manufacturing
Systems
similarity
Systems between
Peer-review product under
- Peer-review families by providing
responsibility design
of the support to
International both, production
Programme committee system
of theplanners
55th CIRPandConference
product designers. An illustrative
on Manufacturing
example
Systems
Keywords: of Wiring
a nail-clipper
Harness;isDeepusedlearning;
to explainPointthe proposed
clouds; methodology.
Synthetic data; DomainAn gapindustrial case study on two product families of steering columns of
thyssenkrupp
Keywords: WiringPresta FranceDeep
Harness; is then carried
learning; outclouds;
Point to give a first industrial
Synthetic evaluation
data; Domain gap of the proposed approach.
© 2017 The Authors. Published by Elsevier B.V.
Peer-review under responsibility of the scientific committee of the 28th CIRP Design Conference 2018.
1. Introduction Automotive wiring harnesses are highly customized
1. Introduction
Keywords: Assembly; Design method; Family identification Automotive
products. Dependingwiringonharnesses are highly customized
the car configuration, which the
The wiring harness is responsible for the information and products. Depending by
customer composes on selecting
the car car configuration, which the
functions during
The wiring
energy flow inharness is responsible
the automotive for and
vehicle the information
ensures safety- and customerprocess,
buying composes eachbywiringselecting car functions
harness consists ofduringdifferentthe
energy
critical carflowfunctions
in the likeautomotive
driving, vehicle and ensures
lane-keeping assistance,safety-
and buying
components.process, each wiring
According to the harness
high consists
degree of of different
product
1.critical
Introduction and of the product range and characteristics high manufactured and/or
brakingcar [1].functions
Nowadays, likea driving, lane-keeping
wiring harness consistsassistance,
of thousands components.
customization,According to theassembly
the wiring harness degree
processof product
is based on
braking [1]. Nowadays, assembled in this system. In this context, the main challenge in
of components, whicha are wiring harness
either consists
rigid of thousands
objects, such as customization,
a one-piece flow themanufacturing
wiring harnessphilosophy
assembly process is based on
with non-repetitive
of Due to
components, the fast
which development
are either in the
rigid objects, domain of modelling and analysis is now not only to cope with single
connectors and fixing elements, or deformable linear such
objects,as asequences
one-pieceof flow manufacturing
assembly tasks. philosophy
The assembly withof non-repetitive
deformable
communication
connectors and and
fixing anand ongoing
tapes, elements,
trend
hosesor[2].
of digitization
deformable and
harness products,
linear objects, sequences a limited
of assemblyproduct range
tasks. or existing product families,
such as wires, In the wiring linear objects, such as cables, isThe assembly
a complex of due
task deformable
to their
digitalization,
such manufacturing
as wires, these
manufacturing, tapes, components enterprises
and hoses [2]. are facing
In the wiring
are gradually important
addedharness
to the but also
linear to be
degreesobjects, able to
such as
of freedom, analyze
cables,
elastic and to compare
is a complex
deformation, products to
task dueand
geometry, todefine
their
size.
challenges
manufacturing, in
assembly board these today’s market
components
at multiple environments:
are gradually
work stations a continuing
added to
which results in the new product
degrees families.
of freedom,
These reasons hamper It can
elastic be observed
the deformation, that
developmentgeometry, classical
of economic existing
and size.and
tendency
tree-like towards
assembly board atreduction
structure multiple
of ofwork
the wiring product development
stations
harness. which timesinand
results
The complexity of product
the These families
efficient reasons
automationare regrouped
hamper insofunction
that up of
the development
solutions to clients
of
90 % or the
economic
of features.
and
state-
shortened
tree-like product
structurelifecycles.
wiring harnesses of vary
can In addition,
the wiring
between smallthere
harness. Theis complexity
an increasing
standardized wiringof However,
efficient assembly
assemblyoriented
of-the-artautomation solutions
process product
so that
is manual families
up
work are
to 90 hardly
[3].% of the
However, to state-
find.
the
demand
wiring
harnesses of and
customization,
harnesses can varybeing
customized between
main at the
body same
small
wiring time in a A
standardized
harnesses. global
wiring
door On the work
of-the-art
manual product
assembly infamily
the level,
process products
is manual
wiring differ
work
harness [3].mainly
However,
manufacturing in twothe
in
competition
harnesses
wiring harness, with
and competitors
customized
which is main allbody
an example overwiring
the
of aworld. Thiscomplex
harnesses.
medium Atrend,
door main manualcharacteristics:
conjunctionwork within the (i)
the thewiring
product number
variance of and
components
harness andvariety
manufacturing
component (ii) the in
is
which
wiringisharness,
inducing
harness, the isdevelopment
iswhich
illustrated aninexample
Fig. 1(a).from macro tocomplex
of a medium micro type of components
conjunction
failure-prone with (e.g.
andthehard
productmechanical,
varianceelectrical,
to monitor. and
Thecomponentelectronical).
automated variety
optical is
markets, results isinillustrated
wiring harness, diminished lot 1(a).
in Fig. sizes due to augmenting Classical methodologies
failure-prone and hard toconsidering
monitor. The mainly single products
automated optical
product varieties (high-volume to low-volume production) [1]. or solitary, already existing product families analyze the
To cope with
2212-8271 this
© 2022 Theaugmenting variety
Authors. Published as well asB.V.
by ELSEVIER to be able to product structure on a physical level (components level) which
This is an open
2212-8271
identify access
© 2022
possible Thearticle under
Authors.
optimization the CCpotentials
Published BY-NC-ND
by ELSEVIER license
the(https://creativecommons.org/licenses/by-nc-nd/4.0)
in B.V. existing causes difficulties regarding an efficient definition and
Peer-review
This under
is an open responsibility
access of the scientific
article under committee
CC BY-NC-ND of the
license 32nd CIRP Design Conference
(https://creativecommons.org/licenses/by-nc-nd/4.0)
production system, it is important to have a precise knowledge comparison of different product families. Addressing this
Peer-review under responsibility of the scientific committee of the 32nd CIRP Design Conference
2212-8271 © 2022 The Authors. Published by Elsevier B.V.
This is an©open
2212-8271 2017access article Published
The Authors. under theby CC BY-NC-ND
Elsevier B.V. license (https://creativecommons.org/licenses/by-nc-nd/4.0)
Peer-reviewunder
Peer-review under responsibility
responsibility of the
of the International
scientific Programme
committee of the 28thcommittee of the
CIRP Design 55th CIRP
Conference Conference on Manufacturing Systems
2018.
10.1016/j.procir.2022.05.142
 1264 Huong Giang Nguyen et al. / Procedia CIRP 107 (2022) 1263–1268
2 Author name / Procedia CIRP 00 (2022) 000–000

inspection (AOI) and recognition of wiring harnesses and 2.1. Point clouds and deep neural networks for point clouds
respective components have been proposed by [4–6].
Nevertheless, there is still a need for flexible and smart quality Point clouds are a set of data points in Euclidian space. For
assurance systems for the wiring harness manufacturing. inspection purposes, they are sampled from surfaces of objects
in a scene and used to represent objects in a scene as 3D
information in semantic data structures. The simplest
description of point clouds is a list with XYZ coordinates of
each point. More valuable information can be added with
descriptors such as surface normal vectors, RGB values, and
intensity values. Point clouds are characterized by irregularity,
unstructuredness, and lack of order [11]. The density of data
points in a scene varies, the distance between data points is
irregular, and point clouds are inherently unorganized.
The characteristics of point clouds complicate the
development of deep learning algorithms for point cloud data
processing. Therefore, early research focuses on data
processing of 3D information by converting point clouds into
alternative representation forms such as voxels or octrees
[12,13]. Recent research advances show that DNNs succeed in
Fig. 1. (a) door wiring harness on an assembly board; (b) 3D model of door
wiring harness with simulated deformable linear objects. extracting features directly from point clouds as unstructured
geometric data. There is no need for data pre-processing to
The following paper introduces a deep learning-based transform the point cloud into a structured grid format. The
approach for the automated optical inspection of wiring most common computer vision tasks using point clouds are
harnesses. The general intent is based on processing point cloud point cloud classification, object detection and tracking, and
data to evaluate the quality of wiring harness assembly states point cloud segmentation [11]. PointNet++ is one of the first
ranging between partially and fully assembled wiring well-known DNNs that implement local feature learners in a
harnesses. The contributions of this paper are threefold. First, a hierarchical structure for the recognition of fine patterns and
synthetic data generation approach for wiring harnesses based aggregation of local features to higher-level features [14].
on CAD engineering data is introduced. Second, a deep learning Other DNNs have since then outdone the performance of the
model trained with synthetic and real point cloud data is PointNet++. Examples are convolutional neural networks such
developed for the automated optical inspection tasks. Lastly, as RS-CNN [14] which have shown that convolutional layers
experiments were conducted to derive success factors for the contribute to learning features of point clouds. Graph CNNs
synthetic data generation. can also extract geometric features in point clouds by
To present the contributions, the subsequent paper is connecting neighboring points to generate a graph [15].
structured as follows. The paper begins with the related work
section for deep learning for point cloud data as well as 2.2. Synthetic point cloud data generation
synthetic point cloud generation approaches. Then, the
developed data processing pipeline are presented. The pipeline Synthetic data are defined as data that are not acquired
consists of the process steps data collection, data pre- through the measurement of physically existing objects but are
processing, data splitting, and model training and evaluation. still highly similar to real data [16]. Synthesizing artificial data
Experiments were conducted with varying parameters and the is becoming increasingly interesting in the research field of
results are analyzed. The paper concludes with the theoretical deep learning because the bottleneck for high-performing
and practical implications as well as the future research agenda. DNNs is a large database for training. The collection and
preparation of real data is a tedious and time-consuming
2. Deep learning for point clouds process. In comparison to real data, synthetic data are easier
and faster to collect, allow higher labeling precision, and are
Deep learning has become a powerful algorithm for less cost-intensive.
computer vision tasks. The strengths of deep neural networks The generation and collection of synthetic data range
(DNN) are the capability to automatically extract features, between simple and advanced approaches. On the one hand,
process structured and unstructured data, and generalize to point clouds can be synthesized from 3D CAD models.
unseen data [7]. Due to these characteristics, there has been a Examples of this approach are the datasets ModelNet40 [17]
significant increase in research on deep learning in the and ShapeNet [18]. On the other hand, elaborate approaches
manufacturing environment [8–10]. To develop the proposed can generate synthetic data through simulation and rendering.
AOI solution for wiring harnesses, the following related work Such techniques not only demand product models but also need
section focuses on DNNs for processing point clouds. high effort for the proper modeling of the optical sensor,
illumination, and other relevant environmental conditions. The
goal is to create models and simulations as close as possible to
 Huong Giang Nguyen et al. / Procedia CIRP 107 (2022) 1263–1268 1265
Author name / Procedia CIRP 00 (2022) 000–000 3

the real environment which occurs during the inference phase. were collected in a laboratory environment. Data capturing was
In an ideal scenario, synthetic data substitutes real data during executed for an assembly line consisting of multiple
the training stage without compromising DNN’s performance workstations similar to the real manufacturing environment. A
when processing unseen real data during the inference stage. structured-light 3D scanner, specifically the Zivid2 3D camera,
However, there is the inherent problem of a domain gap was employed to capture point clouds of different door wiring
between real and synthetic data. To overcome the domain gap, harness configurations, each in multiple assembly states. Due
transfer learning, generative models, and a combination of real to the size of the wiring harness and the specifications of the
and synthetic data are applied. These approaches can camera, more precisely working distance and aperture angle,
significantly increase the accuracy and generalization each point cloud represents either the left or right half of the
performance of DNNs in comparison to training with real point wiring harness. Point clouds are captured and saved as text files
cloud data only [19,20]. containing the XYZ coordinates. Afterward, each point of a
point cloud was manually labeled and the XYZ surface normals
3. Deep learning-based data processing pipeline for the were added using the CloudCompare software [21]. The labels
automated optical inspection of wiring harnesses applied for the segmentation task are ‘untaped wire bundles’
(UWB), ‘fully taped wire bundles’ (TWB), ‘grommet’
The goal of this paper is the development of a data (GROM), ‘clip type 1’ (C1), ‘clip type 2’ (C2), ‘clip type 3’
processing pipeline for the automated optical inspection of (C3), ‘clip type 4’ (C4) and ‘connector’ (CONN).
wiring harness assembly states that occur in the final assembly The starting point of synthetic data generation is the wiring
line. Assembly states are uncompleted wiring harnesses, which harness development, see Fig. 2. The wiring harness with
are work in processes in the assembly line, or finished wiring multiple configurations was designed in CAD. The 3D
harnesses, which occur at the end of the assembly line. For the component models were either created or retrieved from
implementation in this paper, the data processing pipeline is suppliers’ libraries. The next step is composing a scene similar
realized and tested for a door wiring harness, see Fig. 1. The to the physically existing assembly board with wiring harness
automated optical inspection aims to assess, whether the assembly states by arranging the 3D models. The result is a
correct components have been assembled at a specific development artifact that contains the geometric information,
workstation and in the correct position. These quality features name, and layout of components. Then, for each product
are analyzed to check whether the correct product configuration, a 3D CAD model was exported into a simulation
configuration has been built. This is especially important due tool. Mechanical simulation is important for the detection of
to the high product variance and component variety in the deformable linear objects because these objects, especially
wiring harness assembly. To deduct these quality features, a wires, are characterized by many degrees of freedom, no
DNN, specifically PointNet++, for point cloud segmentation is compression strength, and elastic deformation. The simulation
implemented. The proposed data processing pipeline uses real should ensure that the objects look as realistic as possible and
and synthetic depth data and is composed of the process steps that various states of the wiring harness are captured. The
of data collection, data pre-processing, data splitting, and results are 3D simulation models for assembly states of the
model training and evaluation, see Fig. 2. wiring harness, which each reflect the wiring harness as a work
in progress after each work station and finished assembly as an
Wire harness development Synthetic point cloud generation output at the end of the assembly line. For our implementation,
the CAD model was created in the design tool NX [22] and the
3D CAD
3D model library
model
3D simulation
model
Raw synthetic
point cloud
simulations were conducted in IPS Cable Simulation [23]. An
for components
example of the simulation is given in Fig. 1(b). The next step
is generating labeled point clouds for each 3D simulation
Real point cloud generation
model which is done in Blender [24]. Labeling information was
Raw real
Preprocessed syn. added by defining the custom object properties on a component
point cloud
point cloud
Optical sensor Physical wire harness
level. To convert the 3D model into a point cloud data format,
each polygon surface of the model was used to uniformly
Deep neuronal network framework sample data points. Points on these faces were randomly
sampled based on the face-vertex information by a python code
Preprocessed real Training, validation, Trained Pre-trained
point cloud and test dataset PointNet++ PointNet++
in Blender. Analogous to the real data, the synthetic data were
generated for defined wiring harness sections. The output of
this process is raw point cloud text files containing the XYZ
Fig. 2. framework for point cloud segmentation using synthetic and real data. coordinates, XYZ normal coordinates, and labels for each point
in the point cloud. The labeling process for synthetic data is less
time-consuming in comparison to manual labeling. Meta-
3.1. Data collection information is provided from the wiring harness development
and can be directly reused to automatically derive labeling
The data collection step can be differentiated into the information.
generation of real depth data and synthetic depth data. Real data
 1266 Huong Giang Nguyen et al. / Procedia CIRP 107 (2022) 1263–1268
4 Author name / Procedia CIRP 00 (2022) 000–000

3.2. Data pre-processing The DNN models were created through hyperparameter
optimization. Grid search was implemented to find the best
The raw point clouds need to be transformed into an hyperparameter setting for each experiment. Hyperparameters
understandable format for the DNN. The real data, which were were composed of the number of epochs, learning rate,
captured with a physically existing setup, are incomplete and momentum, batch size, decay step, decay rate, and dropout. All
noisy. The conversion of the raw real point clouds into a deep training runs were evaluated regarding the evaluation metrics
learning-relevant format requires the tasks of data cleaning and accuracy, loss, and intersection over union (IoU).
data reduction. The pre-processing tasks for the real point cloud
data were conducted in CloudCompare. Outlier points were Table 1. Overview of experiments, training approaches, and data splitting.
manually deleted to denoise the point clouds. Furthermore, No. Training approach Train/val/test
unnecessary objects, such as the assembly board and fixing E1 Baseline model: Real data train and test 120/15/15
elements that are attached to the formboard, were removed. E2 Ideal model: Synthetic data train, real data test 137/13/15
Normal information was retrieved and added to the text file.
E3 Synthetic data: Synthetic data train and test 262/30/33
The last step is randomly downsampling the number of data
E4 Real data train and synthetic data test 120/15/20
points of each point cloud to 2048. The pre-processing step for
synthetic data entails the task of downsampling because the 3D E5 Synthetic data traing, 262/30/33,
real data fine-tuning and test 120/15/15 or 60/6/15
model of the wiring harnesses already provides clean point
clouds. Similarly to downsampling the real data, the number of E6 Mixed dataset: Real and synthetic data train
414/46/15
(70/30 ratio), real data test
points in the synthetic data was randomly reduced to 2048.
Synthetic data pre-processing is conducted with a python code E7 Mixed dataset: Real and synthetic data train
243/27/15
(50/50 ratio), real data test
in Blender. The result of the data pre-processing step is text
files that contain 2048 rows and seven columns with XYZ
coordinates, XYZ surface normal, and labeling information. 4. Results and discussion

In the following section, the experimental findings and best-

3.3. Data splitting, model training, and evaluation performing PointNet++ models are presented and analyzed.
The different training approaches are discussed to find the most
The starting point of the training was a pre-trained efficient approach to integrate synthetic data for DNN training.
PointNet++ model. The implementation of PointNet++ is based A discussion of the research findings aims to reveal best
on the code provided by Qi et al. on GitHub [25]. The ShapeNet practices for the proposed synthetic data generation process for
[17] dataset was utilized for training from scratch. Transfer point cloud data of wiring harnessed.
learning was then conducted to adapt the weights of the pre-
trained model to the point cloud segmentation task for wiring
harness components. Multiple DNNs were developed based on 4.1. Training with real or synthetic data only
different training approaches to evaluate the synthetic data
generated with our proposed approach. An overview of the The first experiment (E1) aims to achieve the baseline
experiments, datasets, and data splitting ratios are provided in model, which was trained and tested with the real data point
Table 1. The hold-out methodology and a data splitting ratio of cloud. The model reached an accuracy of 96.42 % for training
approx. 90% for training/validation and 10% for testing were and 97.38 % for testing. The mean IoU for all classes during
applied for all experiments. First, a baseline model, which was testing is 93.25 %. This is extremely high considering the small
trained and tested on real data, was created (E1). Training and amount of real training data used for transfer learning. E3 was
testing were also performed for synthetic data only (E3). training and testing with synthetic data. The results were a
Second, experiments were conducted to evaluate the domain training accuracy of 98.78 % and a test accuracy of 99.41 %.
gap between synthetic and real data. The pre-trained The DNN outperforms the baseline model with a mean IoU of
PointNet++ learned exclusively on synthetic data and was then 98.90 %.
tested on real data (E2). This is called the ideal model. This Training PointNet++ to develop the ideal model resulted in
approach is the most desirable for the wiring harness lower accuracies for training as well as testing (E2). Despite
manufacturing because no real data is required for the exhaustive hyperparameter optimization, the mean IoU was
development of the DNN. Additionally, a DNN was trained 45.00 %. Even though the training curve showed a good fit and
with real data and tested on the synthetic dataset (E4). Lastly, increased to a point of stability, the model was not transferable
experiments were carried out using synthetic as well as real to real data. Whereas the training curve for real data reached an
data for training and testing. On the one hand, transfer learning accuracy of 91.16 %, the testing accuracy for synthetic data
with synthetic and fine-tuning with a small amount of real data only achieved 28.42 %. The domain gap from synthetic to real
were conducted (E5). On the other hand, real and synthetic data data is too significant. However, the domain gap from real data
were mixed in varying ratios of 30% to 70% (E6) and 50% to to synthetic data is not as serious. The PointNet++ model
50% for training (E7). trained with real data achieved an accuracy of 97.52 % and
testing with synthetic data reached a mean IoU of 93.48 %
 Huong Giang Nguyen et al. / Procedia CIRP 107 (2022) 1263–1268 1267
Author name / Procedia CIRP 00 (2022) 000–000 5

(E4). E2 and E4 provide valuable insight into the domain gap synthetic to the real training data results in a slight increase in
between real and synthetic datasets. It seems like the data accuracy in comparison to the baseline model. Halving the
distribution of synthetic data is easier to learn for the DNN. A amount of real data caused a slight decrease in accuracy. Fine-
reduction of the XYZ coordinates using T-distributed tuning with 60 point clouds for training and 6 point clouds for
Stochastic Neighbor Embedding (t-SNE) [26] for a synthesized validation resulted in a mean IoU of 92.70 % and testing
as well as real point cloud shows that a clear distinction accuracy of 96.88 %. A drop of 0.48 % mean IoU is not
between the classes is more evident for synthetic data, see significant considering that 50% of the real data was removed
Fig. 3. The t-SNE graph for the real point cloud shows a from the database for fine-tuning.
scattering of the classes. The boundaries between the classes
are blurry and hardly observable. This can especially be
observed for the classes C1 (4) and TWB (2).

Fig. 4. learning curve for best performing model (E6).

The best accuracies were achieved by E6 and E7. A mixing

ratio of 70 % to 30 % led to a mean IoU of 94.62 and a ratio of
50 % to 50 % to a mean IoU of 93.46 %. The former performed
slightly better in comparison to the latter. Nevertheless, both
models provided better results than the benchmark baseline
Fig. 3. (a) t-SNE for a real point cloud, (b) t-SNE for a synthetic point cloud. model. The optimized hyperparameters for the better
performing model are 500 epochs, batch size of 8, learning rate
There is a different data distribution which is inherent due of 0.001, momentum of 0.5, decay step of 10000, and decay
to different data collection strategies and data pre-processing rate of 0.95. The learning curves of E6 increased significantly
steps. Real point clouds are harder to learn as they suffer from in the beginning and reach a point of saturation which signifies
disturbing artifacts such as noise. Contrastingly, synthetic data a good fitting of the model to the data, see Fig. 4. When testing
are perfect data. Even though the point clouds were randomly the PointNet++ model trained on synthetic data to real data, an
downsampled the flawless data distribution of the synthetic accuracy of 97.94 % was reached. This clearly shows that the
data remains. Overall the findings show that it is possible to domain gap between the real and synthetic datasets can be
transfer a model trained on a difficult dataset to a simple overcome by blending real data into the synthetic dataset. Even
dataset, as shown in E4. E2 demonstrated that the opposite though prior experiments led to the conclusion that the data
direction seems to be difficult. Overall the results lead to the distribution of synthetic and real point clouds highly differs,
conclusion that the size of the training dataset is secondary fine-tuning on real data or training with mixed datasets are
important to the data distribution of point clouds. viable approaches to overcome the domain gap.
Further analysis of the best-performing model is conducted
by looking at the confusion matrix for the testing dataset, see
4.2. Training with real and synthetic data Fig. 5. As the test dataset was randomly created, the classes
UWB is not represented. The overall precision was 95 % and
The PointNet++ model in E5 was achieved through transfer recall is 95 %. This demonstrates that most predictions were
learning on synthetic data and fine-tuning on real data. Fine- correct. The lowest recall rates were for class C2 with a recall
tuning was conducted with a training dataset with 135 images, rate of 80 %. The highest and best-predicted classes are C1 and
which were also applied for E1, and a smaller dataset with 66 C4.
point clouds. The test accuracy of the large training dataset was
97.39 % and the mean IoU was 93.28 %. Therefore, adding
1268 Huong Giang Nguyen et al. / Procedia CIRP 107 (2022) 1263–1268
6 Author name / Procedia CIRP 00 (2022) 000–000

5th International Conference on Advanced Robotics and Mechatronics

UWB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 (ICARM). IEEE; 2020 - 2020, p. 329–333.
[5] Gregorio DD, Palli G, Di Stefano L. Let’s Take a Walk on Superpixels
TWB 0.0 12360.0 0.0 28.0 158.0 62.0 0.0 0.0
Graphs: Deformable Linear Objects Segmentation and Model Estimation.
In: Jawahar CV, Li H, Mori G, Schindler K, editors. Computer Vision –
ACCV 2018. Cham: Springer International Publishing; 2019, p. 662–677.
GROM 0.0 20.0 5450.0 0.00 20.0 33.0 28.0 10.0 [6] Ben Abdallah H, Orteu J-J, Dolives B, Jovancevic I. 3D point cloud
Ground truth class

analysis for automatic inspection of aeronautical mechanical assemblies.

C1 0.0 68.0 0.0 6578.0 0.0 0.0 0.0 2.0
In: Cudel C, Bazeille S, Verrier N, editors. Fourteenth International
Conference on Quality Control by Artificial Vision. SPIE; 2019, p. 61.
[7] Shrestha A, Mahmood A. Review of Deep Learning Algorithms and
C2 0.0 95.0 27.0 0.0 707.0 0.0 0.0 0.0 Architectures. IEEE Access 2019;7:53040–65.
[8] Meiners M, Mayr A, Franke J. Process curve analysis with machine
learning on the example of screw fastening and press-in processes. Procedia
C3 0.0 23.0 29.0 0.0 0.0 1739.0 0.0 0.0
CIRP 2021;97:166–71.
[9] Nguyen HG, Meiners M, Schmidt L, Franke J. Deep learning-based
CONN 0.0 0.0 69.0 0.0 0.0 0.0 450.0 0.0 automated optical inspection system for crimp connections. In: 2020 10th
International Electric Drives Production Conference (EDPC). IEEE; 2020
- 2020, p. 1–5.
C4 0.0 5.0 0.0 1.0 0.0 0.0 0.0 2777.0
[10] Mayr A, Kißkalt D, Meiners M, Lutz B, Schäfer F, Seidel R et al. Machine
Learning in Production – Potentials, Challenges and Exemplary
UWB TWB GROM C1 C2 C3 CONN C4 Applications. Procedia CIRP 2019;86:49–54.
Predicted class [11] Guo Y, Wang H, Hu Q, Liu H, Liu L, Bennamoun M. Deep Learning for
Fig. 5. confusion matrix of E6: Mix real and synthetic data 3D Point Clouds: A Survey. IEEE Trans. Pattern Anal. Mach. Intell.
2021;43(12):4338–64.
5. Summary and outlook [12] Kokic M, Stork JA, Haustein JA, Kragic D. Affordance detection for task-
specific grasping using deep learning. IEEE-RAS International Conference
on Humanoid Robots 2017.
The provided research findings underline the importance of [13] Riegler G, Ulusoy AO, Geiger A. OctNet: Learning Deep 3D
synthetic data. Not only the process step of data collection but Representations at High Resolutions. In: 30th IEEE Conference on
also data pre-processing are less time and cost consuming. The Computer Vision and Pattern Recognition: CVPR 2017. IEEE; 2017, p.
generation of synthetic data does not require a physical setup 6620–6629.
[14] Qi CR, Yi L, Su H, Guibas LJ. PointNet++: Deep Hierarchical Feature
of the manufacturing equipment and wiring harness, tedious Learning on Point Sets in a Metric Space. In: Luxburg U von, editor. 31st
data cleaning for high-quality training data, and, first and Annual Conference on Neural Information Processing Systems; 2017, p.
foremost, manual work for labeling. This paper shows that 5105–5114.
CAD and simulations models, which are created during the [15] Wang Y, Sun Y, Liu Z, Sarma SE, Bronstein MM, Solomon JM. Dynamic
development in engineering disciplines anyway, can be used to Graph CNN for Learning on Point Clouds. ACM Trans. Graph.
2019;38(5):1–12.
generate valuable synthetic data. The findings demonstrate that [16] Anderson JW, Kennedy KE, Ngo LB, Luckow A, Apon AW. Synthetic
synthetic data reduces the amount of real data required for data generation for the internet of things. In: Lin J, editor. 2014 IEEE
DNN training and were validated with experimental results for International Conference on Big Data (Big Data 2014). IEEE; 2014, p.
point cloud segmentation of wiring harness components. 171–176.
Future research needs to establish an in-depth understanding [17] Wu Z, Song S, Khosla A, Yu F, Zhang L, Tang X et al. 3D ShapeNets: A
deep representation for volumetric shapes. In: 2015 IEEE Conference on
of the modeling, simulation, and rendering process for the Computer Vision and Pattern Recognition (CVPR). IEEE; 2015 - 2015, p.
creation of synthetic data and investigate the impact of these 1912–1920.
process steps on the performance of DNNs. The modeling of [18] Chang AX, Funkhouser T, Guibas L, Hanrahan P, Huang Q, Li Z et al.
noise in point clouds needs to be researched to overcome the ShapeNet: An Information-Rich 3D Model Repository; 2015.
domain gap between real and synthetic point clouds. A focus [19] Wang F, Zhuang Y, Gu H, Hu H. Automatic Generation of Synthetic
LiDAR Point Clouds for 3-D Data Analysis. IEEE Transactions on
should be given to applications in the manufacturing Instrumentation and Measurement 2019;68(7):2671–3.
environment as the lack of labeled domain-specific data is the [20] Saleh K, Abobakr A, Attia M, Iskander J, Nahavandi D, Hossny M et al.
biggest bottleneck and the initial hurdle for manufacturing Domain Adaptation for Vehicle Detection from Bird's Eye View LiDAR
companies to introduce deep learning-based solutions. Point Cloud Data. In: 2019 IEEE/CVF International Conference on
Computer Vision Workshop (ICCVW). IEEE; 2019 - 2019, p. 3235–3242.
References [21] CloudCompare. CloudCompare V2; Available from:
https://www.danielgm.net/cc/.
[1] Nguyen HG, Kuhn M. The Future of Harness Development and [22] Siemens Software. NX; Available from:
Manufacturing: Results from an Expert Case Study. https://www.plm.automation.siemens.com/global/en/products/nx/.
[2] Frohlig S, Piechulek N, Friedlein M, Sus-Wolf R, Schmidt L, Nguyen [23] fleXstructures. IPS Cable Simulation; Available from:
MKH et al. Innovative signal and power connection solutions for https://flexstructures.com/products/ips-cable-simulation/.
alternative powertrain concepts. In: 2020 10th International Electric Drives [24] Blender Foundation. Blender; Available from: https://www.blender.org/.
Production Conference (EDPC). IEEE; 2020 - 2020, p. 1–7. [25] Qi CR. GitHub repository: PointNet++; Available from:
[3] Nguyen HG, Kuhn M, Franke J. Manufacturing automation for automotive https://github.com/charlesq34/pointnet2.
wiring harnesses. Procedia CIRP 2021;97:379–84. [26] van der Maaten L, Hinton G. Viualizing data using t-SNE. Journal of
[4] Zhou H, Li S, Lu Q, Qian J. A Practical Solution to Deformable Linear Machine Learning Research 2008;9:2579–605
Object Manipulation: A Case Study on Cable Harness Connection. In: 2020

View publication stats