Available online at www.sciencedirect.

com
Available online at www.sciencedirect.com
Available ScienceDirect
online atonline
www.sciencedirect.com
ScienceDirect
Available at www.sciencedirect.com

ScienceDirect
Procedia CIRP 00 (2021) 000–000
ScienceDirect
Procedia CIRP 00 (2021) 000–000
www.elsevier.com/locate/procedia
Procedia Procedia
CIRP 00 CIRP
(2017)103
000–000 www.elsevier.com/locate/procedia
(2021) 262–267
www.elsevier.com/locate/procedia

9th CIRP Global Web Conference – Sustainable, resilient, and agile manufacturing and service operations:
9th CIRP Global Web Conference – Sustainable, resilient,
Lessons from and agile manufacturing and service operations:
COVID-19
Lessons from COVID-19
Image Based 28thQuality Inspection
CIRP Design in Smart
Conference, May 2018,Manufacturing
Nantes, France Systems:
Image Based Quality Inspection in Smart Manufacturing Systems:
A new methodology to analyze A Literature Reviewand physical architecture of
the functional
A Literature Review
existing products for anBabic
Milica assembly
a
oriented
, Mojtaba A. product
Farahania, Thorsten family
Wuest a*
Milica Babica, Mojtaba A. Farahania, Thorsten Wuest a*
identification
a
Industrial and Management Systems Engineering, West Virginia University, Morgantown, WV 26505, USA
a
Paul Stief *, Jean-Yves Dantan, Alain Etienne, Ali Siadat
Industrial and Management Systems Engineering, West Virginia University, Morgantown, WV 26505, USA
* Corresponding author. Tel.: +1-304-293-9439; fax: +1-304-293-4970. E-mail address: [email protected]
Écoleauthor.
* Corresponding Nationale
Tel.:Supérieure d’Arts etfax:
+1-304-293-9439; Métiers, Arts et Métiers
+1-304-293-4970. ParisTech,
E-mail LCFC
address: EA 4495, 4 Rue Augustin Fresnel, Metz 57078, France
[email protected]

* Corresponding author. Tel.: +33 3 87 37 54 30; E-mail address: [email protected]

Abstract
Abstract
Quality inspection is an important component of today’s smart manufacturing systems (SMS). Their prominence stems from the objective of
Quality inspection
manufacturing
Abstract is an important
companies component
to i) deliver of today’s
high-quality products,smart ii)manufacturing systems iii)
inspire brand loyalty, (SMS).
keep Their
withinprominence
regulations,stemsand iv)from the objective
minimize waste of
manufacturing
resources (incl. companies
such as employeeto i) deliver high-quality
and machine time,products, ii) inspire
scrap materials) brand loyalty,
to maximize iii)There
profit. keep is within
a wide regulations,
variety of and iv) inspection
quality minimize waste systemsof
resources on
deployed (incl.
thesuch
shopas floor
employee and different
utilizing machine time, scrap materials)
technologies, ranging to maximize
from humanprofit.
operatorsThere toishigh-fidelity
a wide variety of quality
sensor systemsinspection
to imagesystems
based
Insystems.
today’sIn
deployed business
onthis
thework environment,
shop wefloor
focus onthethetrend
utilizing towards
different
latter, more
technologies,
modern product
image-based variety
ranging and
from
quality customization
human
inspection operators
systems isand
unbroken.
to Due
high-fidelity
processes. tosensor
Vision this development,
basedsystems the need
qualitytoinspection
image of
based
has
agile
seen and
systems. reconfigurable
In this work
an increase weproduction
focus onand
in applications thesystems
academic emerged
latter, modern to over
cope the
image-based
attention withpast
various
quality products
inspection
decade, and
systems
aligned withproduct
anddawn
the families.
processes. To design
Vision
of Industry based
4.0. and optimize
Onquality
the production
oneinspection
hand, has
digital
seen an as
camera
systems increase
systems intoapplications
well as(including
choosethe and
theoptics, academic
optimal sensors,
productattention
and over
connectivity)
matches, the past
product have decade,
become
analysis aligned
more are
methods withneeded.
powerfulthe dawn atofthe
andIndeed, Industry
same of
most 4.0.
time On
themore theaffordable.
known one hand, On
methods digital
aim the
to
camerahand,
other
analyze asystems
product (including
the analytics
or one product the optics,
– namely sensors,
artificial
family and connectivity)
on theintelligence
physical andDifferent
level. haveproduct
machine becomefamilies,
learning more powerful
algorithms – have
however, and at differ
made
may the same
tremendous time
largely more
terms affordable.
inadvancements Onand
in terms
of the number the
of
other
resultshand,
as the
well asanalytics –
accessibility. namely
Most artificial
notably, intelligence
deep neural and machine
networks and learning
deep algorithms
learning have – have
elevated
nature of components. This fact impedes an efficient comparison and choice of appropriate product family combinations for the production made
the tremendous
potential of advancements
compute r vision in terms
in of
quality
results as well
inspection as accessibility.
applications to the Most notably,
next level. In deep neural networks
this paper, and deepa learning
we will conduct have elevated
comprehensive literaturethe review
potentialanalysing
of compute r vision
image based in quality
system. A new
inspection methodology
applications
systems in SMSto the is proposed
overnext lasttodecade.
thelevel. analyze
In thisWe existing
paper,
will we products in viewa on
willparticularly
focus conduct of their functional
comprehensive
the question and
howphysical
literature
of review
image architecture.
based analysing The
image
in-situ quality aimbased
is to quality
cluster
inspection of
these products
inspection
three-dimensionalin new
systems inassembly
partsSMS oriented
over
is currently lastproduct
decade.families
theconducted. Weresults
The forfocus
will the
willoptimization
particularly
provide of
onexisting ofassembly
the question
an overview of how
the differentlinesimage
and the
available creation
based of future
in-situ
image basedquality reconfigurable
qualityinspection
inspectionof
assembly systems.
three-dimensional
approaches, Based
parts ison
their benefits Datum
currently
and Flow Chain,
conducted.
challenges, the
as wellTheas physical
results will
specific structure
provide
application ofareas
theoverview
an products is analyzed.
of the different
and/or industries. Functional
availablesubassemblies
image based are identified,
quality and
inspection
aapproaches, their benefits
functional analysis and challenges,
is performed. Moreover,as well as specific
a hybrid application
functional areas and/or
and physical industries.
architecture graph (HyFPAG) is the output which depicts the
©
© 2021
2021 The
similarity The Authors.
between
Authors. Published
product by ELSEVIER
families
Published by B.V.B.V.
by providing
Elsevier design support to both, production system planners and product designers. An illustrative
© 2021
This
example
This is The
is an
an a Authors.
ofopen accessPublished
access
nail-clipper
open article
is used
article by
under ELSEVIER
the
the CC
to explain
under B.V. license
BY-NC-ND
CC the proposed
BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0)
methodology. An industrial case study on two product families of steering columns of
(https://creativecommons.org/licenses/by-nc-nd/4.0)
This is an open
Peer-review
thyssenkrupp
Peer-review access
under
Presta
under article
responsibility
France under
ofofthe
is then
responsibility carriedCCoutBY-NC-ND
thescientific
scientific committee
tocommittee
give license
a first (https://creativecommons.org/licenses/by-nc-nd/4.0)
of the
ofindustrial
the 9th9th CIRP Global
evaluation
CIRP Global Web
of the
Web Conference
proposed
Conference – Sustainable,
–approach.
Sustainable, resilient,
resilient, andmanufacturing
and agile agile
Peer-review
manufacturing under
and responsibility
service operations:of theLessons
scientificfrom
committee
COVID-19 of the 9th CIRP
(CIRPe 2021)Global Web Conference – Sustainable, resilient, and agile
©manufacturing
2017
and The operations
service Authors. Published
: Lessons
and service by
from
operations:Elsevier
COVID-19B.V.from
Lessons (CIRPe 2021) (CIRPe 2021)
COVID-19
Peer-review under responsibility
Keywords: Machine of the scientific
Vision; Visual Inspection; Qualitycommittee
Inspection;of the 28th
Image CIRP Smart
processing; Design Conference 2018.
Manufacturing; Production; Industry 4.0
Keywords: Machine Vision; Visual Inspection; Quality Inspection; Image processing; Smart Manufacturing; Production; Industry 4.0
Keywords: Assembly; Design method; Family identification
1. Introduction Over time, manufacturing became more complex and
1. Introduction Over time,
efficient, and it manufacturing
became evidentbecame that themore
productcomplex
cannot fully and
In production and manufacturing, quality inspection is a efficient, and it became evident that the
match the sketch as there is always a deviation depending onproduct cannot fully
1. Introduction
In production and manufacturing, qualityofinspection is a of
thethe
match product
the sketch
granularity range
of as and characteristics
there
inspection. is always manufactured
a deviation
In an effort to address thisand/or
depending issue,on
process of checking for conformity a product's
process of checking assembled
the in
granularity this
of system.
inspection. In this
In context,
an effort the
tolerance limits became more prevalent, and human inspectorsto main
address challenge
this issue, in
characteristics with its fororiginal conformity
design ofby a measuring,product's
Due to
characteristics the
examining, testing, fast development
with orits gaging.
originalThis in
design the
process domain
by ismeasuring,
valuableof modelling
tolerance and
limits analysis
became is
more now not only
prevalent,
are being replaced by computer vision systems [1]. With to
and cope
human with single
inspectors
communication
examining, and an
testing,
because it eliminates theongoing
or gaging. trend process
This
risk of supplying of faulty
digitization
isand/or and
valuable
low are beinga limited
products,
increasing replaced
competition, by computer
product the orvision
rangepressure
existing systems
for more [1].
product With
families,
efficient
digitalization,
because it manufacturing
eliminates the risk enterprises
of supplying are
quality parts to customers and may provide valuable insights facing
faulty important
and/or low increasing
but also
processes,to be competition,
able
less to analyze
waste, the
lower and pressure
to
cost,compare
and for more
products
higher efficient
to define
quality of
challenges
quality parts in totoday’s
customers market
and environments:
may
that help to address manufacturing process issues. provide a
valuable continuing
insights processes,
new product
products less
increased waste,
families. It lower
can
across be cost,
observed
all and
sectors.thathigher quality
classical
This led existing
to of
the
tendency
that towards
help
Historically, to reduction
address
quality of product
and development
manufacturing
inspection was times
controlprocess and
issues.
a human products
product
emergence increased
families
of total acrossmanagement,
are regrouped
quality all sectors.of
in function This
clients
lean led to the
or features.
manufacturing,
Historically,
shortened quality
product inspection
lifecycles. In and
addition, control
centric evaluation process. In its simplest form, the finalthere iswas
an a human
increasing emergence
automation,
However, of total
and
assembly massquality
oriented management,
production
product just to
familieslean
name
aremanufacturing,
a few
hardly to- find.
and
centric
demand
product of evaluation be process.
customization,
would being
compared In atits asimplest
to the same time form,in the
blueprint/sketch finala
a global
or automation,
most
On recently and
the productthe mass
Industry
family production
level, just toleveraging
4.0 paradigm
products namemainly
differ a few
the in - two
and
digital
product
previouslywould
competition with be
produced compared
part, andalliftoover
competitors a the
they blueprint/sketch
world. the
matched, or a
Thisproduct
trend, most
main recently theof
transformation
characteristics: Industry the 4.0
(i)the paradigm
manufacturing
number leveraging
of componentsindustry.the (ii)
and digital
These the
previously
which produced
is inducing
was accepted. Thisthe part, and if
development
accounted forthey
a frommatched,
large macrothetoofproduct
amount micro
scrap transformation
requirements
type of components of
put a strain the manufacturing
on the ability
(e.g. mechanical, of human
electrical,industry. These
operators
electronical). to
was
from accepted.
markets, results This
discarded in accounted
diminished
material whichlotfor a large
sizes
added todueamount of costs.
to augmenting
production scrap requirements
keep up with
Classical putthea strain
methodologiesinspectiononconsidering
thetasks
abilityand
of human
mainly operators
the single
probability
products to
of
from discarded material which added to production keep up with the inspection tasks and the probability of
product varieties (high-volume to low-volume production)costs. [1]. or solitary, already existing product families analyze the
To cope with
2212-8271 © 2021this
Theaugmenting variety
Authors. Published as well asB.V.
by ELSEVIER to be able to product structure on a physical level (components level) which
identify
2212-8271 possible
This is an open access
© 2021 optimization
Thearticle under
Authors. the CCpotentials
Published BY-NC-ND
by ELSEVIER inB.V.
the
license existing causes difficulties regarding an efficient definition and
(https://creativecommons.org/licenses/by-nc-nd/4.0)
Peer-review
This under
is an open responsibility
access of the scientific
article under CC BY-NC-ND committee of the
license 9th CIRP Global Web Conference – Sustainable, resilient, and agile manufacturing and service
(https://creativecommons.org/licenses/by-nc-nd/4.0)
production
operations : system,
Lessons fromit is important
COVID-19 to
(CIRPe have
2021) a precise knowledge
Peer-review under responsibility of the scientific committee of the 9th CIRP Globalcomparison Web Conference of– Sustainable,
different resilient,
productandfamilies. Addressing
agile manufacturing this
and service
2212-8271: ©
operations 2021 The
Lessons fromAuthors.
COVID-19 Published by Elsevier B.V.
(CIRPe 2021)
This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0)
Peer-review
2212-8271 underThe
© 2017 responsibility of the scientific
Authors. Published committee
by Elsevier B.V. of the 9th CIRP Global Web Conference – Sustainable, resilient, and agile manufacturing
and serviceunder
Peer-review operations : Lessons
responsibility from
of the COVID-19
scientific (CIRPe
committee of 2021)
the 28th CIRP Design Conference 2018.
10.1016/j.procir.2021.10.042
 Milica Babic et al. / Procedia CIRP 103 (2021) 262–267 263
2 Milica Babic et al. / Procedia CIRP 00 (2019) 000–000

human error increases. 2. Research methodology and data collection

There are many ways to inspect parts for defects. There are
various options available, and categories include in-situ vs. This literature review was conducted to better understand
ex-situ inspection, and 100% inspection vs statistical the research domain of quality inspection and control with a
sampling. A recent study found that human error contributes particular focus on image inspection. The reviewed works
23% towards inaccurate quality control measurements and were identified and accessed through Scopus, ensuring high
70% of organizations concluded their lack of awareness when quality and transparency of the surveyed papers. Because
it comes to equipment management [2]. All studies published quality inspection and control is a process that could be used
between 2010 and 2020 that we have reviewed for this paper as a means of verification, keyword analysis was performed
indicate that machines have more accuracy in providing the for titles, abstracts, and keywords. This approach led to the
defect attributes of a product. Automated [3] quality most results. Image based quality control can be referred to
inspection systems have single 24/7 responsibility unlike using different sets of terms. It was important that we do not
operators that have multiple limited workday tasks. Therefore, exclude any major ones to not omit relevant research from our
human resources can focus better on other tasks. Automated analysis. Therefore, we used “machine” or “computer”,
systems usually have a one-time installation fee and “vision” or “image”, “analysis”, “quality”, and “control” or
periodical maintenance and energy consumption fees; with the “inspection” for the initial search. Since we were focused on
accuracy benefits therefore, in time, the system pays for the quality inspection and control processes in industry, we
itself. added “manufacturing” or “production” as both are used in
Some examples of Automated Quality Control Systems industry. The difference between the two is that
are visual testing, sensor based systems, magnetic particle manufacturing is a process that transforms raw material into
inspection, radiographic testing, ultrasound testing, and finished product and production is a process that uses various
penetrant testing. Among the mentioned methods, image materials [1]. Another factor is the assumed preference for the
based systems are among the most popular because they can term ‘production’ in Europe versus ‘manufacturing’ in the US
be implemented comparably easily and at a lower cost. Even and Asia.
small sized job shops can afford to implement such systems The timeframe for including literature on the Scopus
and improve their traditional quality inspection and control search was influenced by the introduction of Industry 4.0 in
systems and processes. Nowadays, high resolution, industrial 2011. We constrained our search to papers published later
grade camera systems are widely available at a competitive than 2010. We further restricted our search to include only
price point. Furthermore, in a smart manufacturing system engineering and computer sciences as these fields relate to the
they can be integrated into the digital shop floor and purpose of this literature review. Most research is done in or
controlled remotely. translated to English which is also the common language
between the authors of this paper. The last restriction we used
to select our paper sample was limiting our search to
conference papers and articles. This search resulted in a set of
682 papers returned by Scopus. The complete database search
string can be found below:

(TITLE-ABS-KEY(machine OR computer AND vision OR

image AND analysis AND quality AND control OR
inspection) AND (manufacturing OR production)) AND
(LIMIT-TO (SUBJAREA, “COMP”)) AND (LIMIT-TO
(SUBJAREA, “ENGI”)) AND (LIMIT-TO ( LANGUAGE,
“English”)) AND (LIMIT-TO (DOCTYPE, “ar”)) AND
(LIMIT-TO (DOCTYPE, “ar”)) OR LIMIT-TO
(DOCTYPE, "cp")) AND (LIMIT-TO (PUBYEAR, 2020)
Fig. 1. Histogram: number of publications related to Image based Quality OR LIMIT-TO (PUBYEAR, 2019 ) OR LIMIT-TO
Control research (timeframe 1978-2020)
(PUBYEAR, 2018) OR LIMIT-TO
Industry 4.0 utilizes various sensor systems (including (PUBYEAR, 2017) OR LIMIT-TO (PUBYEAR, 2016) OR
camera systems) and artificial intelligence to improve data LIMIT-TO (PUBYEAR, 2015) OR LIMIT-TO (PUBYEAR,
acquisition and analysis. Smart manufacturing requires less 2014) OR LIMIT-TO (PUBYEAR, 2013) OR LIMIT-TO
human intervention at the process level due to the improved (PUBYEAR, 2012) OR LIMIT-TO (PUBYEAR, 2011))
communication between operators and machines in cyber-
physical production systems. This enables human operators to A CSV file with the resulting papers’ details was further
focus on more creative tasks, such as complex process examined. We went through the papers and excluded those
improvements and higher level problem solving. There is an whose topics did not contribute to the topic of this literature
evident increase in the number of Scopus research papers in review following transparent exclusion criteria and the four
the field of ‘Image based Quality Inspection’ with an about eye principle. The first step was to read through the article
87.76% increase in the last five years compared to the titles and evaluate with yes/maybe/no. For this step, the
previous five year term which is almost double (Fig. 1). In papers were split into two evenly sized sets and evaluated
this paper, we aim to better understand the different independently by two of the authors. In the next step, the titles
application areas, prevalent research directions and research evaluated with “maybe” were switched between the authors
gaps. for final assessment. This produced a final set of 323 papers.
264 Milica Babic et al. / Procedia CIRP 103 (2021) 262–267
Milica Babic et al./ Procedia CIRP 00 (2019) 000–000 3

The process was repeated with this set to evaluate the what areas have been well researched and which needs
abstracts, further reducing the set of relevant appears to a final improvement.
set of 224 papers relevant for the literature review. These
papers were analyzed and the basis for the analysis and 3. Results and discussion
discussion of this paper. After reading all the abstracts, we
gained insight into the mutual categories of the identified This section presents the results of our work. The keyword
research works. The categories used to cluster the papers analysis has yielded interesting insights that will be discussed
were: in section 3.1. From reading the research papers, we found
● Industrial domain the research is associated with (e.g., that this topic still has several research gaps that remain to be
manufacturing, food & agriculture) addressed. This includes the cost reduction of image base
● Methodology which is used to perform the quality quality control systems, merging of images from all sides of a
inspection (e.g., ANNs, machine learning, deep (complex) product, and acquisition of images due to difficult
learning, statistical methods) conditions such as a reflective surface of a product.
● Technical and equipment setup used for data collection
(e.g., single- or multi-camera, robots, laser) 3.1 Keywords
● Application of chosen method (e.g. quality control &
inspection, classification, real-time quality control) The keyword analysis on the final 224 papers related to
● Automation level of the solution (e.g. fully automated “Image Based Quality Inspection” resulted in a table with 224
and semi-automated) papers and 4054 keyword rows. The analysis and grouping of
these papers have been done in several steps. After regrouping
Keywords of the included papers were beneficial because the keywords as described in section 2, the final result is that
they indicate the content of a document which can help sort 868 unique keywords are represented in these 224 papers.
the documents into groups. The following work was The most repeated keyword was related to “Quality
completed to sort each paper under the right category. The Control and Inspection“. 160 papers used these keywords
papers were sorted using Tableau Prep Builder based on directly. These keywords were included in the initial search
keywords. Tableau Prep Builder is a software that helps easily string through the database so this result was expected.
combine, shape and clean data for future analysis. We noticed The keywords “Image Processing and Analysis”,
that index keywords included all author keywords as well as “Computer Vision”, “Camera Systems”, “Image
additional ones. Therefore, we decided to solely work with the Segmentation”, and “Computer Image Analysis” were used a
index keywords as we gained more insight on the paper total of 363 times, indicating a growing focus of researches on
contents and not duplicated the author keywords. The output the area of using computerized systems for quality control
was a new CSV file which was then manually edited using versus the old fashioned human inspection process.
Excel because many keywords used in the papers were “Artificial Neural Networks”, “Learning Systems”,
synonyms and there were many duplicates that needed to be “Machine Learning”, “Deep Learning”, “Artificial
removed to avoid double counting. In order to be able to Intelligence” repeated 136 times collectively which further
group the papers into correct groups, we selected one name indicates the increase in usage of AI and machine learning
for the keyword. Each paper category was decided by reading algorithms in quality control and inspection systems in recent
the title, abstract and keywords. For example, “Quality years.
Inspection”, “Quality Assurance”, “Quality Evaluation”, Out of 224 papers, 78 papers used “Defect Detection”
“Qualitycontrol”, “Quality Control”, “Quality of Production”, techniques for quality inspection and control purposes. For
“Quality of Product” etc are all referring to the same area of example, Maggipinto et al. used a deep learning approach for
research so we decided to group all to “Quality Control & defect detection of two dimensional input data in
Inspection”, and remove the duplicate ones. Or in another semiconductor manufacturing [4].
example, we decided to group “Citrus Fruits”, “Dried Fruits”, Implementation of an “Automated Approach” was the
“Freshfruits”, “Fruit and Vegetables”, “Fruit”, and “Fruits” focus of 59 papers for quality control and inspection with the
keywords to “Fruits”. This approach helped us to be able to help of computers and robotic systems. This low result
group all papers in the same area together and to reach a indicates that there is a need for further development of an
better understanding of the works done and the gaps in the automated image based quality inspection.
literature. The focus of 30 papers was to implement a “Real-time
While the keyword sorting on 224 papers was being done, Inspection” system that can be used to make the quality
papers whose topic was “Quality Classification” were inspection endeavour more efficient and smart. Real-time
separated and the remaining 160 papers associated with the inspection tries to complete the quality inspection task during
topic “Quality Control and Inspection” were read completely the manufacturing process to help reduce the production of
to gather detailed insights based on the previously discussed faulty units [5]. ”Automated Approach” and “Real-time
categories, in addition, the main focus and encountered Inspection” keywords seem to have a positive correlation
problems of each research were recorded. While “Quality together and fully automated methods are used for real-time
Classification” papers do give valuable information that can quality inspection as well. For instance, Schmitt et al
aid in better development of quality inspection as both developed an automated real-time machine vision that ensures
processes scan for differences in products, the focus of our the execution of all process steps, i.e. image acquisition,
review is to gather information on what had been done in the processing, and evaluation, under real-time conditions in the
field of “Quality Control and Inspection” in order to showcase automotive industry [6].
 Milica Babic et al. / Procedia CIRP 103 (2021) 262–267 265
4 Milica Babic et al. / Procedia CIRP 00 (2019) 000–000

“Quality Classification” was the focus of 60 papers. They 3.2 Main focus areas
tried to develop solutions to classify inspected products into
different groups based on their different characteristics. A literature review is an overview of past research studies
35 papers were set in “Manufacturing” industries and 45 which aids in forming future ones. It is important to
papers were in the “Agriculture & Food industry”. In total, 80 understand the reasoning behind the past studies because it
papers were related to manufacturing, agriculture, and food provides valuable insight on what areas have been thoroughly
related industries and indicate a high usage of image based researched and which need more contribution. The following
quality inspection in these particular industries. sections discuss the results of the main focus of a thorough
Other than using image based quality inspection techniques paper review on 160 papers previously mentioned in section
in manufacturing, there has been interesting use of image 2.
based quality inspection techniques in the “medical industry”
and the application can be very interesting. Hoshikawa et al
used image analysis algorithms to compensate for the lack of
QC tools for non-invasive monitoring of cultured cells [7].
Habart et al. High-Quality Applied machine learning
algorithms on microscopic images to segment pancreatic islets
and calculate islets volume and purity [8].
To sum up the keyword analysis section, it can be safely
stated that all included research papers have in common that
they are trying to make the procedures more connected,
smarter, and thus their application aligns with the industry 4.0
paradigm.
Table 1. Keywords, their frequency and frequency percentages.

Unique Index Keyword Frequency Frequency (%) Fig. 2. Pie chart: Representation of paper focus area distribution
Quality Control & Inspection 160 71.4
Most of the surveyed research papers were focused on
Image Processing and Analysis 129 57.6
evaluating an existing or new setup of an Image Quality
Computer Vision 113 50.4
Control System. This indicates that there are still many
Defect Detection 78 34.8
components in this topic that require further development and
Quality Classification 60 26.8 investigation. Elmasry, G et al. study provides an overview of
Automated Approach 59 26.3 traditional meat quality assessment techniques in the food
Artificial Neural Network 57 25.4 industry and then present hyperspectral imaging technology as
Agriculture & Food Industry 45 20.1 a non-destructive substitution to those methods. They mention
Camera Systems 42 18.8 traditional methods do not fulfil the modern food processing
Image Segmentation 40 17.9 industry, they are costly, depend on labor and are very time
Computer Image Analysis 39 17.4 consuming. These methods are only capable of sampling
Manufacturing 35 15.6 either a very small area or a very small number of the overall
Real-Time Inspection 30 13.4 products, and thus not efficient. Then they present
Visual Inspection 28 12.5 hyperspectral imaging systems as a rapid and non-invasive
Learning systems 23 10.3 technique which have many advantages over traditional and
even some new techniques. The advantages include reduced
The first set of 15 keywords with a high frequency can be labor costs, elimination of human error during subjective
seen in Table 1. These keywords’ frequency is more than 10% judgment, and the creation of product data in visualized forms
and they can be considered as highly repeated keywords. The in real-time for documentation, traceability, and labelling. All
following list of 17 keywords whose frequency is between 5% of the other papers we examined that compared human
and 10% and can be considered as frequently repeated operators to a fully or semi-automated system mentioned that
keywords: Image Color Analysis, Feature Extraction, the automated system is more accurate and faster but also
Machine Learning, Machine Vision Systems, Algorithm, presents additional problems which we will discuss in more
Deep Learning, Artificial Intelligence, Decision Trees, Image detail later in this review. It is interesting that most of the
Enhancement, Image Texture Analysis, Principal Component computer focused solutions include a human operator which
Analysis, Control Methods, Conventional Methods, Data contributes to the focus of Industry 4.0. The analysis shows
Analysis, High Quality Images, Segmentation, and Robotics. that 32% of the papers compared a human operator to a
An additional set of 29 keywords with a frequency machine and only 4% compared existing systems to the newly
between 3% and 5% can be considered as averagely repeated developed ones. From this, we can deduce that many
keywords. Keywords with a frequency below 3% that are not companies may have not yet implemented this type of
specifically mentioned in this paper account for an additional inspection process and only a small number of companies are
808 unique keywords. Some of these keywords can be already working more in-depth with these systems. Which
discarded because they have no relation between the keyword would indicate that this type of advanced automated image
and our research area. The rest can be analyzed and based quality inspection is still at its early stages when it
considered for future works as it seems there was not enough comes to industrial application.
focus on them.
266 Milica Babic et al. / Procedia CIRP 103 (2021) 262–267
Milica Babic et al./ Procedia CIRP 00 (2019) 000–000 5

3.3 Software and setup reasoning behind this was that the focus on the object defect
was the dimensions of the product from usually one or two
In this section, we discuss the use of software and other sides. This shows that most of the papers focused on a single
supporting tools for the image based quality inspection and product type rather than creating a system that could analyze a
control tasks reported in the identified set of papers. We have whole object with multiple defect types.
found that most researchers either developed their own One of the most interesting ideas that apply to Industry 4.0
software (15%) or did not indicate (8%) which software was was a paper in which researchers developed a phone software
used for their work. The only notable software used were that uses images taken by an employee with a mobile phone
Computer Aided Design (CAD) (10%) and MatLab (6%). The camera to analyze defects [9]. Phone cameras are becoming
CAD suite was used only to merge the acquired images of an more advanced and this paper proves that they are enough to
object into a three dimensional digital representation which is complete an image based quality inspection. By editing the
later employed for image analysis by a different software. image to clear problems such as blurriness and using a trained
Zhou W. et al. proposed a machine vision system for the deep neural network to identify the object in need of
detection of defects occurring on the surface of bottle caps in inspection, the software eliminates the need for a reference
the packaging industry. The setup that is used for capturing image.
the images of caps surface includes a monochrome high-speed
camera, a light source, a sensor and a computer to process the 4. Current research gaps
images. The proposed method is based on using the projection
of the template on the 360 orientations as the template Our work has shown that there is a lot of room for
dictionary and the diagonal matrix form the defect dictionary improvement in this field. Based on the above results where
using a circular region projection histogram (CRPH) and most research designs included human operators, it can be
sparse representation techniques for detecting the defect of seen that indeed Industry 4.0 has influenced the push for
caps surfaces. This process creates many problems such as automated processes and with it image based quality
data overload due to a high number of image pixels or a inspection and control through adaptation of machine vision
wrong image overlap that causes a wrong defect analysis. techniques have risen [10], [11], [12]. Many papers which
Other mentioned software used image analysis to indicate focus on contributing to automation within this topic do so by
defect placement and statistical details about the defect such only partially automating their overall procedure which
as placement on the axis, size, and color. actually contributes to Industry 4.0 because they have human
Chen, T. et al. presented a machine vision system for inclusion, but this is not mentioned in the papers. It is evident
circular can-end inspection in the food and beverage industry, that the knowledge of Industry 4.0 needs to be more
which is used to detect all the surface defects on their bottom widespread.
surface in real-time. The emphasis is put on image processing Many software and high resolution cameras are a possible
algorithms for defect detection across different can-end cost issue if not initially discussed, and along with
regions. A multistep image analysis line including a light complicated setup could be some of the bigger reasons for
source and a single camera is proposed for various defect companies not to implement an Image based Quality
detections across different parts of the can-end surface. First, Inspection. Problems using multiple cameras apart from price
the can end is located by combining entropy rate clustering was putting the pictures together to form a 3D image. High
algorithm with prior shape constraint. Then, a superpixel resolution cameras have proven of less benefit and caused
clustering and selection algorithm is proposed for defect problems such as system data overload. This raises a question
detection in the central panel and a multiscale ridge detection as to how the resolution level should be determined.
algorithm is adopted for defect detection along the other three Image analysis proved to have several key problems that
annular measuring regions. not many papers address. We can separate image contents by
He Z. et al. developed a control charting methodology to the object of focus and background. Both could have
detect multiple visual defect clusters in images of industrial problems such as blurriness, light, colour and background
products. The methodology is suitable for products whose object placement within the image in cases where there is a
quality is characterized by uniformity (e.g. defects in LCD referenced image. Some of the few solutions are applying
monitors) or a specific pattern (e.g. manufactured tiles) that neural networks and object segmentation. Objects with
has to be followed and deviations from this pattern are to be reflective surfaces cause problems due to reflected
detected. To achieve this objective, they proposed a background noise.
multivariate generalized likelihood ratio (MGLR)-based The keyword grouping indicated and the thorough paper
spatiotemporal control charting method for image monitoring. review confirmed that there is not one computer system that
Using this method makes it possible to detect multiple faults dominates the field. On the contrary, many researchers
and also allows to estimate the magnitude and the size of the decided to develop their own image based control computer
fault. They used a single camera setup taking grayscale system. This indicates that there needs to be more emphasis
images to acquire needed images [20]. on software comparison research which would help
Along with these works, the setup of 86% of papers was companies implement this type of quality inspection system
single camera based. The cameras of 20% of papers were and train employees.
operated by humans, and the rest were automatically taking
photos and therefore had to be fixed to a surface or a mount. 5. Conclusion
In our set of papers, 14% focused on obtaining images of
objects from all sides in order to complete a three-dimensional It is evident that a lot of effort has been turned towards
inspection while the rest focused on only specific sides. The advancing quality control in the last decade. Machines are
 Milica Babic et al. / Procedia CIRP 103 (2021) 262–267 267
6 Milica Babic et al. / Procedia CIRP 00 (2019) 000–000

only as smart as the people that made them and for this References
reason, much of the research that has been done under the
claim of the fourth Industrial Revolution concluded that [1] Locht, P., Thomsen, K., & Mikkelsen, P., 1997, Full colour image
analysis as a tool for quality control and process development in the
quality control systems should be developed to aid human food industry, ASAE Annual Int’l Meeting, Paper No. 973006.
operators. Repetitive jobs and especially those which require [2] General Electric , 2016, The Impact of Digital on Unplanned
Downtime, retrieved from:
high attention to detail cause high fatigue and more quickly https://www.ge.com/digital/sites/default/files/download_assets/ge-
decrease a human operator’s cognition [13]. An automated the-impact-of-digital-on-unplanned-downtime.pdf
[3] Diesing, G., 2020, How to Automate your Inspection Process,
system that is made to ease an operator’s work and make it Quality Magazine-online, retrieved from:
more reliable falls under Industry 4.0 focus. It can be seen that https://www.qualitymag.com/articles/96009-how-to-automate-your-
inspection-process
there is a strong push for Fully Automated Artificial [4] Maggipinto, M., Beghi, A., Susto, G.A., 2019, A deep learning-based
Intelligence algorithms used for product inspection in the approach to anomaly detection with 2-dimensional data in
manufacturing, IEEE International Conference on Industrial
research community [14]. Based on our results, automation is Informatics (INDIN) 8972027, pp. 187-192
a strive in this topic but is still an area with a lot of [5] Cho, S-H, Lee, K-T, Kim, S-H, & Kim, J-H, 2019, Image Processing
for Sustainable Remodeling: Introduction to Real-time Quality
improvement possibilities. There is a fine line between full Inspection System of External Wall Insulation Works, Sustainability
automation and Industry 4.0 that has to be treated more 11(4), 1081; doi:10.3390/su11041081
[6] Schmitt R., Fürtjes T., Abbas B., Abel P., Kimmelmann W., Kosse
carefully because if a human operator is fully replaced by a P., Buratti A., 2015, Real-time machine vision system for an
machine, that operator will have idle time that must be automated quality monitoring in mass production of multiaxial non-
crimp fabrics, 15th IFAC Symposium on Information Control
addressed where Industry 4.0 is focused on the solution for Problems in Manufacturing, 28(3), pp. 2393-2398
that. [7] Hoshikawa E., Sato T., Kimori Y., Suzuki A., Haga K., Kato H.,
Tabeta K., Nanba D., Izumi K., 2019, Noninvasive measurement of
There are many image quality control setup ideas and cell/colony motion using image analysis methods to evaluate the
newly developed systems, recently more have been concerned proliferative capacity of oral keratinocytes as a tool for quality
control in regenerative medicine, Journal of Tissue Engineering, 10,
in acquiring 3D imaging and our expectations are that future doi: 10.1177/2041731419881528
research on this topic would be directed towards further [8] Habart D., Švihlík J., Schier J., Cahová M., Girman P., Zacharovová
K., Berková Z., Kříž J., Fabryová E., Kosinová L., Papáčková Z.,
automation of 3D inspection. We have noticed that some of Kybic J., Saudek F., 2016, Automated analysis of microscopic
the greatest problems of this research topic are image images of isolated pancreatic islets, Cell Transplantation 25(12), pp.
2145-2156
acquisition of reflective surfaces, high fixed and variable costs [9] De Mitri, O., Stahl, S., Jauch, C., Distante, C., 2019, Image
of image analysis software and system components, and acquisition, evaluation and segmentation of thermal cutting edges
using a mobile device, Multimodal Sensing: Technologies and
accuracy issues of product position changes due to flexible Applications, 11059, p. 110590U
material use such as rubber. The simplicity of the process [10] Shen, H., Li, S., Gu, D., & Chang, H., 2012, Bearing defect
inspection based on machine vision, Measurement: Journal of the
should also be a factor in further research as employee International Measurement Confederation, 45(4), pp. 719–733.
training is a factor in a company’s decision of new system [11] Manish, R., Venkatesh, A., & Ashok, S. D., 2018, Machine vision
based image processing techniques for surface finish and defect
implementation. inspection in a grinding process, Mater. Today, Proc., 5(5), pp.
The limitations of the conducted study include that some of 12792–12802.
[12] Manzano J.E.M., Bolaños T.A.H., Muñoz M.Á.O., Arenas V.E.P.,
the mentioned contexts and results in this paper are based on Orozco H.P., 2020, Evaluation of a machine vision system applied to
the authors’ understanding of the problems and they were quality control in a liquid filling, lid and labelling line for bottles,
Communications in Computer and Information Science, 1280, pp.
made under certain assumptions, time, and resource 140-154.
limitations. While we developed and followed a transparent [13] Lughofer E., Smith J., Tahir M., Caleb-Solly P., Eitzinger C., Sannen
D., Nuttin M., 2009, Human–Machine Interaction Issues in Quality
methodology and applied the four-eye principle, there might Control Based on Online Image Classification, IEEE Transactions on
still be a bias in the interpretation of the results. Another Systems, Man, and Cybernetics - Part A: Systems and Humans,
39(5), pp. 960-971, doi: 10.1109/TSMCA.2009.2025025.
limitation is that the keyword analysis is not independent of [14] Huang, S.-H. & Pan, Y.-C., 2015, Automated visual inspection in the
the chosen search term and thus is biased to some extent as semiconductor industry: A survey, Comput. Ind., 66, pp. 1–10.
[16] Elmasry, G., Barbin, D.F., Sun, D.-. & Allen, P., 2012, Meat Quality
well. Finally, some additional papers may get indexed after Evaluation by Hyperspectral Imaging Technique: An Overview,
our initial database search so they may not be considered here, Critical reviews in food science and nutrition, 52(8), pp. 689-711.
[17] Zhou, W., Fei, M., Zhou, H. & Li, K., 2014, A sparse representation
and the result may differ under different assumptions. based fast detection method for surface defect detection of bottle
caps, Neurocomputing, 123, pp. 406-414.
[18] Chen, T., Wang, Y., Xiao, C. & Wu, Q.M.J., 2016, A machine vision
Acknowledgments apparatus and method for can-end inspection, IEEE Transactions on
Instrumentation and Measurement, 65(9), pp. 2055-2066.
[19] He, Z., Zuo, L., Zhang, M. & Megahed, F.M., 2016, An image-based
This material is based upon work supported by the U.S. multivariate generalized likelihood ratio control chart for detecting
Department of Energy's Office of Energy Efficiency and and diagnosing multiple faults in manufactured products,
International Journal of Production Research 54(6), pp. 1771-1784.
Renewable Energy (EERE) under the Advanced
Manufacturing Office Award Number DE-EE0007613.
Disclaimer: This report was prepared as an account of
work sponsored by an agency of the United States
Government. Neither the United States government nor any
agency thereof, nor any of its employees, makes any
warranty, express or implied, or assumes any legal liability or
responsibility for the accuracy, completeness or usefulness.
Additionally, this work was supported by the J. Wayne &
Kathy Richards Faculty Fellowship in Engineering at West
Virginia University.