buildings

Article
High-Rise Building Construction Progress Measurement from
Top View Based on Component Detection
Jingguo Xue * and Xueliang Hou

School of Economic and Management, North China Electric Power University, Beijing 102206, China;
[email protected]
* Correspondence: [email protected]

Abstract: The construction progress of a high-rise building is hidden by clutter such as formwork,
wood slats, and rebar, making it difficult to measure its progress through existing automated tech-
niques. In this paper, we propose a method to monitor the construction process of high-rise buildings.
Specifically, by using the target detection technique, unfinished building components are identified
from the top view, and then the identified components are registered to the BIM elements one by one.
This is achieved by comparing the position relationship between the target detection results and the
projection area of the BIM elements on the imaging plane. Finally, the overall construction progress is
inferred by calculating the number of identified and registered components. The method was tested
on a high-rise building construction site. The experimental results show that the method is promising
and is expected to provide a solid basis for the successful automatic acquisition of the construction
process. The use of top view reduces occlusion compared to similar methods, and the identification
of the unfinished component makes the method more suitable for the actual construction sites of
high-rise buildings. In addition, the combination of target detection and rough registration allows
this method to take full advantage of the contextual information in the images and avoid errors
caused by misidentification.






Citation: Xue, J.; Hou, X. High-Rise
Keywords: high-rise building; construction process monitoring; building component detection;
Building Construction Progress building information modeling; top view
Measurement from Top View Based
on Component Detection. Buildings
2022, 12, 106. https://doi.org/
10.3390/buildings12020106 1. Introduction
Academic Editor: Junbok Lee Progress measurement is a daily task in construction management, providing reliable
support for decision making in the field. Traditional construction progress measurement
Received: 28 December 2021 relies heavily on manual work and has been criticized by Architecture, Engineering and
Accepted: 20 January 2022
Construction (AEC) practitioners for its repetitiveness, inefficiency and error-prone na-
Published: 23 January 2022
ture. Over the past decade, many automated technologies have emerged, including radio
Publisher’s Note: MDPI stays neutral frequency identification (RFID), bar codes, ultra-wideband (UWB) tags, global position-
with regard to jurisdictional claims in ing systems (GPS), three-dimensional (3D) laser scanning, image-based modeling, and
published maps and institutional affil- more. These technologies provide a wealth of support for automated monitoring of the
iations. construction process. In particular, 3D reconstruction (including laser-based scanning
and image-based modeling) is favored by researchers as a result of its ability to visually
reproduce the 3D structure of buildings.
In recent years, the number and height of high-rise buildings around the world have
Copyright: © 2022 by the authors.
increased dramatically due to economic development, migration of people from rural to
Licensee MDPI, Basel, Switzerland.
urban areas, and advances in construction technology and materials [1,2]. Estimates have
This article is an open access article
predicted that these trends will continue in the future. Tracking the construction process
distributed under the terms and
of such buildings is an important task. However, the existing literature seems to pay less
conditions of the Creative Commons
Attribution (CC BY) license (https://
attention to it, which may be related to the characteristics of tall building construction. For
creativecommons.org/licenses/by/
example, the same unit is repeatedly constructed in the vertical direction; the horizontal
4.0/).
working plane is very narrow and there is usually no suitable platform to install monitoring

Buildings 2022, 12, 106. https://doi.org/10.3390/buildings12020106 https://www.mdpi.com/journal/buildings

Buildings 2022, 12, x FOR PEER REVIEW 2 of 22
Buildings 2022, 12, 106 2 of 21

working plane is very narrow and there is usually no suitable platform to install monitor-
equipment;
ing equipment; and and
the building
the buildingfacades are blocked
facades are blocked by scaffolding
by scaffolding andandprotective nets,
protective so
nets,
itsoisitdifficult
is difficultto obtain
to obtain thethe
latest construction
latest constructionprogress
progressfrom
fromthetheground
groundlevel.
level. Therefore,
Therefore,
whether
whetherthe theexisting
existingmethod
methodisisapplicable
applicableto tohigh-rise
high-rise buildings
buildings needs
needs further
further discussion.
discussion.
Laser-based
Laser-based scanningscanningallows allows thethe direct
direct acquisition
acquisition of 3D of 3D surface
surface models of models of
buildings
buildings [3–6]. This technique has the advantages of being fast, accurate,
[3–6]. This technique has the advantages of being fast, accurate, and independent of light, and independent
of light, however
however it is not accepted
it is not accepted by most companies
by most companies due to expensive
due to expensive equipment equipment and
and high tech-
high
nicaltechnical
requirementsrequirements
[7–10]. As [7–10].
shown Asinshown
Figurein 1,
Figure 1, the construction
the construction site of asite of a high-rise
high-rise build-
building is chaotic,
ing is chaotic, including
including building
building materials
materials and and construction
construction tools.
tools. More More importantly,
importantly, the
the building under construction is composed of formwork and
building under construction is composed of formwork and steel reinforcement, rather steel reinforcement, rather
than
thanthe theconcrete
concreteelements
elementsthemselves.
themselves. At At this
this time,
time, the
the point
point cloud
cloud scanned
scanned by bythe
thelaser
laser
technology consists of points that are mostly useless. From this perspective,
technology consists of points that are mostly useless. From this perspective, laser scanning laser scanning
technology
technologyisissuitablesuitable forfor scanning
scanning completed
completed components,
components, not not those
those under
under construction.
construction.
Therefore,
Therefore, the laser-based measurements are not suitable for high-risebuildings.
the laser-based measurements are not suitable for high-rise buildings.

Figure1.1.Examples
Figure Examplesof
ofchaotic
chaoticconstruction
constructionsites.
sites.

In
In recent
recentyears,
years,image-based
image-based modeling
modeling has has attracted
attracted increased
increased attention
attention due
due to to the
the
low cost of image acquisition [11–19]. Unlike laser scanning, it generates
low cost of image acquisition [11–19]. Unlike laser scanning, it generates 3D point clouds 3D point clouds
from
fromimages
imagesbasedbasedon onthe
theprinciple
principle ofoftriangulation
triangulation imaging.
imaging. However,
However, in the construction
in the construc-
site
tionofsitea high-rise building,
of a high-rise the concrete
building, the concreteelements are surrounded.
elements are surrounded. As shown
As shownin Figure
in 1,
what
Figure can1,be captured
what can beare formwork,
captured wood strips,wood
are formwork, steel strips,
bars, and
steelother
bars,debris. As with
and other 3D
debris.
laser
As with scanning,
3D laserthescanning,
generatedthe point cloud contains
generated point cloudtoo much
containsuseless information
too much uselessrather
infor-
than the components. Therefore, it is clear that the image-based
mation rather than the components. Therefore, it is clear that the image-based modelingmodeling is not suitable
for such
is not a complex
suitable environment.
for such a complex environment.
Some
Some scholars projectBIM
scholars project BIMmodels
models onto
ontothethe
imaging
imaging plane
plane andand then infer
then construction
infer construc-
progress by identifying materials in the projected area [9,20,21].
tion progress by identifying materials in the projected area [9,20,21]. This approach This approach avoids the
avoids
identification
the identification of components
of componentsfromfromnoisynoisy
pointpoint
clouds. However,
clouds. However,in theinavailable literature,
the available liter-
this technique has not been applied to complex construction sites.
ature, this technique has not been applied to complex construction sites. The reason is that The reason is that it
isit not possible to capture the latest construction progress as the concrete
is not possible to capture the latest construction progress as the concrete components components are
tightly
are tightlywrapped
wrappedby formwork
by formwork and and
protective nets.nets.
protective Therefore, it is difficult
Therefore, to monitor
it is difficult to monitorthe
construction process of high-rise buildings through the materials
the construction process of high-rise buildings through the materials in the projection in the projection area.
area.In conclusion, the common feature of the above methods is the identification of the
completed components.
In conclusion, This is impractical
the common feature ofwhen measuring
the above methodsthe construction progressofofthe
is the identification a
high-rise
completed building. On-site
components. concrete
This pouringwhen
is impractical and continuous
measuring the construction
construction areprogress
character- of
istics of high-rise construction, which means that it is impossible to visualize the latest
a high-rise building. On-site concrete pouring and continuous construction are character-
construction progress.
istics of high-rise construction, which means that it is impossible to visualize the latest
While the above methods do not provide an ideal solution, it has been found that the
construction progress.
construction process of a high-rise building can be observed from an overhead perspective.
While the above methods do not provide an ideal solution, it has been found that the
Moreover, it is easy to collect daily unordered construction images from the tower crane.
construction process of a high-rise building can be observed from an overhead perspec-
Thus, this paper proposes a method to measure the construction progress of high-rise
tive. Moreover, it is easy to collect daily unordered construction images from the tower
buildings from the top view. Specifically, the top view of the construction site is collected
crane. Thus, this paper proposes a method to measure the construction progress of high-
by the tower crane workers; the building components are identified from the top view
rise buildings from the top view. Specifically, the top view of the construction site is col-
and mapped to BIM elements; and the construction progress is inferred from the number
lected by the tower crane workers; the building components are identified from the top
of registered components. The innovations of this study are mainly in the following
view and mapped to BIM elements; and the construction progress is inferred from the
three aspects:
number of registered components. The innovations of this study are mainly in the follow-
1. Identify the auxiliary objects (formwork and reinforcement), not the completed con-
ing three aspects:
crete elements, according to the actual situation;
Buildings 2022, 12, 106 3 of 21

2. Using only the top view to measure construction progress, as monitoring from the
side is not practical for high-rise buildings;
3. The reliability of the results is improved by using rough registration to locate the
target detection results.
The objective of this study is to measure the construction progress of high-rise build-
ings, which is different from existing studies. In order to highlight the uniqueness of this
study, the assumptions are listed as follows:
1. The construction of the high-rise building is carried out by pouring concrete on site,
and the concrete components are wrapped by formwork;
2. The latest construction progress cannot be collected from the ground or sides due to
the obstruction of the protective net;
3. The monitoring object is the latest construction progress of the structure, not the
indoor construction progress.
This paper is structured as follows. First, it starts with an introduction to the related
work about building component detection and construction progress measurement. The
next section outlines the proposed method and explains in detail the three important steps:
target detector training, camera calibration, and registration and reasoning. Next, the
method is validated by tracking and measuring the progress of an actual high-rise building
project. Three results (detection, registration and reasoning) and error cases are analyzed in
detail. Finally, the advantages of the proposed method are summarized and highlighted by
comparing it with similar techniques.

2. Related Work
2.1. Building Component Detection
In the past decade, with the breakthrough of deep learning, target detection technology
has made unprecedented developments. The task of target detection is to find out all
interested targets (objects) within an image and determine their categories and locations,
which is one of the core problems in the field of computer vision. In the AEC industry,
target detection technology has been applied to the detection and tracking of construction
workers [22–25], machines [26,27], equipment [28] and building components [29]. For
example, Park and Brilakis [25] focused on the continuous localization of construction
workers via integration of detection and tracking. Zhu et al. [28] identified and tracked
workforce and equipment from construction-site videos.
As for the recognition of building components, the existing methods can be divided
into three categories:
1. Geometric reasoning (align the 3D point cloud and BIM model, calculate the space
occupied by the point cloud, and infer the components from the geometric point of
view) [11,30];
2. Appearance-based reasoning (the BIM model is projected to the imaging plane, and the
material texture of the projection area is identified to determine the component) [31,32];
3. Target detection (directly using object detection technology to identify building com-
ponents from the image) [29,33].
Each technology has its advantages. Among them, the first two methods have achieved
a lot, while the latter technology is emerging.
At present, building component recognition based on target detection has produced
some results. For example, some scholars use target detection technology to identify con-
crete areas from images. However, since many components are made of concrete, the
detected concrete areas are interconnected, and the building components cannot be subdi-
vided [34,35]. To overcome this limitation, Zhu and Brilakis [36] proposed an automatic
detection method for concrete columns based on visual data. By analyzing the bound-
ary information (such as color and texture) of concrete columns, the structural columns
were separated from the concrete area. Recently, Deng et al. [15] presented a method
that combines computer vision with BIM for the automated progress monitoring of tiles.
Buildings 2022, 12, 106 4 of 21

Wang et al. [37] proposed a novel framework to realize automatically monitoring construc-
tion progress of precast walls. Hou et al. [29] proposed an automatic method for building
structural component detection based on the Deeply Supervised Object Detector (DSOD).
In addition, the point cloud is usually inevitable in the 3D reconstruction of the building
structure. Some scholars use a deep learning method to detect building components from
the point cloud. For example, Bassier et al. [38] presented a method for the automated
classification of point cloud data. The data is pre-segmented and processed by machine
learning algorithms to label the floors, ceilings, roofs, beams, walls and clutter in noisy and
occluded environments.

2.2. Construction Progress Measurement

To improve the accuracy and feedback speed of information in the construction site,
quite a few technologies have been studied to realize automatic construction progress
measurement [39–42]. These methods can be classified into two categories: imaging
techniques and geospatial techniques. The advantages and limitations of these technologies
are shown in Table 1.

Table 1. Automatic construction progress measurement.

Type Method Advantage Limitation References

Automated data collection, Expensive equipment, high
Laser scanning high accuracy, not affected technical requirements, and [3–6,43,44]
by illumination limited texture information
Portable, relatively cheap
3D ranging camera short range applications [20,45,46]
and rich texture information
Imaging Low cost, low technical
techniques requirements, portable, high Sensitivity to lighting conditions
Image-based modeling [11–14,16,17]
resolution, and rich texture and high time complexity
information
Low cost, low technical
Material classification requirements, portable, high Sensitivity to lighting conditions
[31,32,47–49]
of projection area resolution, and rich texture and high time complexity
information
Wide positioning range and Low precision and limited to
GPS [50]
strong adaptability outdoor
Low cost and simple Time consuming and easily
Barcode [51,52]
operation requirements damaged
Geospatial
techniques Time consuming, and error
Wide practicability and no
RFID prone in the presence of metals [53–55]
need for light
and liquids
Reliable signals, longer read
High cost and no mini device or
UWB ranges, provide 3D position [56]
daily-necessity-embedded too
estimate

Imaging techniques include laser scanning, 3D ranging camera, image-based mod-

elling and material classification of projection area. 3D laser scanning technology, based on
laser ranging principle, can quickly reconstruct the 3D model of the measured object by
recording the 3D coordinates, reflectivity and texture of a large number of dense points
on the surface of the measured object [57]. This technology has unique advantages in
efficiency and accuracy, and is not affected by illumination [3–6]. However, there are
also some shortcomings, such as high cost, limited texture information and high technical
requirements for operation [7–10]. Therefore, the small and exquisite 3D ranging camera
is attracting scholars’ attention, and many scholars have carried out research based on
the helmet mounted equipment, especially the Kinect developed by Microsoft [43–46].
The 3D ranging camera is portable, relatively cheap and contains rich texture information.
Buildings 2022, 12, 106 5 of 21

Image-based modelling and material classification of the projection area make full use of
the flexibility and convenience of the camera to obtain information [11–17]. The advantage
of camera measurement lies in the low cost of equipment and the high-speed acquisition
of field data. However, the camera measurement technology requires high visible light
intensity of the environment.
Geospatial techniques include GPS, Barcode, RFID and UWB. GPS is a satellite-based
system widely used in position and navigation. The system receives signals from satellites
to locate the position of specific objects. GPS has been used for equipment and material
tracking and progress tracking [50]. Barcodes are widely used in material tracking, in-
ventory management, construction progress tracking and document management due
to its advantages of low cost, high reliability and easy identification [51,52]. RFID stores
data through chips, and some can even transmit active signals. Therefore, the information
stored in it can be easily read and written [53–55]. UWB is a kind of radio technology for
short-range communication. This has characteristics such as low power requirement, the
ability to work indoors and outdoors, long reading range and so on. It has been applied
to material tracking and activity-based progress tracking, especially in harsh construction
environments [56].

3. Methodology
Based on the above analysis, this section will introduce a method to measure the
construction progress from the top view of the construction site. As shown in Figure 2, the
method includes four main steps:
1. Train building component detector in advance, and identify the building components
from the top view;
2. Establish the mapping relationship between the 3D world (BIM model) and the
imaging plane through the camera calibration;
3. Register the detected components with BIM elements by comparing the projection
position of BIM elements in the imaging plane;
4. Infer the construction progress with the number of detected and registered components.
As mentioned in the introduction section, the construction process of high-rise build-
ings is blocked by scaffolding and a protective net, and can only be collected from the top
view. Therefore, the images of this study are collected daily by the tower crane driver. As
such, these images must overlap each other. On the one hand, these images are inputted
into the target detector to obtain the detection results. On the other hand, they are used to
calculate the external parameters of each image to obtain corresponding projection models.
The remaining steps are discussed in detail in the following sections.

3.1. Target Detector Training

3.1.1. Objects to Be Identified
In the construction process, the formation of a building’s structural component mainly
goes through five stages: binding reinforcement cage, supporting formwork at the periph-
ery, pouring concrete, curing concrete, and removing formwork. Considering the construc-
tion technology, due to the mutual connection of components in same layer, the concrete
pouring, concrete curing and formwork removal processes of different components are
carried out at the same time, while the processes of steel bar binding and formwork support
are staggered. Therefore, it is impossible to observe all states of all components, and most
of the time, what can be photographed from the top view is their semi-finished products.
To identify which components and what states can be identified, the construction site
states of a floor in different periods are combed, as shown in Table 2. The time point after
the concrete pouring of the last layer is taken as the start of the construction of this floor.
Buildings 2022, 12, 106 6 of 21
Buildings 2022, 12, x FOR PEER REVIEW 6 of 22

Figure 2. Overview of the method.

Figure 2. Overview of the method.
3.1. Target
3.1.2. Data Detector Training
Acquisition and Annotation
3.1.1.At
Objects to Be Identified
present, there is no ready-made, open and integrated images dataset of building
components available in the
In the construction AEC industry
process, for building
the formation component
of a building’s recognizer
structural training.
component
To obtain
mainly a data
goes set that
through fivecan be used
stages: to train
binding the component
reinforcement cage,recognizer,
supportingaformwork
DJI Mavicat Air2
the
unmanned aerial vehicle was used to shoot five construction sites from
periphery, pouring concrete, curing concrete, and removing formwork. Considering the top view for 45 days.
Itconstruction
was shot twice a day and
technology, dueato total
the of 450 video
mutual clips were
connection collected. in
of components Tensame
images were
layer, the
intercepted from each
concrete pouring, video curing
concrete as the training set (4500removal
and formwork images processes
in total), and then 500 compo-
of different images
were
nentsintercepted
are carriedas the
out attest
the set.
same Totime,
createwhile
a comprehensive
the processesdata set, bar
of steel different perspectives,
binding and form-
zooms and lighting
work support conditionsTherefore,
are staggered. were considered.
it is impossible to observe all states of all compo-
Based
nents, and on
mosttheofanalysis of the
the time, whatconstruction-site environment,
can be photographed from theit istop
obvious
view that a compo-
is their semi-
nent shows various
finished products. forms in different periods, and it is unrealistic to recognize all states of
all components from the image due to occlusion. Therefore, in this study, only a few types
Buildings 2022, 12, 106 7 of 21

Buildings 2022,
Buildings 2022,
Buildings 12,
12, xxx FOR
2022, 12, FOR PEER
FOR PEER REVIEW
PEER REVIEW
REVIEW 777 of
of 22
of 22
22
Buildings 2022, 12, x FOR PEER REVIEW 7 of 22
of components are labeled and identified to infer the construction progress. The identified
objects include stacked formwork, walls or columns with formwork, beams enclosed by
formwork, Tobeams
To identify
identify with steel
which
which cages, slabs
components
components and
andwith bottom
what
what statesformwork,
states can be
can be slabs with
identified,
be identified,
identified, the steel cages, site
construction
the construction
construction site
To
Toidentify
identify which
which components
components and
andwhat
whatstates
states can
can be identified, the
the construction site
site
and states
pouringof
states of tools.
of aaa floor
floor The
in
floor in training
different
in different set
periods
different periods is annotated
are
periods are combed,
are combed, by
combed, asasthe graphic
shown
as shown
shown inin image
Table
in Table 2.
Table 2. annotation
The
2. The time
The time toolafter
point
time point
point after
states
states[58],
of a generating
floor in different periods are combed, as shown in Table 2. The time point after after
LabelImg
the concrete XML files in PASCAL VOC format.
concrete pouring
the concrete
the pouring ofof
of the last
last layer
the last is
is taken
layer is as
as the
taken as start
start of
the start the
the construction
of the of
of this
construction of floor.
this floor.
floor.
the concretepouring
pouring ofthe the lastlayer
layer istaken
taken asthe
the startof
of theconstruction
construction ofthis
this floor.
2. Construction
TableTable
Table 2. site states.
Construction
2. Construction
Construction site states.
site states.
states.
Table 2. site
Table 2. Construction site states.
Main
Main
Main Work
Main Work
Work Content
WorkContent
Content and
andand Recognizable
Recognizable
Recognizable Components
Components
Components
Stage
Stage
Stage
Stage Example
Exampleof
Example
Example Construction
of Construction
of Site
Construction Site
Construction Site Main WorkContent
Content and
and Recognizable
Recognizable Components
Components
Stage Exampleof
of ConstructionSite
Site Construction
Construction Site
Site
Construction Site
Construction States
States
Site States andand
States
and States
and States
States
Construction SiteStates
States and
and States
States

Work:
Work:
Work: Concrete curing
Concretecuring
curing of
of the
the last
last
Work: Concrete
Work: Concretecuring
Concrete curing ofof
thethe
of lastlast
the last
floor
floor
floor
floor
Start
Start
Start (Concrete
(Concrete
(Concrete
Start floor
Start(Concrete
(Concrete State:
State:
State: The
State:
The
The concrete
The slab
concreteslab
concrete is
slab is
slab flat,
flat, alt-
flat,
is flat, alt- - ----
curing)
curing)
curing)
curing) State: Theconcrete
although concrete
there areslab is flat,alt-
sporadic alt-
curing) hough there
hough there
hough are
there are sporadic
are sporadic protruding
sporadic protruding
protruding
hough there are sporadic
protruding steel bars.protruding
steel bars.
steel bars.
steel bars.
steel bars.

Work:
Work: Molds
Work: Molds
Molds and materials
materials prepara-
and materials prepara-
Work: Moldsand and materialsprepara- prepara-
tion and
Work:
tion and
tion formwork
Molds
and formwork and
formwork removal removal
materials
removal from from
from
tion and
preparation the formwork
andlastformwork removalremoval from
the last
the floor
last floor
floor
the last
from the last floor floor
State:
State:
State: Formwork,
Formwork,
Formwork, steel
steel bars,
bars, sup-
sup-
Preparation
Preparation
Preparation
Preparation State:
State: Formwork,steel
Formwork, steelbars,
steel bars, sup-
bars, sup- Stacked
Stackedformwork
Stacked
Stacked *
formwork
formwork **
Preparation ports
ports
ports and
and
supports other
and other
other
and tools are
toolstools
tools
other areare
are piled
piled
piledpiledup
up on
up on
on Stackedformwork
formwork**
ports and other tools are piled up on
the
theup
the floor,
on the
floor, which
floor, makes
which which makes
makes the construc-
the
the construc-
construc-
thefloor,
floor, which
whichmakes
construction makes
site
the
the construc-
chaotic.
tion site
tion site
tion site
tion site
chaotic.
chaotic.
chaotic.
chaotic.
Work: Steel
Work: Steel
Work: Steel bar bar binding
bar binding
binding and and form-
and form-
form-
Work:
Work: Steel
Steel barbarbinding
bindingand and form-
work
work
work supporting
supporting
supporting for columns
for for
for columns and
and
formwork
work supporting
supporting forcolumnscolumns
columns and
and
and shear
shear
shear walls
walls
shearwallswalls
shear walls
Facade
Facade
Facade State:
State:
State: The
State:
The
TheThe backbackof
back
back of the
ofofthe
the formwork
theformwork
formwork oc-
formwork oc- Walls
oc- Walls or columns
or columns
Walls
Walls withwith
or columns
or columns with
Facade construction
Facade State:
occupies Themost back of of
thethe areaformwork
in the top oc- Walls or columnswith
formwork with
construction
construction
construction cupies
cupies most
cupies most
most of of the
of the area
the area
area in in the
in the top
the top
top formwork
formwork
formwork
construction cupies
view, andmost of the
there are area
manyinbars the top formwork
view,
view,
view, and
andon there
there are
are many
many bars
bars ar-ar-
ar-
view,and
arranged andthere it, soare
there the many
are manybars
construction
bars ar-
ranged
rangedsite
ranged on it,
onisit, so the
it,relatively
so the construction
the construction site
site
rangedon on it, so
so theconstruction
regular.
construction site
site
is relatively
is relatively
is relatively regular.regular.
regular.
is relatively regular.
Work:
Work:
Work: Formwork
Formworkfor for
for slabs
slabs and
and
Work: Formwork
Work: Formworkfor
Formwork for slabs
slabs
slabs and
andand
beams
beams
beams
beams
beams Slabs
Slabs
Slabs
Slabs with
with
with
with bottom
bottom
bottom
bottom formwork
formwork
formwork
formwork
Formwork
Formwork
Formwork for
Formwork for
for
for State:
State:
State: The
The
State: The vertical templates
verticaltemplates
The vertical
vertical templates
templates are
are
areare Slabs with bottom formwork
Formwork
Formwork for
for plane State: The vertical templates are and
andand
and
plane
plane
plane gradually
gradually blocked,
blocked,
gradually blocked,
gradually leaving
leaving
blocked, leaving neatly
neatly
leaving neatly
neatly Beams enclosed and
by formwork
plane gradually
arranged blocked,
flat leaving
templates withneatly Beams
Beams
Beams enclosed
enclosed
enclosed by formwork
by formwork
by formwork
arranged
arranged
arranged flat
flat
flat templates
templates
templates with
with
with obvi-
obvi-
obvi- Beams enclosed by formwork
arranged flat templates
obvious boundaries.with obvi-
ous boundaries.
ous boundaries.
ous boundaries.
ous boundaries.

Work: Steel
Steel bar
Work: Steel
Work: binding
binding for
bar binding slabs
for slabs
slabs
Work: Steelbar bar bindingfor for slabs
State:
State:
State: There
There are steel
arebinding cages
steel cages
cages on the
on the
the
Steel bar
Steel bar
Steel bar Work:There
State: There are
aresteel
Steel bar steel cages on
for slabs
on the Slabs with
Slabs with
Slabs steel
with steel cages
steel cages and
cages and
and
Steel barSteel barin
binding slab,
slab,
slab, and
and
and
State:and the
the
the
There boundaries
boundaries
boundaries
areboundaries between
between
between
steel cages on the Slabs
Slabs with
with steelsteel
cages cages
and and
binding
binding in
binding in plane
in plane
plane slab, the between Beams with
Beams with
Beams steel
with steel cages
steel cages
cages
binding
plane in plane the
the
slab, and the boundaries
the between Beams
Beams withwith
steel steel
cages cages
the formworksbecome
the
become blurred.
blurred.
formworks
formworks
formworks become
become blurred.
formworks becomeblurred. blurred.
Buildings 2022, 12, 106 8 of 21

Table 2. Cont.
Buildings 2022, 12, x FOR PEER REVIEW 8 of 22

Main Work Content and Recognizable Components

Stage Example of Construction Site
Construction Site States and States

Work: Concrete pouring

Work: Concrete pouring
State: There are steel cages on the
State: There are steel cages on the
slab.
slab.With the pouring
With the pouringofofconcrete,
concrete, Slabs
Slabs with
with steel
steel cagescages
and and
Concrete
Concrete pouring the
thesteel
steel cages graduallydisappear,
cages gradually Beams
disappear, Beams with
with steelsteel
cagescages
and and
pouring and the boundaries of the Pouring toolstools
*
and the boundaries of the formwork Pouring *
formwork change from fuzzy
change from fuzzy to clear and then to
clear and then disappear.
disappear.

* To refine the construction progress

* To refine information,
the construction the stacked
progress formwork
information, and formwork
the stacked pouring tools are identified
and pouring tools areto assist reason-
identified to assist
ing. reasoning.

3.1.3.3.1.2.
Detector
Data Selection
Acquisition andand Training
Annotation
There are various target detection
At present, there is no ready-made, algorithms openinandthe integrated
field of computer
imagesvision.
datasetMost tar-
of building
get detection algorithms get initial weights from the pre-trained
components available in the AEC industry for building component recognizer training. model on image databases
like To
ImageNet,
obtain aand databuild thecan
set that finalbemodel
used tothrough
train the fine-tuning.
component There are twoaadvantages:
recognizer, DJI Mavic Air2
one is that many models are open source and can
unmanned aerial vehicle was used to shoot five construction sites be directly used for target
from detection;
top viewthe for 45
otherdays.
is that fine-tuning
It was shot twice cana quickly
day andget the of
a total final
450model, and the
video clips were training
collected.data
Tenrequired
imagesiswere
relatively small. from
intercepted However, the features
each video of building
as the training components
set (4500 images (such as beams,
in total), and then columns,
500 images
etc.) were
are extremely similar, and there are huge differences between them
intercepted as the test set. To create a comprehensive data set, different perspectives, and natural objects.
Therefore,
zoomsinand thelighting
previousconditions
work, Learning Deeply Supervised Object Detectors from Scratch
were considered.
(DSOD) [59] was selected to detect building
Based on the analysis of the construction-site components from site images.
environment, Combining
it is obvious that a com-
DenseNet network [60], the parameters of DSOD model are
ponent shows various forms in different periods, and it is unrealistic to recognize greatly reduced. Moreall im-
states
portantly, it breaks the traditional barrier of detector training
of all components from the image due to occlusion. Therefore, in this study, only a fewbased on pre-training and
fine-tuning,
types of so the highestare
components level of target
labeled anddetector
identified cantobeinfer
obtained by using limited
the construction data The
progress.
sets and training from scratch.
identified objects include stacked formwork, walls or columns with formwork, beams en-
In this study, two RTX 3080 Ti graphics processing units were used to train and test the
closed by formwork, beams with steel cages, slabs with bottom formwork, slabs with steel
DSOD model [61]. The training set was sent into the neural network after data enhancement
cages, and pouring tools. The training set is annotated by the graphic image annotation
(resize, zoom, flip, crop, illumination adjust, hue adjust). Finally, a building component
tool LabelImg [58], generating XML files in PASCAL VOC format.
recognizer is obtained. Using the trained recognizer, the building components (including
type, state and location) can be detected from the construction-site images. Note: In this
3.1.3. Detector Selection and Training
study, the labeled and identified components contain state information, so the recognition
There are
results include three various
kinds target detection type,
of information: algorithms in the
state and field ofFor
location. computer
example, vision.
whenMost
“slab with bottom formwork” is recognized, the component type and state are “slab” anddata-
target detection algorithms get initial weights from the pre-trained model on image
“with bases
bottom likeformwork”,
ImageNet, respectively.
and build the final model through fine-tuning. There are two ad-
vantages: one is that many models are open source and can be directly used for target
detection;
3.2. Camera the other is that fine-tuning can quickly get the final model, and the training
Calibration
data requiredof
The purpose is camera
relatively small. However,
calibration the features
is to determine of building
the mapping components
relationship (such as
between
the BIM model/3D world and the imaging plane. The mapping relationship is a camerathem
beams, columns, etc.) are extremely similar, and there are huge differences between
and natural
projection model,objects.
which isTherefore,
the basis for in the previousthe
establishing work, Learning
mapping Deeply
between theSupervised
BIM elements Object
and the target detection results in this paper. Camera calibration is the process of solving site
Detectors from Scratch (DSOD) [59] was selected to detect building components from
images.
the model Combining
parameters, DenseNet
including network
internal and [60], the parameters.
external parameters of The DSOD
internalmodel are greatly
parameters
reduced.
are related More
to the importantly,ofit the
characteristics breaks
camerathe itself,
traditional
such asbarrier of detector
the focal length andtraining
pixelbased
size, on
while pre-training
the externaland fine-tuning,
parameters so the
are the highest in
parameters level
theof target
world detector system,
coordinate can be obtained
such as by
using limited
the position data sets
and rotation and training
direction of thefrom
camera. scratch.
In this study, two RTX 3080 Ti graphics
To obtain the internal parameters, Zhang’s calibration processing units were
method usedin
was used tothis
train and test
study,
for itthe
hasDSOD
mature model [61]. Thewide
application, training set was range,
application sent into andthemore
neural network
reliable andafter data en-
accurate
hancement
calibration (resize,
results [62]. zoom, a 6 ×crop,
First, flip, illumination
8 square adjust, hue
black-and-white adjust). Finally,
chessboard is made a building
and
componentfrom
photographed recognizer
multiple is angles.
obtained. Using
Then, thesetheimages
trainedare recognizer,
importedthe intobuilding components
the Camera Cal-
ibrator toolbox in
(including MATLAB
type, state and to obtain the camera
location) can be parameters.
detected from Only thethe internal parameters
construction-site images.
are retained.
Note: In this study, the labeled and identified components contain state information, so
the recognition results include three kinds of information: type, state and location. For
example, when “slab with bottom formwork” is recognized, the component type and state
are “slab” and “with bottom formwork”, respectively.
 and pixel size, while the external parameters are the parameters in the world coordinate
system, such as the position and rotation direction of the camera.
To obtain the internal parameters, Zhang’s calibration method was used in this study,
for it has mature application, wide application range, and more reliable and accurate cal-
ibration results [62]. First, a 6 × 8 square black-and-white chessboard is made and photo-
Buildings 2022, 12, 106 graphed from multiple angles. Then, these images are imported into the Camera Calibra- 9 of 21
tor toolbox in MATLAB to obtain the camera parameters. Only the internal parameters
are retained.
To
Toobtain
obtainthe
theexternal
externalparameters,
parameters,ananOpenCV-based
OpenCV-basedprogramprogramisisintroduced,
introduced,which
which
can
canbebe used infer camera
used to infer cameraexternal
externalparameters
parameters through
through feature
feature points
points between
between images
images [63].
[63]. As shown
As shown in Figure
in Figure 3, the3,program
the program consists
consists of theoffollowing
the following steps:
steps:
1.1. ExtractScale
Extract ScaleInvariant
InvariantFeature
FeatureTransform
Transform(SIFT)
(SIFT)points
pointsfrom
fromimages
images[64];
[64];
2.2. Matchfeatures
Match featuresusing
usingk-nearest
k-nearestneighbor
neighbor(KNN)
(KNN)classification
classificationalgorithm
algorithm[65];
[65];
3.3. Calculatethe
Calculate theessential
essentialmatrix
matrixbetween
betweentwo
twoimages
imagesbyby“findEssentialMat”
“findEssentialMat”function;
function;
4.4. Calculate the external parameter matrices by “solvePnPRansac” function.
Calculate the external parameter matrices by “solvePnPRansac” function.

Artificial points EPnP algorithm

Initial external parameters
Initial images SIFT points
External
parameters
Daily images SIFT points KNN “findEssentialMat” “solvePnPRansac”

Figure 3. Calibration of external parameters.

Figure 3. Calibration of external parameters.

In
Inthis
thismethod,
method,thetheparameters
parameters of the initial
of the image
initial needneed
image to beto
known. Therefore,
be known. mul-
Therefore,
tiple known points are manually selected from the initial image, and the initial
multiple known points are manually selected from the initial image, and the initial external external
parameters
parametersare arecalculated
calculatedusing
usingthe
theEfficient
EfficientPerspective-n-Point
Perspective-n-Point(EPNP)(EPNP)algorithm
algorithm[66].[66].
To
Tomake
makethe thecalculation
calculationresults
resultsaccurate,
accurate,the
theinitial
initialimage
imageshould
shouldcontain
containasasmany
manyknownknown
points
pointsas aspossible,
possible,and
andthese
theseknown
knownpoints
pointsshould
shouldnot notbebeononthe
thesame
sameplane
planein inthe
the3D3D
world. In addition, the external parameters are obtained through the corresponding
world. In addition, the external parameters are obtained through the corresponding SIFT SIFT
points
pointsin intwo
twoimages,
images,which
whichisisindependent
independentof ofthe
theshooting
shootingposition.
position.Therefore,
Therefore,the
thelifting
lifting
of
ofthe
thetower
towercrane
cranedoes
doesnot
notaffect
affectthe
theresults
resultsof
ofcamera
camera calibration.
calibration.

3.3.Registration
3.3. Registrationand
andReasoning
Reasoning
3.3.1.BIM-to-Image
3.3.1. BIM-to-ImageRough
RoughRegistration
Registration
Thissection
This sectionestablishes
establishesthethemapping
mappingrelationship
relationshipbetween
betweenthe theBIM
BIMelements
elements(build-
(build-
ing components) and the detected components. It works by calculating
ing components) and the detected components. It works by calculating the positional re- the positional
relationship
lationship between
between thethe
BIMBIM projection
projection area
area and
and the
the detection
detection resultsininthe
results theimaging
imagingplane.
plane.
First, it is necessary to calculate the position of BIM elements in each image
First, it is necessary to calculate the position of BIM elements in each image plane. In plane.
In this study, the coordinates of each corner of BIM element are extracted manually at
this study, the coordinates of each corner of BIM element are extracted manually at once.
once. Through the transformation of the projection model, the coordinates of the corners
Through the transformation of the projection model, the coordinates of the corners of BIM
of BIM elements in the plane can be obtained. As show in Figure 4, each corner of the
elements in the plane can be obtained. As show in Figure 4, each corner of the component
component corresponds to a pixel point, which is named as ‘pixel-corner’. Then, the
corresponds to a pixel point, which is named as ‘pixel-corner’. Then, the exterior pixel-
exterior pixel-corners are connected to form a convex polygon, named ‘envelope frame’, so
corners are connected to form a convex polygon, named ‘envelope frame’, so that all
Buildings 2022, 12, x FOR PEER REVIEW
the
10 of 22
that all the pixel-corners are wrapped in the envelope frame and the component location
pixel-corners are wrapped in the envelope frame and the component location can be rep-
can be represented by these pixel-corners on the envelope frame.
resented by these pixel-corners on the envelope frame.

BIM elements

Pixel-corner Construction-site image

Envelope frame

Detected component

Figure4.4.Envelope
Figure Envelopeframe.
frame.

Then, for each detected component, it is compared with the BIM projection areas. The
comparison method is shown in Figure 5. The outer rectangular box is a location frame
generated by target detection, and the inner polygon is an envelope frame generated by a
BIM element projection, which is a convex polygon composed of six pixel-corners. For
each rectangular frame, the mean-square error between it and the envelope frame k could
Buildings 2022, 12, 106 Figure 4. Envelope frame. 10 of 21

Then, for each detected component, it is compared with the BIM projection areas. The
comparison method
Then, for is shown
each detected in Figure it5.isThe
component, outer rectangular
compared with the BIMboxprojection
is a location frame
areas. The
generated bymethod
comparison target detection,
is shown and the inner
in Figure 5. Thepolygon
outerisrectangular
an envelope frame
box generated
is a location by a
frame
BIM element
generated projection,
by target whichand
detection, is athe
convex
innerpolygon
polygon composed of sixframe
is an envelope pixel-corners.
generatedForby
aeach
BIMrectangular frame, thewhich
element projection, mean-square
is a convex error between
polygon it and theof
composed envelope frame k could
six pixel-corners. For
each by: frame, the mean-square error between it and the envelope frame k could
rectangular
be calculated
be calculated by:
1 n
1 n  d k ,2i 2 , i = 1,2,, n
D =
Dk =k ∑ d , i = 1, 2, . . . , n (1)
(1)
n in=1i =1 k,i
where kk is
where is the
the index
index of
of the
the envelope frame; ddkk,i,i is
envelopeframe; is the
the minimum
minimum distance
distance from
from the
the pixel-
pixel-
corner i of the envelope frame k to the four edges of the rectangular box; and n is the
number of pixel-corners. The BIM element corresponding to the minimum mean-square
error DDkkisischosen,
chosen,and
and thethe registration
registration result
result is is asas follows:
follows:

= ={{kk|minD
KK | min D
k }k
} (2)
where KKrepresents
where represents
thethe envelope
envelope frame
frame with
with index
index k. k.

d k ,2 d k ,3

d k ,1
k
d k ,4

d k ,6 d k ,5

Figure 5. Location Registration.

The purpose of rough registration is to match the building components identified in

images with
with the
theitems
itemsininthe
theBIM
BIMelement
elementprojection library.
projection However,
library. However, forfor
each registration,
each registra-
most BIM envelope frames are useless. Therefore, the envelope frames that
tion, most BIM envelope frames are useless. Therefore, the envelope frames that satisfy satisfy any of
the following conditions would be excluded:
any of the following conditions would be excluded:
1.
1. TheThe envelope
envelope frames
frames ofof components
components that that do
do not
not belong
belong to
to the
the construction
construction layer;
layer;
2. The envelope frames which do not overlap with the rectangular
2. The envelope frames which do not overlap with the rectangular box; box;
3.
3. TheThe envelope
envelope frames
frames with
with horizontal
horizontal andand vertical
vertical dimensions
dimensions greater
greater than
than two times
two times
Buildings 2022, 12, x FOR PEER REVIEW or
or less
less than
than 0.5
0.5 times
times the
the size
size of
of the
the rectangular
rectangular box;
box; 11 of 22
4.
4. TheThe envelope
envelope frames
frames whose
whose type
type are
are inconsistent
inconsistent with
with the
the rectangular
rectangular box.
box.
To
To visualize
visualize the
the BIM-to-Image
BIM-to-Image registration,
registration, a
a component
component is is taken
taken asas an
an example
example to
to
show the experimental
show the experimental process and
process results. The
and results.component is detected from a construction-
construction-site image, which is a rectangular box,The component
as shown in Figureis 6a.
detected
Its BIM from
modela
site image, which is a rectangular box, as shown in Figure 6a. Its BIM model is a hexahedron,
is a hexahedron, as shown in Figure 6b. When the hexahedron is projected onto the con-
as shown in Figure 6b. When the hexahedron is projected onto the construction-site image,
struction-site image, it becomes a polygon and is registered with the rectangular frame, as
it becomes a polygon and is registered with the rectangular frame, as shown in Figure 6c.
shown in Figure 6c.

(a) (b) (c)

Figure
Figure 6.
6. The
Theprocess
processof
ofaacomponent:
component:(a)
(a)Detection
Detectionresult;
result;(b)
(b)BIM
BIMelement;
element;(c)
(c)Registration
Registrationresult.
result.

3.3.2. Progress Reasoning

After the registration, the BIM elements corresponding to the target recognition re-
sults are obtained, however they are not the schedule. It is necessary to further infer the
construction progress with auxiliary information. In addition, due to the widespread oc-
clusion in a construction site, many components are not recognized and registered. There-
Buildings 2022, 12, 106 11 of 21

3.3.2. Progress Reasoning

After the registration, the BIM elements corresponding to the target recognition results
are obtained, however they are not the schedule. It is necessary to further infer the
construction progress with auxiliary information. In addition, due to the widespread
occlusion in a construction site, many components are not recognized and registered.
Therefore, through auxiliary reasoning, the influence of occlusion can be avoided as much
as possible to obtain the actual progress.
Using auxiliary information to assist reasoning can greatly reduce the duplication
of effort in the data collection phase and the ambiguity of recognition results. The aux-
iliary information includes physical relations (aggregation, topological and directional
relationships) and logical relations between objects or geometric primitives. Nuchter and
Hertzberg [67] represented the knowledge model of the spatial relationships with a seman-
tic net. Nguyen et al. [68] automatically derived topological relationships between solid
objects or geometric primitives with a 3D solid CAD model. Braun et al. [30] attributed
these relationships to technological dependencies and represented these dependencies with
graphs (nodes for building elements and edges for dependencies).
In this study, the sequential relationship of different components is used as supple-
mentary information for auxiliary reasoning. For example, when the bottom formworks
of slabs are detected and registered, it can be considered that the steel bar binding and
formwork supporting work of facades have been finished. Figure 7 shows the conditions
under which construction can be judged to have entered a new stage. To avoid accidental
errors, the threshold of stage update is set to three. After the concrete pouring, building
Buildings 2022, 12, x FOR PEER REVIEW
components cannot be identified, and the construction progress would be judged to 12 of 22
enter a
new floor.

Start / Concrete curing

More than three “stacked formwork” are detected.
No components are detected and registered.

Preparation
More than three “walls or columns with formwork” are registered.

Facade construction
More than three “slabs with bottom formwork” or “beams enclosed by formwork” are registered.

Formwork for plane

More than three “slabs with steel cages” or “beams with steel cages” are registered.

Steel bar binding in plane

Pouring tools are detected.

Concrete pouring

Figure conditions.
Figure 7. Stage update 7. Stage update conditions.

4. Experiment and Analysis

4.
4.1.
4.1. Experimental
Experimental Data
Data Collection
Collection and
and Setup
Setup
To
To verify
verify the
the effectiveness
effectiveness of of this
this method,
method, an an experiment
experiment was carried out
was carried out in
in aa high-
high-
rise
rise building construction site. A tower crane driver was employed to take several top
building construction site. A tower crane driver was employed to take several top
views
views of the construction
of the constructionsitesiteevery
everyday.
day.Within
Within4545 days,
days, a total
a total of 117
of 117 images
images withwith a
a res-
resolution
olution of of 4000
4000 × 3000
× 3000 pixels
pixels werewere collected.
collected. TheThe distribution
distribution of 117
of 117 images
images was
was shown
shown in
in
Figure 8. The calibration results of internal parameters were as follows: focal length is
Figure 8. The calibration results of internal parameters were as follows: focal length is
3144.71 ×
3144.71 3178.63 pixels;
× 3178.63 pixels; principal
principal point
point is 2062.56 ××1270.47
is 2062.56 1270.47pixels;
pixels;andandskew −15.48.
skewisis−15.48.
 To verify the effectiveness of this method, an experiment was carried out in a high-
rise building construction site. A tower crane driver was employed to take several top
views of the construction site every day. Within 45 days, a total of 117 images with a res-
olution of 4000 × 3000 pixels were collected. The distribution of 117 images was shown in
Buildings 2022, 12, 106 Figure 8. The calibration results of internal parameters were as follows: focal length12 of 21
is
3144.71 × 3178.63 pixels; principal point is 2062.56 × 1270.47 pixels; and skew is −15.48.

Figure8.8. Image
Figure Imagedistribution.
distribution.

4.2.
4.2. Results
Results and
and Analysis
Analysis
In the process ofobtaining
In the process of obtainingthetheconstruction
construction schedule,
schedule, three
three results
results areare generated:
generated: de-
detection result,
tection result, registration
registration result
result andand process
process reasoning
reasoning result.
result. In In
thethe following,
following, these
these re-
results areanalyzed
sults are analyzedtotoevaluate
evaluatethe
the performance
performance ofof the
the method.
method. The
The collected
collected image
image data
data
belong to different floors and different construction stages, and the content of multiple
belong to different floors and different construction stages, and the content of multiple
images taken at the same time is repeated. To eliminate the influence of sample size, the
images taken at the same time is repeated. To eliminate the influence of sample size, the
images taken at the same time are regarded as a group (64 groups in total), and then these
images taken at the same time are regarded as a group (64 groups in total), and then these
groups are classified into five construction stages. Therefore, taking the construction stage
groups are classified into five construction stages. Therefore, taking the construction stage
as the category and the group as the unit, these data were averaged and analyzed.
as the category and the group as the unit, these data were averaged and analyzed.
The detection results are shown in Table 3, including the detection accuracy of various
The detection results are shown in Table 3, including the detection accuracy of vari-
components and the time-consuming of each image. In the calculation, the repeated
ous components and the time-consuming of each image. In the calculation, the repeated
components in the same group are treated as a component as it only needs to be identified
components in the same group are treated as a component as it only needs to be identified
once to prove that the component already exists.
once to prove that the component already exists.
Table 3. Detection results 1 .

# of Correctly
# of Comp. # of Comp. # of Detected Detection
Stage Component’s Type Detected
in BIM 2 in Images Comp. Accuracy 3
Comp.
Preparation Stacked formwork 0 17.75 17.75 17.75 100.00%
Facade Walls or columns with
82 51.67 37.33 36.25 70.16%
construction formwork
Slabs with bottom
62 44.65 44.35 43.60 97.65%
Formwork for formwork
plane Beams enclosed by
118 81.50 76.75 75.60 92.76%
formwork
Steel bar binding in Slabs with steel cages 62 55.73 41.09 40.00 71.78%
plane Beams with steel cages 118 105.91 81.09 80.73 76.22%
Slabs with steel cages 62 62.00 51.17 50.17 80.91%
Concrete pouring Beams with steel cages 118 118.00 101.80 100.40 85.08%
Pouring tools 0 1.00 1.00 1.00 100.00%
1 The average detection time of each image is 65.43 ms. 2 The number of BIM components in one floor. 3 It is the
ratio between the “# of correctly detected comp.” and the “# of comp. in images”.

It can be seen from Table 3 that the algorithm has a better effect on the recognition
of “Stacked formwork” and “Pouring tools”, and the worst effect on the recognition of
“Walls or columns with formwork” and “Slabs with steel cages”. In the preparation phase,
“Stacked formwork” is almost not occluded, and its boundary is so clear that it is easy to be
detected. In the facade construction stage, the construction site is chaotic, and the occlusion
caused by the interaction between walls or columns is serious, so that the recognition effect
is not ideal. In the plane formwork stage, the bottom formwork of slab blocks the results
of steel bar binding and formwork erection in the lower layer, and the bottom formwork
is orderly arranged with clear boundaries. Although there are some errors caused by
Buildings 2022, 12, 106 13 of 21

occlusion or half formwork, the recognition effect is better than other stages. In the stage of
steel bar binding in plane and concrete pouring, the content of recognition is the slabs or
beams with steel cages, however the steel cages obscure the bottom formwork and make
the boundaries of components fuzzy, so that the recognition effect is poor. In addition, the
average detection time of each image is 65.43 ms.
The registration results are shown in Table 4, including the registration accuracy of
various components and the registration time of each image. The progress collection ratio
is set to evaluate how much progress information the method has collected from the image
during the whole process.

Table 4. Registration results 1 .

# of
# of # of Correctly Progress
Component’s # of Comp. Correctly Registration
Stage Registered Registered Collection
Type in Images Detected Accuracy 2
Comp. Comp. Ratio 3
Comp.
Walls or
Facade
columns with 51.67 36.25 32.25 32.00 88.28% 61.94%
construction
formwork
Slabs with
bottom 44.65 43.60 42.70 42.20 96.79% 94.51%
Formwork formwork
for plane Beams
enclosed by 81.50 75.60 71.80 71.05 93.98% 87.18%
formwork

Steel bar Slabs with steel

55.73 40.00 38.45 38.00 95.00% 68.19%
binding in cages
plane Beams with
105.91 80.73 76.55 76.00 94.14% 71.76%
steel cages
Slabs with steel
62.00 50.17 48.67 48.33 96.35% 77.96%
Concrete cages
pouring Beams with
118.00 100.40 95.80 95.00 94.62% 80.51%
steel cages
Pouring tools 1.00 1.00 - - - -
1 2
The average registration time of each image is 50.25 ms. The ratio between the # of correctly registered comp.
and the # of correctly detected comp. 3 The ratio between the # of correctly registered comp. and the # of comp.
in images.

We can see from Table 4 that the registration accuracy of the column and wall is the
lowest, the registration accuracy of slab and beam is similar, and the registration accuracy
of the slab is higher than that of the beam. From an overall perspective, the progress
collection rate of the facade construction stage is the lowest, and that of the plane formwork
stage is the highest. In the plane formwork stage, the progress collection rate of the slab is
higher than that of the beam, however it is opposite to the plane steel bar binding stage
and concrete pouring stage. Overall, the progress collection rate mainly depends on the
effect of detection accuracy, and is less affected by the registration accuracy.
The progress reasoning results are shown in Figure 9, including the actual process, pro-
cess of registration and process after auxiliary reasoning. These data are used to analyze the
recall rate of construction progress information and the effectiveness of auxiliary reasoning.
 higher than that of the beam, however it is opposite to the plane steel bar binding stage
and concrete pouring stage. Overall, the progress collection rate mainly depends on the
effect of detection accuracy, and is less affected by the registration accuracy.
The progress reasoning results are shown in Figure 9, including the actual process,
process of registration and process after auxiliary reasoning. These data are used to ana-
Buildings 2022, 12, 106 14 of 21
lyze the recall rate of construction progress information and the effectiveness of auxiliary
reasoning.

Start
Preparation
Facade construction
Formwork for plane
Steel bar binding in plane
61.94% Concrete pouring
100% 89.77%
100% 70.53%
100% 79.63%
100%
Actual progress Progress of registration Progress after auxiliary reasoning

Figure 9. Comparison of construction progress.

Figure 9. Comparison of construction progress.
From
From Figure
Figure 10,
10, the
the progress collected is
progress collected is behind
behind the the actual
actual construction
construction progress,
progress, as as
there are not enough components in the top view to prove that the
there are not enough components in the top view to prove that the construction has started, construction has
started, and the threshold
and the threshold of the update
of the progress progress update
is set so as is
toset so as
avoid anytoincorrect
avoid any incorrect
reasoning in rea-
this
soning in this paper. In the process of registration, some components
paper. In the process of registration, some components are not detected, so the progress are not detected, so
the
dataprogress data is Considering
is intermittent. intermittent.theConsidering
number ofthe number ofcannot
components components
be usedcannot be used
to evaluate the
to evaluate the construction progress at the beginning and preparation
construction progress at the beginning and preparation stage, the construction progress stage, the construc-
tion progress ofisregistration
of registration not marked. is not marked.
In other In other construction
construction stages,
stages, the ratio of the
the ratio
number of the
of
number of registered components to the number of components in
registered components to the number of components in images is used as the construction images is used as the
construction progress of Among
progress of registration. registration.
them,Among them,pouring
the concrete the concrete
stagepouring stage
starts from thestarts from
detection
the detection
of pouring of pouring
tools, tools, and
and the progress is the progress
calculated by is
thecalculated by the
missing slabs andmissing
beams with slabs steel
and
beams withFrom
bar cages. steel bar
the cages.
data, itFrom
can the data, it canthat
be concluded be concluded
the progress thatofthe progress
facade of facade
construction,
construction,
steel bar bindingsteelinbar binding
plane in plane and
and concrete concrete
pouring pouring
is fuzzy, while is the
fuzzy, while of
progress theformwork
progress
of
for the plane is clear. After auxiliary reasoning, the construction progress of each stage of
formwork for the plane is clear. After auxiliary reasoning, the construction progress is
each stage is
completed completed
before the nextbefore
stage,the
so next stage,
the final so the finalprogress
construction construction progresswith
is consistent is con-
the
sistent with the actual
actual construction construction
progress progress
in general, in general,
although there are although there are some delays.
some delays.

4.3. Analysis of Error Cases

Error cases are illustrated from
from three
three aspects:
aspects: recognition,
recognition, registration
registration and
andreasoning.
reasoning.
The causes of detection errors are diverse, including occlusion, light intensity, intensity, richness of
Buildings 2022, 12, x FOR PEER REVIEW
training set
set data
data and
and so 15 ofof22
training so on. Figure 10
on. Figure 10 shows
shows fourfour detection
detection error
error cases. The formwork
cases. The formwork of
aa wall
wall in
in Figure
Figure 10a
10a is
is not
not detected
detected asas it
it is
is shaded
shaded byby reinforcement
reinforcement cages. The beams
cages. The beams inin
Figure 10b are not detected as their features are not obvious. In contrast, the support
Figure 10b are not detected as their features are not obvious. In contrast, the support bars bars
ofofaabottom
bottomformwork
formworkininFigure
Figure10c
10careareidentified
identifiedasasa awall.
wall.AAformwork
formworkisisdetected
detectedinin
Figure
Figure 10d, however it is actually a formwork supported for other components,which
10d, however it is actually a formwork supported for other components, whichisis
not expected.
not expected.

(a) (b) (c) (d)

Figure10.
Figure 10.Detection
Detectionerror
errorcases:
cases:(a)
(a)Not
Notdetected
detecteddue
duetotoocclusion;
occlusion;(b)
(b)Not
Notdetected
detectedasasthe
thefeature
featureisis
not obvious; (c) Detected, although of type error; (d) Detected, however it is an unrelated
not obvious; (c) Detected, although of type error; (d) Detected, however it is an unrelated object.object.

Thereason
The reasonfor
forregistration
registration error
error lies
lies in in
thethe inaccurate
inaccurate detection
detection results,
results, andand
the the cam-
camera
calibration has little impact on the registration results. Figure 11 shows three cases ofof
era calibration has little impact on the registration results. Figure 11 shows three cases
registrationerrors.
registration errors.Figure
Figure11a
11ashows
showsthatthatthe theformwork
formworkisisidentified
identifiedasassmall
smallblocks
blocksdue due
totothe
theobvious
obvious interference lines lines inside
insidethe theformwork
formworkafterafterraining. TheThe
raining. formwork
formwork in Fig-
in
ure 11b
Figure 11bis being assembled
is being assembled andand
therefore
therefore identified as small
identified pieces.
as small The recognized
pieces. The recognizedtypes
of these
types components
of these components are correct, however
are correct, however the wrong registration
the wrong occurs
registration due to
occurs the
due tolarge
the
size difference. The three formworks in Figure 11c are actually stacked, although they are
identified as bottom formworks, which lead to improper registration.
 The reason for registration error lies in the inaccurate detection results, and the cam-
era calibration has little impact on the registration results. Figure 11 shows three cases of
registration errors. Figure 11a shows that the formwork is identified as small blocks due
to the obvious interference lines inside the formwork after raining. The formwork in Fig-
Buildings 2022, 12, 106 ure 11b is being assembled and therefore identified as small pieces. The recognized15typesof 21
of these components are correct, however the wrong registration occurs due to the large
size difference. The three formworks in Figure 11c are actually stacked, although they are
identified
large as bottomThe
size difference. formworks, which lead
three formworks to improper
in Figure 11c areregistration.
actually stacked, although they
are identified as bottom formworks, which lead to improper registration.

(a) (b) (c)

Figure11.
Figure 11. Registration
Registration error
error cases:
cases: (a)
(a) The
The interference
interference of
ofmessy
messylines
linesafter
afterrain;
rain;(b)
(b)Incomplete
Incomplete
formwork; (c) Difficult distinction between stacked formwork and bottom formwork.
formwork; (c) Difficult distinction between stacked formwork and bottom formwork.

Thereason
The reasonofofreasoning
reasoningerrorerrorlies
liesinintwo
twoaspects:
aspects:one
oneisisthat
thatsome
somecomponents
componentscannotcannot
be detected, and the other is that the threshold to determine that the progress
be detected, and the other is that the threshold to determine that the progress has changed has changed
isisset.
set. Among
Among them,
them, the the setting
setting of
ofthreshold
thresholdisisthe
thenecessary
necessarycostcostto
toavoid
avoidmore
moreserious
serious
errors. Figure
errors. Figure 12
12 shows
shows two two examples
examples of ofreasoning
reasoningerrors.
errors. NoNo member
member isis identified
identifiedin in
Figure12a,
Figure 12a,so
soititisisassumed
assumedthat thatthe
theconstruction
constructionhas
hasentered
enteredthe theconcrete
concrete curing
curing stage,
stage,
althoughthe
although theopposite
oppositeisistrue.
true.In InFigure
Figure12b, 12b,two
twobottom
bottomformworks
formworksare aredetected,
detected,however
however
since the number is less than three, it is not determined that the construction
since the number is less than three, it is not determined that the construction has entered
Buildings 2022, 12, x FOR PEER REVIEW has entered
16 of 22
theplane
the planeformwork
formworksupportsupportstage.
stage.

(a) (b)
Figure12.
Figure 12.Reasoning
Reasoningerror
errorcases:
cases:(a)
(a)End
Endofofconcreting;
concreting;(b)
(b)Start
Startofofformwork
formworkfor
forplane.
plane.

ToTosum
sumup,up,errors
errorsoccur
occurinindifferent
differentstages,
stages,and
andthe
themain
mainreason
reasonisisthe
thelow
lowdetection
detection
accuracy
accuracycaused
causedby byocclusion
occlusionandandlight.
light.However,
However,most
mostdetection
detectionerrors
errorscan
canbebeeliminated
eliminated
during
duringregistration,
registration,so sothe
theregistration
registrationprocess
processcan
canimprove
improvethetheaccuracy
accuracyof ofthe
thefinal
finalresult.
result.
Although
Althoughtherethereare
areerrors in in
errors thethe
camera calibration
camera process,
calibration these
process, errors
these are very
errors smallsmall
are very and
hardly affectaffect
and hardly the final
theresult. In theIn
final result. progress reasoning
the progress stage, stage,
reasoning auxiliary conditions
auxiliary are used
conditions are
toused
ensure the reliability of the obtained progress information. These auxiliary
to ensure the reliability of the obtained progress information. These auxiliary condi-conditions
will cause
tions will the obtained
cause progress
the obtained to be slightly
progress delayed.
to be slightly The threshold
delayed. set in set
The threshold thisinpaper is
this pa-
three
per iscomponents, which has
three components, an acceptable
which impact impact
has an acceptable on the timeliness of data.of data.
on the timeliness

5.5.Discussion
Discussion
5.1. Why Target Detection?
5.1. Why Target Detection?
The method used to obtain progress information in this paper is one of target detection,
The method used to obtain progress information in this paper is one of target detec-
rather than one in that identifies materials in the back-projected area. Target detection has
many rather
tion, than one
outstanding in that identifies materials in the back-projected area. Target detec-
advantages:
tion has many outstanding advantages:
1. Target detection can identify component categories with texture and context informa-
1. tion
Target detection
in the image.can identify component
In construction, categories with
many components are texture
made ofand contextwhich
concrete, infor-
mation in the image. In construction, many components are made of concrete,
makes it difficult to tell whether the components are beams, slabs, columns, or walls which
makes
only it difficult
according to to
thetell whetherCompared
material. the components
with thearemethod
beams,of
slabs, columns,
inferring or walls
component
only according to the material. Compared with the method of inferring component
types by measuring the space occupied by the point cloud, object detection can make
full use of the texture information of the material in the image. Furthermore, com-
pared with the method of identifying material in the back-projected area, target de-
tection can make full use of the context information in the image;
Buildings 2022, 12, 106 16 of 21

types by measuring the space occupied by the point cloud, object detection can make
full use of the texture information of the material in the image. Furthermore, compared
with the method of identifying material in the back-projected area, target detection
can make full use of the context information in the image;
2. Target detection identifies the actual components rather than the planned ones in the
schedule. When using the back projection method, it is unrealistic to project all the
components into the image, so it is necessary to determine which components are
projected each time, and which is the root of the problem. The general construction
progress can be planned by referring to historical experience, however the specific
implementation of these plans in practice is affected by many factors. When a few
components are projected, there is uncertainty as to whether these components will
be built first and while many components are projected, the occlusion relationship
among these components cannot be determined. Therefore, the number of projected
components and the order of schedule implementation will impact the effect of
schedule tracking. In contrast to this method, target detection directly identifies
the existing components from the image without the planned information, which is
more flexible;
3. Target detection can be extended to the identification of other objects on the construc-
tion site. In this paper, target detection is used to identify formwork, steel cages,
pouring tools and so on, which is necessary for progress reasoning in this paper. In
future research, more components, tools, materials, machinery and personnel can
be identified to enrich the collected construction progress details or expand other
management functions based on this technology.

5.2. The Role of Rough Registration

In fact, the construction progress can be inferred from the target detection results. For
example, the ratio of the number of detected components to the total number of components
can be used as the construction progress, and the appearance of new components, can be
used as the updated condition of the construction stage. However, rough registration is still
used in this paper to specify the schedule to each component. This is case as the authors
want to improve the accuracy of recognition, which is helpful to expand other functions of
this technology in the future. In addition, the method refines the progress to the component
level, which makes it applicable to a flat building structure. Most importantly, the rough
registration can eliminate most of the components that are wrongly detected, which makes
up for the defect of low recognition rate.

5.3. Coherence of Images

This method requires at least one group of data to be collected every day, during
which the construction site changes greatly. Therefore, it is necessary to collect a certain
proportion of the surrounding background to achieve the registration of the front and back
images and obtain the external parameters of new images. When the tower crane is lifted,
the construction site before and after lifting should be recorded to avoid too many changes
affecting the registration effect.

5.4. Advantages
Above all, the advantages of this method are mainly shown in the following five aspects:
1. Top view: The high-rise building is constructed in layers, and the components of
each layer overlap less in the vertical direction, although they overlap seriously in
the horizontal direction. With the help of a tower crane, the construction site can be
photographed clearly with less shielding;
2. Auxiliaries rather than concrete members: The material used to infer construction
progress in this paper is the auxiliary of building components. This includes a form-
work, support, reinforcement cage and pouring tool. Compared with the method
of identifying concrete components, this method is more suitable for the actual con-
Buildings 2022, 12, 106 17 of 21

struction site, the reason being that in the construction site of a cast-in-place concrete
structure it is these auxiliary materials that can be collected rather than the concrete
component itself;
3. Target detection rather than material identification: Compared with material iden-
tification, target detection considers not only the texture of the material, but also
the context information in the image. In this way, the detection results can be spe-
cific to the type of components, which is more accurate and avoids the confusion of
adjacent components;
4. Rough registration: In the process of registration between the target detection results
and BIM model projection, four constraints are set to exclude irrelevant components,
which improves the registration efficiency and accuracy. In addition, rough regis-
tration is not strict registration, which provides a certain fault tolerance space and
ensures the accuracy of registration;
5. Point cloud avoidance: The current 3D reconstruction process mainly relies on the
point cloud, however, there are some inherent shortcomings in point cloud-based 3D
reconstruction. Firstly, it is time-consuming to remove all points of the backgrounds
and the objects of no interest, and there is no guarantee on the completeness of the
point clouds. In addition, the point clouds also have problems such as high noise,
difficulty in segmentation and registration. The proposed method does not need to
infer the progress by calculating the space occupied by feature points, so the trouble
caused by the point cloud is avoided.

5.5. Limitations
The limitations of this method include the following two aspects:
1. External parameter error accumulation: The quality of the lens, the inaccuracy of the
manual estimation of the coordinates of the known points in the initial image, the
registration error between images and other factors will cause the inaccuracy of the
estimated external parameters. After several iterations, the five errors accumulate
continuously. Therefore, external parameters need to be adjusted regularly to keep
them accurate;
2. Incompatibility with indoor scenes: The indoor space is limited, and the camera is
very close to the subject. If you want to include all the scene details as much as
possible, you need to take a lot of photos. The more photos, the lower the accuracy of
camera calibration results. Multiple start scenes must be set to improve the registration
accuracy. However, each initial scenario requires manual participation, which is not
in line with the original intention of automatically collecting progress information.
Therefore, this method needs to be further improved to adapt to indoor scenes in
the future.

5.6. Application
There are many manual working steps for target detector training, camera calibration,
registration and progress reasoning. When this method is applied to other high-rise
building projects in the future, the repeatability and automation of different steps are
different, as shown in Table 5. Most of the steps are automated and the workload of manual
operation is small.
Buildings 2022, 12, 106 18 of 21

Table 5. Repeatability and automation of steps in future applications.

Stage Step Repeatability and Automation

Data can be supplemented, and this step
Image collection and labeling
Target detector training is unnecessary to repeat.
model training Unnecessary repetition
Need to be repeated, and calibrated once
Internal parameter calibration
for each camera
Camera calibration
Need to be repeated, and calibrated once
Initial external parameter acquisition
for each scene
Need to be repeated, and updated
External parameter update
through automated iterations
Projection Need to be repeated, and automated
Registration Screening Need to be repeated, and automated
Rough registration Need to be repeated, and automated
Progress reasoning Progress reasoning Need to be repeated, and automated

6. Conclusions and Future Works

Monitoring the construction process of high-rise buildings is the weak part of the
research as the concrete components are obscured by formwork, wooden strips, steel bars
and other sundries. This paper presents a solution by realizing the automatic tracking
of high-rise building construction process. Using the target detection technology, the
method first identifies the building components in different states from the top view of
the construction site. Secondly, based on the principle of triangular imaging, the mapping
between the 3D world (BIM) and the imaging plane is established to make the 3D BIM
elements and the components in the image comparable. Then, the rough registration
algorithm is used to match the BIM elements with the target detection results one by one.
Finally, the construction progress is inferred from the number of components in each state
in one floor. When compared to similar methods, this paper uses the top view to reduce the
occlusion and identify the auxiliary objects of the construction site instead of the concrete
components, and uses the target detection instead of the material classification, which is
more simple, efficient and practical. In addition, the combination of target detection and
rough registration makes the method make full use of the context information in the image
and avoid the error caused by false recognition.
Nevertheless, there are still some open research challenges that need further investiga-
tion. For instance, in this paper, the detection effect of building components is not ideal.
The reason for this is that the data set used for training is very small, so a larger data set is
needed for recognizer training in the future. In addition, it is a wise choice to use video
instead of manually collected images for monitoring, which can make the construction
progress update in real time. However, the tracked construction progress is still slightly
delayed, so in the future, other clues in the construction site can be used to replace the
number of detected or registered components to update the construction stage.

Author Contributions: Conceptualization, J.X.; investigation, J.X. and X.H.; writing—original draft
preparation, J.X.; writing—review and editing, J.X. All authors have read and agreed to the published
version of the manuscript.
Funding: This research received no external funding.
Institutional Review Board Statement: Not applicable.
Informed Consent Statement: Not applicable.
Data Availability Statement: The data presented in this study are available on request from the
corresponding author. The data are not publicly available as the data also forms part of an ongo-
ing study.
Conflicts of Interest: The authors declare no conflict of interest.
Buildings 2022, 12, 106 19 of 21

References
1. Wua, K.; de Sotob, B.G.; Adeyc, B.T.; Zhanga, F. BIM-based estimation of vertical transportation demands during the construction
of high-rise buildings. Autom. Constr. 2020, 110, 102985. [CrossRef]
2. Do, S.T.; Nguyen, V.T.; Dang, C.N. Exploring the relationship between failure factors and stakeholder coordination performance
in high-rise building projects: Empirical study in the finishing phase. Eng. Constr. Archit. Manag. 2021; ahead-of-print. [CrossRef]
3. Andriasyan, M.; Moyano, J.; Nieto-Julián, J.E.; Antón, D. From point cloud data to building information modelling: An automatic
parametric workflow for heritage. Remote Sens. 2020, 12, 1094. [CrossRef]
4. Adan, A.; Quintana, B.; Prieto, S.A.; Bosche, F. An autonomous robotic platform for automatic extraction of detailed semantic
models of buildings. Autom. Constr. 2020, 109, 102963. [CrossRef]
5. Liu, J.; Xu, D.; Hyyppa, J.; Liang, Y. A survey of applications with combined BIM and 3D laser scanning in the life cycle of
buildings. IEEE J. Sel. Top. Appl. Earth Observ. Remote Sens. 2021, 14, 5627–5637. [CrossRef]
6. Zhang, C.; Arditi, D. Advanced progress control of infrastructure construction projects using terrestrial laser scanning technology.
Infrastructures 2020, 5, 83. [CrossRef]
7. Bechtold, S.; Höfle, B. Helios: A multi-purpose LiDAR simulation framework for research, planning and training of laser scanning
operations with airborne, ground-based mobile and stationary platforms. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci.
2016, III-3, 161–168. [CrossRef]
8. Hong, S.; Park, I.; Lee, J.; Lim, K.; Choi, Y.; Sohn, H.G. Utilization of a terrestrial laser scanner for the calibration of mobile
mapping systems. Sensors 2017, 17, 474. [CrossRef]
9. Rebolj, D.; Pučko, Z.; Babič, N.Č.; Bizjak, M.; Mongus, D. Point cloud quality requirements for Scan-vs-BIM based automated
construction progress monitoring. Autom. Constr. 2017, 84, 323–334. [CrossRef]
10. Xu, Y.; Stilla, U. Toward building and civil infrastructure reconstruction from point clouds: A review on data and key techniques.
IEEE J. Sel. Top. Appl. Earth Observ. Remote Sens. 2021, 14, 2857–2885. [CrossRef]
11. Omar, H.; Mahdjoubi, L.; Kheder, G. Towards an automated photogrammetry-based approach for monitoring and controlling
construction site activities. Comput. Ind. 2018, 98, 172–182. [CrossRef]
12. Yang, J.; Shi, Z.-K.; Wu, Z.-Y. Towards automatic generation of as-built BIM: 3D building facade modeling and material recognition
from images. Int. J. Autom. Comput. 2016, 13, 338–349. [CrossRef]
13. Kropp, C.; Koch, C.; König, M. Interior construction state recognition with 4D BIM registered image sequences. Autom. Constr.
2018, 86, 11–32. [CrossRef]
14. Acharya, D.; Ramezani, M.; Khoshelham, K.; Winter, S. BIM-Tracker: A model-based visual tracking approach for indoor
localisation using a 3D building model. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci. 2019, 150, 157–171. [CrossRef]
15. Deng, H.; Hong, H.; Luo, D.; Deng, Y.; Su, C. Automatic indoor construction process monitoring for tiles based on BIM and
computer vision. J. Constr. Eng. M. 2020, 146, 04019095. [CrossRef]
16. Han, X.-F.; Laga, H.; Bennamoun, M. Image-based 3D object reconstruction: State-of-the-art and trends in the deeplearning era.
IEEE T. Pattern Anal. 2019, 43, 1578–1604. [CrossRef]
17. Liu, Y.-p.; Yan, X.-p.; Wang, N.; Zhang, X.; Li, Z. A 3D reconstruction method of image sequence based on deep learning. J. Phys.
Conf. Ser. 2020, 1550, 032051. [CrossRef]
18. Golparvar-Fard, M.; Peña-Mora, F.; Savarese, S. Automated progress monitoring using unordered daily construction photographs
and IFC-based building information models. J. Comput. Civil Eng. 2015, 29, 04014025. [CrossRef]
19. Golparvar-Fard, M.; Peña-Mora, F.; Arboleda, C.A.; Lee, S. Visualization of construction progress monitoring with 4D simulation
model overlaid on time-lapsed photographs. J. Comput. Civil Eng. 2009, 23, 391–404. [CrossRef]
20. Pučko, Z.; Šuman, N.; Rebolj, D. Automated continuous construction progress monitoring using multiple workplace real time 3D
scans. Adv. Eng. Inform. 2018, 38, 27–40. [CrossRef]
21. Dimitrov, A.; Golparvar-Fard, M. Vision-based material recognition for automated monitoring of construction progress and
generating building information modeling from unordered site image collections. Adv. Eng. Inform. 2014, 28, 37–49. [CrossRef]
22. Lee, Y.-J.; Park, M.-W. 3D tracking of multiple onsite workers based on stereo vision. Autom. Constr. 2019, 98, 146–159. [CrossRef]
23. Park, M.-W.; Brilakis, I. Construction worker detection in video frames for initializing vision trackers. Autom. Constr. 2012, 28,
15–25. [CrossRef]
24. Son, H.; Kim, C. Integrated worker detection and tracking for the safe operation of construction machinery. Autom. Constr. 2021,
126, 103670. [CrossRef]
25. Park, M.-W.; Brilakis, I. Continuous localization of construction workers via integration of detection and tracking. Autom. Constr.
2016, 72, 129–142. [CrossRef]
26. Xiao, B.; Kang, S.-C. Vision-based method integrating deep learning detection for tracking multiple construction machines. J.
Comput. Civil Eng. 2021, 35, 04020071. [CrossRef]
27. Xiao, B.; Lin, Q.; Chen, Y. A vision-based method for automatic tracking of construction machines at nighttime based on deep
learning illumination enhancement. Autom. Constr. 2021, 127, 103721. [CrossRef]
28. Zhu, Z.; Ren, X.; Chen, Z. Integrated detection and tracking of workforce and equipment from construction jobsite videos. Autom.
Constr. 2017, 81, 161–171. [CrossRef]
29. Hou, X.; Zeng, Y.; Xue, J. Detecting structural components of building engineering based on deep-learning method. J. Constr. Eng.
M. 2020, 146, 04019097. [CrossRef]
Buildings 2022, 12, 106 20 of 21

30. Braun, A.; Tuttas, S.; Borrmann, A.; Stilla, U. A concept for automated construction progress monitoring using BIM-based
geometric constraints and photogrammetric point clouds. J. Inf. Technol. Constr. 2015, 20, 68–79.
31. Han, K.; Degol, J.; Golparvar-Fard, M. Geometry-and appearance-based reasoning of construction progress monitoring. J. Constr.
Eng. M. 2018, 144, 04017110. [CrossRef]
32. Han, K.K.; Golparvar-Fard, M. Appearance-based material classification for monitoring of operation-level construction progress
using 4D BIM and site photologs. Autom. Constr. 2015, 53, 44–57. [CrossRef]
33. Rahimian, F.P.; Seyedzadeh, S.; Oliver, S.; Rodriguez, S.; Dawood, N. On-demand monitoring of construction projects through a
game-like hybrid application of BIM and machine learning. Autom. Constr. 2020, 110, 103012. [CrossRef]
34. Son, H.; Kim, C.; Kim, C. Automated color model-based concrete detection in construction-site images by using machine learning
algorithms. J. Comput. Civil Eng. 2012, 26, 421–433. [CrossRef]
35. Zhu, Z.; Brilakis, I. Parameter optimization for automated concrete detection in image data. Autom. Constr. 2010, 19, 944–953.
[CrossRef]
36. Zhu, Z.; Brilakis, I. Concrete column recognition in images and videos. J. Comput. Civil Eng. 2010, 24, 478–487. [CrossRef]
37. Wang, Z.; Zhang, Q.; Yang, B.; Wu, T.; Lei, K.; Zhang, B.; Fang, T. Vision-based framework for automatic progress monitoring of
precast walls by using surveillance videos during the construction phase. J. Comput. Civil Eng. 2021, 35, 04020056. [CrossRef]
38. Bassier, M.; Van Genechten, B.; Vergauwen, M. Classification of sensor independent point cloud data of building objects using
random forests. J. Build. Eng. 2019, 21, 468–477. [CrossRef]
39. Wang, R.; Peethambaran, J.; Chen, D. LiDAR point clouds to 3-D urban models: A review. IEEE J. Sel. Top. Appl. Earth Observ.
Remote Sens. 2018, 11, 606–627. [CrossRef]
40. Wang, Q.; Tan, Y.; Mei, Z. Computational methods of acquisition and processing of 3D point cloud data for construction
applications. Arch. Comput. Method Eng. 2019, 27, 479–499. [CrossRef]
41. Czerniawski, T.; Leite, F. Automated digital modeling of existing buildings: A review of visual object recognition methods. Autom.
Constr. 2020, 113, 103131. [CrossRef]
42. Alizadeh Salehi, S.; Yitmen, İ. Modeling and analysis of the impact of BIM-based field data capturing technologies on automated
construction progress monitoring. Int. J. Civil Eng. 2018, 16, 1669–1685. [CrossRef]
43. Pučko, Z.; Rebolj, D. Automated construction progress monitoring using continuous multipoint indoor and outdoor 3D scanning.
In Proceedings of the Lean and Computing in Construction Congress -Joint Conference on Computing in Construction, Heraklion,
Greece, 4–7 July 2017; pp. 105–112. [CrossRef]
44. Li, Y.; Li, W.; Tang, S.; Darwish, W.; Hu, Y.; Chen, W. Automatic indoor as-built building information models generation by using
low-cost RGB-D sensors. Sensors 2020, 20, 293. [CrossRef] [PubMed]
45. Wasenmüller, O.; Stricker, D. Comparison of kinect v1 and v2 depth images in terms of accuracy and precision. In Proceedings of
the Asian Conference on Computer Vision(ACCV 2016), Taipei, Taiwan, China, 20–24 November 2016; pp. 34–45. [CrossRef]
46. Volk, R.; Luu, T.H.; Mueller-Roemer, J.S.; Sevilmis, N.; Schultmann, F. Deconstruction project planning of existing buildings based
on automated acquisition and reconstruction of building information. Autom. Constr. 2018, 91, 226–245. [CrossRef]
47. Kim, C.; Kim, B.; Kim, H. 4D CAD model updating using image processing-based construction progress monitoring. Autom.
Constr. 2013, 35, 44–52. [CrossRef]
48. Ibrahim, Y.M.; Lukins, T.C.; Zhang, X.; Trucco, E.; Kaka, A.P. Towards automated progress assessment of workpackage components
in construction projects using computer vision. Adv. Eng. Inform. 2009, 23, 93–103. [CrossRef]
49. Han, K.K.; Golparvar-Fard, M. Potential of big visual data and building information modeling for construction performance
analytics: An exploratory study. Autom. Constr. 2017, 73, 184–198. [CrossRef]
50. Song, J.; Haas, C.T.; Caldas, C.H. Tracking the location of materials on construction job sites. J. Constr. Eng. M. 2006, 132, 911–918.
[CrossRef]
51. Navon, R.; Sacks, R. Assessing research issues in Automated Project Performance Control (APPC). Autom. Constr. 2007, 16,
474–484. [CrossRef]
52. Tserng, H.P.; Dzeng, R.J.; Lin, Y.C.; Lin, S.T. Mobile construction supply chain management using PDA and bar codes. Comput.
Aided Civ. Infrastruct. Eng. 2005, 20, 242–264. [CrossRef]
53. Chen, Q.; Adey, B.T.; Haas, C.; Hall, D.M. Using look-ahead plans to improve material flow processes on construction projects
when using BIM and RFID technologies. Constr. Innov. 2020, 20, 471–508. [CrossRef]
54. Oner, M.; Ustundag, A.; Budak, A. An RFID-based tracking system for denim production processes. Int. J. Adv. Manuf. Technol.
2017, 90, 591–604. [CrossRef]
55. Araújo, C.S.; de Siqueira, L.C.; Ferreira ED, A.M.; Costa, D.B. Conceptual framework for tracking metallic formworks on
construction sites using IoT, RFID and BIM technologies. In Proceedings of the 18th International Conference on Computing in
Civil and Building Engineering, São Paulo, Brazil, 18–20 August 2020; pp. 865–878. [CrossRef]
56. Cho, Y.K.; Youn, J.H.; Martinez, D. Error modeling for an untethered ultra-wideband system for construction indoor asset tracking.
Autom. Constr. 2010, 19, 43–54. [CrossRef]
57. Chu, C.-C.; Nandhakumar, N.; Aggarwal, J.K. Image segmentation using laser radar data. Pattern Recognit. 1990, 23, 569–581.
[CrossRef]
58. GitHub. LabelImg: A Graphical Image Annotation Tool. Available online: https://github.com/tzutalin/labelImg (accessed on 15
January 2020).
Buildings 2022, 12, 106 21 of 21

59. Shen, Z.; Liu, Z.; Li, J.; Jiang, Y.-G.; Chen, Y.; Xue, X. DSOD: Learning deeply supervised object detectors from scratch. In
Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 22–29 October 2017; pp. 1937–1945.
[CrossRef]
60. Huang, G.; Liu, Z.; Van Der Maaten, L.; Weinberger, K.Q. Densely connected convolutional networks. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 4700–4708. [CrossRef]
61. GitHub. DSOD: Learning Deeply Supervised Object Detectors from Scratch. Available online: https://github.com/szq0214/
DSOD (accessed on 15 January 2020).
62. Zhang, Z. Flexible camera calibration by viewing a plane from unknown orientations. In Proceedings of the IEEE International
Conference on Computer Vision (ICCV), Kerkyra, Greece, 20–27 September 1999; pp. 666–673. [CrossRef]
63. GitHub. Sfm-Python. Available online: https://github.com/adnappp/Sfm-python (accessed on 15 January 2020).
64. Lowe, D.G. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 2004, 60, 91–110. [CrossRef]
65. Denœux, T. A k-nearest neighbor classification rule based on Dempster-Shafer theory. IEEE Trans. Syst. Man Cybern. 1995, 219,
804–813. [CrossRef]
66. Lepetit, V.; Moreno-Noguer, F.; Fua, P. EPnP: An accurate O(n) solution to the PnP problem. Int. J. Comput. Vision 2009, 81,
155–166. [CrossRef]
67. Nuchter, A.; Hertzberg, J. Towards semantic maps for mobile robots. Robot. Auton. Syst. 2008, 56, 915–926. [CrossRef]
68. Nguyen, T.-H.; Oloufa, A.A.; Nassar, K. Algorithms for automated deduction of topological information. Autom. Constr. 2005, 14,
59–70. [CrossRef]