Journal of Asian Architecture and Building Engineering

ISSN: (Print) (Online) Journal homepage: www.tandfonline.com/journals/tabe20

BIM performance assessment system using a K-means

clustering algorithm

Hyeon-Seung Kim, Sung-Keun Kim & Leen-Seok Kang

To cite this article: Hyeon-Seung Kim, Sung-Keun Kim & Leen-Seok Kang (2021) BIM
performance assessment system using a K-means clustering algorithm, Journal of Asian
Architecture and Building Engineering, 20:1, 78-87, DOI: 10.1080/13467581.2020.1800471

To link to this article: https://doi.org/10.1080/13467581.2020.1800471

© 2020 The Author(s). Published by Informa

UK Limited, trading as Taylor & Francis
Group on behalf of the Architectural
Institute of Japan, Architectural Institute of
Korea and Architectural Society of China.

Published online: 31 Aug 2020.

Submit your article to this journal

Article views: 2984

View related articles

View Crossmark data

Citing articles: 7 View citing articles

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=tabe20
JOURNAL OF ASIAN ARCHITECTURE AND BUILDING ENGINEERING
2021, VOL. 20, NO. 1, 78–87
https://doi.org/10.1080/13467581.2020.1800471

CONSTRUCTION MANAGEMENT

BIM performance assessment system using a K-means clustering algorithm

Hyeon-Seung Kima, Sung-Keun Kimb and Leen-Seok Kangc
a
BIM Development Team, Seoyeong Engineering Co. Ltd., Seoul, Korea; bDepartment of Civil Engineering, Seoul National University of
Science & Technology, Seoul, Korea; cDepartment of Civil Engineering, Engineering Research Institute, Gyeongsang National University,
Jinju, Korea

ABSTRACT ARTICLE HISTORY

Currently, various guidelines regarding building information modelling (BIM) technology policy Received 18 January 2019
are being developed in different countries. However, for many companies, the cost- Accepted 16 July 2020
effectiveness of BIM investment remains unclear. Some studies suggest a return on investment KEYWORDS
(ROI) as the result of cost-effective analysis calculations, which can be obtained by the BIM; BIM performance
introduction of BIM. However, a lack of research has led to inconsistent metrics being applied assessment; K-means
to the calculation of BIM-ROI for various types of projects. The purpose of this study is to clustering; ROI
develop a system to evaluate the performance of BIM using a K-means clustering algorithm
and ROI analysis to reflect the cost-effectiveness of BIM investment. The proposed system also
includes methods for determining best-case projects with high similarities from existing case
projects and benchmarking their evaluation know-how, and its usability was verified through
experienced BIM users.

1. Introduction ROI on the value of BIM introduction and benchmark­

ing their assessment capabilities.
Recently, many governments have proposed policies
For many construction companies, including com­
for the development of building information model­
panies without BIM experience, this system will be
ling (BIM) delivery guidelines or the mandatory intro­
useful for planning further improvement and deter­
duction of BIM. Nevertheless, McGraw Hill Construction
mining whether to introduce BIM by considering spe­
(2014) indicated that, owing to the BIM performance
cific projects and company environments. To validate
assessment system, the corporate sector is far from
the reliability and accuracy of the proposed system,
obtaining a clear expectation value and objective
a practical survey was conducted by BIM experts.
opportunity cost, which complicates decision-making
regarding the introduction and operation of BIM for
many construction companies. 2. Research background and contribution
Traditional BIM performance assessment tools are
2.1. Previous research for BIM performance
focused on qualitative aspects rather than the financial
assessment
assessment aspects. These tools can measure how
successfully a BIM project is progressing or the ability In terms of research into BIM capability assessments,
of a company to successfully complete a BIM project Lee and Yu. (2013) proposed the relationships among
(Won and Lee 2016). Although qualitative BIM perfor­ different metrics related to BIM acceptance by utilizing
mance assessment tools are effective for determining the technology acceptance model (TAM). Kim and Park
BIM capabilities and for establishing an operational (2012) suggested necessary process improvements
strategy, whereas the cost-effectiveness of BIM invest­ and a change of organization for construction projects
ment (i.e., the financial benefit expected from BIM introducing BIM. They compared BIM promotion poli­
introduction) is often difficult to determine. cies between the USA and the UK as references for the
To resolve this problem, this study suggests a BIM domestic introduction and promotion policy of BIM.
performance assessment system that enables both the Succar (2009) and Succar and Kassem (2015) derived
assessment of the current BIM execution ability and assessment areas for technology, process, and institu­
prediction of the cost-effectiveness before and after tional aspects of the BIM capability assessment, and
implementation of BIM projects through classified BIM capability into three phases: object-
a comparison with other projects. The method is based modelling, model-based collaboration, and net­
based on a K-means clustering algorithm and ROI ana­ work-based integration. Becerik-Gerber and Rice
lysis. This includes finding the most similar case project (2010) analysed the substantial benefits and character­
among many case projects, predicting the expected istics of BIM from the perspective of cost-effectiveness,

CONTACT Leen-Seok Kang [email protected] Gyeongsang National University, Jinju 52828, Korea
© 2020 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group on behalf of the Architectural Institute of Japan, Architectural Institute of Korea and
Architectural Society of China.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits
unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
 JOURNAL OF ASIAN ARCHITECTURE AND BUILDING ENGINEERING 79

proving that a greater BIM performance experience development environment; i.e., nationality, organiza­
leads to greater profitability and cost-cutting effects tion, and entity (Hoffer 2014; Poirier, Staub-French, and
due to the reduced duration. Chien, Wu, and Huang Forgues 2015; Neelamkavil and Ahamed 2012). For this
(2014) analysed the risk factors required to consider reason, the ROI analysis results of BIM projects have
a swift counterstrategy for various risk elements and a large range of 229–39,900% (Azhar 2011). This is
increase the possibility of BIM project success, then because the metrics utilized for ROI analysis are not
suggested 13 risk factors including aspects of technol­ quite consistent. Most are developed through surveys
ogy, management, human resources, finance, and or expert interviews or result from the application of ROI
legal procedure. Bryde, Broquetas, and Volm (2013) metrics utilized in the field of information technology.
derived standard metrics for determining the success Therefore, it is necessary to develop specific metrics for
of a BIM-based project, reporting both positive and BIM. As various BIM functions are utilized across the
negative effects that will follow the introduction of entire life cycle of a project, the revenue structure can
BIM. Lee et al. (2013) proposed an ROI predictive ana­ differ depending on the phase of application. Therefore,
lysis calculation formula that reflects controllable and tangible effects from the BIM functions, such as clash
uncontrollable factors when evaluating the value of detection and reduction of the construction period, and
BIM introduction regarding a change of plan. the intangible effects, such as image improvement of
Regarding the cost-effectiveness analysis of BIM, the corporation, should be converted into financial gain.
Barlish and Sullivan (2012) defined the return metrics
as change orders, requests for information (RFIs), and
schedules, and the investment metrics as design costs 3. Development of ROI measure items for BIM
and contractor costs, while suggesting a calculation performance assessment
method for each index. Suermann (2009) developed 3.1. Total cost of ownership for BIM-ROI
six performance indicators, including quality control
(rework reduction), schedule conformance (delay Recently, due to the lower cost of BIM-related hard­
reduction), total cost (cost reduction), unit (square ware and the shift away from permanent licensing to
feet)/man-hour, cost/unit (square feet), and safety paying for software annually, operating costs are
(delayed man-hour reduction), and calculated the higher than the initial acquisition costs. For this reason,
cost and effect of BIM. Love et al. (2014) suggested the valuation factors for BIM investment cost using
a framework for the investment value evaluation of Total Cost of Ownership (TCO), which takes into
BIM comprising four essential questions that reveal account the operating costs of hardware and software,
how a property owner can earn business values from are selected in this study.
investment in BIM through benefit realization manage­ In this study, the metrics for BIM investment are
ment (BRM). Sen (2012) derived the factors of BIM and largely classified into direct and indirect costs. Direct
virtual design and construction (VDC) that influence costs comprise outsourcing, development of applica­
the ROI through interviews with experts, case studies, tions and the database, and construction of BIM func­
and research results, and suggested a finance model tions, as well as personnel expenses related to the
for ROI calculation. Giel and Issa (2011) conducted introduction of BIM. In particular, the introduction of
a comparative analysis between an actual project BIM functions includes modelling, calculation of supply
case where BIM was applied and a project case where and costs, interference review, and simulation. This
BIM was not applied. The BIM construction cost was allows us to calculate the cost of BIM introduction,
assumed to be 0.5% of the total cost, and the effect of which is predominantly equipped with a specific func­
BIM considered the cost of duration reduction per day tionality. Indirect costs are classified into one-time
and interest rates for the duration reduction. indirect costs and durable indirect costs. One-time
Reizgevičius et al. (2018) presented the methods and indirect costs include the costs for purchasing HW/
factors of ROI evaluation for small design companies SW and other equipment, alterations to the office
and performed a comparative analysis of Autodesk’s environment due to BIM operation, and modification
ROI assessment methods. costs, as well as the costs for education and purchasing
assets necessary for introducing the BIM system. The
durable indirect costs include the costs for upgrading
2.2. Contribution comparing with previous HW/SW and other equipment for the efficient opera­
research tion of BIM, updating applications, and continual
Traditional research on BIM performance assessment development of BIM applications.
defines the performance capability required for an orga­
nization or an individual, then develops a maturity
3.2. Tangible/intangible effects of BIM-ROI
model and metrics for evaluating said organization’s
or individual’ capability. However, the metrics sug­ In traditional IT-ROI analyses, the benefit of the IT
gested by traditional research differ with the project is calculated as a cost through the analysis of
80 H.-S. KIM ET AL.

tangible and intangible effects (Carratta, Dadayan, and As shown in Figure 1, this study uses the K-means
Ferro 2006). The tangible effect can be converted to algorithm in the process of searching for the similar
financial profits and is diversified into the improve­ project group.
ment of labor productivity, improvement of process A K-means algorithm is the most widely used one in
productivity, and so forth. The intangible effect is cluster analysis and is known to present quick and
a potentially beneficial nonfinancial profit that can be stable results. Because the K-means algorithm is effec­
diversified into superior brand value and intellectual tive in identifying the characteristics of different types
property, for example. Based on these diversifications, of assessment items, it can be used for identifying the
we developed metrics for the tangible and intangible characteristics of the type of data for this study.
effects of BIM investment. The tangible effects of BIM The algorithm is processed by inputting project infor­
include a reduction of duration, a request for informa­ mation, setting metrics and weighted values, calculating
tion (RFI) and process modification, and reduction of similarities, searching for similar projects, and extracting
the construction costs. The intangible effect enables similar projects. The K-means clustering algorithm is used
calculation of the strategic superiority of a company to classify the case projects into multiple clusters. The
and intellectual property with regard to the employees type of data used for these clusters is then determined by
and includes the improvement of customer preference the similarity ratio between the case project and target
by the introduction of BIM, expansion of the BIM mar­ project. The similarity ratio defined in this research is the
ket, increased employee productivity, etc. ratio of similar parts derived through a comparison
In general, the benefits of these tangible and intan­ between case project and target project characteristics.
gible effects are difficult to objectively calculate. The Therefore, in this study, relevant articles were analysed
costs estimated by construction managers with a high when suggesting metrics in order to enable comparison
understanding of BIM projects are used in this study. of project characteristics. When the clustering analysis is
conducted using the similarity ratio, the case projects are
provided in categories. Among these, the cluster with the
4. K-means clustering algorithm-based model highest similarity ratio was determined to derive the
for searching similar project project with the highest similar ratio, termed the best
project, which was utilized for benchmarking. Through
4.1. K-means clustering algorithm-based process
various integrated analysis processes, different types of
BIM has a wide range of applications from design to information were provided for the target project to refer
maintenance phases. Therefore, efficient implementa­ to, including the metrics, measured values, costs, and
tion of BIM requires knowledge of appropriate pro­ usage frequency.
cesses for the requirements of the user, the purpose
of BIM introduction, its scope of application, the orga­
nization in charge of its implementation, and the types 4.2. Metric development for searching similar
of software employed. Considering this, a reasonable projects
evaluation method should be applied in accordance The metrics derived in this study will evaluate the simi­
with various types of BIM implementation at the BIM larity of BIM environments between the case project
performance management level. Using the searching and the target project. The metrics can contain informa­
process shown in Figure 1, evaluation methods and tion on projects and organizations including the scale of
metrics for conducting ROI in various BIM project envir­ the project, composition of the organization, and
onments are analysed, and the most similar projects retained technologies. This information is included in
are extracted to act as a benchmark. the BIM guidelines and BIM implementation plan (e.g.

Figure 1. Process for searching of similar projects using the K-means clustering algorithm.
 JOURNAL OF ASIAN ARCHITECTURE AND BUILDING ENGINEERING 81

Table 1. Metrics for analysing the BIM environment. can be applied differently according to the country
Metrics Description and region. Second, items such as the purpose of
Project information Project name, project type, project cost, BIM, functions requiring BIM and their effects, improve­
country, region, etc.
Goal of BIM Improvement of productivity, schedule ment of productivity, and the risk reduction are input
introduction reduction, cost reduction, etc. for the purpose of BIM introduction. Third, in the BIM
BIM tool Revit architecture, Bentley architecture,
ArchiCAD, Navisworks, Vico control, etc. tool, technologies retained by the company and their
BIM application Design phase-conceptual, design phase-detail, utilization capabilities are evaluated through the BIM
phase construction phase, etc.
Performance Organization, technology, management
S/W and other equipment information such as
capability maturity Autodesk, Navisworks, and Bentley Architecture.
Fourth, in the BIM application phase, a BIM information
exchange system and BIM process specifically modi­
BIM Project Execution Planning Guide V2.1). During fied for detailed tasks were reflected by designating
construction of the BIM environment, a phase is the specific step of a task or specific task requiring BIM
required in which the project’s information and purpose application. Finally, the performance capability utilizes
for introducing BIM are analysed to clarify the user’s the BIM maturity model to evaluate the intellectual
primary goal. These preconditions are determined by capability of the staff or the technological level of the
the performance capability and technologies owned by organization in order to perform BIM.
the target company and determine whether the com­
pany is indeed capable of introducing BIM. Through
these results, deficiencies related to conducting BIM 4.3. Data organization for K-means
are finalized and complemented to establish an efficient algorithm-based clustering analysis
BIM implementation plan.
Table 1 shows five types of metrics derived from the As shown in the top panel of Figure 2, the information
analysis of the reference and BIM implementation input to the metrics exhibits various forms shown by
plans (AEC 2012; CIC Research Group 2011; Building letters and numbers.
and Construction Authority 2012; Autodesk 2014). The This data must be standardized into numerical data
metrics are classified into project information, goal of for application to the K-means clustering algorithm. The
BIM introduction, BIM tool, BIM application phase, and method for digitizing the input values employs the
performance capability. These were determined after frequency number or ratios occurring in similar projects
a discussion with working-level staff in order to effi­ compared with the target project. However, when using
ciently comprehend the performance capability of the values calculated using identical frequency numbers or
target company and the user requirements. ratios without considering the characteristics of each
First, basic project information is input to the pro­ item, the correlation between the metrics cannot be
ject name, including the type of project such as bridge, reflected. For example, the types of software input to
road, or dam, and total cost of the project. Notably, the the BIM tool cannot be compared; i.e., the effect of the
location of the project can be specified by the level of software on BIM performance is difficult to compre­
country and region (Kang 2012) and the ROI metrics hend. To solve this problem, the data for each metric

Figure 2. Calculation process of similarity ratio.

82 H.-S. KIM ET AL.

are expressed in a unified unit, which is defined by the Ci ¼ jZmaturityi Zmaturitymain j

similarity to which the target project is compared.
First, among the six metrics, four variables used for Matusri : similarity of performance competency for
the clustering analysis are selected (type of project, organization; Xmaturityi ; Ymaturityi ; Zmaturityi
cost of project, purpose of BIM introduction, and per­
: organization capability; technical capability
formance capability maturity) and their weighted
and management capability for case project
values are input. Then, similarity ratios are calculated
for each variable. The calculation formula for the simi­ iXmaturitymain ; Ymaturitymain ; Zmaturitymain
larity ratio between the case project and target project : organization capability; technical capability
is categorized into three types. and management capability for target project

Csri ¼ Costmain � Costi � 100%; Costmain Costi < 0 Third, Formula 4 is used to evaluate the similarity
Csri ¼ Costi � Costmain � 100%; Costmain Costi > 0 rate between the case project and target project
Csri : cost similarity; Costi : cost of case project i regarding the performance capability of the organiza­
tion, which is categorized into organizational capabil­
Costmain : cost of target project
ity, technical capability, and management capability.
(1)
These are derived from the maturity model developed
The project cost information is utilized to calculate the in the previous research. The similarity ratio is input
similarity of project scale (Formula 1). The cost of the from 0 to 5 points according to each capability, based
project is expected to exhibit greater differences on the difference between the case project and the
between the case and target projects than the other target project. Finally, the similarity ratio between each
variables. However, due to the fact that the difference metric is then calculated in the range of 0 to 100 and
would be shown in the type of interval-based variable, can be integrated, along with the application of
for example, the similarity in this factor is simply calcu­ a weighted value (determined in accordance with the
lated using the ratio of costs between both projects. importance of weights), depending on its degree of
This problem can be partially resolved by conducting importance. The integrated data are then applied to
a relative correction in the weighted-value calculation the K-means clustering algorithm.
phase.

Facsri ¼ ComItemi � TotItemi � 100% 5. Development and application of BIM

Facsri : Item similarity performance assessment system
ComItemi : number of items commonly included in
(2) 5.1. System process according to assessment
the case and target project timing
TotItemi : total number of items in the case and
In this section, a BIM performance assessment system
target project; Exceptforduplicateitems
was developed based on previously suggested
Second, to measure the similarity ratio of the variable methods.
that represents various items, such as the purpose of Application of the BIM performance assessment can
BIM introduction, Formula 2 is used, which reflects differ according to the purpose and timing of the
detailed items. For variables that are input with assessment; it is classified into a first phase (i.e. validity
detailed items, specific software may be used, as for of BIM introduction) and a second phase (i.e. effective­
the metric of the BIM tool. This means that the input ness analysis of BIM implementation), as shown in
items are correlated with each other, which is useful Figure 3.
when searching for an item that is highly correlated The first-phase assessment is conducted to evaluate
through two coincidental items. Therefore, this study the expected value that can be obtained by the intro­
uses the support calculation formula (Formula 3) to duction and investment of BIM, prior to conducting
calculate the similarity of the items. a project. This can help support reasonable decision-
making among stockholders who decide the introduc­
Number of transactions that tion and investment of BIM. This phase is utilized when
include both items A and B
SupportðA ) BÞ ¼ (3) no BIM-related experience exists and when relevant
Total transactions
information is lacking. By inputting only minimal infor­
mation, the validity of the introduction and investment
Matusri ¼ ½15 ðAi þ Bi þ Ci Þ�=15 � 100% (4)
of BIM can be approximately determined.
The second phase assessment is conducted to eval­
Ai ¼ jXmaturityi Xmaturitymain j; uate whether BIM management is properly promoted
during the operation of BIM. Continuous BIM perfor­
Bi ¼ jYmaturityi Ymaturitymain j; mance assessment is conducted during BIM operation
 JOURNAL OF ASIAN ARCHITECTURE AND BUILDING ENGINEERING 83

Figure 3. Assessment process by BIM operation phase.

to monitor the achievement of the planned BIM objec­ study, at least a certain level of data is required. But,
tive. As such, the requirements of the site can be BIM-related information in the case projects alone
determined and relevant feedback can be provided, makes it difficult to obtain the appropriate data for
thereby gradually establishing a system for effective the evaluation items. For this reason, hypothetical
BIM performance management. data were generated and used for the case study by
constructing information from the case projects to suit
the evaluation items through consultation with
5.2. First-phase case study experts.
The first-phase assessment aims to derive similar case First, the characteristic information of the BIM
projects and secure approximate BIM performance Introduction environment is input to reflect the envir­
information. Therefore, this case was analysed to deter­ onmental conditions (Figure 4). The project character­
mine the expected construction costs, tangible effects, istic information is applied to the calculation of the
intangible effects, etc., for a company wishing to intro­ similarity ratio used as basic data for clustering the
duce BIM. For the case study of the developed system, similar projects. As the clustering analysis differs with
the previously collected data of case projects are orga­ the similarity ratio, care should be taken to best reflect
nized into a data set that is appropriately categorized the characteristics of the target project.
into the evaluation items of BIM-ROI and BIM imple­ There are six metrics in the calculation of similarity
mentation environment. To perform a reliable case ratios, which allow the user to select data types.

Figure 4. Input data and similarity rate analysis.

84 H.-S. KIM ET AL.

Therefore, the metrics that will be used for the analysis similar projects, can be determined using the graph
of the BIM introduction environment can be selected shown in the bottom panel of Figure 5. Types of
and require a process for assigning weighted values to metrics are then considered for introducing and con­
each metric. As shown in Figure 4, the metrics for structing the BIM, and the types of metrics that gen­
project characteristics are selected as “BIM tools” and erate investment and revenue in the similar projects
“BIM goal,” and the similarity ratio calculation is con­ can be determined.
ducted by inputting the weighted values, resulting in In this case study, the individual BEP (the best break-
a similarity ratio for “BIM tools” and “BIM goal.” even point of 0.9 years) and integrated BEP (average
Through this, the case project that is most similar to break-even point of 1.32 years) for 18 similar projects
these two metrics can be confirmed, while excluding are shown in Table 2. In addition, the average cost and
projects with low similarity ratios from the cluster. metric frequency (maximum 18, minimum 1) of the
After inputting the number of cluster k and conducting Total Cost of Ownership (TCO) were identified. For
the clustering analysis, the result of the K-means clus­ the tangible and intangible effects of BIM, the profit
tering analysis can be confirmed, as shown in the costs and metric frequency (maximum 18, minimum 2)
upper right corner of Figure 5. This information is were identified
then used to obtain the characteristics of the cluster Considering that this result is calculated by only
and distribution of individual projects. The results inputting minimum information regarding “BIM tools”
prove that the best of the three clusters is cluster No. and “BIM goal,” it is relatively specific. This information
2. This is because the clustering analysis was con­ is therefore considered sufficient for nonprofessional
ducted on the basis of the similarity ratio; therefore, corporations with no BIM experience to decide
the cluster that is farthest from the origin is considered whether to introduce BIM.
the best. However, as this cluster has a higher average
distance within the cluster and it includes the farthest
case, this anomaly should be calibrated and 5.3. Second phase case study
reconsidered. This case study is performed to consistently evaluate
When the similar projects are derived through the the effect of BIM during BIM operation in a large-scale
cluster analysis, approximate information, including project with multiple on-site construction zones,
the major metrics and costs of each item related to where the BIM is analysed and integrated for each

Figure 5. Results of the K-means clustering and ROI analysis.

Table 2. Results of the first-phase case study.

BEP Avg. 1.32 (max. 0.9)
Investment cost ($) Avg. 10,619
ROI 145% (3 years)
Major Investment cost Sub-constrictors cost (18 times),
metrics HW cost, SW cost, etc.
Tangible and Planned duration vs. actual duration (three times), number of design
intangible changes [unit],
improvement of labor productivity, etc.
 JOURNAL OF ASIAN ARCHITECTURE AND BUILDING ENGINEERING 85

Figure 6. Analysis of ROI and integration of common metrics.

construction zone. The basic information of the project usage frequency, as shown in Table 3. The decision on
to be created is input in a module screen (Figure 6) and whether to use these as common metrics for each site
can be utilized as version management information for was made through consultation with practitioners.
further consistent BIM performance management. Through these efficient discussions, the planned con­
After inputting the basic information, the metrics of struction duration and actual duration, a number of
TCO and BIM benefits for BIM performance assessment reviews requested, and a number of rework occurrences
are selected. The metrics can be selected through dis­ were obtained as common metrics. These metrics were
cussions among internal staff or manually by creating delivered to all sites of the project so that BIM perfor­
new metrics. Moreover, as mentioned in the first-phase mance could be consistently monitored using identical
analysis, the similar projects derived from the K-means metrics at each site. Likewise, the metrics with high
clustering analysis can be utilized to select the metrics. usage frequency can be consistently reviewed and
Then, the values of each metric are input, and ROI designated as common. Repeating this will eventually
analysis is performed. The measured values of ROI constitute an appropriate package for the project.
comprise quantity and unit price. To ensure the cred­
ibility of these data, criteria are required for the calcu­
6. Reliability and accuracy of the developed
lation. After inputting the measured values, the return
system
flow, NPV, ROI, BEP, etc., are analysed to review the
validity of investment. To analyse the reliability and accuracy of the BIM per­
Table 3 shows that the break-even point for this formance assessment system proposed in this study,
case study is 1.59 years, which means that the cumu­ 30 practitioners (average experience of 11 years, more
lative revenues are higher after approximately 1 year than two previous experiences with BIM projects and
and 7 months and the cost-effectiveness is highest at ROI) were surveyed. A total of 11 questions were asked
2400% in Sector No. 2. about system reliability, system convenience, and
Figure 6 (lower panel) shows how a general manager overall accuracy (Table 2), and a frequency and relia­
can aggregate the results of each site to improve BIM bility analyses were conducted using SPSS 21. Each
performance management across the entire project. item employed the Likert 5-point scale, where 5 point
The data of each site (November 2014–January 2015) represents “high agreement” and 1 point represents
were collected and analysed to derive metrics with high “poor agreement”. As a result of analysing the reliabil­
ity by questionnaire, Cronbach’s alpha was found to be
more reliable than 0.860 (Table 4).
Table 3. Results of the second-phase case study. The system reliability, convenience, and overall
BEP Avg. 1.59 (max. 0.5, Sector No. 2)
accuracy were 3.97, 3.92, and 3.95, respectively. In
Investment cost ($) 30,619
ROI 350% (Max. 2400%, Sector No. 2) particular, the value of the metrics and the objectivity
Frequently Investment User education cost and re-education of assessment timing were the highest at 4.0 in the
used cost cost,
metrics HW cost, SW cost, etc.
system reliability. In terms of system convenience satis­
Tangible and Planned duration vs. actual duration, faction, the assessment procedure satisfaction was the
intangible number of reviews for opinions [unit], highest at 3.93. For the overall accuracy, benchmarking
number of reworks [unit] etc.
information was the highest at 4.00 or higher. These
86 H.-S. KIM ET AL.

Table 4. Reliability and accuracy results.

Cronbach’s
Type Description α
System Reliability of BIM-ROI analysis procedure and analysis timing (first and second phase), reliability of BIM-ROI metrics 0.912
reliability
System Satisfaction of BIM-ROI analysis procedure and BIM-ROI metrics, etc. 0.860
convenience
Overall Overall BIM-ROI accuracy 0.886
accuracy (vs. existing ROI)

results indicate that the proposed methodology and implementation environment and ROI evaluation
system are highly useful as a benchmarking tool for items were required, but, in reality, it was difficult to
BIM performance and investment analysis. collect such case data. This is because the BIM-ROI
information provided by most existing case projects is
different and inconsistent. In the future research, it is
7. Conclusion necessary to obtain enough case data based on the
evaluation items presented in this study and system­
In this study, an assessment system for BIM perfor­
atize evaluation items and measurement methods for
mance was suggested to predict and evaluate the cost-
the types of BIM projects. And, it is also necessary to
effectiveness of the introduction and operation of BIM.
suggest key evaluation items and to objectify their
First, to suggest an ROI analysis model for BIM,
weights through the analysis of the association among
a traditional IT-ROI technique was utilized to develop
the evaluation items.
a BIM-ROI evaluation model, employing metrics sug­
gested in traditional BIM performance analysis and in
projects or research related to BIM-ROI analysis. By
utilizing the traditional TCO model, we developed Disclosure statement
metrics for analysing the construction costs of BIM No potential conflict of interest was reported by the authors.
and the tangible and intangible effects of BIM.
Second, the K-means clustering algorithm was
applied to benchmark projects that are most similar
References
to the environment of BIM introduction of the target AEC (UK). 2012. “BIM Protocol Project BIM Execution Plan.”
project. The metrics used for evaluating the similarity AEC (UK).
ratio of the BIM introduction environment between the Autodesk. 2014. “ BIM for Infrastructure.” Autodesk.
Azhar, S. 2011. “Building Information Modeling (BIM): Trends,
target project and the case project were constructed in
Benefits, Risks, and Challenges for the AEC Industry.”
accordance with various BIM working-level conducting Leadership and Management in Engineering 11 (3):
guidelines. The BIM performance analysis information 241–252. doi:10.1061/(ASCE)LM.1943-5630.0000127.
was derived from the similar case projects. The BIM Barlish, K., and K. Sullivan. 2012. “How to Measure the
performance assessment, which differs from the user’s Benefits of BIM – A Case Study Approach.” Automation in
purpose and time of the performance assessment, was Construction 24: 149–159. doi:10.1016/j.
autcon.2012.02.008.
enabled by suggesting the process for each applica­ Becerik-Gerber, B., and S. Rice. 2010. “The Perceived Value of
tion phase. Building Information Modeling in the US Building
Third, by utilizing the suggested methods, a BIM Industry.” Journal of Information Technology in
performance assessment system was developed, Construction (Itcon) 15 (15): 185–201.
which provides and enables the benchmarking of var­ Bryde, D., M. Broquetas, and J. M. Volm. 2013. “The Project
Benefits of Building Information Modelling (BIM).”
ious actual assessment methods and metrics related to
International Journal of Project Management 31 (7):
BIM project performance in various environments. 971–980. doi:10.1016/j.ijproman.2012.12.001.
Through a survey of reliability, convenience and accu­ Building and Construction Authority. 2012. “ Singapore BIM
racy with the proposed system, users verified that they Guide.” Singapore BIM Guide.
had a higher degree of reliability, convenience and Carratta, T., L. Dadayan, and E. Ferro. 2006. “ROI Analysis in
accuracy than for ROI analysis with existing BIM E-Government Assessment Trials: The Case of Sistema
Piemonte.” In International Conference on Electronic
models. Government, 329–340. Kraków, Poland.
The proposed method for assessing BIM perfor­ Chien, K.-F., Z.-H. Wu, and S.-C. Huang. 2014. “Identifying and
mance provides actual BIM performance information, Assessing Critical Risk Factors for BIM Projects: Empirical
thereby providing a benchmark tool for evaluating the Study.” Automation in Construction 45: 1–15. doi:10.1016/j.
validity of BIM introduction and operation for users autcon.2014.04.012.
CIC Research Group. 2011. “ BIM Project Execution Planning
with no experience in BIM performance assessment.
Guide V2.1.” Department of Architectural Engineering, the
In order to reliably evaluate BIM performance by Pennsylvania State University.
evaluation items for BIM-ROI and project similarity, Giel, B. K., and R. R. A. Issa. 2011. “Return on Investment
numerous data for similar projects in the BIM Analysis of Using Building Information Modeling in
 JOURNAL OF ASIAN ARCHITECTURE AND BUILDING ENGINEERING 87

Construction.” Journal of Computing in Civil Engineering 27 Ottawa, Canada: Institute for Research in Construction,
(5): 511–521. doi:10.1061/(ASCE)CP.1943-5487.0000164. National Research Council of Canada.
Hoffer, E. 2014. “Measuring the Value of BIM: Achieving Poirier, E. A., S. Staub-French, and D. Forgues. 2015.
Strategic ROI.” Autodesk. “Measuring the Impact of BIM on Labor Productivity in
Kang, T. W. 2012. “ Planning Report for Development BIM a Small Specialty Contracting Enterprise through
Maturity Assessment Framework and Model.” Korea Action-research.” Automation in Construction 58: 74–84.
Institute Construction Technology. doi:10.1016/j.autcon.2015.07.002.
Kim, K. P., and S. H. Park. 2012. “Comparative Analysis of the Reizgevičius, M., L. Ustinovičius, D. Cibulskienė, V. Kutut, and
BIM Status in the UK and US for Improving the Efficiency of L. Nazarko. 2018. “Promoting Sustainability through
Construction Project Management Process in Korea.” Investment in Building Information Modeling (BIM)
Project Management Review Korea 2 (2): 1–16. Technologies: A Design Company Perspective.”
Lee, D. M., J. Sang, K. Ahn, H. Park, and S. Y. Chin. 2013. Sustainability 10 (3): 600. doi:10.3390/su10030600.
“A Study on the Process and Elements of Expected ROI Sen, S. 2012. “The Impact of BIM/VDC on ROI: Developing
Analysis for Estimating Value by Adopting BIM.” KIBIM a Financial Model for Savings and ROI Calculation of
Annual Conference 3 (1): 81–82. Construction Projects.” Master´s thesis, KTH, School of
Lee, S.-K., and J.-H. Yu. 2013. “Key Factors Affecting BIM Architecture and the Built Environment.
Acceptance in Construction.” Journal of the Architectural Succar, B. 2009. “Building Information Modelling Framework:
Institute of Korea Planning & Design 29 (8): 79–86. A Research and Delivery Foundation for Industry
doi:10.5659/JAIK_PD.2013.29.8.79. Stakeholders.” Automation in Construction 18 (3):
Love, P. E. D., J. Matthews, I. Simpson, A. Hill, and 357–375. doi:10.1016/j.autcon.2008.10.003.
O. A. Olatunji. 2014. “A Benefits Realization Management Succar, B., and M. Kassem. 2015. “Macro-BIM Adoption:
Building Information Modeling Framework for Asset Conceptual Structures.” Automation in Construction 57:
Owners.” Automation in Construction 37: 1–10. 64–79. doi:10.1016/j.autcon.2015.04.018.
doi:10.1016/j.autcon.2013.09.007. Suermann, P. C. 2009. “Evaluating the Impact of Building
McGraw Hill Construction. 2014. “The Business Value of BIM Information Modeling (BIM) on Construction.”
for Construction in Major Global Markets: How Contractors Ph.D. dissertation, Florida Univ. Gainesville Graduate
around the World Are Driving Innovation with Building School.
Information Modeling.” Smart market report. Won, J., and G. Lee. 2016. “How to Tell If a BIM Project Is
Neelamkavil, J., and S. Ahamed. 2012. “The Return on Successful: A Goal-driven Approach.” Automation in
Investment from BIM-driven Projects in Construction.” Construction 69: 34–43. doi:10.1016/j.autcon.2016.05.022.