diagnostics

Article
Deep Learning Model Based on You Only Look Once Algorithm
for Detection and Visualization of Fracture Areas in
Three-Dimensional Skeletal Images
Young-Dae Jeon 1 , Min-Jun Kang 2 , Sung-Uk Kuh 2, *, Ha-Yeong Cha 3 , Moo-Sub Kim 3 , Ju-Yeon You 3 ,
Hyeon-Joo Kim 3 , Seung-Han Shin 4 , Yang-Guk Chung 4 and Do-Kun Yoon 3, *

1 Department of Orthopedic Surgery, University of Ulsan College of Medicine, Ulsan University Hospital,
Ulsan 44033, Republic of Korea; [email protected]
2 Department of Integrative Medicine, College of Medicine, Yonsei University of Korea,
Seoul 03722, Republic of Korea; [email protected]
3 Industrial R&D Center, KAVILAB Co., Ltd., Seoul 06675, Republic of Korea; [email protected] (H.-Y.C.);
[email protected] (M.-S.K.); [email protected] (J.-Y.Y.); [email protected] (H.-J.K.)
4 Department of Orthopedic Surgery, Seoul St. Mary’s Hospital, College of Medicine, The Catholic University
of Korea, Seoul 06591, Republic of Korea; [email protected] (S.-H.S.); [email protected] (Y.-G.C.)
* Correspondence: [email protected] (S.-U.K.); [email protected] (D.-K.Y.)

Abstract: Utilizing “You only look once” (YOLO) v4 AI offers valuable support in fracture detection
and diagnostic decision-making. The purpose of this study was to help doctors to detect and diagnose
fractures more accurately and intuitively, with fewer errors. The data accepted into the backbone are
diversified through CSPDarkNet-53. Feature maps are extracted using Spatial Pyramid Pooling and a
Path Aggregation Network in the neck part. The head part aggregates and generates the final output.
All bounding boxes by the YOLO v4 are mapped onto the 3D reconstructed bone images after being
resized to match the same region as shown in the 2D CT images. The YOLO v4-based AI model was
Citation: Jeon, Y.-D.; Kang, M.-J.; Kuh,
evaluated through precision–recall (PR) curves and the intersection over union (IoU). Our proposed
S.-U.; Cha, H.-Y.; Kim, M.-S.; You, J.-Y.;
system facilitated an intuitive display of the fractured area through a distinctive red mask overlaid
Kim, H.-J.; Shin, S.-H.; Chung, Y.-G.;
on the 3D reconstructed bone images. The high average precision values (>0.60) were reported as
Yoon, D.-K. Deep Learning Model
0.71 and 0.81 from the PR curves of the tibia and elbow, respectively. The IoU values were calculated
Based on You Only Look Once
Algorithm for Detection and
as 0.6327 (tibia) and 0.6638 (elbow). When utilized by orthopedic surgeons in real clinical scenarios,
Visualization of Fracture Areas in this AI-powered 3D diagnosis support system could enable a quick and accurate trauma diagnosis.
Three-Dimensional Skeletal Images.
Diagnostics 2024, 14, 11. Keywords: YOLO v4; fracture detection; deep learning; three dimensional (3D) reconstructed image;
https://doi.org/10.3390/ tibia and elbow
diagnostics14010011

Academic Editors: Dechang Chen and

Magalie Viallon
1. Introduction
Received: 2 November 2023 From a medical engineering perspective, “artificial intelligence (AI)”-based diagnostic
Revised: 30 November 2023 technology has developed greatly. AI has been applied to various platforms from large
Accepted: 10 December 2023 medical devices to portable mobile devices. For example, diagnoses using AI are used in
Published: 20 December 2023
various places in the medical field, such as AI models to check the bone density status
or predict fracture risk from medical images [1]. According to the most recent research,
there have been many studies on the diagnosis of fractures from X-ray images using
Copyright: © 2023 by the authors.
AI. Kuo et al. demonstrated the usefulness of AI for predicting fractures from radiation
Licensee MDPI, Basel, Switzerland. images [2]. Kolanu et al. present a study on diagnosing fractures using the “X-Ray Artificial
This article is an open access article Intelligence Tool (XRAIT)” [3]. All of the above cases are methods for the diagnosis of
distributed under the terms and bone fractures from X-ray images with a high accuracy. However, most studies only
conditions of the Creative Commons showed the performances of AI for diagnosing bone fractures using two dimensional
Attribution (CC BY) license (https:// (2D) images. Naturally, the 2D images included a lot of information which can provide
creativecommons.org/licenses/by/ a basis for the diagnosis of a bone fracture. However, some factors that could reveal the
4.0/). characteristics of the bone fracture may be lost due to dimensional limitations. Moreover,

Diagnostics 2024, 14, 11. https://doi.org/10.3390/diagnostics14010011 https://www.mdpi.com/journal/diagnostics

Diagnostics 2024, 14, 11 2 of 19

because the overlapped object in 2D X-ray images can disturb the identification of the
affected region, an intuitive diagnosis can be made using 3D images instead of 2D images.
In particular, identifying the impacted region within an image of a comminuted fracture
has been challenging. An inaccurate diagnosis in the case of a bone fracture can create
critical problems for patients, and it can be one of the major factors that lengthens the
operation time [4].
However, AI can assist in the detection of the fracture region and to make a decision
for the diagnosis and surgical strategy. The diagnosis of bone fractures is classified as an
object detection task in the field of computer vision. There are many deep learning models
for object detection in medical images. And, a dedicated dataset can make the model
show better performance for the object detection task. Because the fracture detection in the
medical images is difficult, models which can show an excellent detection performance are
essential. The representative model for object detection is the “you only look once (YOLO)”
model. The first YOLO model (YOLO v1) was presented in 2016 by Joseph et al. [5,6]. The
latest model is known as YOLO v8 which was presented in 2023. YOLO v4 was developed
by Alexey et al. in 2020. YOLO v4 was constructed to train the model using low-level
hardware conditions (e.g., GTX 2080 Ti). Moreover, it uses the Bag of Freebies and the Bag
of Specials methods with spatial pyramid pooling (SPP) and a path aggregation network
(PAN) [7,8]. In addition, YOLO v4 was developed to become YOLO v7 which can detect
objects in real time [9,10].
As an example, a research endeavor employed YOLO v4 to proficiently detect lung
nodules [11]. The findings demonstrated the effectiveness of YOLO v4 within a hospital
hardware environment. In one of the more recent investigations, object recognition using
YOLO v4 was scrutinized, revealing substantial enhancements in both detection speed
and accuracy. Notably, a recent study harnessed YOLO v4 to explore the classification and
detection capabilities using extensive medical data, and demonstrated a commendable
performance [12]. The ongoing refinement of YOLO v4’s performance is evident, with
continuous updates and user-driven feedback contributing to its improvement. Through
these investigations, it can be discerned that YOLO v4’s performance has experienced
consistent advancement, underscoring the ongoing research endeavors in the realm of
object-based detection. And, YOLO v4 is one of the proper models that can detect bone
fractures using medical images [13]. YOLO v4 can detected the fractured region using a
bounding box on 2D computed tomography (CT) images, one slide at a time; bounding
boxes in several image slides can show an intuitive fractured region in the 3D reconstructed
image for the bone region. In this case, a more accurate diagnosis can be obtained, and errors
in diagnosis will be reduced before the surgeons begin the operation. Moreover, leveraging
YOLO v4, the system can intuitively guide diagnoses, irrespective of the surgeon’s skill
level and experience, further enhancing its efficacy [10]. The purpose of this study was to
evaluate the performance of the YOLO v4-based fracture detection method, and to show
the possibility of intuitive diagnoses through a combination of 3D image reconstruction
and fracture detection. In 3D space, the precise location of the wound can be identified,
allowing for the accurate identification of fragmented or cracked bones [14]. This study
is different from previous studies and significant in that it detects the fracture region
more intuitively and accurately through 3D visualization [15,16]. To date, no studies have
formulated a model capable of automatically reconstructing and identifying the fracture site
in three dimensions. The imperative nature of such surgeries underscores the significance
of this study.

2. Materials and Methods

2.1. Overall Workflow and Data Pre-Processing
The methods in this study were conducted according to the relevant guidelines and reg-
ulations of the Clinical Trial System of the Catholic Medical Center (CTSC). The experimen-
tal protocols in this study were approved by the Institutional Review Board (IRB) at Seoul
St. Mary’s Hospital, the Catholic University of Korea (approval number: KC20RISI1034).
Diagnostics 2024, 14, 11 3 of 19

The data pre-processing was performed by using MATLAB (2022b, Mathworks, Natick,
MA, USA), which is software optimized for analyzing experimental data and processing
data [17]. There are five steps for pre-processing the data in this study. Figure 1 shows the
work flow for the overall process in this study [18]. MATLAB offers numerous supplemen-
tary modules tailored for researchers, providing the benefit of expediting model design.
This acceleration is achieved by leveraging pre-implemented functions for preprocessing,
training, validation, and testing. MATLAB holds considerable prevalence in medical re-
Diagnostics 2024, 14, x FOR PEER REVIEW 4 of 19
search and is a robust tool, as demonstrated by its application in recent drug-related studies
leading to FDA approval [19].

Figure 1.
Figure 1. Overview
Overviewofof overall algorithm
overall algorithmof the working
of the process.
working (a) Data
process. (a) preprocessing was per-
Data preprocessing was
formed with the help of a surgeon using MATLAB. After upload of the computed tomography
performed with the help of a surgeon using MATLAB. After upload of the computed tomography
(CT) Digital Imaging and Communications in Medicine (DICOM) file including the fracture re-
(CT) Digital Imaging and Communications in Medicine (DICOM) file including the fracture region,
gion, each bounding box (yellow box) was added according to the fracture region in the CT image.
each bounding
(b) The box (yellow
model was box)on
built based was added
YOLO v4according to the fracture
and was trained region in
with a dataset the CT
made image.pre-
through (b) The
processing.
model (c) The
was built model
based on that
YOLO hasv4
been
andtrained is tested
was trained and
with validated.
a dataset (d) through
made The results of detection
pre-processing.
obtained
(c) by test
The model data
that haswere
beenevaluated
trained is by confirming
tested the loss(d)
and validated. function, precision–recall
The results of detectioncurve, andby
obtained
intersection
test data wereover union, by
evaluated andconfirming
additionalthe
data preprocessing
loss or optimization
function, precision–recall of the
curve, andmodel was per-
intersection over
formedand
union, according to the
additional evaluation
data resultsorwhen
preprocessing it needed.
optimization of the model was performed according to
the evaluation results when it needed.
2.2. Modeling and Training
The
The first step for data
construction pre-processing
of the deep learningwas the preparation
models of theusing
was performed data.MATLAB.
The formatThe
of
all the data was the CT Digital Imaging and Communications in Medicine (DICOM) image
hardware for the training models used two graphics processing units (GPUs) as NVIDIA
type; the CT images for the 155 cases included cases of comminuted fractures for several
GeForce RTX 3090 with 24 GB GPU memory, and 2.10 GHz dual Intel(R) Xeon(R) Silver
regions such as the tibia and elbow. In total, 250,000 individual CT images were used for
processors with 128 GB RAM (Intel, Santa Clara, CA, USA).
training via data augmentation. The data augmentation was conducted 10 times using
The initial learning rate was established at 0.001, coupled with a predefined set of
random shifts along the X and Y axes. The second step is the upload of the CT DICOM files
drop periods affecting the learning rate. The drop period, denoting the epoch interval for
to the image labeler of MATLAB. After uploading the DICOM files, the bounding boxes
reducing the learning rate, was fixed at 20, accompanied by a 0.5 drop factor. Despite the
initial maximum epoch being set at 50, the minibatch size was configured at 4. The chosen
learning approach involved iterative training utilizing adaptive moment estimation
(ADAM). The structure of YOLO v4 consists of a “Input”, “Backbone”, “Neck”, and
“Head”. The overall structure of YOLO v4 can be seen in Figure 2. The Input part accepts
Diagnostics 2024, 14, 11 4 of 19

were manually added on the fracture region of the CT image, one by one (the third step).
The bounding box has four kinds of information to demonstrate the right position on the
CT image: X-coordinate of the starting point for the bounding box, Y-coordinate of the
starting point for the bounding box, the number of pixels on the X-axis for the horizontal
line, and the number of pixels on the Y-axis for the vertical line. All the information of the
bounding boxes were stored by in format of a 4 by N matrix (the fourth step). It was used
for as training images to train the network. Lastly, the DICOM images were converted into
the Portable Network Graphics (PNG) format.
The model was constructed based on the YOLO v4 architecture, and training involved
feeding preprocessed datasets for both the tibia and elbow into the designed model. Follow-
ing training, the model underwent testing and validation using separately curated datasets
for the tibia and elbow. The evaluation of the 3D reconstructed bone and fracture bounding
box, obtained during testing and validation, encompassed metrics such as the loss function,
Precision–Recall Curve (P-R curve), and Intersection over Union (IoU). Subsequent to
this evaluation, iterations of additional data preprocessing or model optimization were
conducted based on the assessment results.
In the dataset construction phase, the data underwent categorization based on the
anatomical regions of the tibia and elbow. The categorized dataset underwent prepro-
cessing, transforming it into a training dataset. Post MATLAB-based preprocessing for
fracture site identification, the data were organized into datasets corresponding to the tibia
or elbow region. Within the dataset, each entry encapsulated details about the fracture
sites’ location, size, and label names, which were assigned during the preprocessing phase.
Out of the complete dataset, 60% was allocated for training, 10% for validation, and the
remaining 30% for testing. Each dataset earmarked for training, validation, and testing
incorporated critical information concerning the location and size of the fracture labels,
which are integral for the mapping of fracture sites onto masks in the 3D reconstructed
bone images.

2.2. Modeling and Training

The construction of the deep learning models was performed using MATLAB. The
hardware for the training models used two graphics processing units (GPUs) as NVIDIA
GeForce RTX 3090 with 24 GB GPU memory, and 2.10 GHz dual Intel(R) Xeon(R) Silver
processors with 128 GB RAM (Intel, Santa Clara, CA, USA).
The initial learning rate was established at 0.001, coupled with a predefined set of
drop periods affecting the learning rate. The drop period, denoting the epoch interval
for reducing the learning rate, was fixed at 20, accompanied by a 0.5 drop factor. Despite
the initial maximum epoch being set at 50, the minibatch size was configured at 4. The
chosen learning approach involved iterative training utilizing adaptive moment estima-
tion (ADAM). The structure of YOLO v4 consists of a “Input”, “Backbone”, “Neck”, and
“Head”. The overall structure of YOLO v4 can be seen in Figure 2. The Input part accepts a
PNG file transformed from a DICOM file. The data flowing into the backbone of the cross
stage partial connection DarkNet-53 (CSPDarkNet-53) is augmented. Data augmentation
primarily involved implementing 2D affine transformations on both the labeled bounding
box and CT images. The images underwent X-axis flipping or random scaling transfor-
mations ranging from 1 to 1.3 times. The decision to exclude Y-axis flipping stems from
the regular shape of medical images, eliminating the necessity for such transformations.
In contrast, flipping along the X-axis was adopted to enhance the learning experience
related to the tibia or elbow, aligning with the right and left perspectives. Serving as both a
convolutional neural network and a backbone for object detection, CSPDarknet-53 employs
a split and merge strategy, promoting an increased gradient flow within the network [20].
The CSPDarkNet-53 model in Figure 3 is a model that dramatically reduces the amount
of computation by combining the cross-stage hierarchy method of the layer based on
DenseNet [21]. There are multiple ‘CSP Blocks’ which meticulously structured to allow for
flexible output sizing (Figure 3a). The initial CSP Block comprises a singular layer, yielding
 CSPDarknet-53 employs a split and merge strategy, promoting an increased gradient flow
within the network [20]. The CSPDarkNet-53 model in Figure 3 is a model that dramati-
cally reduces the amount of computation by combining the cross-stage hierarchy method
of the layer based on DenseNet [21]. There are multiple ‘CSP Blocks’ which meticulously
Diagnostics 2024, 14, 11 structured to allow for flexible output sizing (Figure 3a). The initial CSP Block comprises 5 of 19
a singular layer, yielding an output dimension of 256 × 256 × 128. The second CSP Block
incorporates two layers, resulting in an output size of 128 × 128 × 128. The third CSP Block
generates
an output an output size
dimension of ×
of 256 64256× 64××128.
256.TheLikewise,
second the
CSPfourth
Block CSP Block produces
incorporates an
two layers,
output
resultingof in
32an× 32 × 512.
output Concluding
size of 128 × 128 the×sequence,
128. The the
thirdfifth
CSPCSP Block,
Block comprising
generates four
an output
layers, delivers
size of 64 × 64 ×an256. output of 16 ×the
Likewise, 16fourth
× 1024. Within
CSP Blockthe confines
produces anofoutput
the CSP block,
of 32 × 32convo-
× 512.
lution is succeeded
Concluding by batch
the sequence, thenormalization
fifth CSP Block, and activationfour
comprising through thedelivers
layers, mish function, con-
an output of
tributing
16 × 16 ×to1024.
the reduction
Within theofconfines
the lossof value (Figure
the CSP 3b).convolution
block, The Input part goes through
is succeeded the
by batch
Backbone
normalizationnetwork
and and is transformed
activation through into a feature
the mish map with
function, semantics.
contributing toThe
the Neck part
reduction
consists
of the lossof value
Spatial(Figure
Pyramid 3b).Pooling
The Input(SPP) andgoes
part a Path Aggregation
through Network
the Backbone (PAN).
network andThe is
SPP divides the
transformed intolast featuremap
a feature mapwithof the convolution
semantics. layer part
The Neck into consists
a fixed-size grid and
of Spatial aver-
Pyramid
ages the (SPP)
Pooling valuesandfrom the feature
a Path map toNetwork
Aggregation obtain a fixed-size
(PAN). The representation,
SPP divides the andlast
PAN uses
feature
bottom-up path augmentation
map of the convolution layer into to aeasily pass grid
fixed-size low-level featuresthe
and averages to values
high-level
fromfeatures for
the feature
more
map to accurate
obtain alocalization [22,23]. In particular,
fixed-size representation, and PAN the uses
Neckbottom-up
part connectspaththe Backbone and
augmentation to
Head, and refines and reconfigures the feature map. The Head part processes the aggre-
easily pass low-level features to high-level features for more accurate localization [22,23]. In
gated features
particular, usingpart
the Neck a one-stage
connects detector method
the Backbone andand
Head,predicts the bounding
and refines box, object
and reconfigures the
score,
featureand classification
map. The Head part score.processes the aggregated features using a one-stage detector
method and predicts the bounding box, object score, and classification score.

Figure
Figure 2.2.Overall
Overallstructure forfor
structure YOLO
YOLOv4.v4.
YOLO YOLO consists of four
consists mainmain
of four parts:parts:
Input,Input,
Backbone,
Backbone,
Neck,
Neck, and
and Head.
Head.Preprocessed
Preprocesseddatadataare
arefed
fedinto
intothe backbone
the backbone from
fromthethe
input.
input.TheThe
data accepted
data accepted
into
into the
the backbone
backbone(purple
(purplebox)
box)arearediversified
diversifiedthrough
throughcross
crossstage
stagepartial
partialconnection
connection DarkNet-53
DarkNet-53
(CSPDarkNet-53).
(CSPDarkNet-53). Feature
Featuremaps
maps are extracted
are extracted using Spatial
using Pyramid
Spatial PyramidPooling (SPP)
Pooling and aand
(SPP) Path
a Path
Aggregation
Aggregation Network (PAN) in the neck part (red box). The head part (blue box) aggregatesthree
Network (PAN) in the neck part (red box). The head part (blue box) aggregates three
YOLO v3 models and generates the final output.
YOLO v3 models and generates the final output.

The
The Bag
Bag of
of Freebies
Freebies constitutes
constitutes an
an amalgamation
amalgamation of of diverse
diverse training
training techniques
techniquesde-de-
signed to enhance the accuracy of object detectors without incurring additional inference
signed to enhance the accuracy of object detectors without incurring additional inference
costs [24,25].
costs [24,25]. This
This methodology
methodology represents
represents an
an offline
offline training
training approach
approach aimed
aimed atat elevating
elevating
the overall accuracy without a corresponding increase in overall inference costs.
the overall accuracy without a corresponding increase in overall inference costs. Conversely,
the Bag of Specials encompasses an assortment of plugins and pre-processing modules that
marginally augment the inference cost but can also substantially enhance the accuracy of
the object detection [26]. The utilization of plugin modules and pre-processing to slightly
elevate inference costs while significantly boosting accuracy is referred to as the Bag of
Specials method. Typically, these plugins are engineered to augment specific attributes,
such as expanding the receptive field, introducing attention mechanisms, or fortifying
feature integration capabilities.
 modules that marginally augment the inference cost but can also substantially enhance
the accuracy of the object detection [26]. The utilization of plugin modules and pre-pro-
cessing to slightly elevate inference costs while significantly boosting accuracy is referred
to as the Bag of Specials method. Typically, these plugins are engineered to augment spe-
Diagnostics 2024, 14, 11 cific attributes, such as expanding the receptive field, introducing attention mechanisms,6 of 19
or fortifying feature integration capabilities.

Figure
Figure3. 3.
Detailed
Detailedlayers
layers forfor
backbone.
backbone.Detailed
Detailed layers in in
layers thethe
overall structure
overall of YOLO
structure of YOLO v4 v4
(a)(a)
andand
the detailed structure for cross stage partial (CSP) blocks (b). In Figure 3a, there are multiple
the detailed structure for cross stage partial (CSP) blocks (b). In (a), there are multiple CSP Blocks, CSP
Blocks, andCSP
and each eachBlock
CSP Block is organized
is organized so thatso that
the sizethe
ofsize of the output
the output can be can be varied.
varied. The
The first first
CSP CSPhas
Block
Block has one layer and produces an output size of 256 × 256 × 128, the second
one layer and produces an output size of 256 × 256 × 128, the second CSP Block has two layers CSP Block has twoand
layers and produces an output size of 128 × 128 × 128, the third CSP Block has eight layers and
produces an output size of 128 × 128 × 128, the third CSP Block has eight layers and produces an
produces an output size of 64 × 64 × 256, the fourth CSP Block has eight layers and produces an
output size of 64 × 64 × 256, the fourth CSP Block has eight layers and produces an output size of 32
output size of 32 × 32 × 512, and the fifth CSP Block has four layers and produces an output size of
16×× 32
16 × 512, and
× 1024. thethe
Inside fifth
CSPCSP Blockafter
block, has convolution,
four layers and produces
batch an output
normalization andsize of 16
mish × 16 × 1024.
activation
are performed
Inside the CSP to block,
reduceafterthe loss value. batch normalization and mish activation are performed to
convolution,
reduce the loss value.
2.3. Evaluation and Analysis
2.3. Evaluation and Analysis
The YOLO v4-based AI model was evaluated through “Precision-Recall curve (PR
The YOLO v4-based AI model was evaluated through “Precision-Recall curve (PR
curve)”. The PR curve follows the basic attributes of classification evaluation metrics, and
curve)”. The PR curve follows the basic attributes of classification evaluation metrics, and
there are four basic attributes of classification evaluation metrics: true positive (TP), false
there are four basic attributes of classification evaluation metrics: true positive (TP), false
positive (FP), true negative (TN), and false negative (FN). The “TP” attribute indicates that
positive (FP), true negative (TN), and false negative (FN). The “TP” attribute indicates that
the AI model successfully predicted a positive classification, the “TN” attribute indicates
the AI model successfully predicted a positive classification, the “TN” attribute indicates
that the AI model successfully predicted a negative classification, and the “FP” attribute
that the AI model successfully predicted a negative classification, and the “FP” attribute
indicates
indicatesthat the
that AIAI
the model
modelpredicted
predicteda positive classification
a positive which
classification was
which actually
was negative,
actually negative,
and the “FN” attribute indicates that the AI model predicted a negative classification
and the “FN” attribute indicates that the AI model predicted a negative classification which
was actually positive. Precision is the percentage of “positive” predictions made by the AI
model that are actually true. The precision was calculated as follows (Equation (1)):

true positive (TP)

(Precision) = (1)
true positive (TP) + false positive (FP)
Diagnostics 2024, 14, 11 7 of 19

Recall is the percentage of predictions that are actually true and also have a “positive”
prediction. The recall was calculated as follows (Equation (2)):

true positive (TP)

(Recall) = (2)
true positive (TP) + false negative (FN)

Although precision and recall are both ratios of how well the AI obtains the correct
answer, precision is focused on lowering the value of FP and recall is focused on lowering
the value of FN. According to the PR curve, a good AI model should have high precision
and recall values. In other words, a PR curve is a graph for evaluating the performance
of a model that shows the relationship between the AI’s precision and recall. To show the
correlation between the precision and recall of the YOLO v4 model using a PR curve, the
YOLO v4 model was used to predict the tibia and elbow fracture sites that the YOLO v4
model had been previously trained with.
The IoU serves as a metric that indicates the success of the detection for an individual
object. The derivation of the IoU is as follows (Equation (3)):

Area of Intersection
(intersection over Union) = (3)
Area of Union
The term “Area of Intersection” refers to the region corresponding to the correct
answer, while “Area of Union” pertains to the predicted area. The IoU can achieve a
maximum value of 1, with a higher value signifying a more precise prediction [27].

2.4. Three-Dimensional Image Reconstruction and Plotting Mask

Initially, details such as the width and height, slice thickness, and resolution of the
DICOM image were extracted from the header of the DICOM file [28,29]. Given that the
3D conversion of DICOM images necessitates exclusively retrieving bone-related image
information, a blank matrix was populated with 1 based on the origin coordinate axis,
aligning with the Threshold set derived from the Hounsfield Unit (HU) range of 1150–1250.
Subsequently, the populated matrix was transformed into a representation isolating only
the bone segments from a single DICOM file. Equivalent curves were then derived from
the finalized binary matrix to construct the bone surface.
To envelop the outer epidermis of the bone, composed of dots and lines, a layer of
cotton was applied, with the upper and lower sections capped to fashion a complete bone
structure. For the introduction of bounding boxes onto the 3D reconstructed bones, the
DICOM image’s pixel information in the DICOM header was translated into distances
along the X-axis and Y-axis. Calculating the starting point of the bounding box from the
origin, the bounding box was subsequently mapped from this starting point onto the X-axis,
Y-axis, and thickness, strategically placed within the fracture area.
Figure 4 is a diagram that briefly shows the process of positioning the bounding box
onto the bone reconstructed in 3D. The DICOM header stores pixel information, height
and width of the image, and threshold information of the HU range for the bone. In
order to create a bounding box, pixel information from the DICOM file was converted
into a 3D distance, and the starting position of the bounding box was drawn based on the
coordinate axis. The thickness of the bounding box was constructed using the information
on thickness in the DICOM file. In Figure 4, the red squares are the bounding boxes located
in the fracture area.
Diagnostics 2024, 14, x FOR PEER REVIEW 8 of 19
Diagnostics 2024, 14, 11 8 of 19

Figure 4. Utilizing
Figure both both
4. Utilizing the bounding box’sbox’s
the bounding location andand
location sizesize
data, along
data, with
along thethe
with DICOM
DICOM file’s
file’s header
header information,
information, thethe bounding
bounding boxbox is positioned
is positioned onto
onto thethe
3D 3D reconstructed
reconstructed bone.
bone. By converting
By converting the pixel
the pixel information from the DICOM file’s header into a distance concept based on the
information from the DICOM file’s header into a distance concept based on the 3D reconstructed 3D recon-
structed bone,
bone, the the bounding
bounding box’s
box’s placement
placement involves
involves calculating
calculating its its width,
width, height,
height, and
and thickness
thickness from the
from the origin axis.
origin axis.

3. Results
3. Results
3.1. Performance of Model
3.1. Performance of Model
Figure 5 displays
Figure the outcome
5 displays of the validation
the outcome loss post-training.
of the validation On the Y-axis
loss post-training. On theandY-axis
X-axis, the loss and iteration are depicted, signifying the count of validations for the com-
and X-axis, the loss and iteration are depicted, signifying the count of validations for the
pact datasets.
compact Within
datasets.theWithin
curve,thethecurve,
blue line
theillustrates the fluctuation
blue line illustrates in loss during
the fluctuation train-
in loss during
ing, while thewhile
training, blackthe
line represents
black the actual
line represents the validation loss variation.
actual validation WhenWhen
loss variation. analyzing
analyzing
the loss
the fluctuation basedbased
loss fluctuation on iteration usingusing
on iteration CT images featuring
CT images tibia tibia
featuring region fractures,
region fractures,
the ultimate validation loss amounted to 4.87. Similarly, examining the loss variation after after
the ultimate validation loss amounted to 4.87. Similarly, examining the loss variation
validation basedbased
validation on iteration usingusing
on iteration CT images featuring
CT images elbow
featuring region
elbow fractures
region yielded
fractures a
yielded a
final validation loss of 3.90.
final validation loss of 3.90.
Diagnostics2024,
Diagnostics 2024,14,
14,11
x FOR PEER REVIEW 99 of 19
of 19

Figure 5. Result of validation loss after training. The Y-axis and X-axis show the loss and iteration
Figure 5. Result of validation loss after training. The Y-axis and X-axis show the loss and iteration
which mean the number of validations for the minibatch. The blue line represents the variation in
which mean the number of validations for the minibatch. The blue line represents the variation in
loss
loss for
for training
training and
and the black line
the black line shows
shows the
the variation
variation inin the
theloss
lossin
inthe
theactual
actualvalidation.
validation.(a)(a)Loss
Loss
variation according
variation according to
to iteration
iteration using
using CT
CTimages
imagesincluding
includingfracture
fracturecases
casesinintibia
tibiaregion;
region;the
thefinal
final
validation loss was 4.87.
validation 4.87. (b)
(b)Variation
Variationininthe
theloss
lossafter
aftervalidation
validation according
according to to
iteration using
iteration usingCTCT
images including
images including fracture
fracture cases
cases in
in elbow
elbow region;
region;the
thefinal
finalvalidation
validationloss
losswas
was3.90.
3.90.

Figure 66illustrates
Figure illustratessample
sampleoutcomes
outcomesdepicting
depictingthe the detection
detection ofof fractured
fractured areas
areas utiliz-
utilizing
ing YOLO
YOLO v4 and
v4 and 3D reconstructed
3D reconstructed images.
images. The The left segment
left segment displays
displays the outcomes
the outcomes re-
related
lated to a tibia case, while the right part exhibits the findings concerning an elbow case.
to a tibia case, while the right part exhibits the findings concerning an elbow case. The
The initial
initial imageimage showcases
showcases the representations
the 2D 2D representations of both
of both thethe tibia
tibia andand elbow,
elbow, highlight-
highlighting
ing the fractured regions. The second image showcases the fracture site predictions
the fractured regions. The second image showcases the fracture site predictions mademade
by
YOLO v4. The third image exhibits the 3D reconstructed images derived
by YOLO v4. The third image exhibits the 3D reconstructed images derived from the CT from the CT series,
incorporating the CT the
series, incorporating imageCT from
imagethe firstthe
from illustration. The fourth
first illustration. The image
fourthpresents the 3D
image presents
reconstructed images, images,
the 3D reconstructed featuring a red mask
featuring a redeffectively highlighting
mask effectively the fractured
highlighting regions.
the fractured
regions.
Diagnostics 2024, 14, x FOR PEER REVIEW 10 of 19
Diagnostics 2024, 14, 11 10 of 19

Figure 6. Example results for detection of fractured regions by YOLO v4 and 3D reconstructed images.
Figure 6. Example results for detection of fractured regions by YOLO v4 and 3D reconstructed
The left side shows the results for the tibia case, and the right side shows the results for the elbow
images. The left side shows the results for the tibia case, and the right side shows the results for
case.
the elbow(a) case.
Two (a)
dimensional (2D) images
Two dimensional (2D) for bothfor
images tibia
bothand elbow
tibia andcases
elbowincluding fractured
cases including regions.
frac-
tured regions.
(b) Results of(b) Resultsfor
detection of fractured
detection regions
for fractured
using regions
images inusing images YOLO
(a) through in (a) through
v4. The YOLO
yellow v4.
box is
The
theyellow
boundingbox box
is the
tobounding box
specify the to specify
fractured the fractured
region region
on the image. (c)on the image. (c) Three-dimen-
Three-dimensional reconstructed
sional
imagesreconstructed images
using CT series using CT
including CTseries
image including
in (a). (d)CT image in (a). (d) Three-dimensional
Three-dimensional 3D
3D reconstructed images
reconstructed
including redimages including
mask which can red mask which
effectively showcanthe effectively show the fractured regions.
fractured regions.

InIn
Figure
Figure7, the representative
7, the results
representative are presented
results to demonstrate
are presented the detection
to demonstrate per-
the detection
formance for fractured
performance regions
for fractured in theintibia,
regions including
the tibia, the fibula,
including from from
the fibula, CT images. The en-
CT images. The
tity detection
entity network
detection YOLO
network YOLOv4 delineated
v4 delineated the fracture site site
the fracture on the CT CT
on the images using
images a
using
yellow bounding box. This bounding box not only pinpoints the exact location of the frac-
a yellow bounding box. This bounding box not only pinpoints the exact location of the
tured area but
fractured areaitbut
alsoitprovides a score
also provides based
a score on the
based onprobability of anofaccurate
the probability detection.
an accurate detection.
Diagnostics 2024, 14, x FOR PEER REVIEW 11 of 19
Diagnostics 2024, 14, 11 11 of 19

Figure
Figure7.7.Representative
Representative results to to
results show
showthethe
detection performance
detection for fractured
performance region
for fractured in thein the
region
computed
computedtomography
tomography(CT) (CT)images
imagesforfortibia
tibiaincluding
includingfibula.
fibula.The
Theyellow
yellow bounding
bounding box which
box whichis is
generated
generated byby object
object detection
detection network
network YOLO
YOLO v4 v4 is
is specifying
specifying the fractured region
the fractured region in
in the
the CT
CT images.
im-
ages. The yellow
The yellow bounding
bounding box indicates
box indicates not the
not only only the specific
specific location
location of the of the fractured
fractured regionregion butthe
but also
also the score which is based on the probability for a correct
score which is based on the probability for a correct detection. detection.

Figure
Figure 88 exhibits
exhibitsrepresentative
representative results
results indicating
indicatingthe thedetection
detectionperformance
performancefor forfrac-
frac-
tured
tured regions
regions in
in the
the elbow
elbow (humerus,
(humerus, radius,
radius, and
and ulna)
ulna) inin CT
CT images.
images. TheThe various
various yellow
yellow
bounding
bounding boxes
boxes specify
specify the
the fractured
fractured areas
areas in in the
the elbow
elbow region
region in the CT
in the CT images.
images. The
The
creation of bounding
creation of boundingboxesboxesvaried
variedaccording
according to to
thethe
sizesize of the
of the fracture.
fracture. In instances
In instances of
of com-
comminuted fractures affecting the entire structure, as visible in the CT image,
minuted fractures affecting the entire structure, as visible in the CT image, the bounding the bound-
ing boxes
boxes covered
covered a wider
a wider range,
range, indicating
indicating the the extensive
extensive impact
impact of the
of the fracture.
fracture.
Diagnostics 2024, 14, x FOR PEER REVIEW 12 of 19
Diagnostics 2024, 14, 11 12 of 19

Figure
Figure 8.8. Representative
Representativeresults
resultsto
toshow
showthe
thedetection
detectionperformance
performanceforforthe
thefractured
fracturedregion
regionininthe
the
computed tomography (CT)
computed tomography (CT) images
images of
ofthe
theelbow
elbow(humerus,
(humerus,radius,
radius,and
andulna).
ulna).The
Theyellow
yellowbounding
bound-
ing boxes
boxes are specifying
are specifying the the fractured
fractured region
region of the
of the elbow
elbow in the
in the CT CT images.
images. TheThe relatively
relatively large
large box is
box is specifying the whole bone region in the CT images. In that case, the fracture type
specifying the whole bone region in the CT images. In that case, the fracture type was a comminuted was a
comminuted fracture for the whole structure which is shown in the CT image.
fracture for the whole structure which is shown in the CT image.

3.2.
3.2. Three-Dimensional
Three-Dimensional Image Reconstruction and
Image Reconstruction and Visualization
Visualization of
of Fractured
Fractured Area
Area
Figure
Figure 99 shows
shows representative
representative results
results featuring
featuring 3D
3D reconstructed
reconstructed images
images of
of fractured
fractured
tibia
tibia and
and fibula
fibula bones,
bones, with
with red
red masks
masks indicating
indicating the
the fractured
fractured regions.
regions. These
These images
images were
were
generated from aa CT
generated from CT series,
series, comprising
comprising multiple CT slides.
multiple CT slides. YOLO
YOLO v4 v4 successfully
successfully detected
detected
the fractured regions,
the fractured regions, specified
specified by
by the
the bounding
bounding boxes.
boxes. Although
Although these
these bounding
bounding boxes
boxes
were essentially 2D representations on individual CT images, the 3D depiction (red mask)
were essentially 2D representations on individual CT images, the 3D depiction (red mask)
was achieved
was achieved byby amalgamating
amalgamating multiple
multiple bounding
bounding boxes
boxes from
from various
various CT
CT images
images within
within
a series. By
By stacking
stacking the
the bounding
bounding boxes
boxes in
in multiple
multiple layers,
layers, the
the fracture area could be
intuitively identified.
Diagnostics 2024, 14,2024,
Diagnostics x FOR14,PEER
11 REVIEW 13 of 19 13 of 19

Figure Figure
9. Representative results results
9. Representative (tibia and fibula)
(tibia to showtothe
and fibula) showthree
thedimensional (3D) recon-
three dimensional (3D) reconstructed
structed image
image forfor the
the fractured
fractured bone
bone with
with redred mask
mask specifying
specifying thethe fractured
fractured region.
region. TheThe fractured
fractured bone was
bone was reconstructed as 3D images from the computed tomography (CT) series
reconstructed as 3D images from the computed tomography (CT) series which included many CTwhich included
many CT image
image slides.
slides. Moreover,
Moreover, YOLOYOLO v4 detected
v4 detected the fractured
the fractured region
region andregions
and the the regions
werewerespecified by the
specified by the bounding box. Although the bounding box is basically a
bounding box. Although the bounding box is basically a 2D box on the individual 2D box on the individual
CT image, the
CT image, the 3D expression (red mask) was possible by stacking the bounding boxes from several
3D expression (red mask) was possible by stacking the bounding boxes from several CT images in
CT images in one series.
one series.
FigureFigure
10 illustrates representative
10 illustrates resultsresults
representative relatedrelated
to the elbow (humerus,
to the elbow radius,radius,
(humerus, and and
ulna), ulna),
whichwhich
are 3Darereconstructed images of fractured bones with red masks indicating
3D reconstructed images of fractured bones with red masks indicating the
the specific fractured
specific fractured regions.
regions.Given
Giventhe
theelbow’s
elbow’scomplex
complexstructure
structureinvolving
involvingmultiple
multiple bone
bone fragments, the red mask provides an intuitive representation of the fracturedarea
fragments, the red mask provides an intuitive representation of the fractured areaininthe 3D
the 3Dreconstructed
reconstructed image.
image. All
All regions
regions affected
affected by
by the
thefracture
fracturewere
wereprominently
prominentlymarked
marked with
with the red mask.
the red mask.
Diagnostics 2024, 14, x2024,
Diagnostics FOR14,
PEER
11 REVIEW 14 of 19 14 of 19

Figure 10. Representative

Figure results (elbow;
10. Representative results humerus, radius, and
(elbow; humerus, ulna)
radius, to ulna)
and show to
theshow
threethe
dimen-
three dimensional
sional (3D)
(3D)reconstructed
reconstructedimage
imageforforthe
thefractured
fracturedbone
bonewith
withaaredredmask
maskspecifying
specifying the
the fractured
fractured region. The
region. The
red red
maskmask intuitively
intuitively shows
shows the fractured
the fractured region
region in thein3D
thereconstructed
3D reconstructed
imageimage
of theoffractured
the elbow.
fracturedThe
elbow. Theshow
results results
theshow the multiple
multiple fracturedfractured regions
regions using usingred
several several red masks.
masks.

3.3. Evaluation and Analysis

3.3. Evaluation and Analysis
Figure 11 shows
Figure 11two Precision–Recall
shows (PR) curves,
two Precision–Recall (PR)illustrating the precision
curves, illustrating theand re-
precision and
call of the fractured region detection facilitated by YOLO v4. The Y-axis delineates preci-
recall of the fractured region detection facilitated by YOLO v4. The Y-axis delineates
sion, while the X-axis
precision, whilerepresents
the X-axis recall. In Figure
represents recall. 10a,b, these
In Figure PR these
10a,b, curvesPRwere generated
curves were generated
utilizingutilizing
distinctdistinct
datasets tailored
datasets for the
tailored fortibia andand
the tibia elbow, respectively.
elbow, respectively.Notably,
Notably,these
these curves
revealed average precisions of 0.71 and 0.81 in Figure 10a and 10b, respectively,
curves revealed average precisions of 0.71 and 0.81 in Figure 10a and Figure 10b, respec- providing
a comprehensive overview of the detection accuracy achieved by
tively, providing a comprehensive overview of the detection accuracy achieved by the the YOLO v4 model.
YOLO v4 model.
 registered at 0.6638. The standard deviation of the IoU of the elbow was 0.2709, and 0.2754
for the tibia. Notably, the IoU demonstrated relatively high values despite the limited
availability of learning data. Typically, when the IoU reaches 0.5 or above, the model is
considered to exhibit compliant performance. This is because an IoU surpassing 0.5 im-
Diagnostics 2024, 14, 11 plies effective coverage of the ground truth area, equivalent to two-thirds of the region
15 of 19
[30,31].

Figure 11. Precision–Recall (PR) curves to show the accuracy of detection of fractured regions by
Figure
YOLO11. v4.Precision–Recall (PR)
The Y-axis and the curves
X-axis to showthe
represent theprecision
accuracyand
of detection
the recall, of fractured regions
respectively. The PR by
curves
YOLO v4. The Y-axis and the X-axis represent the precision and the recall, respectively. The PR
in (a) and (b) were acquired by using datasets for the tibia and elbow, respectively. The average
curves in (a) and (b) were acquired by using datasets for the tibia and elbow, respectively. The
precisions were 0.71 and 0.81 from (a) and (b), respectively.
average precisions were 0.71 and 0.81 from (a) and (b), respectively.
In the case of the tibia, the IoU yielded a value of 0.6327, while for the elbow, the IoU
registered at 0.6638. The standard deviation of the IoU of the elbow was 0.2709, and 0.2754
for the tibia. Notably, the IoU demonstrated relatively high values despite the limited
availability of learning data. Typically, when the IoU reaches 0.5 or above, the model is
considered to exhibit compliant performance. This is because an IoU surpassing 0.5 implies
effective coverage of the ground truth area, equivalent to two-thirds of the region [30,31].

4. Discussion
In this study, we demonstrated the accurate visualization of fractured regions using
deep learning and 3D reconstructed images for the fractured bone from CT images. In
Diagnostics 2024, 14, 11 16 of 19

other words, the purpose of this study was to evaluate the performance of the YOLO
v4-based fracture detection method, and to show the possibility of an intuitive diagnosis by
combining 3D image reconstruction and fracture detection [32]. The possibility of finding
the fractured region from the 2D/3D CT images can be decreased according to internal
or external conditions. In particular, there are several limitations in 2D images, such as
low contrast, a lot of noise, etc. In the case of 3D reconstructed CT images, although
most cases provide better visualization than the 2D images, there are still difficulties in
detecting the fractured regions due to the following reasons: over-processing to smooth
mesh, occluded objects, unclear fracture lines, etc. For these reasons, there are always
factors affecting the accurate diagnosis of orthopedic trauma from 2D/3D images. In
this study, the methodology to generate a 3D reconstructed bone image with automatic
detection of the fractured regions which can be easy to miss (such as Figures 8 and 9) was
suggested to make up for the weakness of the conventional method for the diagnosis of
orthopedic trauma. Additionally, it is imperative to conduct more nuanced comparative
studies to determine the necessity of utilizing YOLO v4. Despite the proliferation of YOLO
model versions, YOLO v4 has demonstrated notable efficacy and accuracy even in resource-
constrained environments. A noteworthy observation is that YOLO v4 not only exhibited
compliant performance but also has credibility due to the availability of relevant research
papers. When opting for a model, careful consideration should be given to selecting the
most suitable one based on its accuracy and the specific context of its application. The
accompanying table provides a reference for the requisite conditions related to performance
and utilization [33].
We explored the latest methodologies for diagnosing fractures, uncovering a study that
employed the YOLOv4 model to detect and classify hip fracture types in X-ray images [34].
The findings of this study indicated that YOLOv4 achieved an accuracy of 95%, sensitivity
of 96.2%, and surgeon sensitivity ranging from 69.2% to 96.2%. YOLOv4 exhibited a
fracture prediction performance comparable to or better than that of a first-year resident
when compared to a surgeon. However, this investigation was limited to a 2D environment
as it utilized X-rays. In contrast, our study operated within a 3D environment, offering
intuitive fracture extent markings for surgeons. Additionally, we came across a recent study
discussing the paradigm shift in fracture detection methods through Deep Supervised
Learning [35]. The study noted a significant increase in fracture detection studies utilizing
artificial intelligence, with nearly 5000 conducted from 2017 to the present, marking a
substantial shift from the pre-2017 period with fewer than 1000 studies. Notably, the
YOLOv4 model we employed incorporates DenseNet, a feature found in only 5% of recent
studies. Given DenseNet’s superior deep network performance, our model stands out from
the remaining 95%. Another noteworthy discovery was a study comparing various papers
focused on bone fracture detection using AI models [36]. After reviewing around 40 reliable
papers, it became evident that these studies were predominantly detecting fracture sites in
2D environments, with a lack of emphasis on 3D visualization. The majority of the existing
studies predominantly focused on detecting fractures in a 2D environment. In contrast,
our approach stands out by automatically identifying the fracture site and providing a 3D
visualization.
Although the aspects examined for the diagnosis of orthopedic trauma can vary
according to the surgeons’ experiences and skill level, orthopedic surgeons can obtain assis-
tance for a quick and clear diagnosis of orthopedic trauma regardless of their experience or
skill level, by using the proposed combination of 3D reconstructed images and automatic
detection of the fractured regions. Moreover, the biggest benefit of the proposed system is
that it could reduce human errors, and misdiagnoses can be decreased in the clinic [37,38].
Clearly, the deep learning used in this study did not need to be trained with a lot of data.
Nevertheless, the results showed the relatively good performance with an over 0.60 average
precision, as seen in the PR curve in Figure 10. Naturally, the performance can be increased
by training using additional data. In Figures 8 and 9, some cases showed red masks located
Diagnostics 2024, 14, 11 17 of 19

at a normal region without a fracture. This kind of error can be also dramatically decreased
by additional training with more data.
This study has clear limitations in that the deep learning roughly detected the fractured
region, not the individual fracture fragments. Because this study is a basic level study to
show the feasibility of intuitive diagnosis support for orthopedic trauma, the use of YOLO
v4 with hardware limitations and insufficient data are also other limitations of this study.
Nevertheless, we found a reason why we used YOLO v4 from Table 1. YOLO v5 and v8
were not validated by the peer review paper, while the origin of YOLO v6 was YOLO
v5. YOLO v7 is difficult to use in hospitals because the model requires high hardware
performance. For these reasons, there are a lot of studies that have focused on YOLO
v4-based object detection for medical images. For example, a research endeavor employed
YOLO v4 to proficiently detect lung nodules [39]. The findings indicated the effectiveness of
YOLO v4 within a hospital hardware environment. In one of the more recent investigations,
object recognition using YOLO v4 was scrutinized, revealing substantial enhancements in
both detection speed and accuracy. In the future, the upgraded system for the diagnosis
support for orthopedic trauma can be developed with a better deep learning model and a
larger dataset to detect even individual fracture fragments [40–43].

Table 1. Comprehensive comparison table of YOLO series. Table 1 is a comparison table summarizing
the YOLO v1 to v8 models’ publication date, average accuracy, whether it is a paper, and the minimum
GPU performance required to run the model.

Average Precision
Version Date Paper Minimum Specification GPU Performance
(%)
YOLOv1 2016 YES 63.4 Very High
YOLOv2 2017 YES 63.4 Very High
YOLOv3 2018 YES 36.2 Medium
YOLOv4 2020 YES 43.5 Low
YOLOv5 2020 NO 55.8 Low
YOLOv6 2022 YES 52.5 Very High
YOLOv7 2022 YES 56.8 Medium
YOLOv8 2023 NO 53.9 Medium

5. Conclusions
We confirmed the feasibility and the performance of the combination between YOLO
v4-based fracture detection with intuitive visualization and 3D reconstructed bone images
from the CT images. If the orthopedic surgeons use this AI-based 3D diagnosis support
system for orthopedic trauma in the clinic, the surgeons can conduct easy and fast trauma
diagnoses with a high accuracy regardless of their experience and skill level.
Moving forward, our strategy involves leveraging additional CT data to enhance the
sophistication of our AI model. The plan is to broaden the scope of training beyond the
tibia and elbow, encompassing more extensive anatomical regions. Looking ahead, our
future endeavors include subjecting the implemented model to clinical trials to validate its
reliability and usability in 3D visualization.

Author Contributions: In this study, D.-K.Y., Y.-D.J. and M.-J.K. designed the method and drafted the
manuscript. D.-K.Y. and H.-Y.C. wrote the code. S.-H.S., Y.-G.C. and D.-K.Y. acquired the approval
from the IRB and collected the data, respectively, M.-J.K. and Y.-D.J. processed the dataset including
making the annotations for all the data. H.-J.K., J.-Y.Y. and M.-S.K. evaluated the results of the
annotations. H.-Y.C. and Y.-D.J. trained the models and analyzed them. S.-U.K. and Y.-D.J. performed
the test for clinical support ability. Y.-D.J., D.-K.Y. and S.-U.K. coordinated and supervised the whole
work. All authors have read and agreed to the published version of the manuscript.
Funding: This work was supported by the Program of Support for R&D of companies specializing in
AI in Seoul, R&D Support Center, Seoul Business Agency (grant number: CY220040).
Diagnostics 2024, 14, 11 18 of 19

Institutional Review Board Statement: All methods in this study were performed in accordance
with the relevant guidelines and regulations of the Clinical Trial System of the Catholic Medical
Center (CTSC) in the Catholic University of Korea. All experimental protocols were approved by the
Institutional Review Board (IRB) at the Seoul St. Mary’s Hospital, the Catholic University of Korea
(approval number: KC20RISI1034).
Informed Consent Statement: Informed consent was obtained from all subjects (and/or their legal
guardians) involved in this study.
Data Availability Statement: We have deposited the sample code we authored on GitHub (https:
//github.com/Louis-Youn/Code_Storage.git) to facilitate exploration by the wider community. The
sample code shared via GitHub mirrors the structure of our original model. Because the full datasets
are still protected by privacy issues and regulation policies, the training model can be acquired by
contacting to corresponding author (D.-K.Y., E-mail: [email protected]).
Conflicts of Interest: Authors H.-Y.C., M.-S.K., J.-Y.Y., H.-J.K. and D.-K.Y. were employed by the
company KAVILAB Co. Ltd. The remaining authors declare that the research was conducted in the
absence of any commercial or financial relationships that could be construed as a potential conflict of
interest.

References
1. Hsieh, C.I.; Zheng, K.; Lin, C.; Mei, L.; Lu, L.; Li, W.; Kuo, C.F. Automated bone mineral density prediction and fracture risk
assessment using plain radiographs via deep learning. Nat. Commun. 2021, 12, 5472. [CrossRef] [PubMed]
2. Kuo, R.Y.; Harrison, C.; Curran, T.A.; Jones, B.; Freethy, A.; Cussons, D.; Stewart, M.; Collins, G.S.; Furniss, D. Artificial intelligence
in fracture detection: A systematic review and meta-analysis. Radiology 2022, 304, 50–62. [CrossRef] [PubMed]
3. Kolanu, N.; Brown, A.S.; Beech, A.; Center, J.R.; White, C.P. Natural language processing of radiology reports for the identification
of patients with fracture. Arch. Osteoporos. 2021, 16, 6. [CrossRef] [PubMed]
4. Raisuddin, A.M.; Vaattovaara, E.; Nevalainen, M.; Nikki, M.; Järvenpää, E.; Makkonen, K.; Pinola, P.; Palsio, T.; Niemensivu, A.;
Tervonen, O.; et al. Critical evaluation of deep neural networks for wrist fracture detection. Sci. Rep. 2021, 11, 6006. [CrossRef]
[PubMed]
5. Juan, R.; Diana, M. A comprehensive review of YOLO: From YOLOv1 and beyond. arXiv 2023, arXiv:2304.00501.
6. Durve, M.; Orsini, S.; Tiribocchi, A.; Montessori, A.; Tucny, J.M.; Lauricella, M.; Camposeo, A.; Pisignano, D.; Succi, S. Benchmark-
ing YOLOv5 and YOLOv7 models with DeepSORT for droplet tracking applications. Eur. Phys. J. E 2023, 46, 32. [CrossRef]
7. Ma, W.; Yu, H.; Fang, W.; Guan, F.; Ma, D.; Guo, Y.; Zhang, Z.; Wang, C. Crop Disease Detection against Complex Background
Based on Improved Atrous Spatial Pyramid Pooling. Electronics 2023, 12, 216. [CrossRef]
8. Liu, S.; Qi, L.; Qin, H.; Shi, J.; Jia, J. Path aggregation network for instance segmentation. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018.
9. Wang, C.-Y.; Bochkovskiy, A.; Liao, H.Y.M. YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object
detectors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada,
18–22 June 2023.
10. Zhang, L.; Zhang, C.; Zhao, J.; Guan, J.; Zhou, S. Meta-ZSDETR: Zero-shot DETR with Meta-learning. In Proceedings of the
IEEE/CVF International Conference on Computer Vision, Paris, France, 30 September–6 October 2023; pp. 6845–6854.
11. Mei, S.; Jiang, H.; Ma, L. YOLO-lung: A practical detector based on imporved YOLOv4 for Pulmonary Nodule Detection. In
Proceedings of the International Congress on Image and Signal Processing. BioMedical Engineering and Informatics, Shanghai,
China, 23–25 October 2021; pp. 1–6.
12. Wu, H.; Wang, Y.; Zhao, P.; Qian, M. Small-target weed-detection model based on YOLO-V4 with improved backbone and neck
structures. Precis. Agric. 2023, 24, 2149–2170. [CrossRef]
13. Bochkovskiy, A.; Wang, C.-Y.; Liao, H.-Y.M. Yolov4: Optimal speed and accuracy of object detection. arXiv 2020, arXiv:2004.10934.
14. Fishman, E.K.; Kuszyk, B. 3D imaging: Musculoskeletal applications. In 3D Imaging in Medicine, 2nd ed.; CRC Press: Boca Raton,
FL, USA, 2023; pp. 301–327.
15. Lavdas, E.; Papaioannou, M.; Papanikolaou, P.; Papageorgiou, E.; Sakkas, G.; Tsikrika, A.; Tsagkalis, A.; Pappas, E.; Michail, C.;
Roka, V.; et al. Visualization of meniscus with 3D axial reconstructions. J. Med. Imaging Radiat. Sci. 2021, 52, 519–526. [CrossRef]
16. Wang, C.; Zhu, Y.; Long, H.; Lin, Z.; Zhao, R.; Sun, B.; Zhao, S.; Cheng, L. Three-dimensional mapping of distal humerus fracture.
J. Orthop. Surg. Res. 2021, 16, 545. [CrossRef] [PubMed]
17. Shardt, Y.A.W. Introduction to MATLAB® ; Springer Nature: Cham, Switzerland, 2023; pp. 1–7.
18. Padilla, R.; Netto, S.L.; Da Silva, E.A. A survey on performance metrics for object-detection algorithms. In Proceedings of the
2020 International Conference on Systems, Signals and Image Processing (IWSSIP), Niteroi, Brazil, 1–3 July 2020; pp. 237–242.
19. Arias-Serrano, I.; Velásquez-López, P.A.; Avila-Briones, L.N.; Laurido-Mora, F.C.; Villalba-Meneses, F.; Tirado-Espin, A.;
Cruz-Varela, J.; Almeida-Galárraga, D. Artificial Intelligence based glaucoma and diabetic retinopathy detection using
MATLAB—Retrained AlexNet convolutional neural network. F1000Research 2023, 12, 14. [CrossRef]
Diagnostics 2024, 14, 11 19 of 19

20. Wang, C.Y.; Liao, H.Y.M.; Wu, Y.H.; Chen, P.Y.; Hsieh, J.W.; Yeh, I.H. CSPNet: A new backbone that can enhance learning
capability of CNN. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle,
WA, USA, 14–19 June 2020.
21. Shen, D.; Wu, G.; Suk, H.-I. Deep learning in medical image analysis. Annu. Rev. Biomed. Eng. 2017, 19, 221–248. [CrossRef]
[PubMed]
22. Zhu, Y.X.; Li, Y.; Wu, S.T.; Liu, W.D.; Song, R.R.; Li, W.; Wang, Y. Application of ROC and PR curves in the evaluation of clinical
diagnostic testing. Zhonghua Yu Fang Yi Xue Za Zhi [Chin. J. Prev. Med.] 2022, 56, 1341–1347. [PubMed]
23. Parra-Cabrera, G.; Pérez-Cano, F.D.; Jiménez-Delgado, J.J. Fracture pattern projection on 3D bone models as support for bone
fracture simulations. Comput. Methods Programs Biomed. 2022, 224, 106980. [CrossRef] [PubMed]
24. Kaushik, V.; Singh, G.; Jain, P. Pothole Detection System: A Real-Time Solution for Detecting Potholes. In Proceedings of the 2023
3rd Asian Conference on Innovation in Technology (ASIANCON), Ravet IN, India, 25–27 August 2023; pp. 1–6.
25. Shadid, W.G.; Willis, A. Bone fragment segmentation from 3D CT imagery. Comput. Med. Imaging Graph. 2018, 66, 14–27.
[CrossRef] [PubMed]
26. Redmon, J.; Divvala, S.; Girshick, R.; Farhadi, A. You only look once: Unified, real-time object detection. In Proceedings of the
IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 26 June–1 July 2016.
27. Radke, K.L.; Kors, M.; Müller-Lutz, A.; Frenken, M.; Wilms, L.M.; Baraliakos, X.; Wittsack, H.J.; Distler, J.H.; Abrar, D.B.;
Antoch, G.; et al. Adaptive IoU Thresholding for Improving Small Object Detection: A Proof-of-Concept Study of Hand Erosions
Classification of Patients with Rheumatic Arthritis on X-ray Images. Diagnostics 2023, 13, 104. [CrossRef]
28. Mukherjee, S.; Korfiatis, P.; Khasawneh, H.; Rajamohan, N.; Patra, A.; Suman, G.; Singh, A.; Thakkar, J.; Patnam, N.G.; Trivedi,
K.H.; et al. Bounding box-based 3D AI model for user-guided volumetric segmentation of pancreatic ductal adenocarcinoma on
standard-of-care CTs. Pancreatology 2023, 23, 522–529. [CrossRef]
29. Anitha, A.; Shivakumara, P.; Jain, S.; Agarwal, V. Convolution Neural Network and Auto-Encoder Hybrid Scheme for Automatic
Colorization of Grayscale Images; Springer International Publishing: Cham, Switzerland, 2023; pp. 253–271.
30. Caron, R.; Londono, I.; Seoud, L.; Villemure, I. Segmentation of trabecular bone microdamage in Xray microCT images using a
two-step deep learning method. J. Mech. Behav. Biomed. Mater. 2023, 137, 105540. [CrossRef]
31. AboElenein, N.M.; Piao, S.; Zhang, Z. Encoder–Decoder Network with Depthwise Atrous Spatial Pyramid Pooling for Automatic
Brain Tumor Segmentation. Neural Process. Lett. 2023, 55, 1697–1713. [CrossRef]
32. Iyoho, A.E.; Young, J.M.; Volman, V.; Shelley, D.A.; Ng, L.J.; Wang, H. 3D Tibia Reconstruction Using 2D Computed Tomography
Images. Mil. Med. 2019, 184 (Suppl. S1), 621–626. [CrossRef]
33. Shuo, W. Research Towards Yolo-Series Algorithms: Comparison and Analysis of Object Detection Models for Real-Time UAV
Applications. J. Phys. Conf. Ser. 2021, 1, 012021.
34. Twinprai, N.; Boonrod, A.; Boonrod, A.; Chindaprasirt, J.; Sirithanaphol, W.; Chindaprasirt, P.; Twinprai, P. Artificial intelligence
(AI) vs. human in hip fracture detection. Heliyon 2022, 8, e11266. [CrossRef]
35. Meena, T.; Roy, S. Bone fracture detection using deep supervised learning from radiological images: A paradigm shift. Diagnostics
2022, 12, 2420. [CrossRef]
36. Su, Z.; Adam, A.; Nasrudin, M.F.; Ayob, M.; Punganan, G. Skeletal Fracture Detection with Deep Learning: A Comprehensive
Review. Diagnostics 2023, 13, 3245. [CrossRef]
37. Read, G.J.; Shorrock, S.; Walker, G.H.; Salmon, P.M. State of science: Evolving perspectives on ‘human error’. Ergonomics 2021, 64,
1091–1114. [CrossRef]
38. Gawron, V.J.; Drury, C.G.; Fairbanks, R.J.; Berger, R.C. Medical error and human factors engineering: Where are we now? Am. J.
Med. Qual. 2006, 21, 57–67. [CrossRef]
39. Wu, D.; Lu, T.; Li, X. An improved attention mechanism based YOLOv4 structure for lung nodule detection. In Proceed-
ings of the 2022 IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence
and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress
(DASC/PiCom/CBDCom/CyberSciTech), Falerna, Italy, 12–15 September 2022; pp. 1–6.
40. Rizwan. YOLOv7 and YOLOv5 Comparison on Embedded Devices and Computer Systems. 2022. Available online: https:
//www.plugger.ai/blog/yolov7-and-yolov5-comparison-on-embedded-devices-and-computer-systems (accessed on 10 October
2023).
41. Byrne, M.F.; Parsa, N.; Greenhill, A.T.; Chahal, D.; Ahmad, O.; Bagci, U. (Eds.) AI in Orthopaedic Surgery. In AI in Clinical
Medicine: A Practical Guide for Healthcare Professionals; John Wiley & Sons: Hoboken, NJ, USA, 2023; pp. 266–281.
42. Miyagami, T.; Watari, T.; Harada, T.; Naito, T. Medical Malpractice and Diagnostic Errors in Japanese Emergency Departments.
Am. J. Med. Qual. 2023, 24, 340. [CrossRef]
43. Chappard, D. La microarchitecture du tissu osseux. Bull. L’académie Natl. Méd. 2010, 194, 1469–1481. [CrossRef]

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual
author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to
people or property resulting from any ideas, methods, instructions or products referred to in the content.