2688 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 39, NO.

8, AUGUST 2020

Deep Learning COVID-19 Features on CXR

Using Limited Training Data Sets
Yujin Oh , Sangjoon Park, and Jong Chul Ye , Fellow, IEEE

Abstract — Under the global pandemic of COVID-19, diagnosis and assessment of disease progression. Chest com-
the use of artificial intelligence to analyze chest X-ray (CXR) puted tomography (CT) screening on initial patient presenta-
image for COVID-19 diagnosis and patient triage is becom- tion showed outperforming sensitivity to RT-PCR [2] and even
ing important. Unfortunately, due to the emergent nature of
the COVID-19 pandemic, a systematic collection of CXR data confirmed COVID-19 infection on negative or weakly-positive
set for deep neural network training is difficult. To address RT-PCR cases [1]. Accordingly, recent COVID-19 radiological
this problem, here we propose a patch-based convolutional literature primarily focused on CT findings [2], [3]. However,
neural network approach with a relatively small number of as the prevalence of COVID-19 increases, the routine use
trainable parameters for COVID-19 diagnosis. The proposed of CT places a huge burden on radiology departments and
method is inspired by our statistical analysis of the potential
imaging biomarkers of the CXR radiographs. Experimen- potential infection of the CT suites; so the need to recognize
tal results show that our method achieves state-of-the-art COVID-19 features on chest X-ray (CXR) is increasing.
performance and provides clinically interpretable saliency Common chest X-ray findings reflect those described by
maps, which are useful for COVID-19 diagnosis and patient CT such as bilateral, peripheral consolidation and/or ground
triage. glass opacities [2], [3]. Specifically, Wong et al. [4] described
Index Terms — COVID-19, chest X-ray, deep learning, seg- frequent chest X-ray (CXR) appearances on COVID-19.
mentation, classification, saliency map. Unfortunately, it is reported that chest X-ray findings have
a lower sensitivity than initial RT-PCR testing (69% versus
I. I NTRODUCTION 91%, respectively) [4]. Despite this low sensitivity, CXR
abnormalities were detectable in 9% of patients whose initial
C ORONAVIRUS disease 2019 (COVID-19), caused by
severe acute respiratory syndrome coronavirus 2 (SARS-
CoV-2), has become global pandemic in less than four months
RT-PCR was negative.
As the COVID-19 pandemic threatens to overwhelm health-
care systems worldwide, CXR may be considered as a tool
since it was first reported, reaching a 3.3 million confirmed
for identifying COVID-19 if the diagnostic performance with
cases and 238,000 death as of May 2nd, 2020. Due to its
CXR is improved. Even if CXR cannot completely replace
highly contagious nature and lack of appropriate treatment and
the RT-PCR, the indication of pneumonia is a clinical mani-
vaccines, early detection of COVID-19 becomes increasingly
festation of patient at higher risk requiring hospitalization, so
important to prevent further spreading and to flatten the curve
CXR can be used for patient triage, determining the priority
for proper allocation of limited medical resources.
of patients’ treatments to help saturated healthcare system in
Currently, reverse transcription polymerase chain reaction
the pandemic situation. This is especially important, since
(RT-PCR), which detects viral nucleic acid, is the golden
the most frequent known etiology of community acquired
standard for COVID-19 diagnosis, but RT-PCR results using
pneumonia is bacterial infection in general [5]. By excluding
nasopharyngeal and throat swabs can be affected by sampling
these population by triage, limited medical resource can be
errors and low viral load [1]. Antigen tests may be fast, but
spared substantially.
have poor sensitivity.
Accordingly, deep learning (DL) approaches on chest X-ray
Since most COVID-19 infected patients were diagnosed
for COVID-19 classification have been actively explored [6]–
with pneumonia, radiological examinations may be useful for
[12]. Especially, Wang and Wong [6] proposed an open source
Manuscript received April 23, 2020; revised May 2, 2020; accepted deep convolutional neural network platform called COVID-
May 5, 2020. Date of publication May 8, 2020; date of current version Net that is tailored for the detection of COVID-19 cases
July 30, 2020. This work was supported by the National Research
Foundation of Korea under Grant NRF-2020R1A2B5B03001980. (Yujin from chest radiography images. They claimed that COVID-
Oh and Sangjoon Park are co-first authors.) (Corresponding author: Net can achieve good sensitivity for COVID-19 cases with
Jong Chul Ye.) 80% sensitivity.
The authors are with the Department of Bio and Brain Engi-
neering, Korea Advanced Institute of Science and Technology Inspired by this early success, in this paper we aim to further
(KAIST), Daejeon 34141, South Korea (e-mail: [email protected]; investigate deep convolutional neural network and evaluate
[email protected]; [email protected]). its feasibility for COVID-19 diagnosis. Unfortunately, under
Color versions of one or more of the figures in this article are available
online at http://ieeexplore.ieee.org. the current public health emergency, it is difficult to collect
Digital Object Identifier 10.1109/TMI.2020.2993291 large set of well-curated data for training neural networks.

© IEEE 2020. This article is free to access and download, along with rights for full text and data mining, re-use and analysis

Authorized licensed use limited to: IEEE Xplore. Downloaded on October 30,2022 at 17:38:56 UTC from IEEE Xplore. Restrictions apply.
OH et al.: DL COVID-19 FEATURES ON CXR USING LIMITED TRAINING DATA SETS 2689

Fig. 1. Overall architecture of the proposed neural network approach: (a) Segmentation network, and (b) Classification network.

Therefore, one of the main focuses of this paper is to develop Furthermore, by extending the idea of the gradient-weighted
a neural network architecture that is suitable for training with class activation map (Grad-CAM) [14], yet another important
limited training data set, which can still produce radiologically contribution of this paper is a novel probabilistic Grad-CAM
interpretable results. Since most frequently observed distribu- that takes into account of patch-wise disease probability in
tion patterns of COVID-19 in CXR are bilateral involvement, generating global saliency map. The resulting class activation
peripheral distribution and ground-glass opacification (GGO) map clearly show the interpretable results that are well corre-
[13], a properly designed neural network should reflect such lated with radiological findings.
radiological findings.
To achieve this goal, we first investigate several imag- II. P ROPOSED N ETWORK A RCHITECTURE
ing biomarkers that are often used in CXR analysis, such The overall algorithmic framework is given in Fig. 1. The
as lung area intensity distribution, the cardio-thoracic ratio, CXR images are first pre-processed for data normalization,
etc. Our analysis found that there are statistically significant after which the pre-processed data are fed into a segmentation
differences in the patch-wise intensity distribution, which is network, from which lung areas can be extracted as shown
well-correlated with the radiological findings of the local- in Fig. 1(a). From the segmented lung area, classification
ized intensity variations in COVID-19 CXR. This findings network is used to classify the corresponding diseases using
lead us to propose a novel patch-based deep neural network a patch-by-patch training and inference, after which the final
architecture with random patch cropping, from which the decision is made based on the majority voting as shown in
final classification result are obtained by majority voting Fig. 1(b). Additionally, a probabilistic Grad-CAM saliency
from inference results at multiple patch locations. One of the map is calculated to provide an interpretable result. In the
important advantages of the proposed method is that due to following, each network is described in detail.
the patch training the network complexity is relative small
and multiple patches in each image can be used to augment
training data set, so that even with the limited data set the A. Segmentation Network
neural network can be trained efficiently without overfitting. Our segmentation network aims to extract lung and heart
By combining with our novel preprocessing step to normalize contour from the chest radiography images. We adopted an
the data heterogeneities and bias, we demonstrate that the extended fully convolutional (FC)-DenseNet103 to perform
proposed network architecture provides better sensitivity and semantic segmentation [15]. The training objective is
interpretability, compared to the existing COVID-Net [6] with
argmin L() (1)
the same data set. 

Authorized licensed use limited to: IEEE Xplore. Downloaded on October 30,2022 at 17:38:56 UTC from IEEE Xplore. Restrictions apply.
2690 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 39, NO. 8, AUGUST 2020

where L() is the cross entropy loss of multi-categorical a classification network. Classification network were imple-
semantic segmentation and  denotes the network parameter mented in two different versions: global and local approaches.
set, which is composed of filter kernel weights and biases. In the global approach, the masked images were resized to
Specifically, L() is defined as 224 × 224, which were fed into the network. This approach is
 focusing on the global appearance of the CXR data, and was
L() = − λs 1(y j = s)log( p (x j )) (2) used as a baseline network for comparison. In fact, many of
s j
the existing researches employs similar procedure [6]–[9].
where 1(·) is the indicator function, p (x j ) denotes the In the local patch-based approach, which is our proposed
softmax probability of the j -th pixel in a CXR image x, and y j method, the masked images were cropped randomly with a size
denotes the corresponding ground truth label. s denotes class of 224 × 224, and resulting patches were used as the network
category, i.e., s ∈ {background, heart, left lung, right lung}. inputs as shown in Fig. 1(b). In contrast to the global approach,
λs denotes weights given to each class category. various CXR images are resized to a much bigger 1024×1024
CXR images from different dataset resources may induce image for our classification network to reflect the original pixel
heterogeneity in their bits depth, compression type, image distribution better. Therefore, the segmentation mask from
size, acquisition condition, scanning protocol, postprocessing, Fig. 1(a) are upsampled to match the 1024 × 1024 image size.
etc. Therefore, we develop a universal preprocessing step To avoid cropping the patch from the empty area of the masked
for data normalization to ensure uniform intensity histogram image, the centers of patches were randomly selected within
throughout the entire dataset. The detailed preprocessing steps the lung areas. During the inference, K -number of patches
are as follows: were randomly acquired for each image to represent the entire
1) Data type casting (from uint8/uint16 to float32) attribute of the whole image. The number K was chosen to
2) Histogram equalization (gray level = [0, 255.0]) sufficiently cover all lung pixels multiple times. Then, each
3) Gamma correction (γ = 0.5) patch was fed into the network to generate network output,
4) Image resize (height, width = [256, 256]) and among K network output the final decision was made
Using the preprocessed data, we trained FC-DenseNet103 based on majority voting, i.e. the most frequently declared
[15] as our backbone segmentation network architecture. class were regarded as final output as depicted in Fig. 1(b). In
Network parameters were initialized by random distribution. this experiments, the number of random patches K was set to
We applied Adam optimizer [16] with an initial learning 100, which means that 100 patches were generated randomly
rate of 0.0001. Whenever training loss did not improve by from one whole image for majority voting.
certain criterion, the learning rate was reduced by factor For network training, pre-trained parameters from Ima-
10. We adopted early stopping strategy based on validation geNet are used for network weight initialization, after which
performance. Batch size was optimized to 2. We implemented the network was trained using the CXR data. As for opti-
the network using PyTorch library [17]. mization algorithm, Adam optimizer [16] with learning rate
of 0.00001 was applied. The network were trained for
100 epochs, but we adopted early stopping strategy based
B. Classification Network on validation performance metrics. The batch size of 16 was
The classification network aims to classify the chest X-ray used. We applied weight decay and L 1 regularization to
images according to the types of disease. We adopted the prevent overfitting problem. The classification network was
relatively simple ResNet-18 as the backbone of our classi- also implemented by Pytorch library.
fication algorithm for two reasons. The first is to prevent from
overfitting, since it is known that overfitting can occur when
using an overly complex model for small number of data. C. Probabilistic Grad-CAM Saliency Map Visualization
Secondly, we intended to do transfer learning with pre-trained
weights from ImageNet to compensate for the small training We investigate the interpretability of our approach by visu-
data set. We found that these strategy make the training stable alizing a saliency map. One of the most widely used saliency
even when the dataset size is small. map visualization methods is so-called gradient weighted class
The labels were divided into four classes: normal, bac- activation map (Grad-CAM) [14]. Specifically, the Grad-CAM
terial pneumonia, tuberculosis (TB), and viral pneumonia saliency map of the class c for a given input image x ∈ Rm×n
which includes the pneumonia caused by COVID-19 infec- is defined by
tion. We assigned the same class for viral pneumonia from
  
other viruses (e.g. SARS-cov or MERS-cov) with COVID-19, 
since it is reported that they have similar radiologic features l (x) = U P σ
c
αkc f (x)
k
∈ Rm×n (3)
even challenging for the experienced radiologists [18]. Rather, k
we concentrated on more feasible work such as distinguishing
bacterial pneumonia or TB from viral pneumonia, which show where f k (x) ∈ Ru×v is the k-th feature channel at the last
considerable differences in the radiologic features and are still convolution layer (which corresponds to the layer 4 of ResNet-
useful for patient triage. 18 in our case), U P(·) denotes the upsampling operator from a
The pre-processed images were first masked with the lung u×v feature map to the m×n image, σ (·) is the rectified linear
masks from the segmentation networks, which are then fed into unit (ReLU) [14]. Here, αkc is the feature weighted parameter

Authorized licensed use limited to: IEEE Xplore. Downloaded on October 30,2022 at 17:38:56 UTC from IEEE Xplore. Restrictions apply.
OH et al.: DL COVID-19 FEATURES ON CXR USING LIMITED TRAINING DATA SETS 2691

for the class c, which can be obtained by TABLE I

S EGMENTATION D ATASET R ESOURCES
1  ∂yc
uv
αkc = (4)
Z
i=1
∂ fik

for some scaling parameter Z , where yc is the score for the

class c before the softmax layer and f ik denotes the i -th pixel
value of f k (x). The Grad-CAM map l c is then normalized to
have value in [0, 1]. The Grad-CAM for the global approach
is used as a baseline saliency map.
TABLE II
However, care should be taken in applying Grad-CAM to
C LASSIFICATION D ATA S ET R ESOURCES
our local patch-based approach, since each patch has different
score for the COVID-19 class. Therefore, to obtain the global
saliency map, patch-wise Grad-CAM saliency maps should
be weighted with the estimated probability of the class, and
their average value should be computed. More specifically,
our probabilistic Grad-CAM with respect to the input image
x ∈ Rm×n has the following value at the i -th pixel location:
 K
  1  c 
l prob (x) =
c
r (x k )Qk l c (x k ) (5)
i Ki
k=1 i
TABLE III
where x k ∈ R p×q denotes the k-th input patch, Qk : R p×q → D ISEASE C LASS S UMMARY OF THE D ATA S ET
Rm×n refers to the operator that copies the p × q-size k-th
patch into the appropriate location of a zero-padded image in
Rm×n , and l c (x k ) ∈ R p×q denotes the Grad-CAM computed
by (3) with respect to the input patch x k ∈ R p×q , and K i
denotes the number of the frequency of the i -th pixel in the
total K patches. Additionally, the class probability r c (x k )
for the k-th patch can be readily calculated after the soft-
max layer. Accordingly, the average probability of each pixel 2) Classification Dataset: The dataset resources for the clas-
belonging to a given class can be taken into consideration in sification network are described in Table II. Specifically, for
Eq. (5) when constructing a global saliency map. normal cases, the JSRT dataset and the NLM dataset from the
segmentation validation dataset were included. For comparing
III. M ETHOD COVID-19 from normal and different lung diseases, data were
also collected from different sources [22], [23], including addi-
A. Dataset tional normal cases. These datasets were selected because they
We used public CXR datasets, whose characteristics are are fully accessible to any research group, and they provide the
summarized in Table I and Table II. In particular, the data in labels with detailed diagnosis of disease. This enables more
Table I are used for training and validation of the segmenta- specific classification of pneumonia into bacterial and viral
tion networks, since the ground-truth segmentation masks are pneumonia, which should be classified separately because of
available. The curated data in Table II are from some of the their distinct clinical and radiologic differences.
data in Table I as well as other COVID-19 resources, which In the collected data from the public dataset [22], over
were used for training, validation, and test for the classifi- 80% was pediatric CXR from Guangzhou Women and Chil-
cation network. More detailed descriptions of the dataset are dren’s Medical Center [24]. Therefore, to avoid the network
follows. from learning biased features from age-related characteristics,
1) Segmentation Network Dataset: The JSRT dataset was we excluded pediatric CXR images. This is because we aim
released by the Japanese Society of Radiological Technology to utilize CXR radiography with unbiased age distribution for
(JSRT) [19]. Total 247 chest posteroanterior (PA) radiographs more accurate evaluation of deep neural networks for COVID-
were collected from 14 institutions including normal and 19 classification.
lung nodule cases. Corresponding segmentation masks were Total dataset was curated into five classes; normal, TB,
collected from the SCR database [20]. The JSRT/SCR dataset bacterial pneumoia, viral pneumonia, COVID-19 pneumonia.
were randomly split into training (80%) and validation (20%). The numbers of each disease class from the data set are sum-
For cross-database validation purpose, we used another public marized in Table III. Specifically, a total of 180 radiography
CXR dataset: U.S. National Library of Medicine (USNLM) images of 118 subjects from COVID-19 image data collection
collected Montgomery Country (MC) dataset [21]. Total were included. Moreover, a total of 322 chest radiography
138 chest PA radiographs were collected including normal, images from different subjects were used, which include 191,
TB cases and corresponding lung segmentation masks. 54, and 20 images for normal, bacterial pneumonia, and

Authorized licensed use limited to: IEEE Xplore. Downloaded on October 30,2022 at 17:38:56 UTC from IEEE Xplore. Restrictions apply.
2692 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 39, NO. 8, AUGUST 2020

TABLE IV
D ATASET FOR C OMPARISON W ITH COVID-N ET

viral pneumonia (not including COVID-19), respectively. The

combined dataset were randomly split into train, validation,
and test sets with the ratio of 0.7, 0.1, and 0.2. Fig. 2. (a) Segmentation result. Each lung and heart segment are
overlapped on CXR coloring in blue and red, respectively. Green line
3) Dataset for Comparison With COVID-Net: We prepared a represent the ground truth. (b) Extracted lung areas, and (c) correspond-
separate dataset to compare our method with existing state- ing lung area pixel intensity histogram.
of-the art (SOTA) algorithm called COVID-Net [6]. COVID-
19 image data collection was combined with RSNA Pneu-
monia Detection Challenge dataset as described in [6] for a diagnosis cardiomegaly [25], [26]. We hypothesized that
fair comparison between our method and COVID-Net. The if cardiothoracic boundary become blurred by rounded
reason we separately train our network with the COVID-Net opacities or consolidation in COVID-19 CXR [2]–[4],
data set is that RSNA Pneumonia Detection Challenge dataset distinct off-average CTR value can be utilized as an
provide only the information regarding the presence of pneu- abnormality alarm.
monia, rather than the detailed diagnosis of disease, so that Statistical analysis for the potential biomarkers was per-
the labels were divided into only three categories including formed using MATLAB 2015a (Mathworks, Natick). Kol-
normal, pneumonia, and COVID-19 as in Table IV. More mogorov Smirnov test was used to evaluate the normal dis-
specifically, there were 8,851 normal and 6,012 pneumonia tribution of marker candidates. For non-normally distributed
chest radiography images from 13,645 patients in RSNA Pneu- variables, Wilcoxon signed rank test was used to compare seg-
monia Detection Challenge dataset, and these images were mentation performance with identical data size, and Wilcoxon
combined with COVID-19 image data collection to compose rank sum test was used to compare COVID-19 marker can-
a total dataset. Among these, 100 normal, 100 pneumonia, and didates to those of other classes with different data sizes.
10 COVID-19 images were randomly selected for validation Statistical significance (SS) levels were indicated as asterisks;
and test set, respecitvely as in [6]. Although we believe our * for p < 0.05, ** for p < 0.01 and *** for p < 0.001.
categorization into normal, bacterial, TB, and viral+COVID-
19 cases is more correlated with the radiological findings and C. Classification Performance Metrics
practically useful in clinical environment [18], we conducted
The performance of the classification methods was evalu-
this additional comparison experiments with the data set in
ated using the confusion matrix. From the confusion matrix,
Table IV to demonstrate that our algorithm provides com-
true positive (TP), true negative (TN), false positive (FP), and
petitive performance compared to COVID-Net in the same
false negative (FN) values were obtained, and 5 metrics for
experiment set-up.
performance evaluation were calculated as below:
1) Accuracy = (T N + T P)/(T N + T P + F N + F P)
B. Statistical Analysis of Potential CXR
2) Precision = T P/(T P + F P)
COVID-19 Markers
3) Recall = T P/(T P + F N)
The following standard biomarkers from CXR image analy- 4) F1 score = 2(Precision × Recall)/(Precision + Recall)
sis are investigated. 5) Specificity = T N/(T N + F P)
• Lung Morphology: Morphological structures of the seg-
Among these, the F1 score was used as the evaluation metric
mented lung area as illustrated in Fig. 2(b) was evaluated for early stopping. The overall metric scores of the algorithm
throughout different classes. were calculated by averaging each metric for multiple classes.
• Mean Lung Intensity: From the segmented lung area,
we calculated mean value of the pixel intensity within
IV. E XPERIMENTAL R ESULTS
the lung area as shown in Fig. 2(c).
• Standard Deviation of Lung Intensity: From the intensity A. Segmentation Performance on Cross-Database
histogram of lung area pixels, we calculated one standard Segmentation performance of anatomical structure was eval-
deviation which is indicated as the black double-headed uated using Jaccard similarity coefficient. Table V presents
arrow in Fig. 2(c). the Jaccard similarity coefficient of each contour on the
• Cardiothoracic Ratio (CTR): CTR can be calculated by validation dataset. The results confirmed our method provides
dividing the maximal transverse cardiac diameter by the comparable accuracy to previous works using the JSRT dataset
maximal internal thoracic diameter annotated repectively and the NLM(MC) dataset [27], [28].
as red and blue double-headed arrows in Fig. 2(a). To evaluate segmentation performance on cross-database,
Cardiothoracic Ratio (CTR) is a widely used marker to we tested either original or preprocessed images of the

Authorized licensed use limited to: IEEE Xplore. Downloaded on October 30,2022 at 17:38:56 UTC from IEEE Xplore. Restrictions apply.
OH et al.: DL COVID-19 FEATURES ON CXR USING LIMITED TRAINING DATA SETS 2693

TABLE V
CXR S EGMENTATION R ESULTS

NLM dataset as inputs. The result shows that our univer-

sal preprocessing step for data normalization contributes to
the processing of cross-database with statistically significant
improvement on segmentation accuracy (Jaccard similarity
coefficients from 0.932 to 0.943, p < 0.001). This result
indicates that preprocessing is crucial factor to ensure seg-
mentation performance in cross-database.

B. Morphological Analysis of Lung Area

To analyze morphological characteristics in the segmented
lung area, a representative CXR radiograph for each class
was selected for visual evaluation. Lung contour of each
class showed differentiable features and showed mild tendency.
In normal and TB cases (the first and the second row of
Fig. 3, respectively), overall lung and heart contour were
well-segmented. In the bacterial case, however, the segmented
lung area was deformed due to wide spread opacity of bacterial
pneumonia as shown in the third row of Fig. 3, and both
the right cardiac and thoracic borders were lost. In overall
bacterial infection cases, similar findings were occasionally
observed which caused degraded segmentation performance.
This suggests that abnormal morphology of the segmentation
masks may be a useful biomarker to differentiate the severe
infections. In the fourth row of Fig. 3, viral infection caused Fig. 3. Preprocessed images, corresponding segmentation results, and
bilateral consolidations [29], thus partial deformation of lung the extracted lung contours are shown along with the column-axis. Each
row depicts different categorical class.
area was observed. In the COVID-19 case of the fifth row of
Fig. 3, despite the bi-basal infiltrations [30], lung area was
fully segmented. In overall cases of the viral and the COVID- TABLE VI
19 classes, lung areas were either normally or partially- L UNG A REAS I NTENSITY S TATISTICS
imcompletely segmented, so morphological features of the
segmentation masks may not be sufficiently discriminatory
markers for viral and COVID-19 classes. Based on these
morphological findings in segmented lung area, we further
investigated other potential COVID-19 biomarkers.

C. Statistical Significancy of Potential

COVID-19 Bio-Markers
We hypothesized that CXR appearance influenced by con-
solidations or infiltration of COVID-19 may be reflected in 2) Lung Areas Intensity Variance: Standard deviation of pixel
intensity of the radiograph. Thus, intensity-related COVID- intensity of each lung area are scattered in plot in Fig. 4(b).
19 marker candidates were extracted and compared. For both the COVID-19 and the viral cases, the variance values
1) Lung Areas Intensity: Mean pixel intensity of each lung were higher than other classes with statistical significance
area is shown in the scatter plot of Fig. 4(a). COVID-19 cases ( p < 0.001 for all). Table VII describes the corresponding
showed lower mean intensity compared to other cases with statistical result.
statistical significance level ( p < 0.001 for normal and bacte- To investigate the effect of scanning protocol on statistics,
rial, p < 0.01 for TB). Table VI describes the corresponding we performed additional study by excluding anteroposterior
statistical result. Despite the statistical significance, the scatter (AP) Supine radiographs from entire dataset with docu-
plot showed broad overlap between several classes. mented patient information. Recall that AP Supine protocol

Authorized licensed use limited to: IEEE Xplore. Downloaded on October 30,2022 at 17:38:56 UTC from IEEE Xplore. Restrictions apply.
2694 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 39, NO. 8, AUGUST 2020

TABLE VII
L UNG A REAS I NTENSITY VARIANCE S TATISTICS

TABLE VIII
L UNG A REAS I NTENSITY VARIANCE S TATISTICS BY E XCLUDING AP
S UPINE R ADIOGRAPHS

TABLE IX
C ARDIOTHORACIC R ATIO S TATISTICS

broad overlap between several classes. Table IX describes the

corresponding statistical result.
Based on the statistical analysis of potential bio-marker
candidates, we found that intensity distribution pattern within
the lung area may be most effective in the diagnosis, which
highly reflects the reported chest X-ray (CXR) appearances
of COVID-19, i.e., multi-focally distributed consolidation and
GGO in specific region such as peripheral and lower zone
[2]–[4].
Fig. 4. Scatter plot (Left) and corresponding mean values with one
standard deviation error bars (Right). All the parameter values are However, care should be taken, since not only the locally
normalised to an arbitrary unit. concentrated multiple opacities can cause uneven intensity
distribution throughout entire lung area, but also different
texture distribution within CXR may cause the similar intensity
is an alternative to standard PA or AP protocol depending on variations. For example, multi-focally distributed consolidation
patient condition. Since AP Supine protocol is not common from COVID-19 could make the intensity variance differenti-
in normal cases, supine scanning with different acquisition ating factor from other classes, but also bacterial pneumonia
condition may have potential for considerable heterogeneity in generates opacity as well, whose feature may lead to the simi-
data distribution, causing biased results in statistical analysis, lar intensity distributions as results of different characteristics
so we investigated this issue. The result shown in Table VIII of opacity spreading pattern.
compared to Table VII showed minor difference in both the To decouple these compounding effects, we further inves-
COVID-19 and the viral and cases. The result indicates that tigated the local and global intensity distribution. For the
for both the COVID-19 and viral classes, the highly intensity- correctly classified patches from our classification network,
variable characteristic in the lung area is invariant to scaning we computed their mean intensity and standard deviation
protocol. (STD) values. We refer to the distribution of mean inten-
3) Cardiothoracic Ratio: CTR values of each lung area is sity of each patch as the inter-patch intensity distribution
scattered in Fig. 4(c). Despite there exist statistical differences (Fig. 5(a)) and the STD of each patch as intra-patch intensity
between the COVID-19 cases to other classes ( p < 0.001 for distribution (Fig. 5(b)). As shown in Fig. 5(a), the inter-
normal and TB, p < 0.05 for Bacteria), the scatter plot showed patch intensity distribution of the unified COVID-19 and viral

Authorized licensed use limited to: IEEE Xplore. Downloaded on October 30,2022 at 17:38:56 UTC from IEEE Xplore. Restrictions apply.
OH et al.: DL COVID-19 FEATURES ON CXR USING LIMITED TRAINING DATA SETS 2695

TABLE X
C LASSIFICATION R ESULTS F ROM THE G LOBAL A PPROACH AND THE
P ROPOSED PATCH -B ASED C LASSIFICATION N ETWORK

TABLE XI
S ENSITIVITY OF THE G LOBAL A PPROACH AND THE L OCAL
PATCH -B ASED C LASSIFICATION N ETWORK

global method and the (b) local patch-based method are shown
in Fig. 6. The proposed local patch-based approach showed
consistently better performance than global approach in all
metrics. In particular, as depicted in Table XI, our method
showed the sensitivity of 92.5% for COVID-19 and viruses,
which was acceptable performance as a screening method,
considering the fact that the sensitivity of COVID-19 diagnosis
by X-ray image is known to be 69% even for clinical experts
and that the current gold standard, RT-PCR, has sensitivity
of 91% [4]. Moreover, compared to the global approach,
the sensitivity of other classes are significantly high, which
confirms the efficacy of our method.

Fig. 5. Scatter plot (Left) and corresponding mean values with one E. Interpretability Using Saliency Map
standard deviation error bars. Each scatter depicts a patch which was
correctly classified to the ground truth label. All the parameter values Fig. 7 and Fig. 8 illustrate the examples of visualization of
were normalised to an arbitrary unit. Statistically differentiable classes saliency map. As shown in Fig. 7(a), the existing Grad-CAM
from the COVID-19 and viral cases (p < 0.001) are marked at each error
bar. method for global approach showed the limitation that it only
focuses on the broad main lesion so that it cannot properly
class showed distict lower intensity values ( p < 0.001 for differentiate multifocal lesions within the image. On the other
all) to other classes and highly intensity-variant characteris- hand, with the probabilistic Grad-CAM, multifocal GGOs
tics which can be represented as the large error bar. This and consolidations were visualized effectively by our local
result is in accordance with the result of lung area intensity patch-based approach as shown in Fig. 7(c), which was in
and intensity variance (Fig. 4(a), (b)). Intra-patch intensity consistent with the findings reported by clinical experts. In
distribution, however, showed no difference compared to the particular, when we compute the probabilistic Grad-CAM for
normal class ( p > 0.05). From these intra- and inter-patch the COVID-19 class using patient images from various classes
intensity distribution results, we can infer that intra-patch (i.e., normal, bacterial, TB, and COVID-19), a noticeable
variance, which represents local texture information, was not activation map was observed only in the COVID-19 patient
crucially informative, whereas the globally distributed multi- data set, whereas almost no activations were observed in
focal intensity change may be an important discriminating patients with other diseases and conditions as shown in Fig. 8.
feature for COVID-19 diagnosis, which is strongly correlated These results strongly support our claim that the probabilistic
with the radiological findings. Grad-CAM saliency map from our local patch-based approach
One common finding among the marker candidates was is more intuitive and interpretable compared to the existing
no difference between the COVID-19 and the viral case methods.
( p > 0.05 for all the markers), which is also correlated
with radiological findings [18]. Therefore, in the classification V. D ISCUSSION
network, the COVID-19 and viral classes were integrated into
A. COVID-19 Features on CXR
one class.
In the diagnosis of COVID-19, other diseases mimick-
ing COVID-19 pneumonia should be differentiated, including
D. Classification Performance community-acquired pneumonia such as streptococcus pneu-
The classification performances of the proposed method monia, mycoplasma and chlamydia related pneumonia, and
are provided in Table X. The confusion matrices for the (a) other coronavirus infections.

Authorized licensed use limited to: IEEE Xplore. Downloaded on October 30,2022 at 17:38:56 UTC from IEEE Xplore. Restrictions apply.
2696 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 39, NO. 8, AUGUST 2020

Fig. 7. Examples of saliency maps for COVID-19 patient. (a) Grad-CAM

saliency map for the global approach, (b) original X-ray image, and (c) our
probabilistic Grad-CAM saliency map for local patch-based approach.

acquired pneumonia is bacterial infection [5]. Specifically,

most studies reported that S. pneumoniae is the most fre-
quent causative strain (15 − 42%), after which H. influenza
(11 − 12%) and viral pneumonia follow as the second and the
third most common causes of pneumonia, respectively [5].
In addition, depending on the geological region, substantial
proportion of pneumonia may be diagnosed as TB (up to 10%)
[5]. Summing these up, the proportion of bacterial pneumonia
and TB is suspected to be still large even in this pandemic
situation of COVID-19. In this respect, the disease such as
bacterial pneumonia or TB as well as normal condition can
be excluded primarily, to preserve limited medical resources
such as RT-PCR or CT only for those who suspected to be
infected with COVID-19.
The detailed triage workflow that utilizes the proposed
algorithm is described in Fig. 9. Specifically, our neural
network is trained to classify other virus and COVID-19 in the
Fig. 6. Confusion matrices for the (a) global approach, and (b) the
same class. This is not only because it is strongly correlated
proposed local patch-based approach. with the radiological findings [18], but also useful as a triage.
More specifically, by excluding normal, bacterial pneumonia,
and TB at the early stage, we can use RT-PCR or CT for only
In radiological literature, most frequently observed dis- those patients classified as other virus and COVID-19 cases for
tribution patterns of COVID-19 are bilateral involvement, final diagnosis. By doing this procedure, we can save limited
peripheral distribution and GGO [13]. Wong et al. [4] found medical resources such as RT-PCR or CT to those patients
that consolidation was the most common finding (30/64, 47%), whose diagnosis by CXR is difficult even by radiologists.
followed by GGO (21/64, 33%). CXR abnormalities had a
peripheral (26/64, 41%) and lower zone distribution (32/64,
50%) with bilateral involvement (32/64, 50%), whereas pleural C. Training Stability
effusion was uncommon (2/64, 3%).
In order to investigate the origin of the apparent advantages
Our statistical analysis of the intensity distribution clearly
of using local patch-based training over the global approach,
showed that the globally distributed localized intensity vari-
we investigate the training dynamics to investigate the pres-
ation is a discriminatory factor for COVID-19 CXR images,
ence of overfitting. This is especially important, given that
which was also confirmed with our saliency map. This clearly
the training data is limited due to the difficulty of systematic
confirmed that the proposed method clearly reflects the radi-
CXR data collection for COVID-19 cases under current public
ological findings.
health emergency.
Fig. 10 shows the curves for accuracy and F1 score of
B. Feasibility as a ‘Triage’ for COVID-19 (a) the global approach and (b) the proposed local patch-
In pandemic situation of infectious disease, the distribution based approach for each epoch. Note that both approaches
of medical resources is a matter of the greatest importance. use the same number of weight parameters. Still, thanks to the
As COVID-19 is spreading rapidly and surpassing the capacity increasing training data set from the random patch cropping
of medical system in many countries, it is necessary to make across all image area, our local patch-based algorithm did not
reasonable decision to distribute the limited resources based on showed any sign of overfitting even with the small numbers
the ‘triage’, which determine the needs and urgency for each of training data, while the global approach showed significant
patient. In general, the most common cause of community overfitting problem. This clearly indicates that with the limited

Authorized licensed use limited to: IEEE Xplore. Downloaded on October 30,2022 at 17:38:56 UTC from IEEE Xplore. Restrictions apply.
OH et al.: DL COVID-19 FEATURES ON CXR USING LIMITED TRAINING DATA SETS 2697

Fig. 8. Examples of probabilistic Grad-CAM of COVID-19 class for (a) normal, (b) bacterial, (c) tuberculosis, and (d) COVID-19 pneumonia patients.

Fig. 9. Potential triage workflow that utilizes the proposed algorithm in

the diagnosis of COVID-19 patient.

data set the patch-based neural network training may be a

promising direction.

D. Comparison With COVID-Net

Since the proposed patch-based neural network architecture
is designed by considering limited training data set, we investi-
gated any potential performance loss in comparison with other
SOTA deep learning approach that has been developed without
such consideration. Specifically, COVID-Net [6] is one of the
most successful approaches in COVID-19 diagnosis, so we Fig. 10. Training and validation accuracy and F1-score for each
chose it as the SOTA method. epoch. (a) Global approach, and (b) the proposed patch-based approach.
The comparison between our method and COVID-Net is
shown in Table XII. With the same dataset, our method compared to the COVID-Net. In addition, it is also remarkable
showed overall accuracy of 91.9 % which is comparable to that our method uses only about 10% number of parameters
that of 92.4 % for COVID-Net. Furthermore, our method (11.6 M) compared to that of COVID-Net (116.6 M), because
provided significantly improved sensitivity to COVID-19 cases the proposed algorithm is developed based on less complex

Authorized licensed use limited to: IEEE Xplore. Downloaded on October 30,2022 at 17:38:56 UTC from IEEE Xplore. Restrictions apply.
2698 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 39, NO. 8, AUGUST 2020

TABLE XII
C OMPARISON OF O UR M ETHOD W ITH COVID-N ET

TABLE XIII
L UNG S EGMENTATION R ESULTS C OMPARISON

network architecture without increasing the complexity of the

model. This may bring the advantages not only in the aspect
of computational time but also in the aspect of performance
and stability with small-sized dataset.

E. Cross-Database Generalization Capability

We are aware that the current study has limitations due to
the lack of well-curated pneumonia CXR dataset. Specifically,
our CXR data set come from a single or at most two sources
(see Table II). Moreover, publicly available COVID-19 dataset
[23] are largely extracted from online publications, website,
etc, so they are not collected in a rigorous manner.
To mitigate the issue of potential bias from the limita-
tion of the database, we employed a universal preprocessing Fig. 11. Intensity distribution of segmented anatomies of (a) normal,
step for data normalization for the entire dataset as dis- (b) original COVID-19, and (c) preprocessed COVID-19 CXRs. Repre-
sentative intensity histogram of each (d) normal, (e) original COVID-19,
cussed before. We investigated the effects of our preprocess- and (f) preprocessed COVID-19 CXR.
ing step on cross-database generalization by investigating
the COVID-19 dataset, which poses the most severe intra-
TABLE XIV
dataset heterogeneity. As shown in Fig. 11(b), each original C LASSIFICATION R ESULTS W ITH D IFFERENT S EGMENTATION
CXRs of the COVID-19 class showed highly-varying intensity M ETHODS
characteristics among each segmented anatomies. Thanks to
our preprocessing step, the mean pixel intensity distribution
between lung and heart regions of the preprocessed COVID-
19 dataset (see Fig. 11(c)) became similar to the normal
class in Fig. 11(a). The problem of heterogeneity can be also
mitigated as shown in the intensity histograms (see Fig. 11(d)-
(f)). The results confirmed that the original COVID-19 data
better classification performance in all metrics than the U-
could be well preprocessed to have comparable intensity
Net. When compared with the FC-DenseNet67, which has
distribution to that of well-curated normal data.
smaller number of parameters (3.5 M) [15], the performance
improvement by our method is significant. Given the better
F. Segmentation Network Analysis trade-off between the complexity versus performance, we
1) Comparison With U-Net: Recall that we chose FC- adopted FC-DenseNet103 as our segmentation network.
DenseNet103 as a backbone segmentation network architecture 2) Effect of Trainset Size: To demonstrate the robustness
thanks to its higher segmentation performance with smaller of the proposed segmentation network with limited training
number of parameters (9.4 M) [31]. To demonstrate the dataset, we performed the ablation study by reducing training
effectiveness of CXR segmentation by the FC-Densenet103, dataset size. Lung segmentation performance was evaluated
we trained U-Net [32] under identical training conditions on the cross-database NLM(MC) dataset. For the preprocessed
and compared the results. There was no significant difference NLM(MC) dataset, Jaccard similarity coefficients remain sta-
between the networks result. ble until 50% of trainset was used for training as shown in
We further analyzed the effect of the different segmentation Table XV; however, in the original NLM(MC) dataset with-
methods on classification performance. The proposed segmen- out preprocessing step, segmentation performance decreased
tation method with the FC-DenseNet103 resulted consistently steeply as the size of trainsets decreased. This results support

Authorized licensed use limited to: IEEE Xplore. Downloaded on October 30,2022 at 17:38:56 UTC from IEEE Xplore. Restrictions apply.
OH et al.: DL COVID-19 FEATURES ON CXR USING LIMITED TRAINING DATA SETS 2699

TABLE XV TABLE XVI

E FFECTS OF T RAINSET S IZE C LASSIFICATION R ESULTS W ITH AND W ITHOUT S EGMENTATION M ASK

that proposed segmentation network endures limited train-

ing dataset size by matching intensity distribution of cross-
database CXRs thanks to our preprocessing step.
3) Segmentation Effects on Marker Analysis and Classifica-
tion: The proposed segmentation network was trained with
normal subject set that have segmentation mask as shown
Table V, XV, and showed comparable performance with the
SOTA for the normal subjects. However, when CXR images
with severe consolidation are used, segmentation performance
degradation is unavoidable, since such images have been never
observed during the training.
For example, the radiograph from a bacterial pneumonia
case in Fig. 3 was under-segmented due to widely spread
severe opacity. To further investigate this issue, we exam-
ined all cases of under-segmentation by defining the under-
segmentation as a segmentation mask in which over 1/4 of the Fig. 12. Accuracy of the algorithm according to the dataset size in
entire lung region is deformed. Our investigation showed that (a) absolute and (b) relative scales.
the under-segmentations are only outliers that are observed in TABLE XVII
some patient data set (7 of 54 bacterials cases (13%,), 2 of C LASSIFICATION R ESULTS U SING D IFFERENT PATCH S IZES
57 cases for TB (3.5%), and 5 of 200 cases for COVID-19 and
viral pneumonia (2.5%)), whereas no under segmentation were
observed from 191 healthy subjects.
To confirm that the difference in the segmentation results
can be a morphological marker for classification, we evalu-
ated whether it is possible to distinguish normal and abnor-
G. Classification Network Analysis
mal images (including bacterial pneumonia, TB, COVID-19
and other viruses) using only binary segmentation masks 1) Effect of Patch Size on the Performance: To evaluate the
(not X-ray images). With a separately trained neural network effect of the patch size on the performance of the classification
using only binary masks, it was possible to distinguish between algorithm, we tested various patch sizes, such as 112 × 112,
normal and abnormal images with 86.9% sensitivity. This con- and 448×448. Using half-sized (112×112) patches, the results
firms that the morphology of the segmentation mask is a dis- were worse as shown in Table XVII. With double-sized (448×
criminatory biomarker between normal and the patient groups. 448) patches, the results were not better than those with patch
Then, we conducted additional experiments to evaluate how size of 224 × 224, as depicted in Table XVII. In summary,
classification performance is affected by excluding or includ- there seems to be a clear drawback in reducing the patch size,
ing under-segmentation cases. Although the differences in and there was also no benefit with increasing the patch size.
other labels were not significant, the overall sensitivity for bac- Therefore, we chose the value in between.
terial pneumonia were better when the under-segmented sub- 2) Effects of Trainset Size: We analyzed the effect of dataset
jects were excluded. Therefore, the under-segmentation still size in terms of classification performance, since we aimed
has some effects on the classification between the diseases. to develop the method that has the advantage of maintaining
Finally, we performed an additional experiment for the robustness even when only limited data are available. The
comparison of classification with and without segmentation classification performances with decreasing dataset sizes are
masks. The results in Table XVI clearly confirmed that despite provided in Fig. 12. The global approach using whole image,
the under-segmented outliers the use of segmentation mask which is similar to most classification methods, showed promi-
significantly improved the classification performance on the nent decrease in accuracy with decreasing dataset size, but the
whole. This suggests that there are rooms to improve the proposed local patch-based method showed less compromised
performance of the proposed method, if the segmentation performance, showing the robustness to the reduced dataset
network could be further trained using patient cases with size as shown in Fig. 12(a). These results were more prominent
correct segmentation labels. when comparing them by relative scale as shown in Fig. 12(b).

Authorized licensed use limited to: IEEE Xplore. Downloaded on October 30,2022 at 17:38:56 UTC from IEEE Xplore. Restrictions apply.
2700 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 39, NO. 8, AUGUST 2020

VI. C ONCLUSION [10] I. D. Apostolopoulos, S. Aznaouridis, and M. Tzani, “Extracting possibly

representative COVID-19 biomarkers from X-ray images with deep
In the rapidly evolving global pandemic of COVID-19, learning approach and image data related to pulmonary diseases,” 2020,
the use of CXR for COVID-19 diagnosis or triage for patient arXiv:2004.00338. [Online]. Available: http://arxiv.org/abs/2004.00338
management has become an important issue to preserve [11] M. Farooq and A. Hafeez, “COVID-ResNet: A deep learning framework
for screening of COVID19 from radiographs,” 2020, arXiv:2003.14395.
limited medical resources and prevent further spreading of [Online]. Available: http://arxiv.org/abs/2003.14395
the virus. However, the current diagnostic performance with [12] P. Afshar, S. Heidarian, F. Naderkhani, A. Oikonomou, K. N. Plataniotis,
and A. Mohammadi, “COVID-CAPS: A capsule network-based frame-
CXR is not sufficient for routine clinical use, so the need work for identification of COVID-19 cases from X-ray images,” 2020,
of artificial intelligence to improve diagnostic performance of arXiv:2004.02696. [Online]. Available: http://arxiv.org/abs/2004.02696
CXR is increasing. Unfortunately, due to the emergent nature [13] S. Salehi, A. Abedi, S. Balakrishnan, and A. Gholamrezanezhad, “Coro-
navirus disease 2019 (COVID-19): A systematic review of imaging
of COVID-19 global pandemic, systematic collection of the findings in 919 patients,” Amer. J. Roentgenol., pp. 1–7, Mar. 2020.
large data set for deep neural network training is difficult. [14] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and
To address this problem, we investigated potential biomark- D. Batra, “Grad-CAM: Visual explanations from deep networks via
gradient-based localization,” Int. J. Comput. Vis., vol. 128, no. 2,
ers in the CXR and found the globally distributed localized pp. 336–359, Feb. 2020.
intensity variation can be an discrimatory feature for the [15] S. Jegou, M. Drozdzal, D. Vazquez, A. Romero, and Y. Bengio, “The
COVID-19. Based on this finding, we propose a patch-based one hundred layers tiramisu: Fully convolutional DenseNets for semantic
segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit.
deep neural network architecture that can be stably trained Workshops (CVPRW), Jul. 2017, pp. 11–19.
with small data set. Once the neural network was trained, [16] D. P. Kingma and J. Ba, “Adam: A method for stochastic
optimization,” 2014, arXiv:1412.6980. [Online]. Available:
the final decision was made based on the majority voting from http://arxiv.org/abs/1412.6980
multiple patches at random locations within lungs. We also [17] A. Paszke et al., “Automatic differentiation in PyTorch,” in Proc. NIPS,
proposed a novel probabilistic Grad-CAM saliency map that is 2017, pp. 1–4.
[18] S. H. Yoon et al., “Chest radiographic and CT findings of the 2019 novel
tailored to the local patch-based approach. Our experimental coronavirus disease (COVID-19): Analysis of nine patients treated in
results demonstrated that the proposed network was trained korea,” Korean J. Radiol., vol. 21, no. 4, pp. 494–500, 2020.
stably with small data set, provided comparative results with [19] J. Shiraishi et al., “Development of a digital image database for chest
radiographs with and without a lung nodule,” Amer. J. Roentgenol.,
the SOTA method, and generated interpretable saliency maps vol. 174, no. 1, pp. 71–74, Jan. 2000, doi: 10.2214/ajr.174.1.1740071.
that are strongly correlated with the radiological findings. [20] B. van Ginneken, M. B. Stegmann, and M. Loog, “Segmentation of
anatomical structures in chest radiographs using supervised methods:
A comparative study on a public database,” Med. Image Anal., vol. 10,
ACKNOWLEDGMENT no. 1, pp. 19–40, Feb. 2006.
The authors would like to thank the doctors and medical [21] S. Jaeger, S. Candemir, S. Antani, Y.-X. J. Wáng, P.-X. Lu, and
G. Thoma, “Two public chest X-ray datasets for computer-aided screen-
professionals from Korea and all around the world who have ing of pulmonary diseases,” Quant. Imag. Med. surgery, vol. 4, no. 6,
dedicated their time and efforts to treat COVID-19 patients p. 475, 2014.
[22] Praveen. Corona Hack: Chest X-Ray-Dataset. Accessed: Mar. 21, 2020.
and protect the health of citizens during this pandemic. [Online]. Available: https://www.kaggle.com/praveengovi/coronahack-
Source codes can be downloaded from https://github.com/ chest-xraydataset
jongcye/Deep-Learning-COVID-19-on-CXR-using-Limited- [23] J. Paul Cohen, P. Morrison, and L. Dao, “COVID-19 image data
collection,” 2020, arXiv:2003.11597. [Online]. Available: http://arxiv.
Training-Data-Sets. org/abs/2003.11597
[24] D. S. Kermany et al., “Identifying medical diagnoses and treat-
able diseases by image-based deep learning,” Cell, vol. 172, no. 5,
R EFERENCES pp. 1122–1131.e9, Feb. 2018.
[1] X. Xie, Z. Zhong, W. Zhao, C. Zheng, F. Wang, and J. Liu, “Chest CT [25] C. S. Danzer, “The cardiothoracic ratio: An index of cardiac enlarge-
for typical 2019-nCoV pneumonia: Relationship to negative RT-PCR ment,” Amer. J. Med. Sci., vol. 157, no. 4, p. 513, 1919.
testing,” Radiology, Feb. 2020, Art. no. 200343. [26] K. Dimopoulos et al., “Cardiothoracic ratio from postero-anterior chest
[2] Y. Fang et al., “Sensitivity of chest CT for COVID-19: Comparison to radiographs: A simple, reproducible and independent marker of disease
RT-PCR,” Radiology, Feb. 2020, Art. no. 200432. severity and outcome in adults with congenital heart disease,” Int.
[3] T. Ai et al., “Correlation of chest CT and RT-PCR testing in coronavirus J. Cardiol., vol. 166, no. 2, pp. 453–457, Jun. 2013.
disease 2019 (COVID-19) in China: A report of 1014 cases,” Radiology, [27] S. Candemir and S. Antani, “A review on lung boundary detection in
Feb. 2020, Art. no. 200642. chest X-rays,” Int. J. Comput. Assist. Radiol. Surgery, vol. 14, no. 4,
[4] H. Y. F. Wong et al., “Frequency and distribution of chest radi- pp. 563–576, Apr. 2019.
ographic findings in COVID-19 positive patients,” Radiology, Mar. 2020, [28] E. Jangam and A. Rao, “Public datasets and techniques for segmentation
Art. no. 201160. of anatomical structures from chest x-rays: Comparitive study, current
[5] P. D. Brown and S. A. Lerner, “Community-acquired pneumonia,” The trends and future directions,” in Communications in Computer and
Lancet, vol. 352, no. 9136, pp. 1295–1302, 1998. Information Science. RTIP2R, vol. 1036. Singapore: Springer, 2018,
[6] L. Wang and A. Wong, “COVID-net: A tailored deep convolu- pp. 321–331.
tional neural network design for detection of COVID-19 cases from [29] N. S. Paul et al., “Radiologic pattern of disease in patients with severe
chest X-ray images,” 2020, arXiv:2003.09871. [Online]. Available: acute respiratory syndrome: The toronto experience,” RadioGraphics,
http://arxiv.org/abs/2003.09871 vol. 24, no. 2, pp. 553–563, Mar. 2004.
[7] A. Narin, C. Kaya, and Z. Pamuk, “Automatic detection of coron- [30] I. Thevarajan et al., “Breadth of concomitant immune responses prior to
avirus disease (COVID-19) using X-ray images and deep convolu- patient recovery: A case report of non-severe COVID-19,” Nature Med.,
tional neural networks,” 2020, arXiv:2003.10849. [Online]. Available: vol. 26, no. 4, pp. 453–455, Apr. 2020.
http://arxiv.org/abs/2003.10849 [31] D. Lobo Torres et al., “Applying fully convolutional architectures for
[8] I. D. Apostolopoulos and T. A. Mpesiana, “Covid-19: Auto- semantic segmentation of a single tree species in urban environment on
matic detection from X-ray images utilizing transfer learning high resolution UAV optical imagery,” Sensors, vol. 20, no. 2, p. 563,
with convolutional neural networks,” Phys. Eng. Sci. Med., 2020.
Apr. 2020. [32] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional net-
[9] E. El-Din Hemdan, M. A. Shouman, and M. Esmail Karar, “COVIDX- works for biomedical image segmentation,” in Proc. Int. Conf. Med.
net: A framework of deep learning classifiers to diagnose COVID- Image Comput. Comput.-Assist. Intervent. (MICCAI), in Lecture Notes
19 in X-ray images,” 2020, arXiv:2003.11055. [Online]. Available: in Computer Science, vol. 9351. Cham, Switzerland: Springer, 2015,
http://arxiv.org/abs/2003.11055 pp. 234–241.

Authorized licensed use limited to: IEEE Xplore. Downloaded on October 30,2022 at 17:38:56 UTC from IEEE Xplore. Restrictions apply.