Measurement 145 (2019) 511–518

Contents lists available at ScienceDirect

Measurement
journal homepage: www.elsevier.com/locate/measurement

Identifying pneumonia in chest X-rays: A deep learning approach

Amit Kumar Jaiswal a, Prayag Tiwari b, Sachin Kumar c, Deepak Gupta d, Ashish Khanna d,
Joel J.P.C. Rodrigues e,f,g,⇑
a
Institute for Research in Applicable Computing, University of Bedfordshire, United Kingdom
b
Department of Information Engineering, University of Padova, Padova, Italy
c
Department of System Programming, South Ural State University, Chelyabinsk, Russia
d
Maharaja Agrasen Institute of Technology, Delhi, India
e
National Institute of Telecommunications (Inatel), Santa Rita do Sapucaí, MG, Brazil
f
Instituto de Telecomunicações, Portugal
g
Federal University of Piauí, Teresina, PI, Brazil

a r t i c l e i n f o a b s t r a c t

Article history: The rich collection of annotated datasets piloted the robustness of deep learning techniques to effectuate
Received 11 March 2019 the implementation of diverse medical imaging tasks. Over 15% of deaths include children under age five
Received in revised form 1 May 2019 are caused by pneumonia globally. In this study, we describe our deep learning based approach for the
Accepted 21 May 2019
identification and localization of pneumonia in Chest X-rays (CXRs) images. Researchers usually employ
Available online 4 June 2019
CXRs for the diagnostic imaging study. Several factors such as positioning of the patient and depth of
inspiration can change the appearance of the chest X-ray, complicating interpretation further. Our iden-
2010 MSC:
tification model (https://github.com/amitkumarj441/identify_pneumonia) is based on Mask-RCNN, a
00-01
99-00
deep neural network which incorporates global and local features for pixel-wise segmentation. Our
approach achieves robustness through critical modifications of the training process and a novel post-
Keywords: processing step which merges bounding boxes from multiple models. The proposed identification model
Chest X-ray achieves better performances evaluated on chest radiograph dataset which depict potential pneumonia
Medical imaging causes.
Object detection Ó 2019 Elsevier Ltd. All rights reserved.
Segmentation

1. Introduction images can help in accurate and better understanding of the

X-ray images.
Pneumonia is one of the leading causes of death among children X-ray image analysis is considered as tedious and crucial tasks
and old age people around the world. It is an infection caused by a for radiology experts. Therefore, researchers have proposed several
virus, bacteria or other germs. Pneumonia results in inflammation computer algorithms to analyze X-ray images [1,2]. Also, several
in lungs which can be life threatening if not diagnosed in time. computer assisted diagnosis tools [3–5] have been developed to
Chest X-ray is an important pneumonia diagnosis method world- provide an insight of X-ray images. However, these tools are not
wide. However, an expert knowledge and experience is required able to provide sufficient information to support doctors in making
to read the X-ray images carefully. Therefore, the process of pneu- decisions [6]. Machine learning is a promising approach in the field
monia detection by reading X-ray images can be time consuming of artificial intelligence. A plenty of research works have been car-
and less accurate. The reason is that several other medical condi- ried out to investigate the chest and lung diseases using machine
tions i.e. lung cancer, excess fluid etc. can also show similar opac- learning. Vector quantization, regression neural networks has been
ities in images. Therefore, accurate reading of images is highly used to investigate chest disease [7]. In another study [8], chronic
desirable. The power of computing is world known and developing pneumonia disease was analyzed and its diagnosis was imple-
an identification model for finding pneumonia causes in clinical mented using neural networks. Another study [9] used chest radio-
graphic images for the detection of lung diseases. They applied
histogram equalization for image pre-processing and further feed
⇑ Corresponding author. forward neural network was used for classification. Although the
E-mail addresses: [email protected] (A.K. Jaiswal), prayag.tiwar- above mentioned studies have performed efficiently, however
[email protected] (P. Tiwari), [email protected] (D. Gupta), [email protected].
lacks in terms of higher accuracy, computational time and error
in (A. Khanna), [email protected] (J.J.P.C. Rodrigues).

https://doi.org/10.1016/j.measurement.2019.05.076
0263-2241/Ó 2019 Elsevier Ltd. All rights reserved.
512 A.K. Jaiswal et al. / Measurement 145 (2019) 511–518

rate. Deep learning has already been proved an effective approach object in the image should be detected carefully and segmentation
in object detection and segmentation, image classification, natural of each instance should be done precisely. Therefore, a different
language processing etc. Further, deep learning has also shown its approach is required to handle both instance segmentation and
potential in medical image analysis for object detection and seg- object detection. Such powerful methods are faster region based
mentation such as radiology image analysis in order to study CNN (F-RCNN) [21] and FCN (Fully Convolutional Network) [22].
anatomical or pathological structures of human body [10–12]. Moreover, F-RCNN can be extended with an additional branch
Also, deep learning provided higher accuracy than traditional neu- for segmentation mask prediction on each region of interest along
ral network architectures. with existing branches for classification task. This extended net-
In the remainder of this article, we first review the literature work is called Mask R-CNN and it is better than F-RCNN in terms
related to pneumonia identification in chest X-ray images in Sec- of efficiency and accuracy. Kaiming He et al. [23] presented Mask
tion 2 followed by proposed model architecture in Section 3 detail- R-CNN approach for object instance segmentation. They compared
ing algorithm and training steps in different stages. We have their results with best models from COCO 2016 [24,25]. Luc et al.
detailed our extensive analysis of RSNA dataset in Section 4 with [26] extended their approach by introducing an instance level seg-
image augmentation steps including the result from cleaned data, mentation by predicting convolutional features.
and evaluation metrics followed by evaluation result in Section 5 of
our proposed model as well as ensembles of our model. Finally, we 3. Proposed architecture
conclude our work in Section 6 along with future work.
In this section, we formulate and explore the problem pipeline
followed by our model based on Mask-RCNN in detecting pneumo-
2. Literature survey nia symptoms from chest X-ray images.1.

Roth et al. [13] demonstrated the power of deep convolutional

3.1. Problem settings
neural network (CNN) to detect the lymph node in clinical diagnos-
tic task and obtained drastic results even in the presence of low
The problem consists of binary classification of chest X-rays on
contrast surrounding structures obtained from computer tomogra-
three different classes of lung opacities such as opacity, no opacity
phy. In another study, Shin et al. [14] addressed the problems of
and not normal. The major issue is dissimilarity in quality X-rays in
thoraco-abdominal lymph detection and interstitial lung disease
terms of brightness, resolution and position region of interest. To
classification using deep CNN. They developed different CNN archi-
model such task, we describe our algorithm that can detect the
tectures and obtained promising results with 85 percent sensitivity
visual signal for pneumonia in medical chest radiographs, and out-
at three false positives per patient. Ronneburger et al. [15] devel-
put either pneumonia positive or negative, and if positive it also
oped a CNN approach with the use of data augmentation. They sug-
returns predicted bounding boxes around lung opacities.
gested that even trained on small samples of image data obtained
from transmitted light microscopy; the developed model was able
to capture high accuracy. Jamaludin et al. [16] applied CNN archi- 3.2. Modeling
tecture to analyze the data obtained from spinal lumber magnetic
resonance imaging (MRI). They developed an efficient CNN model In this section, we describe our modeling approach based on
to generate radiological grading of spinal lumber MRIs. Mask-RCNN [23] which aims to identify lung opacity that are likely
All these studies have performed well on radiological data to depict pneumonia. Mask-RCNN is a deep neural network devel-
except that the size of the data was restricted to few hundred sam- oped to solve instance segmentation in particular. Initially, we
ples of patients. Therefore, a detailed study is required to use the illustrate how we employed faster region based convolutional net-
power of deep learning over thousand samples of patients to work [21] with pixelwise instance segmentation [27] for classifica-
achieve the accurate and reliable predictions. Kallianos et al. [17] tion and localisation to build our model. We first input an image
presented a state of art review stating the importance of artificial from chest X-ray sample data which goes through ROIAlign classi-
intelligence in chest X-ray image classification and analysis. Wang fier extracting features from the input radiograph, and then F-
et al. [18] addressed this issue and prepared a new database RCNN model which then instantiated for pixelwise segmentation
ChestX-ray8 with 108,948 front view X-ray images of 32,717 and makes a bounding box of the input image. It returns the pre-
unique patients. Each of the X-ray images could have multiple dicted labels of image reported in Fig. 1. We use ResNet101 as a
labels. They used deep convolutional neural networks to validate backbone detector in Mask-RCNN model and also compared it by
the results on this data and obtained promising results. They men- replacing the detector with ResNet50 which is a convolutional
tioned that chestX-ray8 database can be extended by including network.
more disease classes and would be useful for other research In perspective of pneumonia identification, Mask-RCNN model
studies. takes chest X-ray image as an input and predicts the bounding
Rajpurkar et al. [19] developed a 121 layer deep convolutional boxes of the image, label, mask including classes. It extends the
layer network chestX-ray14 dataset. This dataset is publically algorithm of F-RCNN by adding a branch which induces binary
available with more than 0.1 million front view X-ray images with mask predicting whether the given image pixel contributes to
14 disease labels. They mentioned that their algorithm is capable the given part of the object or not. Also, it is easy to train a Mask
to predict all 14 disease categories with high efficiency. Irvin R-CNN and it adds a small overhead in terms of running time
et al. [20] stated that large labeled dataset is the key to success which is negligible. So, we may consider Mask R-CNN as an
for prediction and classification tasks. They presented a huge data- advanced faster R-CNN. We trained a RetinaNet [28] model which
set that consists of 224,316 chest radiographic images of 65,240 is a classic approach for object detection. However, the approach
patients. They named this dataset as CheXpert. Then they used does not work well in all scenarios especially in the case on non-
convolutional neural networks to assign labels to them based on vertical/horizontal objects. With Mask R-CNN this issue can be
the probability assigned by model. Model used frontal and lateral resolved. Our Mask-RCNN based model gives more accurate pixel-
radiographs to output the probabilities of each observation. Fur- wise semantic segmentation than faster-RCNN for pneumonia
ther, they released the dataset as a benchmark dataset. Besides
the availability of a large dataset, it is highly desirable that every 1
Chest X-ray images are in DICOM format:https://en.wikipedia.org/wiki/DICOM
 A.K. Jaiswal et al. / Measurement 145 (2019) 511–518 513

Fig. 1. Mask R-CNN based model for opacity identification and pixel-wise disease segmentation.

prone region in lungs around the rectangular bounding boxes. For Table 1
instance, refer to the input image and prediction sample in Fig. 1. List of parameters in post-processing stage.

Type of threshold Value

3.3. Algorithm Maximum overlap [Phase 0] 0.05

Confidence [Phase 0] 0.94
Minimum average confidence [Phase 0] 0.65
Our algorithmic approach for identifying potential pneumonia Class probability [Phase 1] 0.175
causes is devised by Faster-RCNN [21]. We also tried several other Confidence [Phase 2] 0.975
object detection techniques such as You Look Only Once (YOLO3) Below threshold confidence [Phase 3] 0.69
[29] and U-Net [15] image detection architectures but it fails to
produce better predictions, from our tests, we found that Mask-
RCNN performing better in prediction tasks. We implemented the confidence score for each and every bounding boxes before the
the base network of Mask-RCNN pre-trained on COCO weights2 post-processing stage using the following equation:
using typical residual convolutional neural network (i.e., ResNet50
1 X
and ResNet101 [30]) for extracting features of actual human lungs S^c ¼ Sc;i ð1Þ
and ROIAlign as a classifier and bounding box as a regressor. We per- F i
formed pixel-wise segmentation of lung opacity selected by ROI clas-
sifier which helps scaling during inferencing and losses. We used where S^c;i is the confidence score for every ith bounding box for the
multi-task loss [31] for training our model (identification and classi- cth set, F depicts a scaling factor and S^c is the ensemble of confi-
fication) and estimated hyperparameters based on a 10% stratified dence score for the cth set. The bounding box from the ensembling
sample from the validation set of the training data. We employed of cth set is calculated by
stochastic gradient descent (SGD) with an initial learning rate of
0.00105 for training, the overall training time is 11.2 h for 20 epochs P^cl ¼ medianfPcl g þ a:rcl ð2Þ
with batch size and image size of 16 and 512
512. Initially, we
where Pcl denotes the group of pixel locations of the corner l (top-
trained our model on positive images only followed by fine-tuning
right, top-left, bottom-right, or bottom-left) of bounding boxes for
the model on all images, in which the foreground IoU threshold sets
each cth set, a depicts the scaling factor with value 0.1, rcl depicts
to 0.3. We also performed augmentation3 during the training pro-
cess. Our final prediction set is generated by post-processing fol- the standard deviation of P cl , and P^cl depicts pixel location of the
lowed by ensemble of two trained models on the entire training ensembled bounding box for corner l. We discard those ensembled
data. In every model, we generated unique predictions as our pri- bounding boxes from the final prediction set which has a confi-
mary focus is on bounding boxes which ‘‘hits”, which was post- dence score less than 0.25.
processed, the parameter values are given in Table 1: The post-
processing step is performed in three phases in which the very initial 3.4. Training
Phase 0 applies non-maximum suppression4 to fold prediction for 2
varied 5-fold assignments which takes three parameters as reported We employ the RSNA pneumonia dataset5 which is a subset of
in Table 1 which identifies maximum sustainable overlap, confidence NIH CXR14 dataset [18]. For identification of pneumonia task, we
threshold and minimum average confidence followed by Phase 1 which have the following examples6 or features from the stage 1 and stage
solicits classification probability to the output of phase 0 which 2 dataset: In Table 2, we have less examples in Stage 1 in comparison
consist of a set containing patient ID and their corresponding to Stage 2 dataset, though the exact number of lung opacity feature
predictions. In Phase 2, we have the confidence threshold of our in Stage 1 dataset is 8964, which is the total number of ground truth
model which adds high confidence outcomes from phase 1 to the bounding boxes, and 5659 is the total number of patients with pneu-
result of our identified set of pneumonia (from our identification monia. Each of these patients has 1–4 ground-truth bounding boxes.
model). Finally, in Phase 3, we convert the Mask-RCNN confidence Similarly, in Stage 2 dataset we have 6012 patient which is caused by
to fitted class probability by taking maximum confidence for every pneumonia and a slightly greater number of lung opacity feature
patient and then merge it with class predictions. We aggregate which is 9555 than Stage 1 dataset.

2 5
https://github.com/matterport/Mask_RCNN/releases/download/v2.0/mask_rc- The RSNA pneumonia dataset can be found athttps://www.kaggle.com/c/rsna-
nn_coco.h5. pneumonia-detection-challenge/data.
3 6
The augmentation step discussed in later section. Here examples represent the patients‘ chest examination during data collection
4
https://github.com/jrosebr1/imutils/blob/master/imutils/object_detection.py. from expert team of radiographers
514 A.K. Jaiswal et al. / Measurement 145 (2019) 511–518

Table 2 Table 3
Features of RSNA dataset. RSNA training and test image set.

# Features Stage 1 Stage 2 # Images Stage 1 Stage 2

Normal 8525 8851 Train Set 25684 26684
Lung Opacity 5659/8964 6012/9555 Test Set 1000 3000
Abnormal 11,500 11,821

that the isolated bounding box is valid then the box will endures
a positive minority belief, in other cases it will be discarded. Ini-
Also, we give a brief overview on training and test data of RSNA
tially, they assigned a confidence score to the bounding boxes.
dataset in these two stages reported in Table 3: The detailed infor-
Also, a low confidence bounding boxes was discarded and high/
mation about the difference between number of images in Stage 1
intermediate boxes was aggregated into a group of appropriate
and 2 is given in later section below.
pneumonia. Given a low probability based bounding box, they dis-
We have examined the training data classifying the positive and
card the box and check whether the labeling is abnormal or no lung
negative features among patients‘ of different age reported in
opacity. The opposed bounding boxes was adjudicated by one of
Fig. 2, whereas the various feature class among patients‘ of differ-
two thoracic radiology practitioners in multi-read cases which
ent age group reported in Fig. 3.
does not consent. Also, the practitioners found that the annotations
of all three readers in adjudicated case is more than 15%. They used
intersection for the rest of the bounding boxes in case of at least
4. Experimental evaluation
50% coincide by one of the bounding boxes, this step has ample
effect in discarding few pixel data for multiple readers including
4.1. Data preparation and augmentation
positive pixels. They used 1500 read cases out of the 4500 triple
cases into the training set to average out few probable distinction
Dataset: We used a large publicly available chest radiographs
among single and multi-read cases. Rest 3000 triple read cases
dataset from RSNA7 which annotated 30,000 exams from the origi-
allocated to the test set. The majority vote is used to distinguish
nal 112,000 chest X-ray dataset [18] to identify instances of potential
weak labels. The radiologists followed the below requirement dur-
pneumonia as a training set and STR8 approximately generated con-
ing data collection:
sensus annotations for 4500 chest X-rays to be used as test data. The
annotated collection contains participants ground truth which fol-
1. Bounding box (Lung Opacity): A patient’s chest radiograph
lows training our algorithm for evaluation. The sets containing
includes finding of fever and cough for potential signs of
30,000 samples is actually made up of 15,000 samples with pneumo-
pneumonia.
nia related labels such as ‘Pneumonia’, ‘Consolidation’, and ‘Infiltra-
2. They made few conjectures during the sabbatical of lateral
tion’, where 7500 samples are chosen randomly with ‘No Findings’
radiography, serial examination and clinical information.
label, and another randomly selected 7500 samples without the
3. Based on Fleischner’s [32] work, they considered every region
pneumonia related labels and ‘No Findings’ label. They created a
which was more opaque than the neighbouring area.
unique identifier for each of those 30,000 samples.
4. They also excluded area such as nodule(s), evident mass(es),
Annotation: Samples were annotated using a proprietary web
linear atelectasis and lobar collapse.
based annotation system and permanently inaccessible to the
other peoples’ annotations. Every radiologists practitioners who Data augmentation: We performed augmentation on lung opac-
took part in training initially executed on the similar set of 50 ities and images data with random scaling including shifting in
exemplary chest X-rays in a hidden manner, and then were visibly coordinate space ððx1 ; y1 Þ; ðx2 ; y2 ÞÞ as well as increasing/decreasing
annotated to the other practitioners for the similar 50 chest X-rays brightness and contrast including blurring with Gaussian blur
for evaluations, as it enable for questions such as does an X-ray under batches. Following these image augmentation, we found
with healed rib fractures enumerate and no enumeration as ‘Nor- images after augmentation reported in Fig. 4.
mal’ and for preliminary calibration. The final sets of label com- Considering the outcome in Figs. 2 and 3 signifies status of
prises of as given in Table 4: There are ‘Question’ labels to patient class and labels from the X-ray images, as we see a highly
suggest questions which will be answered by a chest radiologist imbalanced dataset. The imbalance of training and test dataset
practitioner. Overall coequal distribution of 30,000 human lungs among too many negatives and too few positives generates a crit-
is annotated by six radiologists experts to assess whether the col- ical issue, as we want high recall but the model could predict all
lected images of lungs opacities equivocal for pneumonia with negatives to attain high accuracy and the recall significantly
their analogical bounding box to set forth the status. Also, other undergo. In this case, it is unsure whether or not the present imbal-
twelve experts from STR collaborated in annotating fairly 4500 ance is acceptable or not. We test whether balancing the class dis-
human lungs. Out of 4500 triple read conditions, we divided these tribution would yield any improvement in this case. To do this, we
chest X-rays into three sets containing 1500 human lungs in train- have trained our model on two training data sets, one balanced and
ing set, 1000 in test set (initial stage) and rest 2000 in test set at the other not. We then create the balanced dataset by augmenting
final stage. However, the test sets is double checked by five radiol- more images to the negative (0) class. We discussed previously the
ogist practitioners including six other radiologists from the first augmentation steps which includes flipping, rotating, scaling, crop-
group. ping, translating, and noise adding. Introducing the images in the
Primary consideration: We discuss the adjudicate during the current negative class can possibly create radically new feature
data collection for such sophisticated task. A bounding box is that does not exist in the other class. For example, if we choose
assessed as isolated in multi-read case provided that it does not to flip every negative-class image, then we have in the negative
coincide with the bounding boxes of the other two readers i.e., class a set of images that have the right and left parts of the bodies
these two readers fails to flag that particular area of the image as switched while the other class does not have this feature. This is
being unsure for pneumonia. Whenever the adjudicator concurs not desirable because, the network may learn unnecessary (and
incorrect) features such as the image with the left part of the body
7
Radiological Society of North America. being to a certain side is more likely to exhibit non-pneumonia. So
8
Society of Thoracic Radiology.
 A.K. Jaiswal et al. / Measurement 145 (2019) 511–518 515

Fig. 2. Positive and negative features among patients’ of different age group.

Fig. 3. Feature class of patients’ among different age group.

Table 4 that there are more chest X-ray images of males than females, both
List of labels. genders have the most classes of ‘‘No Lung Opacity/Not Normal”,
Probability Opacity No opacity Abnormal however other than this fact the men are more likely to have a
High Yes No No
class of ‘‘Lung Opacity” where as women are by proportion less
Intermediate Yes No No likely. This clearly explains about the class probability ranking
Moderate Yes No No among the men and women.

we scaled (cropping a little then resizing to the original size) the 4.2. Performance measures
images.
We also classify the distribution of positional view features We employ the mean of the intersection over union (IoU) of
which is a radiographic view allied with the patient position given pairing ground truth bounding boxes and prediction at varied
in training and test data as in Table 5: thresholds. The IoU can be computed from the paired threshold
Data cleaning: We have performed an extensive data cleaning which is the region of the predicted bounding boxes and ground-
on Stage 2 dataset and have explored the class probability ranking truth bounding boxes as an evaluation metric for pneumonia iden-
among males and females which is reported in Fig. 5 this shows tification task. It follows the below formula for IoU:

Fig. 4. Augmentation on chest X-ray images.

516 A.K. Jaiswal et al. / Measurement 145 (2019) 511–518

Table 5
Distribution of positional features in RSNA dataset.

Positional feature Stage 1 Stage 2

Train Set Test Set Train Set Test Set
Anterior/Posterior 20714 468 12173 1382
Posterior/Anterior 15161 532 14511 1618

Fig. 5. Distribution of patient age and sex.

T
Bpredicted Bgroundtruth 1 X
Image
IoU region ðBpredicted ; Bgroundtruth Þ ¼ S ð3Þ MSdataset ¼ MSi ð6Þ
Bpredicted Bgroundtruth jImagej i
The IoU determines a true-positive during pairing of predicted
where Image in the dataset can be either a predicted bounding box
object with ground-truth object above the threshold which ranges
or ground-truth bounding box.
from 0.4 to 0.75 at a step size of 0.05 to classify ‘‘misses” and ‘‘hits”.9
Pairing among predicted bounding boxes and ground-truth bound-
ing boxes is assessed in descending order of the predictions and 5. Evaluation results
strictly injective which is based on their confidence levels.
Given any threshold value, the mean threshold value (MTV) We report our prediction result in this section followed by
over the outcomes for a particular threshold can be computed fol- results from ensemble model.
lowing the counts of true positives ðcTP Þ, false negatives ðcFN Þ, and We perform ensembling in Stage 2 due to labelled dataset,
false positives ðcFP Þ whereas the dataset in Stage 1 was highly imbalanced. The vari-
ance in the dataset is due to radiologists are overlooked with read-
cTPðtÞ ing high volumes of images every shift. We have discussed this in
MTVðtÞ ¼ ð4Þ
cTPðtÞ þ cFPðtÞ þ cFNðtÞ earlier section of this article. In Fig. 6, we overlay the probabilities
of ground truth labels to check whether it is flipped or not. This
Also, we compute mean score (MS) for every image over all thresh-
also shows the successful predictions depicting inconsistency
old values:
between ground-truth and prediction bounding boxes. We trained
1 X our proposed model in Stage 2 on NVIDIA Tesla P100 GPU and
MSi ¼ MTVðtÞ ð5Þ Tesla K80 in Stage 1, which also depicts that one needs an efficient
jThresholdj t
computing resources to model such task on highly imbalanced
Therefore, we can compute the mean score for the dataset as dataset.
follows: The prediction outcome of our model at given threshold is
reported in Table 6, in which the best prediction set of bounding
9
A predicted box hits when it reaches at a threshold of 0.5 provided its IoU with a boxes and ground-truth boxes results in Stage 2. Also, the pre-
ground-truth box is greater than 0.5. dicted sample set depicting pneumonia showing the position
 A.K. Jaiswal et al. / Measurement 145 (2019) 511–518 517

Fig. 6. The results from stage 2 dataset. The probability overlaid on few images which includes all patient classes and labels. Green, orange, blue and red overlays shows
predictions and ground truth labels, respectively. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Table 6 Table 7
Result: prediction at given threshold. Ensemble model results.

Model Threshold Stage 1 Stage 2 Model Stage 2

Mask-RCNN (ResNet50) 0.30 0.098189 0.183720 Mask-RCNN (ResNet50 + ResNet101) 0.218051
Mask-RCNN (ResNet101) 0.97 0.100155 0.199352

With the usage of image augmentation, dropout and L2 regulariza-

(point) and the bounding box for each of the different image types tion prevented the overfitting, but are obtained something weaker
is reported in Fig. 6. The position (point) and the bounding box for results on the training set with respect to the test. Our model can
each of the different image types are reported in The training loss be improved by adding new layers, but this would introduce even
of our proposed model is reported in Fig. 7. more hyperparameters that should be adjusted. We intend to
extend our model architecture in other areas of medical imaging
with the usage of deep learning and computer vision techniques.
5.1. Ensembles

Acknowledgements
Our proposed approach, as illustrated in the beginning of this
section, implies a typical ensemble model after post-processing
We would like to acknowledge the Radiological Society of North
step which is then employed to obtain prediction set of patients’
America for the chest-Xray dataset and Kaggle for computing
having pneumonia. We ensembled our Mask-RCNN based model
infrastructure support.
developed on ResNet50 and ResNet101 and the result is reported
Amit Kumar Jaiswal and Prayag Tiwari has received funding
in Table 7.
from the European Union’s Horizon 2020 research and innovation
programme under the Marie Sklodowska-Curie grant agreement
6. Conclusion and future work No 721321.
Sachin Kumar has received funding from Ministry of Education
In this work, we have presented our approach for identifying and Science of Russian Federation (government order
pneumonia and understanding how the lung image size plays an 2.7905.2017/8.9).
important role for the model performance. We found that the dis- Joel J. P. C. Rodrigues has received funding by the National
tinction is quite subtle for images among presence or absence of Funding from the FCT – Fundação para a Ciência e a Tecnologia
pneumonia, large image can be more beneficial for deeper informa- through the UID/EEA/50008/2019 Project; by RNP, with resources
tion. However, the computation cost also burden exponentially from MCTIC, Grant No. 01250.075413/2018-04, under the Centro
when dealing with large image. Our proposed architecture with de Referência em Radiocomunicações – CRR project of the Instituto
regional context, such as Mask-RCNN, supplied extra context for Nacional de Telecomunicações (Inatel), Brazil; by Brazilian
generating accurate results. Also, using thresholds in background National Council for Research and Development (CNPq) via Grant
while training tuned our network to perform well in the this task. No. 309335/2017-5.

Fig. 7. Training loss: identification of pneumonia in Stage 2.

518 A.K. Jaiswal et al. / Measurement 145 (2019) 511–518

References [16] A. Jamaludin, T. Kadir, A. Zisserman, Spinenet: automatically pinpointing

classification evidence in spinal mris, in: International Conference on Medical
Image Computing and Computer-Assisted Intervention, Springer, 2016, pp.
[1] U. Avni, H. Greenspan, E. Konen, M. Sharon, J. Goldberger, X-ray categorization
166–175.
and retrieval on the organ and pathology level, using patch-based visual
[17] K. Kallianos, J. Mongan, S. Antani, T. Henry, A. Taylor, J. Abuya, M. Kohli, How
words, IEEE Trans. Med. Imaging 30 (3) (2011) 733–746.
far have we come? artificial intelligence for chest radiograph interpretation,
[2] P. Pattrapisetwong, W. Chiracharit, Automatic lung segmentation in chest
Clin. Radiol..
radiographs using shadow filter and multilevel thresholding, in: 2016
[18] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, R.M. Summers, Chestx-ray8: hospital-
International Computer Science and Engineering Conference (ICSEC), IEEE,
scale chest x-ray database and benchmarks on weakly-supervised
2016, pp. 1–6.
classification and localization of common thorax diseases, in: Proceedings of
[3] S. Katsuragawa, K. Doi, Computer-aided diagnosis in chest radiography,
the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp.
Comput. Med. Imaging Graph. 31 (4–5) (2007) 212–223.
2097–2106.
[4] Q. Li, R.M. Nishikawa, Computer-aided Detection and Diagnosis in Medical
[19] P. Rajpurkar, J. Irvin, K. Zhu, B. Yang, H. Mehta, T. Duan, D. Ding, A. Bagul, C.
Imaging, Taylor & Francis, 2015.
Langlotz, K. Shpanskaya, et al., Chexnet: radiologist-level pneumonia detection
[5] C. Qin, D. Yao, Y. Shi, Z. Song, Computer-aided detection in chest radiography
on chest x-rays with deep learning. arXiv:1711.05225..
based on artificial intelligence: a survey, Biomed. Eng. Online 17 (1) (2018)
[20] J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurea-Ilcus, C. Chute, H. Marklund, B.
113.
Haghgoo, R. Ball, K. Shpanskaya, et al., Chexpert: a large chest radiograph
[6] A.A. El-Solh, C.-B. Hsiao, S. Goodnough, J. Serghani, B.J. Grant, Predicting active
dataset with uncertainty labels and expert comparison. arXiv:1901.07031..
pulmonary tuberculosis using an artificial neural network, Chest 116 (4)
[21] S. Ren, K. He, R. Girshick, J. Sun, Faster r-cnn: towards real-time object
(1999) 968–973.
detection with region proposal networks, Adv. Neural Inf. Process. Syst. (2015)
[7] O. Er, N. Yumusak, F. Temurtas, Chest diseases diagnosis using artificial neural
91–99.
networks, Expert Syst. Appl. 37 (12) (2010) 7648–7655.
[22] J. Long, E. Shelhamer, T. Darrell, Fully convolutional networks for semantic
[8] O. Er, C. Sertkaya, F. Temurtas, A.C. Tanrikulu, A comparative study on chronic
segmentation, in: Proceedings of the IEEE conference on computer vision and
obstructive pulmonary and pneumonia diseases diagnosis using neural
pattern recognition, 2015, pp. 3431–3440.
networks and artificial immune system, J. Med. Syst. 33 (6) (2009) 485–492.
[23] K. He, G. Gkioxari, P. Dollár, R. Girshick, Mask r-cnn, in: Proceedings of the IEEE
[9] S. Khobragade, A. Tiwari, C. Patil, V. Narke, Automatic detection of major lung
International Conference on Computer Vision, 2017, pp. 2961–2969.
diseases using chest radiographs and classification by feed-forward artificial
[24] J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, Z. Wojna,
neural network, in: 2016 IEEE 1st International Conference on Power
Y. Song, S. Guadarrama, et al., Speed/accuracy trade-offs for modern
Electronics, Intelligent Control and Energy Systems (ICPEICES), IEEE, 2016,
convolutional object detectors, in: Proceedings of the IEEE Conference on
pp. 1–5.
Computer Vision and Pattern Recognition, 2017, pp. 7310–7311.
[10] G.G.R. Schramek, D. Stoevesandt, A. Reising, J.T. Kielstein, M. Hiss, H. Kielstein,
[25] A. Shrivastava, R. Sukthankar, J. Malik, A. Gupta, Beyond skip connections: top-
Imaging in anatomy: a comparison of imaging techniques in embalmed
down modulation for object detection. arXiv:1612.06851..
human cadavers, BMC Med. Educ. 13 (1) (2013) 143.
[26] P. Luc, C. Couprie, Y. Lecun, J. Verbeek, Predicting future instance segmentation
[11] J. Li, Z. Liang, S. Wang, Z. Wang, X. Zhang, X. Hu, K. Wang, Q. He, J. Bai, Study on
by forecasting convolutional features, in: Proceedings of the European
the pathological and biomedical characteristics of spinal cord injury by
Conference on Computer Vision (ECCV), 2018, pp. 584–599.
confocal raman microspectral imaging, Spectrochim. Acta Part A Mol. Biomol.
[27] A. Arnab, P.H. Torr, Pixelwise instance segmentation with a dynamically
Spectrosc. 210 (2019) 148–158.
instantiated network, in: Proceedings of the IEEE Conference on Computer
[12] D.J. Winkel, T. Heye, T.J. Weikert, D.T. Boll, B. Stieltjes, Evaluation of an ai-
Vision and Pattern Recognition, 2017, pp. 441–450.
based detection software for acute findings in abdominal computed
[28] T.-Y. Lin, P. Goyal, R. Girshick, K. He, P. Dollár, Focal loss for dense object
tomography scans: toward an automated work list prioritization of routine
detection, in: Proceedings of the IEEE International Conference on Computer
ct examinations, Invest. Radiol. 54 (1) (2019) 55–59.
Vision, 2017, pp. 2980–2988.
[13] H.R. Roth, L. Lu, A. Seff, K.M. Cherry, J. Hoffman, S. Wang, J. Liu, E. Turkbey, R.M.
[29] J. Redmon, A. Farhadi, Yolov3: an incremental improvement. arXiv:1804.
Summers, A new 2.5 d representation for lymph node detection using random
02767..
sets of deep convolutional neural network observations, in: International
[30] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in:
Conference on Medical Image Computing and Computer-assisted Intervention,
Proceedings of the IEEE Conference on Computer Vision and Pattern
Springer, 2014, pp. 520–527.
Recognition, 2016, pp. 770–778.
[14] H.-C. Shin, H.R. Roth, M. Gao, L. Lu, Z. Xu, I. Nogues, J. Yao, D. Mollura, R.M.
[31] R. Girshick, Fast r-cnn, in: Proceedings of the IEEE International Conference on
Summers, Deep convolutional neural networks for computer-aided detection:
Computer Vision, 2015, pp. 1440–1448.
Cnn architectures, dataset characteristics and transfer learning, IEEE Trans.
[32] D.M. Hansell, A.A. Bankier, H. MacMahon, T.C. McLoud, N.L. Muller, J. Remy,
Med. Imaging 35 (5) (2016), 1285–computing1298.
Fleischner society: glossary of terms for thoracic imaging, Radiology 246 (3)
[15] O. Ronneberger, P. Fischer, T. Brox, U-net: convolutional networks for
(2008) 697–722.
biomedical image segmentation, in: International Conference on Medical
Image Computing and Computer-assisted Intervention, Springer, 2015, pp.
234–241.