International Journal of Computer Assisted Radiology and Surgery

https://doi.org/10.1007/s11548-017-1696-0

ORIGINAL ARTICLE

Agile convolutional neural network for pulmonary nodule

classification using CT images
Xinzhuo Zhao1 · Liyao Liu1 · Shouliang Qi1 · Yueyang Teng1 · Jianhua Li1 · Wei Qian1,2

Received: 2 June 2017 / Accepted: 20 December 2017

© CARS 2018

Abstract
Objective To distinguish benign from malignant pulmonary nodules using CT images is critical for their precise diagnosis
and treatment. A new Agile convolutional neural network (CNN) framework is proposed to conquer the challenges of a
small-scale medical image database and the small size of the nodules, and it improves the performance of pulmonary nodule
classification using CT images.
Methods A hybrid CNN of LeNet and AlexNet is constructed through combining the layer settings of LeNet and the
parameter settings of AlexNet. A dataset with 743 CT image nodule samples is built up based on the 1018 CT scans of LIDC
to train and evaluate the Agile CNN model. Through adjusting the parameters of the kernel size, learning rate, and other
factors, the effect of these parameters on the performance of the CNN model is investigated, and an optimized setting of the
CNN is obtained finally.
Results After finely optimizing the settings of the CNN, the estimation accuracy and the area under the curve can reach 0.822
and 0.877, respectively. The accuracy of the CNN is significantly dependent on the kernel size, learning rate, training batch
size, dropout, and weight initializations. The best performance is achieved when the kernel size is set to 7 × 7, the learning
rate is 0.005, the batch size is 32, and dropout and Gaussian initialization are used.
Conclusions This competitive performance demonstrates that our proposed CNN framework and the optimization strategy
of the CNN parameters are suitable for pulmonary nodule classification characterized by small medical datasets and small
targets. The classification model might help diagnose and treat pulmonary nodules effectively.

Keywords Lung cancer · Nodule classification · Deep learning · Convolutional neural network

Introduction
Liyao Liu: Joint first author. Lung cancer is the leading cause of cancer deaths in the world
B Shouliang Qi [1]. It has become the first killer among cancers in China,
[email protected] partially due to the asymptomatic growth of this cancer [2,3].
Xinzhuo Zhao In the majority of cases, it is too late for successful therapy
[email protected] once the patient develops the first symptoms. However, there
Liyao Liu is a survival rate of 47% if the lung cancer is detected early
[email protected] according to the American Cancer Society. Therefore, early
Yueyang Teng determination of whether a pulmonary nodule is benign or
[email protected] malignant is important.
Jianhua Li Computed tomography (CT) scanners can provide contin-
[email protected] uous high-resolution, near-isotropic thin sections throughout
Wei Qian the lungs in a single-breath hold. These CT images delineate
[email protected] the location, size and shape of the suspicious pulmonary nod-
1 Sino-Dutch Biomedical and Information Engineering School,
2 College of Engineering, University of Texas at El Paso,
Northeastern University, Life Science Building, 500 Zhihui
Street, Hun’nan District, Shenyang 110169, China 500 West University Avenue, El Paso, Texas 79968, USA

123
 International Journal of Computer Assisted Radiology and Surgery

ules [4]. Via imaging processing techniques, some computer (SDAE). Cheng et al. [18] developed an SDAE CAD system
aided diagnosis (CAD) systems have been implemented to with an accuracy of 94.40%. Kumar et al. [19] utilized the
estimate the malignance of the detected pulmonary nodules autoencoder to extract image features and used the decision
[5]. In these systems, after the lung nodules are segmented, tree to realize classification. Wei et al. [20] utilized multi-
various types of image features (e.g., intensity) are extracted scale CNNs to capture features from raw nodule patches and
[6]. Then, machine-learning classifiers are used to predict the classified the nodules with SVM, achieving an accuracy of
malignance [7]. However, several challenges have faced these 86.84%. They further proposed a multi-crop CNN to increase
handcrafted feature-based CAD systems. First, the hand- the accuracy to 87.14% [21].
crafted features depend on the segmentation of the lung To further increase the accuracy of classification of lung
nodule. However, this step is challenging and contentious nodules, the new CNN network architecture and optimiza-
because whether there is ground truth is open to debate and tion strategy for the learning parameters are required. In this
the reproducibility of the segmentation is contingent [8]. paper, the Agile CNN architecture, which is suitable for small
Second, the handcrafted features are based on prior knowl- datasets of lung nodule CT images, is proposed and imple-
edge, which is dependent on the ability of the designers of mented. In the Agile CNN, which has only two convolutional
the CAD system. These challenges make the handcrafted layers, a small number of kernels (20 kernels in C1 and 1000
feature-based CAD systems difficult for clinical applications. kernels in C2) are adopted. Compared with those famous
Deep learning, especially the convolutional neural net- deep or deeper structures, such as the GoogleNet, ResNet,
work (CNN), might have the potential to address the afore- and VGGNet, the Agile structure has relatively fewer layers
mentioned challenges, considering its significant success in (only 2 convolutional layers), and thus, it is called the “Agile”
object recognition and localization in nature images [9]. CNN. Additionally, the number of parameters to calculate for
One of the advantages of the CNN is that it can be fed the training is determined by the number of the layers, the
raw images without previous image preprocessing, which number of the kernels, and the sizes of the kernels. In addi-
is highly amenable to image analysis. Deep learning consists tion, more parameters for training require more input data,
of increased numbers of layers, which permits higher levels which is not always feasible in medical applications. Thus,
of abstraction and improved predictions from data [10]. the strategy for the optimization of learning parameters of
Many deep learning networks with more layers and flexi- CNNs is clarified to increase the accuracy of the classification
ble structures have been proposed since LeNet-5 [11]. For and to avoid overfitting at the same time. Finally, a classifica-
example, AlexNet [12] contains eight learned layers, and tion model for classifying the malignant pulmonary nodules
VGG-VD [13] has 16-layer and 19-layer CNN structures. from the others based on CT scan images is obtained.
GoogLeNet [14], a 22-layer deep network that contains
inception architectures, is proposed to manage the con-
tradiction between increasing the training parameters and Materials and methods
overfitting. ResNet [15] is approximately 20 times deeper
than AlexNet and 8 times deeper than VGGNet. By increas- Dataset of lung nodule CT images
ing the depth, the network can better approximate the target
function with increased nonlinearity and achieve better fea- The images in the current study are generated from the
ture representations. Lung Image Database Consortium image collection (LIDC-
The applications of the CNN to medical images are quite IDRI) [22–24]. So far, it contains 1018 cases. Each subject
different from those for nature images in several respects. includes images from a clinical thoracic CT scan and an asso-
CNN requires a large number of labeled training data acting ciated XML file that records the results of a two-phase image
as ImageNet. However, large datasets are not always avail- annotation process performed by four radiologists. Each radi-
able because of the extremely expensive expert annotations ologist independently reviewed each CT scan and marked
and scarcity of the disease images [9]. Moreover, instead lesions that belonged to one of three categories (“nodule >
of containing RGB channels as in natural images, medical or = 3 mm”, “nodule < 3 mm” and “non-nodule >
images are grayscale images. or = 3 mm”). Then, the nodules are marked with 5 malig-
These important differences between medical and nature nancy levels, from 1 to 5.
images have prompted investigators to study whether CNNs To generate the training dataset, several steps are implied.
can be used effectively for lung nodule classification. Hua First, we select nodules that are larger than 3 mm for this
et al. [16] first introduced the CNN to nodule classification study. Since each nodule is labeled by four radiologists, those
in CT images and found that it outperforms the conven- that are recognized by fewer than three of the radiologists are
tional handcrafted feature-based CAD frameworks. Sun et al. eliminated. At the same time, we label the nodules according
[17] found that the deep belief networks (DBN) performed to their malignancy levels, i.e., the average rating of the four
best followed by CNNs and stacked denoising autoencoder radiologists. Levels 1 to 2.5 are considered to be benign, and

123
International Journal of Computer Assisted Radiology and Surgery

Fig. 1 Schematic structure of

the proposed Agile CNN, LeNet
and AlexNet. a Presents the
schematic structure of the
proposed agile CNN. It is a
hybrid structure of LeNet and
AlexNet, combining the layer
settings of LeNet and the
parameter settings of AlexNet. b
Shows the schematic structure
of LeNet. LeNet has two
convolutional layers, two
pooling layers, and two fully
connected layers, and it is
designed for the image size of
28 × 28. c AlexNet is designed
for the image size of 256 × 256.
It has more convolutional layers,
with the layers of ReLU, LRN,
and dropout in this framework

levels 3.5 to 5 are denoted as malignant, and there is elimina- nodule areas are annotated based on the union of the radiolo-
tion of all of the intermediate cases (level 3). In total, there are gists’ truth files, obtaining the minimum bounding rectangle
743 nodules left, with 375 malignant nodules and 368 benign of each slice. The size of the cropped patch of slices is fixed at
nodules. Then, because of the varying image resolutions, the 53 by 53. Instead of centralizing the nodules, they are located
nodules are resampled using spline interpolation with a fixed at random positions of the patch. To utilize the background
resolution with 0.5 mm/voxel along two axes [25]. Third, the information, the surrounding pixels are preserved. Since the

123
 International Journal of Computer Assisted Radiology and Surgery

Table 1 Computer environment Inspired by AlexNet, two convolutional layers in our

Computer environment Detail framework are followed by a rectified linear unit (ReLU)
layer and a local response normalization (LRN) layer [27].
Computer HP Z840 A deep CNN with ReLU trains several times faster than
GPU NVIDIA Quadro M2000 4 GB those with sigmoid and other logistic functions. ReLU is a
CPU Intel Xeon E5-2640v3 2.60 GHz non-saturating neuron, which avoids the problem of gradient
Memory RAM 64G vanishing [12]. The non-saturating nonlinearity of ReLU can
Operating system Linux ubuntu 16.04 64-bit be shown as
CUDA CUDA 8.0
cuDNN cuDNN 5.1 ϕ (x) = max (0, x) (1)

In addition to the ReLU layer, the LRN scheme aids the gen-
eralization of the network. The performance of LRN appears
CNN used in this paper is a 2D structure, each slice of a to be a type of “lateral inhibition.” At the LRN layer, each
nodule is cropped as a patch. i
input ax,y is divided by an expression:

  α   β
bix,y = ax,y
i
/ 1+ x2 (2)
CNN experiment n i i

i
Here, ax,y is the input neuron at position (x, y) applied by
CNN architecture
kernel i, where the sum runs over the adjacent kernel maps at
The CNN architecture consists of a number of convolutional the same spatial position. There are two modes of LRN [28]:
and pooling layers optionally followed by fully connected one is the in-channel mode and the other is the cross-channel
layers. The convolutional layer is composed of several small mode. Here, the cross-channel mode is selected with n = 5,
matrices or “kernels” that are convolved throughout the α = 0.0001 and β = 0.75 as in AlexNet’s set.
whole input image, which work as filters. The output of this Motivated by Srivastava [29], the dropout is applied in F1,
convolution is called a “feature map”. These feature maps are while setting the output hidden neuron to zero with a proba-
the input for the pooling layer, which aggregates contiguous bility of 0.5. The role is to reduce the complex coadaptations
values to one scalar with functions such as mean or max [26]. of the neurons in F1.
In the following parts, the convolutional layers are labeled
Cx, the pooling layers Px, and the fully connected layers Fx, Parameters for optimization
where x is the layer index.
The Agile CNN framework is proposed in the current After determining the architecture of the CNN, another
study, as shown in Fig. 1. It is a hybrid structure of LeNet and important task is to optimize the parameters to improve the
AlexNet, which combines the layer settings of LeNet and the performance of the proposed CNN for lung nodule classifi-
parameter settings of AlexNet. In other words, we start from cation. There are four main parts: (1) kernel size; (2) learning
the LeNet framework, add the layers of ReLU, LRN, and rate; (3) batch size; and (4) weight initialization.
dropout into this framework, and construct the Agile CNN.
A. Kernel size
Inspired by LeNet, the proposed CNN has two convolutional
The kernel size and the kernel number are two significant
layers, two pooling layers, and two fully connected layers.
parameters that affect the learning efficiency of the system.
Layer C1 has 20 feature maps. Every unit in each feature
The number of parameters to be learned is proportional to
map is connected to a 7 × 7 neighborhood in the input. The
the kernel size, the previous kernel number and the current
size of the input patch is 53 × 53, and the size of the feature
kernel number. In our architecture, the number of learned
maps in C1 is 47 × 47, which prevents a connection from
parameters can be calculated as 7 × 7 × 20 × 50, which is
the input from falling out of the boundary. In P1, every unit
49,000. The number of learned parameters must adapt to the
in each feature map is connected to a 2 × 2 neighborhood in
number of training images, which not only guarantees the
the corresponding feature map in C1. Then, layer C2 has 50
richness of image features but also avoids overfitting.
feature maps. The other settings are the same as the previous
layers. Finally, F1 and F2 follow after layer P2. The number B. Learning rate
of neuron units in F1 and F2 is 500 and 2, respectively. Beyond choosing a single global learning rate, it is clear
The experiment environment is listed in Table 1. The that picking a different learning rate η can improve the con-
platform that we work on is Caffe 1.0 (Convolutional Archi- vergence. Whenever the loss function stops to decay, the
tecture for Fast Feature Embedding). learning rate is multiplied by a factor γ . During the whole

123
International Journal of Computer Assisted Radiology and Surgery

training experiment, the learning rate has decayed for several Table 2 Summary of the optimized parameters
times. In each instance of decay, the learning rate η must be Parameter Variation
multiplied by γ . In Caffe, the decay time of the learning rate
is defined as a “step.” Kernel size C1 C2
3×3 3×3
C. Batch size
5×5 5×5
The ability to perform generalization by a network also
7×7 7×7
relates to the batch size [30]. CNNs with large batch sizes tend
9×9 9×9
to make the training and testing functions converge to sharp
9×9 7×7
minimizers, which leads to the neural network to having poor
7×7 5×5
generalizability. In contrast, CNNs with small batch sizes
5×5 3×3
consistently converge to flat minimizers.
Learning rate 0.01
D. Weight initialization 0.005
The initial values of the weights have a significant effect 0.001
on the training process. If the randomly chosen weights are 0.0005
all very large, then the ReLU will saturate, which results 0.0001
in small gradients that make learning slow. If the randomly Decay times (learning rate) 4
chosen weights are very small, then the gradients will also 4.5
be very small. Intermediate weights with a Gaussian distri-
5
bution with a mean of 0 and a standard deviation of 0.01
5.5
have two advantages: (1) the gradients are sufficiently large
6
that learning can proceed, and (2) the network will learn the
Batch size 32
linear part of the mapping before the more difficult nonlinear
64
part.
Weight initialization Xavier
Gaussian 0.001
Training, testing, and parameter optimization
0.005
We train and evaluate CNNs using tenfold cross-validation. Bias initialization Constant 1
The 743 nodules are split into training, validation, and testing Constant 0
datasets. In each fold of the cross-validation, 10% patients Dropout Yes
are used to test the architecture. To augment the training No
and validating datasets, each slice is cropped four times ran- The bold indicates the optimized parameters which are adopted in the
domly and rotated three times with the angles of 90, 180, current study
and 270 degrees. Each of them is flipped horizontally and
vertically. Data augmentation is not applied to the testing
For the weights of C1, C2, and F1, an experiment is per-
dataset. To evaluate the effect of each parameter on the per-
formed to compare the Xavier and Gaussian initialization
formance of the CNN, several groups of control experiments
methods. The standard deviation of the Gaussian is also
were designed. During the experiments, the same conditions
shifted from 0.001 to 0.005 for further experiments. In addi-
were maintained except for in one particular factor, and then,
tion, a constant (0 or 1) is used to initialize the bias of C1,
the effect of this varied factor was evaluated.
C2, F1 and F2.
To study the effect of the kernel size on the performance
of the proposed CNN, it was varied from 3 × 3 to 9 × 9 while
freezing the other parameters. The variation of the kernel size Results
is described in detail in Table 2. This group of experiments
is used to observe the effect of different convolution kernel CNN classification performance
sizes on the performance of our architecture.
The effect of the learning rate and the number of steps After the comparison of the layers and parameter settings,
is also investigated. The variation in the learning rate and the Agile CNN structure was optimized. It contains two con-
the decay times (steps) of the learning rate during the train volutional layers and two fully connected layers, all of which
iteration process are exhibited in Table 2. The decay factor are followed by ReLU and LRN. Finally, we set two convo-
γ is set to 0.1. Moreover, the effect of the batch size is also lutional layer kernel sizes to be 7 × 7. Dropout is used in
studied while altering it from 32 to 64. Along with the batch the fully connected layer. The base learning rate is 0.0005,
size, the influence of dropout is also checked. with a 5.5 times reduction to diminish the learning speed.

123
 International Journal of Computer Assisted Radiology and Surgery

of LeNet, which is 0.648, and that of AlexNet, which is

0.782. The T -test and Wilcoxon signed ranks test are used
to compare the classification accuracy of the Agile CNN
structure with that of LeNet and AlexNet. The significance
test results are shown in Table 3. It was found that the test
accuracy of the Agile CNN structure is significantly higher
than that of LeNet (T -test, p < 0.05) and that of AlexNet
(Wilcoxon signed ranks test, p < 0.08). Figure 5 presents
the Receiver Operating Characteristic (ROC) curve of the
test result of the Agile CNN structure for nodule classifica-
Fig. 2 Visualization of the 20 kernels in the C1. There are 20 kernels
with the size of 7 × 7 in the first convolutional layer. The kernels are tion.
presented in 4 rows and 5 columns without any particular order Table 4 summarizes the methods and accuracy reported
in recently published papers using deep learning algorithms
based on the LIDC dataset. The accuracy of our method is
The batch size is 32. The weights of the C1, C2 and F2 are higher than that of Kumar et al. [19] and Sun et al. [17], but
initialized by Gaussian, with a standard deviation of 0.01. lower than that of Wei et al. [20,21].
For the initialization of the weights of the F1, the standard
deviation of the Gaussian is 0.001. The bias of the C2 and The effect of different parameter settings
F1 is initialized as the constant of 0.1, and the bias of the C1
and F2 is initialized as the constant of 0.0. Figure 6 shows the accuracy of validation with iterations for
The CNN algorithm helps the computer learn its own fea- seven structures with different kernel sizes. The kernel size
tures, instead of using handcrafted features. The visualization of 7 × 7 for two convolutional layers is the best, as indicated
of the weights is one of the important methods of evaluating by the red crossline. The relationship between the accuracy
the features. Figure 2 shows the visualization of the weights and kernel size is not monotonic: the accuracy increases with
in C1. These kernels not only present a noise pattern but the kernel size while it is smaller than 9 × 9, it reaches the
also exhibit a high correlation. Additionally, they also lack peak at the size of 7 × 7, and it decreases with increasing
structural patterns. These characteristics implicate under- size.
training. The features in C2, as shown in Fig. 3, obviously Table 5 shows the validation accuracy of the proposed
perform better, displaying a smoother pattern, containing CNN with different learning rates and steps. On the left side
more shape information, and demonstrating that C2 is well of Table 5, the learning rate (Lr) remains unchanged (constant
trained. learning speed) when the learning “step” equals zero during
After 50,000 iterations, 10% of the nodules are utilized one whole iteration. It is found that the validation accuracy
to evaluate the performance of our framework. We obtain of the last row (Lr = 0.0001) is higher compared with the
the final classification model with a test accuracy of 0.822 former rows. However, the loss of the last row is also larger,
and an AUC of 0.877. Figure 4 shows the test accuracy of which indicates that the validation process starts overfitting.
the Agile CNN structure, LeNet, and AlexNet. The Agile The right side of Table 5 shows the performance of CNN
CNN has an accuracy of 0.822, which is higher than that with different steps and a constant learning rate of 0.0005.

Fig. 3 Visualization of the 1000 kernels in the C2. There are 1000 kernels with the size of 7 × 7 in the second convolutional layer. The kernels are
presented in 20 rows and 50 columns without any particular order

123
International Journal of Computer Assisted Radiology and Surgery

Table 4 Summary of the methods and accuracy of the recently pub-

lished papers using LIDC as a dataset
Experiments Year Method Accuracy (%)

Wei et al. [21] 2017 Multi-crop CNN 87.14

Multi-scale CNN 86.53
CNN 86.32
Sun et al. [17] 2016 CNN 79.76
CNN 81.19
SDAE 79.29
Wei et al. [20] 2015 Multi-scale CNN to 86.84
extract feature with
the SVM classifier
Kumar et al. [19] 2015 Autoencoder to 75.01
Fig. 4 Test accuracy of the Agile CNN structure, LeNet and AlexNet. extract feature with
The Agile CNN has an accuracy of 0.822, which is higher than that of the decision tree
LeNet, which is 0.648, and AlexNet, which is 0.782 classifier
Our method CNN 82.23
Table 3 Significance tests of the classification accuracy of the Agile
CNN structure, LeNet, and AlexNet
Model T test Wilcoxon signed ranks or without dropout. There are mainly four observations. First,
test it is shown that dropout can reduce the loss value and prevent
Our CNN structure p = 5.52 × e − 05 p = 2.45 × e − 04 overfitting. Second, the CNN with a batch size of 32 performs
versus LeNet better than that with a batch size of 64. Third, Xavier per-
Our CNN structure p = 0.135 p = 0.076 forms worse than the Gaussian, and thus, a Gaussian with a
versus AlexNet constant bias should be used to initialize the weights. Fourth,
for the initialization of the weight of F1, a standard devia-
tion of the Gaussian of 0.001 is better than 0.005, as set in
With each reduction step, the learning rate is multiplied by AlexNet. For the initialization of the bias of the C2 and F1,
0.1. Finally, the base learning rate of 0.0005 and step of 5.5 the constant of 0 is better than 1, as set in ImageNet.
achieves the best result, which not only obtains the highest
accuracy but also reduces the loss value.
Figure 7 shows the validation accuracy of the experi- Discussion and future work
mented CNNs with different validation batch sizes and with
In this paper, we proposed the Agile CNN for pulmonary nod-
ule malignancy classification using CT images. This Agile
CNN is designed to be a feature extractor and classifier. The
parameters of the CNN are optimized through a series of
experiments to obtain a high accuracy and low loss. The
results demonstrate that the Agile CNN presents competitive
performance. From our viewpoint, three main contributions
are included in the current work. First, a method is proposed
through combining LeNet and AlexNet into the Agile CNN.
Second, a strategy for optimizing the parameters in the CNN
is developed. Third, a model for classification is obtained for
discriminating the benign and malignant lung nodules using
CT images.

CNN framework and performance

For the CNN layer setting, the hybrid structure is based on

Fig. 5 Receiver operating characteristic (ROC) curve of the test accu- the improved LeNet [11] and AlexNet [12]. The LeNet CNN
racy for the proposed nodule classification model model is applied to check the handwriting and is suitable

123
 International Journal of Computer Assisted Radiology and Surgery

Fig. 6 Validation accuracy of various kernel sizes in the proposed Fig. 7 Validation accuracy of various batch sizes, dropout and weights
nodule classification model. The red cross line achieves the highest and bias initialization in the proposed nodule classification model. Com-
accuracy, which means that the kernel size of 7 × 7 is the best. Based paring the red cross line with the dark blue line, it is found that dropout
on all of the lines, there is a tendency that the accuracy increases with helps to increase the validation accuracy. The batch size of 64 (black
the size of the kernel, and it reaches the highest accuracy. Furthermore, line) underperforms the batch size of 32 (dark blue). The pink line
by observing the red cross line, it is found that the accuracy increases presents the Xavier initialization, which underperforms that initialized
quickly at the beginning. After a small fluctuation, the accuracy tends by the Gaussian method (the red cross line). The azure line (the stan-
to be stable. With more iterations, the accuracy does not decrease. This dard deviation of the Gaussian is 0.001) and the green line (the standard
finding illustrates that the model does not overfit deviation of the Gaussian is 0.0.005) are used to show the influence of
the standard deviation of the Gaussian. It is proven that the standard
deviation of the Gaussian being 0.001 is better than 0.005
Table 5 Validation accuracy of the proposed CNN with different learn-
ing rates and steps
et al. [20,21]. There are three possible reasons: (1) The 3D
Learning rate (Step = 0) Accuracy Step Accuracy
(Learning rate = 0.0005) CNN and multi-scale strategy is used [20,21]; (2) The more
complicated network with a multi-crop strategy is adopted;
0.01 0.7576 4 0.8424 (3) more features, including histogram of oriented gradient
0.005 0.7576 4.5 0.8462 (HOG) and local binary patterns (LBP), are extracted and
0.001 0.8245 5 0.8404 combined with the features extracted using CNN.
0.0005 0.8408 5.5 0.8564
0.0001 0.8449 6 0.8503 CNN parameter settings
The bold indicates the optimized parameters which are adopted in the
current study To further improve the performance, we optimized the
parameter settings of the CNN. For the kernel size, Sun et
for a small dataset and images with a small size, such as al. [17] utilized kernel sizes of 12, 8, and 6. Setio et al. [31]
our experimental dataset. However, the AlexNet model uti- designed their structure with 5, 3, and 3. Compared with the
lizes a deeper CNN that is suitable for a large dataset with three-layer CNN structure, our two-layer structure uses larger
large input images. For the CNN structure used in medical kernels, which are 7 and 7. The large size of the kernels can
image classification tasks, Sun et al. [17], Setio et al. [31], create a wider receptive field. It is also different from Shin et
and He et al. [15] have all built architectures with three con- al. [32], which set the batch size at 50, Sun et al. [17] at 100,
volutional layers. Based on our experiment with a two-layer and Hinton et al. [33] and Setio et al. [31] at 128, and our
CNN, three layers are more likely to overfit. Additionally, a smaller batch size of 32, as demonstrated above, performs
structure with two convolutional layers followed by two fully better. For the initialization of the weights, Xavier does not
connected layers can obtain a higher accuracy. This proposed have an effect on the network convergence compared with
structure performs well for a small-scale and small-size med- the Gaussian initialization, based on our observations.
ical dataset.
The deep learning framework can have an important Evaluation of the misclassified samples
impact on the accuracy of the lung nodule classification. Our
results have shown that the proposed CNN achieves better Figure 8 gives some patches of true negative (TN), false pos-
performance than the Autoencoder to extract features with itive (FP), true positive (TP), and false negative (FN) results.
the Decision Tree classifier [19] and DBNs, as well as SDAE The first row is TN, and the third row is TP. One general
[17]. Even using the CNNs, the framework of our model is feature can be found: the small-size nodules with regular
more suitable for lung nodule classification than that of Sun surroundings belong to the negative (or benign) class; the
et al. [17], which has three convolutional levels. However, it large size nodules with irregular surroundings have a higher
is noted that our methods cannot reach the accuracy of Wei probability of belonging to the positive (or malignant) class.

123
International Journal of Computer Assisted Radiology and Surgery

Fig. 8 Examples for the lung

nodule classification results. The
first and the third rows are
examples of correctly classified
and the second and fourth rows
are misclassified

The nodules in the second row, which are labeled benign, are learning. Further research, such as transfer learning and fine-
classified as malignant by mistake. Except for the nodules in tuning, is needed.
the second and third columns, the other nodules in this row Finally, performing diagnosis based on only medical
are very small. The last row is for the FN samples. The com- images has its own limitations. Usually, doctors reach a con-
mon feature of these six images is that their nodule sizes are clusion based on many types of medical information. It is an
all small, which is easily misidentified as benign nodules. impossible mission to affirm whether the nodule is benign or
As seen in Fig. 8, the size of the nodule appears to be the malignant solely based on medical images.
prime reason for misclassification. To validate this observa- In addition to the methods mentioned above to conquer
tion, the average sizes of the bounding rectangle of the four the drawbacks, more innovative studies will be performed
classified categories are measured. The average size of the for further research. First, we prefer to utilize the multi-
bounding rectangle of TN, FP, TP, and FN is 10.9, 11.3, 22.3, modality strategy, which combines the general CT scan with
and 14.8. The difference in the average size of the bounding the contrast-enhanced CT scan to determine the malignance
rectangle between TN and FP is not very obvious, yet there of a nodule. The contrast-enhanced images usually contain
is a trend in that the larger size benign nodules have a greater more information about the vessel distribution and can dis-
probability of being classified as malignant and the small- tinguish tissues from lung effusion. These ample input data
size malignant nodules are more likely to be classified as contribute to a more accurate result. Second, instead of using
benign. the CNN, many other supervised deep learning methods can
be utilized to classify a nodule or tumor, such as deep rein-
forcement learning, generative adversarial nets. These deep
learning models can be used to extract the features of medi-
Limitations and future work cal images, which are significant for image analysis. Third,
features that are gathered from different training models can
One of the limitations of this work is that the three chan- be fused as input data to SVMs or other classifiers. Finally, if
nels of input are homogeneous. Both LeNet and AlexNet are data acquisition is available, then the medical imaging infor-
designed for color images, while our medical images are gray mation will combine genomic knowledge to contribute to
scale images, which results in an inability to make full use better diagnosis.
of all channels.
Another limitation is that the current CNN classifier uti-
lizes the independent 2D patch as the input.
The misclassified patches shown in Fig. 8 are also difficult Conclusions
for a radiologist to diagnose, because the candidate nodules
are diagnosed based on the information in the front and back In this paper, we constructed one new Agile CNN for pul-
slices. As a result, 2.5D or 3D input will be used in the future. monary nodule classification using CT images, on which we
Moreover, the original sample patients in the LIDC dataset investigated the effects of kernel size, learning rate, training
total to only 1018. Compared with the natural images in Ima- batch size, dropout, and weight initialization on the accuracy
geNet, it has too small a number to be calculated by deep and loss of the proposed CNN model. This Agile structure

123
 International Journal of Computer Assisted Radiology and Surgery

achieves a relatively high accuracy of 0.822 and an AUC of 8. Gillies RJ, Kinahan PE, Hricak H (2015) Radiomics: images are
0.877, and it is less prone to overfitting, which is associated more than pictures, they are data. Radiology 278(2):563. https://
doi.org/10.1148/radiol.2015151169
with other redundant CNN structures. The results have shown 9. Greenspan H, van Ginneken B, Summers RM (2016) Guest edi-
that the proposed CNN framework and the optimization strat- torial deep learning in medical imaging: overview and future
egy for the CNN parameters might be suitable for pulmonary promise of an exciting new technique. IEEE Trans Med Imaging
nodule classification when characterized by small medical 35(5):1153–1159. https://doi.org/10.1109/TMI.2016.2553401
10. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature
datasets and small targets. The classification model might 521(7553):436–444. https://doi.org/10.1038/nature14539
help to diagnose and treat pulmonary nodules effectively, 11. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based
and the strategy of optimizing the CNN parameters should learning applied to document recognition. Proc IEEE 86(11):2278–
be referential for other medical applications that could use 2324. https://doi.org/10.1109/5.726791
12. Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classifi-
CNNs. Further research will explore the three-dimensional cation with deep convolutional neural networks. In: NIPS 2012, pp
input data, transfer learning and fine-tuning. 1097–1105
13. Simonyan K, Zisserman A (2014) Very deep convolutional
Acknowledgements The authors acknowledge the National Cancer networks for large-scale image recognition. Comput Sci.
Institute and the Foundation for the National Institutes of Health and arXiv:1409.1556
their critical role in the creation of the free publicly available LIDC- 14. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan
IDRI Database used in this study. D, Vanhoucke V, Rabinovich A (2015) Going deeper with convo-
lutions. In: CVPR 2015, pp 1–9. https://doi.org/10.1109/CVPR.
Funding This study was funded by the National Natural Science Foun- 2015.7298594
dation of China under Grant (Nos. 81671773, 61672146). 15. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for
image recognition. In: CVPR, pp 770–778
16. Hua KL, Hsu CH, Hidayati SC, Cheng W, Chen Y (2015)
Compliance with ethical standards Computer-aided classification of lung nodules on computed tomog-
raphy images via deep learning technique. Onco Targets Ther
8:2015–2022. https://doi.org/10.2147/OTT.S80733
Conflict of interest The authors declare that they have no conflict of 17. Sun W, Zheng B, Qian W (2016) Computer aided lung can-
interest. cer diagnosis with deep learning algorithms. In: SPIE Medical
Imaging 9785 2016:97850Z-97850Z-8. https://doi.org/10.1117/
Ethical approval All procedures performed in these studies involving 12.2216307
human participants were in accordance with the ethical standards of 18. Cheng JZ, Ni D, Chou YH, Qin J, Tiu CM, Chang YC, Huang
the institutional and/or national research committee and with the 1964 CS, Chen CM (2016) Computer-aided diagnosis with deep learn-
Helsinki Declaration and its later amendments or comparable ethical ing architecture: applications to breast lesions in us images and
standards. For this type of study, formal consent is not required. pulmonary nodules in CT scans. Sci Rep 6:24454. https://doi.org/
10.1038/srep24454
19. Kumar D, Wong A, Clausi DA (2015) Lung nodule classification
using deep features in CT images. Comput Robot Vis 2015:133–
138. https://doi.org/10.1109/CRV.2015.25
References 20. Shen W, Zhou M, Yang F, Yang C, Tian J (2015) Multi-scale con-
volutional neural networks for lung nodule classification. IPIM
1. Siegel R, Naishadham D, Jemal A (2013) Cancer statistics. CA- 2015:588–599. https://doi.org/10.1007/978-3-319-19992-4_46
Cancer J Clin 63(1):11–30. https://doi.org/10.3322/caac.21166 21. Shen W, Zhou M, Yang F, Yu D, Dong D, Yang C, Zang Y,
2. Chen D, Zheng R, Peter D, Baade PD, Zhang S, Zeng H, Bray F, Tian J (2017) Multi-crop convolutional neural networks for lung
Jemal A, Yu X, He J (2015) Cancer statistics in China. CA-Cancer nodule malignancy suspiciousness classification. Pattern Recognit
J Clin 66(2):115–132. https://doi.org/10.3322/caac.21338 61:663–673. https://doi.org/10.1016/j.patcog.2016.05.029
3. Valente IR, Cortez PC, Neto EC, Soares JM, De Albuquerque VH, 22. Armato SG, McLennan G, Bidaut L, McNitt-Gray MF, Meyer CR,
Tavares JM (2016) Automatic 3D pulmonary nodule detection in Reeves AP (2015) Data from LIDC-IDRI. The Cancer Imaging
CT images: a survey. Comput Methods Prog Biomed 124(C):91– Archive. https://doi.org/10.7937/K9/TCIA.2015.LO9QL9SX
107. https://doi.org/10.1016/j.cmpb.2015.10.006 23. Armato SG, McLennan G, Bidaut L, McNitt-Gray MF, Meyer CR,
4. Gridelli C, Rossi A, Carbone DP, Guarize J, Karachaliou N, Mok T, Reeves AP, Zhao B, Aberle DR, Henschke CI, Hoffman EA, Kaze-
Petrella F, Spaggiari L, Rosell R (2015) Non-small-cell lung cancer. rooni EA, MacMahon H, Van Beeke EJ, Yankelevitz D, Biancardi
Nat Rev Dis Primers 2(T3):N1. https://doi.org/10.1038/nrdp.2015. AM, Bland PH, Brown MS, Engelmann RM, Laderach GE, Max
9 D, Pais RC, Qing DP, Roberts RY, Smith AR, Starkey A, Batrah
5. Elbaz A, Beache GM, Gimelfarb G, Suzuki K, Okada K, Elnakib P, Caligiuri P, Farooqi A, Gladish GW, Jude CM, Munden RF,
A, Soliman A, Abdollahi B (2013) Computer-aided diagnosis sys- Petkovska I, Quint LE, Schwartz LH, Sundaram B, Dodd LE, Fen-
tems for lung cancer: challenges and methodologies. Int J Biomed imore C, Gur D, Petrick N, Freymann J, Kirby J, Hughes B, Casteele
Imaging 1:942353–942353. https://doi.org/10.1155/2013/942353 AV, Gupte S, Sallamm M, Heath MD, Kuhn MH, Dharaiya E, Burns
6. Doi K (2007) Computer-aided diagnosis in medical imaging: R, Fryd DS, Salganicoff M, Anand V, Shreter U, Vastagh S, Croft
historical review, current status and future potential. Com- BY (2011) The lung image database consortium (LIDC) and image
put Med Imag Gr 31(4–5):198–211. https://doi.org/10.1016/j. database resource initiative (IDRI): a completed reference database
compmedimag.2007.02.002 of lung nodules on CT scans. Med Phys 38(2):915–931. https://doi.
7. Parmar C, Grossmann P, Bussink J, Lambin P, AertsH J (2015) org/10.1118/1.3528204
Machine learning methods for quantitative radiomic biomarkers. 24. Clark K, Vendt B, Smith K, Freymann J, Kirby J, Koppel P, Moore
Sci Rep 5:13087. https://doi.org/10.1038/srep13087 SM, Phillips S, Maffitt DR, Tarbox L, Prior F (2013) The cancer

123
International Journal of Computer Assisted Radiology and Surgery

imaging archive (TCIA): maintaining and operating a public infor- 30. Keskar NS, Mudigere D, Nocedal J, Smelyanskiy M, Tang PTP
mation repository. J Digit Imaging 26(6):1045–1057. https://doi. (2016) On large-batch training for deep learning: generalization
org/10.1007/s10278-013-9622-7 gap and sharp minima. arXiv:1609.04836
25. Sun W, Tseng TLB, Zhang J, Qian W (2016) Enhancing deep con- 31. Setio AAA, Ciompi F, Litjens G, Gerke PK, Jacobs C, van Riel S,
volutional neural network scheme for breast cancer diagnosis with Wille MMW, Naqibullah M, Sanchez CI, van Ginneken B (2016)
unlabeled data. Comput Med Imaging Gr 57:4–9. https://doi.org/ Pulmonary nodule detection in CT images: false positive reduction
10.1016/j.compmedimag.2016.07.004 using multi-view convolutional networks. IEEE Trans Med Imag-
26. Arevalo J, González FA, Ramos-Pollán R, Oliveira JL, Lopez ing 35(5):1160–1169. https://doi.org/10.1109/TMI.2016.2536809
MAG (2016) Representation learning for mammography mass 32. Shin HC, Roth HR, Gao M, Lu L, Xu Z, Nogues I, Yao J,
lesion classification with convolutional neural networks. Comput Mollura D, Summers RM (2016) Deep convolutional neural net-
Methods Programs Biomed 127:248–257. https://doi.org/10.1016/ works for computer-aided detection: CNN architectures, dataset
j.cmpb.2015.12.014 characteristics and transfer learning. IEEE Trans Med Imaging
27. Nair V, Hinton GE (2010) Rectified linear units improve restricted 35(5):1285–1298. https://doi.org/10.1109/TMI.2016.2528162
boltzmann machines (ICML-10), pp 807–814 33. Hinton GE, Srivastava N, Krizhevsky A, Sutskever I, Salakhut-
28. Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, dinov R (2012) Improving neural networks by preventing co-
Guadarrama S, Darrell T (2014) Caffe: convolutional architecture adaptation of feature detectors. arXiv:1207.0580
for fast feature embedding. ACMMM 2014:675–678. https://doi.
org/10.1145/2647868.2654889
29. Srivastava N (2013) Improving neural networks with dropout. Uni-
versity of Toronto. http://www.cs.toronto.edu/~nitish/msc_thesis.
pdf. Accessed 18 Feb 2013

123