Hindawi

Journal of Healthcare Engineering

Volume 2020, Article ID 8860011, 11 pages
https://doi.org/10.1155/2020/8860011

Research Article
A Novel Algorithm for Breast Mass Classification in Digital
Mammography Based on Feature Fusion

Qian Zhang ,1 Yamei Li ,2,3 Guohua Zhao ,2,3 Panpan Man ,2,3 Yusong Lin ,3,4,5
and Meiyun Wang 6
1
School of Computer Science, Zhongyuan University of Technology, Zhengzhou 450007, China
2
School of Information Engineering, Zhengzhou University, Zhengzhou 450001, China
3
Collaborative Innovation Center for Internet Healthcare, Zhengzhou University, Zhengzhou 450052, China
4
School of Software, Zhengzhou University, Zhengzhou 450002, China
5
Hanwei IoT Institute, Zhengzhou University, Zhengzhou 450002, China
6
Department of Radiology, People’s Hospital of Zhengzhou University, Zhengzhou 450003, China

Correspondence should be addressed to Meiyun Wang; [email protected]

Received 30 April 2020; Revised 7 December 2020; Accepted 13 December 2020; Published 22 December 2020

Academic Editor: Aiping Liu

Copyright © 2020 Qian Zhang et al. This is an open access article distributed under the Creative Commons Attribution License,
which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Prompt diagnosis of benign and malignant breast masses is essential for early breast cancer screening. Convolutional neural
networks (CNNs) can be used to assist in the classification of benign and malignant breast masses. A persistent problem in current
mammography mass classification via CNN is the lack of local-invariant features, which cannot effectively respond to geometric
image transformations or changes caused by imaging angles. In this study, a novel model that trains both texton representation
and deep CNN representation for mass classification tasks is proposed. Rotation-invariant features provided by the maximum
response filter bank are incorporated with the CNN-based classification. The fusion after implementing the reduction approach is
used to address the deficiencies of CNN in extracting mass features. This model is tested on public datasets, CBIS-DDSM, and a
combined dataset, namely, mini-MIAS and INbreast. The fusion after implementing the reduction approach on the CBIS-DDSM
dataset outperforms that of the other models in terms of area under the receiver operating curve (0.97), accuracy (94.30%), and
specificity (97.19%). Therefore, our proposed method can be integrated with computer-aided diagnosis systems to achieve precise
screening of breast masses.

1. Introduction digital mammography images on radiologists and improve

their diagnostic efficiency. Amongst various breast abnor-
Breast cancer ranks first in morbidity and mortality amongst malities (e.g., masses, microcalcifications, architectural
all diseases that affect females [1]. Recent research from the distortions, and asymmetry), breast masses are difficult to
International Agency for Research on Cancer indicates that distinguish from similar backgrounds because of their
the incidence of breast cancer in China is gradually in- variable size and low contrast, both of which affect the
creasing, with more than 300,000 women diagnosed with diagnostic results.
breast cancer every year [2]. Mammography is widely used Images obtained from X-rays display various body
in the early screening of breast cancer because of its high postures or different imaging angles, and effectively iden-
diagnostic sensitivity for small lesions [3]. tifying the texture to be measured at different angles is
For radiologists, the detection or interpretation of breast important when performing texture analysis on the mass [5].
masses via digital mammography is a time-consuming task Convolutional neural networks (CNNs) can directly extract
[4]. Computer-aided diagnosis (CAD) systems are utilized in objective features from images without relying on feature
breast cancer diagnosis to reduce the burden of reading extraction and manual selection [6]. A persistent problem in
2 Journal of Healthcare Engineering

current mammography mass classification via CNN is the dimensional discrete wavelet transform with matrices to
lack of local-invariant features, which cannot effectively extract features from mammographic images, Beura et al.
respond to geometric image transformations or changes [12] achieved an accuracy of 97.4%. Texton is an effective
caused by imaging angles. This challenge can only be alle- tool for texture analysis. It is usually obtained through a filter
viated by manually manipulating the image rotation to set-based feature extraction approach that characterizes
augment the dataset, which is not effective for fine rotation. various pixel relationships in a specific area of an image
Image texture is defined as a function of the spatial [13–15]. Acharya et al. [16] applied the MR filter bank to
variation in pixel intensity (grey value) [7]. Texture analysis convolve with images to generate textons. This approach
can systematically characterize complex visual patterns. The attained 96% accuracy, demonstrating the effectiveness of
maximum response (MR) filter bank used in our study can the textons generated by the MR filter bank for classifying
deal with the rotation invariance of local images [8]. breast datasets. However, this approach does not consider
Moreover, the MR filter bank can effectively capture slight deeper image features. A single texture feature cannot fully
changes in the texture of images. Learned representations describe deep image features. Furthermore, the settings of
based on the MR filter bank can precisely model multiscale the initial parameters of these traditional methods heavily
and multidirectional information that is important for breast rely on experience.
mass diagnosis. However, a single texture feature cannot With the rapid development of deep learning, con-
describe deep image features. volutional neural networks (CNNs) can directly extract
In summary, the feature representations of these two objective features from images without relying on feature
approaches are integrated into a single model. A novel extraction and manual selection [17]. A single deep learning
method harnessing the complementary ability of fused model is effective in fields involved in disease diagnosis, such
rotation-invariant filters and deep learning for breast as radiology and ophthalmology [18]. A previous study
mass classification is proposed in this work. The MR filter reported that deep learning outperforms physicians in
bank is convolved with the images to generate textons, classifying benign and malignant breast lesions [19]. Car-
which are then fused with the feature representation neiro et al. [20] showed that pre-trained deep learning
extracted by ImageNet pre-trained CNN. The discrimi- models can be applied to medical imaging. An area under the
native ability of the rotation-invariant filter banks and curve (AUC) of 0.90 was achieved in various mammogram
deep learning features in classifying benign and malig- datasets (e.g., INbreast and DDSM). Qiu et al. [21] recog-
nant masses is tested. Direct fusion and fusion after nized features from breast images through CNN and max
reduction approaches are implemented to compare and pooling concepts. A prior study implemented a CNN along
select the best classification model for breast mass with intensity information and a decision mechanism to
diagnosis. classify breast masses [22]. The increasing availability of
The proposed method has the following advantages: large medical datasets facilitates the satisfactory perfor-
mance of CNNs in assisting breast cancer diagnosis [23–25].
(1) Given that body postures or imaging angles vary in
However, CNNs cannot explicitly realize rotation invariance
mammography mass images, a rotation-invariant
of local images and thus cannot effectively respond to
filter set is used to analyze the texture of mass images
geometric image transformations or changes caused by
(2) This study is the first to harness the complementary imaging angles.
discriminative power of rotation-invariant and deep Some researchers sought to develop a methodology that
learning representations for breast mass classification combines texture analysis and deep learning for feature
(3) The fusion after implementing the reduction ap- extraction. Wang et al. [26] explored a breast CAD method
proach can harness better the complementarity be- based on feature fusion with CNN deep features, texture
tween two groups of features and markedly improve features, and density features. He et al. [27] established a
the performance of breast mass classification classification model on the basis of extracted textures and
deep CNN features for evaluating diagnostic performance
2. Related Work on differentiating malignant masses. They proved that the
deep learning classification model for breast lesions, which
Texture analysis can systematically characterize complex was established according to image texture characteristics,
visual patterns. Via this approach, suspected regions can be can effectively differentiate malignant masses. These fusion
examined by analyzing texture features. Haralick et al. [9] methods are merely simple extensions at the feature level,
proposed the method of grey level co-occurrence matrices and they do not consider the characteristics of mammog-
(GLCM), which is extensively used in image recognition and raphy mass images.
classification. Da Rocha et al. [10] combined diversity indices
with GLCM as a way of describing the texture of breast 3. Materials and Methods
tissues. Through this combination, they obtained an accu-
racy of 88.31%. Abdalla et al. [11] adopted the GLCM to The framework of the proposed method in this work is
extract the texture features of images from the Digital shown in Figure 1. The MR filter bank is convolved with the
Database for Screening Mammography (DDSM) and images to generate textons, which are then fused with the
achieved an accuracy of 91.67%. Co-occurrence matrices are feature representation extracted by ImageNet pre-trained
also the main tools for texture analysis. By combining two- CNN. Direct fusion and fusion after reduction approaches
Journal of Healthcare Engineering 3

Texton generation

Input image
Local binary pattern extraction Feature
Textons-based
feature
extraction

Benign

Fine-tuning Feature
Deep CNN-based
feature
extraction

Malignant
Fusion

Prediction and evaluation Feature fusion

Diret
fusion
Fusion
after
reduction

Figure 1: Proposed method framework for mammogram mass classification.

are implemented to compare and select the best classification

model for breast mass diagnosis.
Orientation filter
(first derivative)
3.1. Learning Representation from Texton. Texton can be
generated using a filter set-based feature extraction approach
that characterizes various pixel relationships in a specific area
of an image. In past decades, many effective filter banks, such
as Leung–Malik [28], Schmid [15], and MR filters [8], have
been used to generate textons. The current study uses the MR Orientation filter
filter bank containing isotropic and anisotropic filters to (second derivative)
produce a satisfactory response to directional textures. Several
studies have indicated that the MR filter bank can obtain
textons with powerful discrimination [7, 29].
Gaussian filter
3.1.1. MR Filter Bank. The MR filter bank consists of 38
filters. As shown in Figure 2, six orientations at three scales Laplacian of gaussian filter
exist for two oriented filters in the first and second deriv-
atives, thereby forming 36 anisotropic filters. The two iso- Figure 2: MR filter bank.
tropic filters are the Gaussian and Laplacian of Gaussian
(LOG) filters. Gaussian and LOG filters. Using the MR8 filter bank to
If G is a Gaussian kernel function, then the first and convolve with the images reduces the 38 filter responses to 8.
second derivatives Gaussian filter can be defined as This step not only reduces the dimensionality of the re-
G′ � Gx cos θ + Gy sin θ. sponses but also implies rotation invariance. Compared with
(1) the traditional rotation invariance filter, the MR8 filter bank
G″ � Gxx cos2 θ + Gyy sin2 θ − 2Gxy cos θ sin θ. can calculate the statistical information of high-order
symbiosis in the relevant direction. Such information can
LOG can be defined as help distinguish textures that are visually similar to the mass
in its surrounding area.
LOG � ∇2 G � Gxx + Gyy . (2)

The MR8 filter bank is used to achieve rotational in- 3.1.2. Local Binary Pattern Extraction. After the MR8 filter
variance. It yields eight responses: six responses from the bank is used to generate textons, the local binary pattern
three scales for two filters and two responses from the (LBP) is then employed to extract features. LBP is a simple
4 Journal of Healthcare Engineering

method that can effectively describe local image features by 112 0

quantifying differences between the grey value of the 118 67 0 0
neighborhood and the central pixel [30]. The texture de-
scriptor is rotation-invariant and is not affected by bright-
ness fluctuations during recognition, which can avoid the 152 132 189 1 1
variation caused by different angles or imaging times of the
mammography images. LBP can be defined as 72 95 0 0
234 1
⎧ P−1
⎪
⎪
⎨ 􏽘 s gn − gc 􏼁, U􏼐LBPP,R 􏼑 ≤ 2,
LBPP,R 􏼐xc, yc 􏼑 � ⎪ n�0 (3) Figure 3: LBP.
⎪
⎩
P + 1, otherwise,
feature extractor. To use the pre-trained weights for fine-
where tuning, we repeat the grey value matrix of the image in three
1, x ≥ 0, different channels of RGB to match the input of the
s(x) � 􏼨 (4) retraining architecture. We utilize cross-entropy as the
0, x < 0,
objective function and set the learning rate to be less than the
P is the number of equally spaced points on the cir- initial learning rate pre-trained by the ImageNet. This step
cumference with radius R, gc is the pixel intensity at the ensures that the network will not completely forget the
centre point, and U(LBPP,R ) is a measure of uniformity features learned from the original dataset. The last softmax
applied to calculate the number of 0–1 transformations (i.e., layer of the network is removed and the 1024-dimensional
from 0 to 1 or vice versa). The working principle of LBP is feature vector extracted from the layer with the largest re-
illustrated in Figure 3. The pixel values on the circumference ceptive field (i.e., previous layer of the classification layer,
are compared with the central pixel value to generate a that is, pool5) is selected as the final output features. This
binary value of “0” or “1” to extract the local contrast layer includes all different learning modes in the previous
information. layer and can obtain the features with a strong discrimi-
The MR8 filter bank is used for the convolution with the natory ability for classification [17].
images; each filter generates eight filter responses. The LBP
algorithm is then applied to extract 36-dimensional feature
vectors from each filter response to obtain a texton-based 3.3. Fusing Texton and Deep CNN Features. The idea of
feature representation. A total of 288 (36 × 8) dimensional feature fusion comes from the early information fusion
features are extracted from each image. field, which is used for multisensor fusion in military
applications [31]. Feature fusion methods are widely used
in image recognition to achieve feature complementation
3.2. Learning Representation from Deep CNN. Given that and address the shortcomings of a single feature vector
large-scale training for medical tasks cannot be performed [32]. Two fusion strategies are designed to determine how
because of the lack of a medical dataset, a pre-trained the complementary information of the two features can be
network is introduced in this study. The InceptionV3 net- utilized.
work is applied to the deep feature extraction. The com-
putational cost and memory requirements of this network
are lower than those of Residual Neural Network 50, Visual 3.3.1. Direct Fusion. The most direct way to fuse two sets of
Geometry Group Network, and other networks. The main feature vectors is to use cascade fusion [33], which can be
feature of the inception architecture is the calculation of defined as
nonlinear weighted sum modules (σ(Wx)) in each layer, XF � 􏼂XMR , XCNN 􏼃, (6)
which can be defined as
M where XMR and XCNN are the texton- and deep CNN-based
⎝􏽘 w x + b⎞
σ⎛ ⎠, (5) feature vectors, respectively; XF is the fusion feature vector;
j j
j�1 and dim(XF ) � dim(XMR ) + dim(XCNN ). As described in
Sections 3.1 and 3.2, dim(XMR ) � 288 and
where M is the number of neurons in this layer, wj ∈ W, W dim(XCNN ) � 1024. The dimension of the deep CNN features
is the weight matrix, x is the input vector, b is the deviation is more than three times the feature vectors obtained from
term, and σ(·) is the activation function. The module uses MR8. The classification may focus on the deep CNN features
factorization to decompose 5 × 5 convolutions into two 1D and ignore the supplementary information in the textons.
(1 × 5 and 5 × 1) and compress the input or the dimension of Therefore, we design a fusion strategy after feature reduction.
the output of the previous layer, thereby effectively reducing
the complexity and computational cost of the model. The
experiment verifies that the effect of using random initial- 3.3.2. Fusion after Reduction. In this strategy, feature se-
ization to retrain the weights in the network is not as good as lection is performed on two sets of feature vectors before
that using the ImageNet pre-trained network. Hence, the cascade fusion is executed. Random forest is used for feature
ImageNet pre-trained InceptionV3 model is used as the selection, which can analyze complex interactive features
Journal of Healthcare Engineering 5

and is extremely robust to noisy and redundant data [34, 35]. sample. In this study, the SVM based on radial basis function
On the basis of the feature importance measurement (RBF) kernel is used, and the features fused after reduction
method, which uses the classification accuracy of Out-of-Bag are used as inputs to obtain the probability of classifying the
(OOB) [36], feature subsets are selected according to the masses as benign or malignant.
sequential forward selection (SFS) method. Algorithm 1 shows the workflow of the method pro-
The feature importance ranking method based on the posed here.
classification accuracy of the OOB can be expressed as
follows. 4. Results and Discussion
If the feature dimension is N, then bootstrap is adopted
to extract M datasets. M OOB datasets are also generated 4.1. Image Databases and Preprocessing. In our study, we
accordingly. utilized three digital databases for screening mammography
images, namely, Curated Breast Imaging Subset of DDSM
Step 1. m � 1 initialized and a decision tree Tm is created on (CBIS-DDSM) [39], INbreast [40], and Mammographic
the training set. Image Analysis Society (mini-MIAS) [41] to evaluate per-
formance of the proposed method.
Step 2. The classification accuracy of the mth OOB dataset
Aoob
m is calculated. 4.1.1. CBIS-DDSM. The CBIS-DDSM dataset is the curated
breast imaging subset of DDSM. It consists of 861 mass cases
Step 3. The feature xi (i � 1, 2, · · · , N)) is disturbed in the and full mammography images, including mediolateral
OOB dataset, and the accuracy Aoob m,i is recalculated. oblique and craniocaudal views of mammograms (i.e., 912
benign and 784 malignant masses).
Step 4. Steps 2 and 3 are repeated for m � 2, 3, · · · , M.
4.1.2. INbreast. The INbreast dataset was created by the
Step 5. The importance of xi is calculated using Breast Research Group, INESCPorto, Portugal. It contains
1 M images of 115 patients for a total of 410 images, including
oob oob
Di � 􏽘 􏼐A − Am,i 􏼑. (7) images of masses, calcifications, and other abnormalities. It
M m�1 m
contains a total of 112 masses (i.e., 36 benign and 76 ma-
lignant masses).
Step 6. It is sorted in descending order. A high feature
ranking indicates high importance. 4.1.3. Mini-MIAS. The mini-MIAS, which is provided by the
Fivefold cross-validation is used to select more effective Mammographic Image Analysis Society, London, UK,
features. Subsequently, the OOB dataset is utilized to obtain dataset contains 322 mammogram images obtained from
the rank of importance and calculate accuracy. The sorted set 161 women. It contains a total of 70 available mass images
of results with the most satisfactory classification effect is (i.e., 40 benign and 30 malignant masses).
then selected, and the optimal feature subset is obtained Given that the sample sizes of INbreast and mini-MIAS
using the SFS method. Finally, the cascade fusion of the two datasets are too small, we merge them into one dataset.
sets of features is executed. Therefore, these three databases are divided into two groups
for evaluating the proposed method (Table 1). To render the
dataset suitable for the pre-trained network and reduce the
3.4. Classification. The classifier is used to determine the
running cost, we extract 300 × 300 patches centered at
relationship amongst the sets of attributes to predict the
masses in the three databases to build our dataset. Next, an
possible attribution results [37]. After the classifier is trained,
adaptive histogram equalization [42] is applied to balance
the test data are fed into the network to predict the category
the contrast. For CBIS-DDSM, similar to other medical
and evaluate the performance of the algorithm. The fol-
image classification experiments, the affine transformation is
lowing classifiers are used to classify benign and malignant
used to rotate the images by 0°, 90°, 180°, and 270° and reflect
masses.
them along the horizontal axes to augment the dataset and
For the direct fusion, the softmax in InceptionV3 is used
avoid overfitting. For INbreast and mini-MIAS, each mass
as the classifier and the fused feature as its input. A dropout
patch is augmented by the aforementioned affine transfor-
is added to the classification layer to enhance the robustness
mation, and then these four images are flipped from left to
of the network. The stochastic gradient descent is used to
right to generate eight images for each patch as the second
minimize cross-entropy cost function.
dataset. Finally, each dataset is split into training (60%),
For the fusion after reduction, a support vector machine
validation (10%), and test (30%) sets.
(SVM) is utilized to distinguish benign and malignant
masses on the basis of low-dimensional features. SVM is a
supervised machine learning method widely used in sta- 4.2. Experiment Settings. The MR8 filter bank is operated in
tistical classification and regression analyses [38]. This MATLAB and convolve with the mass images to generate
technique can identify the best compromise between textons. The InceptionV3 model based on Keras is used to
learning accuracy and learning ability of a specific training transfer the pre-trained weights from ImageNet to the mass
6 Journal of Healthcare Engineering

(1) Input: mammography mass image Ii , i � 1, 2, . . . , N

(2) Output: diagnosis results matrix Y ∈ RN
(3) Calculate the deep feature matrix XCNN ∈ RN×n1
(4) Get textons by convoluting MR8 filter bank with the Ii
(5) Calculate the textons-based feature matrix XMR ∈ RN×n2 through LBP
(6) if use the direct fusion approach then
(7) Fusion feature XF � [XMR , XCNN ], XF ∈ RN×(n1 +n2 )
(8) Train softmax classifier with XF and predict Ii
(9) Return Y
(10) end if
(11) if use the fusion after reduction approach then
(12) Calculate the subset X′CNN ∈ RN×n3 of XCNN by random forest
(13) ′ ∈ RN×n4 of XMR by random forest
Calculate the subset XMR
(14) Fusion feature XF � [X’MR , X’CNN ], XF ∈ RN×(n3 +n4 )
(15) Train SVM classifier with XF and predict Ii
(16) Return Y
(17) end if

ALGORITHM 1: Mammography mass benign-malignant classification algorithm.

Table 1: Digital mammogram dataset.

Database Number of benign images Number of malignant images Total number of images
1 CBIS-DDSM 912 784 1696
INbreast 36 76
2 182
Mini-MIAS 40 30

dataset. Given that mammography mass images are vastly In benign and malignant mass classification, if the
different from ImageNet images, we propose to fine-tune our malignant mass is classified as malignant, then the result will
models to adjust the features of the last convolutional blocks be true positive (TP). The result will become true negative
and make them more data-specific. We utilize stochastic (TN) if the benign mass is classified as benign. Similarly, if
gradient descent to fine-tune the network and set the initial the benign mass is classified as malignant, then the result will
learning rate to 10−5. We divide the initial learning rate by 10 be false positive (FP), which will become false negative (FN)
each time the validation error stops improving. Moreover, to if the malignant mass is classified as benign.
improve the results and avoid overfitting, we perform L2 The k-fold cross-validation [43] method is adopted to
regularization and dropout. When training the SVM model, evaluate the performance of the proposed method. The
we employ the train and validation sets to fine-tune the C evaluation metrics in this study are derived from the fivefold
parameter for the SVM classifier. After tuning the models and cross-validation method.
choosing the best hyperparameters, we train each final model
by using a stratified fivefold cross-validation with all the data
4.4. Results and Analysis
and evaluate each model’s performance.
4.4.1. Direct Fusion
4.3. Evaluation Metrics. In the diagnostic results of medical
images, accuracy (Acc), sensitivity (Sens), and specificity (1) MR8 Features Only. First, an MR8 filter bank is built, and
(Spec) are the commonly used objective evaluation metrics. the filter responses are collected by convolving them with the
The area under the receiver operating characteristic curve images. Second, the LBP algorithm is used to extract the 36-
(ROC) (i.e., AUC score) is another important metric used to dimensional feature vectors from each filter response. Fi-
evaluate the performance of diagnostic results. These eval- nally, the 288-dimensional feature vectors based on MR8 are
uation metrics are calculated as follows: obtained and used to train the softmax classifier. Fivefold
cross-validation is applied to evaluate the average perfor-
NR mance of this classifier in benign and malignant mass
Acc � ,
N classification. As shown in Table 2, the AUC score and
accuracy obtained by the MR8 features for classification are
TP
Sens � , (8) 0.79 and 70.21%, respectively.
TP + FN
TN (2) Deep CNN Features Only. The average accuracy obtained
Spec � . by the InceptionV3 model by using the initial weight is
FP + TN
72.21%. When the ImageNet pre-trained InceptionV3 is
Journal of Healthcare Engineering 7

Table 2: Comparison of MR8 features, deep CNN features, and also confirm that fusion after reduction can effectively
direct fusion with CBIS-DDSM. combine the advantages of the two features, and the feature
representation based on MR8 can provide supplementary
Methods AUC Acc information to facilitate the CNN in classifying benign and
MR8 features only 0.7974 0.7021 malignant masses.
Deep CNN features only 0.8711 0.7934 We also construct the fusion after reduction approach on
Direct fusion of MR8 and deep CNN features 0.9204 0.8002 INbreast and mini-MIAS. The classification results are
summarized in Table 3. The AUC and accuracy of training
used to extract the 1024-dimensional feature vectors and the classifier by using the MR8 feature subset only are 0.88
train the softmax classifier described in Section 3.4, the and 88.47%, respectively, which are slightly higher than
classification results demonstrated improvements. The re- those of the classification performance by using CNN fea-
sults are shown in the third row of Table 2, where the AUC tures. This result is obtained because these two databases are
score is 0.87 and the accuracy is 79.34%. too small despite the fact that we have already augmented the
data. CNNs cannot obtain additional effective features from
(3) Direct Fusion. The two features are directly fused using a limited database. In spite of the limited number of datasets,
the cascade fusion method to train the softmax classifier. The training the classifier with fusion features still improves the
classification results in the fourth row of Table 2 indicate that performance of the classifier (AUC is 0.93 and accuracy is
the AUC score is 0.92 and the accuracy is 80.02%. 93.59%). This result suggests that our method can achieve
Although the classification results after direct fusion are high performance even when sample sets are small and
slightly better than those after using a single feature, the image bases are heterogeneous.
accuracy is almost the same as that when only deep CNN Three other machine learning classifiers are used to
features are applied. This finding might be attributed to the verify the classification performance of the fused features
excessively large feature dimension of the fusion, and the after reduction. Figure 5 shows the classification results by
feature dimension of deep CNN being more than three times using k-nearest neighbor classifier (kNN), SVM based on
the feature obtained from MR8. Therefore, the classifier linear function kernel (SVM-linear), and extreme gradi-
prefers the information contained in the deep CNN features ent boosting (XGBoost). Fusion features improve classi-
during the classification, which is why the fusion method fication performance under all three classifiers (AUC
after feature reduction is developed. scores are 0.89, 0.93, and 0.96, respectively). The three
classifiers reflect the superiority of the fusion features after
reduction. The confusion matrices using XGBoost as
4.4.2. Fusion after Reduction. Random forest and SFS are displayed in Figure 6 indicate that the number of mis-
used to select the feature subsets from the two groups of classified benign and malignant masses after fusion is
features. The OOB dataset and fivefold cross-validation are substantially reduced. Specifically, the number of malig-
implemented to obtain the importance ranking and select nant masses incorrectly classified as benign is reduced by
the best set of features for the classification results, re- nearly half.
spectively. A total of 47 dimensional features are obtained,
where 17 are obtained from the feature representation based
on MR8 and 30 are obtained from the deep CNN features. 4.4.3. Comparative Analysis. To prove the complementary
The fused features are then fed into the SVM classifier. To capabilities of MR8 features for CNNs, we adopt two popular
obtain an effective comparison of the classification results of deep learning models, namely, ResNet50 and Efficient-B7, to
the fused features, we train the same SVM classifier by using replace the InceptionV3 model in our method. The structure
the two feature subsets. A comparison of the classification and depth of these models are suitable for medical image
results before and after fusion is shown in Table 3. The AUC classification tasks with few training samples. MR8 + Res-
score and accuracy of the MR8 feature subset only are 0.89 Net50 and MR8 + EfficientNet-B7 represent the use of fusion
and 80.42%, respectively, in classifying CBIS-DDSM mass after reduction approach for fusing both MR8 and deep
images. By comparison, the AUC score and accuracy of the CNN features. As shown in Table 4, the fused features
deep CNN feature subset only are 0.92 and 88.67%, re- improve the performance of ResNet50 (ACC and AUC
spectively. After implementing the reduction strategy, the increased by 5% and 0.02, respectively) and Efficient-B7
fusion reaches an accuracy of 94.30% and an average AUC of (ACC and AUC increased by 8.55% and 0.05, respectively).
0.97, an increase of 0.05 and 14.28%, respectively, compared Therefore, the features obtained from the MR8 filter can
with those of the direct fusion strategy. This result suggests effectively compensate for the shortcomings of CNNs in
that training the classifier with the fusion features after feature extraction.
reduction can better harness the complementarity of these Various methods have been devised for classifying be-
two sets of features. nign and malignant masses. The best case achieved by the
Figures 4(a) and 4(b) show the ROC curves of the direct method proposed herein is further compared with that of
fusion and the fusion after reduction, respectively. The three some recently developed classification methods (Table 4).
different color curves in each picture reveal that the area The performance of our method is superior to that of tra-
under the yellow ROC curve is the largest, which represents ditional textural analyses and other machine learning
the classification result using the fusion features. The curves methods [44, 45]. The performance of two deep learning
8 Journal of Healthcare Engineering

Table 3: Comparison of MR8 features, deep CNN features, and fusion after reduction.
Dataset Methods AUC Acc
MR8 features only 0.8964 0.8042
CBIS–DDSM Deep CNN features only 0.9262 0.8867
Fusing MR8 and deep CNN features 0.9795 0.9430
MR8 features only 0.8812 0.8847
INbreast + mini–MIAS Deep CNN features only 0.8553 0.8728
Fusing MR8 and deep CNN features 0.9383 0.9359

Receiver operating characteristic curve Receiver operating characteristic curve

1.0 1.0

0.8 0.8
True positive rate

True positive rate

0.6 0.6

0.4 0.4

0.2 0.2

0.0 0.0
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
False positive rate False positive rate

Random Random
ROC only MR8 features (AUC = 0.79740) ROC only MR8 features (AUC = 0.89644)
ROC only CNN features (AUC = 0.87110) ROC only CNN features (AUC = 0.92622)
ROC fusing MR8 and CNN (AUC = 0.92047) ROC fusing MR8 and CNN (AUC = 0.97953)
(a) (b)

Figure 4: ROC curves of direct fusion and fusion after reduction. (a) The ROC curve of the average performance using direct fusion
approach. (b) The ROC curve of the average performance using fusion after reduction approach.

0.96
MR8

0.82 0.80 0.91 0.93

0.90
Feature
CNN

0.86 0.88 0.93

0.87
MR8 + CNN

0.84
0.89 0.93 0.96
0.81
kNN SVM-linear XGBoost
Classifier
Figure 5: Heat map of AUC score under kNN, SVM-linear, and XG boost using fusion after reduction with CBIS-DDSM.

methods [46, 47] is also compared with that of our method. better than that of these two deep learning-based ap-
As shown in Table 4, these two methods achieve high proaches. Moreover, the performance of methods described
sensitivity (98.00% and 93.83%). However, their specificity is in [10, 11], which integrate multiple features to classify
substantially lower than that of our method, suggesting that benign and malignant masses, is slightly lower than that of
they may misclassify more negative masses compared with our method. The results establish the superiority and ro-
our method. Overall, the performance of our method is bustness of our proposed method.
Journal of Healthcare Engineering 9

Only MR8 features Only CNN features

1000 1000
Benign

Benign
1076 79 800 1050 105 800
True label

True label
600 600

Malignant
Malignant

400 400
201 534 230 505

200 200

Benign Malignant Benign Malignant

Predicted label Predicted label
(a) (b)
Fusing MR8 and CNN features

1000
Benign

1112 43
800
True label

600
Malignant

400
116 619
200

Benign Malignant
Predicted label
(c)

Figure 6: Confusion matrix obtained by XGBoost using fusion after reduction approach with CBIS-DDSM. (a) Only MR8 features. (b) Only
deep CNN features. (c) Fusing MR8 and CNN after feature reduction.

Table 4: Comparison of proposed method with other mass classification methods.

Dataset Methods Sens Spec Acc AUC
ResNet50 77.31% 82.07% 79.50% 0.86
MR8 + ResNet50 83.17% 85.94% 84.50% 0.88
CBIS-DDSM
EfficientNet-B7 80.47% 81.05% 80.75% 0.80
MR8 + EfficientNet-+B7 89.88% 88.02% 89.30% 0.85
DDSM M. Hussain et al. [44] – – 85.53% 0.87
BCDR L. Fangyi et al. [45] 88.93% 93.41% 91.65% 0.96
INbreast N. Dhungel [46] 98.00% 70.00% 90.00% –
CBIS-DDSM C. Yuanqin [47] 93.83% 92.17% 93.15% 0.95
DDSM S. V. da rochaa [10] 85.00% 91.89% 88.31% 0.88
DDSM Q. Abbas [11] 92.00% 84.20% 91.00% 0.91
CBIS-DDSM Proposed method 89.97% 97.91% 94.30% 0.97

5. Conclusions features. Hence, they can be difficult to differentiate. In this

study, a novel method based on texton fusion and CNNs for
Body postures or imaging angles vary in mammography extracting mass features and classifying benign and malig-
masses. Malignant and benign masses may show similar nant masses is proposed. Two fusion strategies, namely,
10 Journal of Healthcare Engineering

direct fusion and fusion after reduction, are employed to screening,” Ca-a Cancer Journal for Clinicians, vol. 69, no. 13,
fuse texton-based feature representations with deep CNN pp. 184–210, 2019.
features. Moreover, these fusion strategies are adopted to [5] H. D. Cheng, X. J. Shi, R. Min, L. M. Hu, X. P. Cai, and
explore how the complementary discrimination ability of H. N. Du, “Approaches for automated detection and classi-
two groups of features can be applied to mass classification fication of masses in mammograms,” Pattern Recognition,
tasks. These strategies are tested on the public databases vol. 39, no. 4, pp. 646–668, 2006.
[6] A. Esteva, A. Robicquet, B. Ramsundar et al., “A guide to deep
CBIS-DDSM, INbreast, and mini-MIAS. Results show that
learning in healthcare,” Nature Medicine, vol. 25, no. 1,
the fused features can provide useful supplementary in-
pp. 24–29, 2019.
formation for extracting mass features via CNN. By com- [7] M. Varma and A. Zisserman, “A statistical approach to texture
parison, the fusion after reduction approach can harness classification from single images,” International Journal of
better the complementarity of features extracted from the Computer Vision, vol. 62, no. 1-2, pp. 61–81, 2005.
MR8 filter and deep CNN. Thus, this approach can achieve [8] J. Geusebroek, A. W. M. Smeulders, and J. Van De Weijer,
an accurate classification of benign and malignant masses. “Fast anisotropic Gauss filtering,” Institute of Electrical and
Experimental results demonstrate that our method out- Electronics Engineers Transactions on Image Processing,
performs other state-of-the-art methods without pixel-level vol. 12, no. 8, pp. 938–943, 2003.
annotation. Given that our method does not require any user [9] R. M. Haralick, K. Shanmugam, and I. H. Dinstein, “Textural
interaction, it can be easily integrated into CAD systems for features for image classification,” Institute of Electrical and
breast cancer. Electronics Engineers Transactions on Systems, Man, and
However, mammography images have many patholog- Cybernetics, vol. 3, no. 6, pp. 610–621, 1973.
ical classifications, such as microcalcifications and structural [10] S. V. Da Rocha, G. Braz Junior, A. C. Silva, A. C. De Paiva, and
M. Gattass, “Texture analysis of masses malignant in mam-
distortions. At present, although this method has achieved
mograms images using a combined approach of diversity
good results in the classification of benign and malignant
index and local binary patterns distribution,” Expert Systems
masses, it has not been tested in classifing and diagnosing with Applications, vol. 66, pp. 7–19, 2016.
other pathological classifications. With the expansion of our [11] Q. Abbas, “Deep CAD: A computer-aided diagnosis system
database, we will be able to optimize our method for other for mammographic masses using deep invariant features,”
pathological classifications of breast images. Computers, vol. 5, no. 4, 2016.
[12] S. Beura, B. Majhi, and R. Dash, “Mammogram classification
Data Availability using two dimensional discrete wavelet transform and gray-
level co-occurrence matrix for detection of breast cancer,”
The data used to support the findings of this study are Neurocomputing, vol. 154, pp. 1–14, 2015.
available from the corresponding author upon request. [13] F. Jurie and B. Triggs, “Creating efficient codebooks for visual
recognition,” in Proceedings of Tenth Ieee International
Conflicts of Interest Conference on Computer Vision, pp. 604–610, Institute of
Electrical and Electronics Engineers Computer Society, Los
The authors declare that they have no conflicts of interest. Alamitos, CA, USA, October 2005.
[14] U. R. Acharya, H. Fujita, V. K. Sudarshan et al., “An integrated
Acknowledgments index for identification of fatty liver disease using radon
transform and discrete cosine transform features in ultra-
This work was supported by National Natural Science sound images,” Information Fusion, vol. 31, pp. 43–53, 2016.
Foundation of China under Grant no. 81772009, Scientific [15] L. Zhang, X. Ye, T. Lambrou, W. Duan, N. Allinson, and
and Technological Research Project of Henan Province N. J. Dudley, “A supervised texton based approach for au-
under Grant no. 182102310162, and Collaborative Innova- tomatic segmentation and measurement of the fetal head and
tion Major Project of Zhengzhou under Grant no. femur in 2D ultrasound images,” Physics in Medicine and
20XTZX06013. Biology, vol. 61, no. 3, pp. 1095–1115, 2016.
[16] U. R. Acharya, K. M. Meiburger, J. E. Wei Koh et al., “A novel
algorithm for breast lesion detection using textons and local
References configuration pattern features with ultrasound imagery,”
[1] R. Siegel, C. DeSantis, and A. Jemal, “Colorectal cancer sta- Institute of Electrical and Electronics Engineers Access, vol. 7,
tistics, 2014,” CA: A Cancer Journal for Clinicians, vol. 64, pp. 22829–22842, 2019.
no. 2, pp. 104–117, 2014. [17] L. Zhou, Z. Zhang, Y.-C. Chen, Z.-Y. Zhao, X.-D. Yin, and
[2] J. Ferlay, M. Colombet, I. Soerjomataram et al., “Estimating H.-B. Jiang, “A deep learning-based radiomics model for
the global cancer incidence and mortality in 2018: GLO- differentiating benign and malignant renal tumors,” Trans-
BOCAN sources and methods,” International Journal of lational Oncology, vol. 12, no. 2, pp. 292–300, 2019.
Cancer, vol. 144, no. 8, pp. 1941–1953, 2019. [18] D. S. Kermany, M. Goldbaum, W. Cai et al., “Identifying
[3] C. Lerman, M. Daly, C. Sands et al., “Mammography ad- medical diagnoses and treatable diseases by image-based deep
herence and psychological distress among women at risk for learning,” Cell, vol. 172, no. 5, pp. 1122–1131, 2018.
breast cancer,” JNCI Journal of the National Cancer Institute, [19] R. K. Samala, H.-P. Chan, L. Hadjiiski, M. A. Helvie, J. Wei,
vol. 85, no. 13, pp. 1074–1080, 1993. and K. Cha, “Mass detection in digital breast tomosynthesis:
[4] R. A. Smith, K. S. Andrews, D. Brooks et al., “Cancer screening deep convolutional neural network with transfer learning
in the United States, 2019: a review of current American from mammography,” Medical Physics, vol. 43, no. 12,
Cancer Society guidelines and current issues in cancer pp. 6654–6666, 2016.
Journal of Healthcare Engineering 11

[20] G. Carneiro, J. Nascimento, and A. P. Bradley, “Unregistered analysis of multiple data types,” in Proceedings of the 2006
multiview mammogram analysis with pre-trained deep Institute of Electrical and Electronics Engineers Symposium on
learning models,” in Medical Image Computing And Com- Computational Intelligence in Bioinformatics and Computa-
puter-Assisted Intervention, pp. 652–660, Springer Interna- tional Biology, p. 171, Institute of Electrical and Electronics
tional Publishing Ag, Cham, Switzerland, 2015. Engineers, Kunming, China, April 2006.
[21] Y. Qiu, Y. Wang, S. Yan et al., “An initial investigation on [36] A. Verikas, A. Gelzinis, and M. Bacauskiene, “Mining data
developing a new method to predict shor-tterm breast cancer with random forests: a survey and results of new tests,”
risk based on deep learning technology,” in Medical Imaging Pattern Recognition, vol. 44, no. 2, pp. 330–349, 2011.
2016, Computer-Aided Diagnosis, San Diego, CA, USA, 2015. [37] S. Aydin, N. Arica, E. Ergul et al., “Classification of obsessive
[22] Z. Jiao, X. Gao, Y. Wang, and J. Li, “A deep feature based compulsive disorder by EEG complexity and hemispheric
framework for breast masses classification,” Neurocomputing, dependency measurements,” International Journal of Neural
vol. 197, pp. 221–231, 2016. Systems, vol. 25, no. 3, 2015.
[23] F. Gao, H. Yoon, T. Wu et al., “A feature transfer enabled [38] C. Cortes and V. Vapnik, “Support-vector networks,” Ma-
multi-task deep learning model on medical imaging,” Expert chine Learning, vol. 20, no. 3, pp. 273–297, 1995.
Systems with Applications, vol. 143, Article ID 112957, 2020. [39] R. S. Lee, F. Gimenez, A. Hoogi et al., “A curated mam-
[24] W. Sun, T.-L. Tseng, B. Zheng et al., “A Preliminary Study on mography data set for use in computer-aided detection and
Breast Cancer Risk Analysis Using Deep Neural Network,” in diagnosis research,” Scientific Data, vol. 4, Article ID 170177,
Proceedings of the 13th International Workshop on Breast 2017.
Imaging, IWDM 2016, pp. 385–391, Malmö, Sweden, June [40] I. C. Moreira, I. Amaral, I. Domingues et al., “INbreast:
2016. Toward a Full-Field Digital Mammographic Database,” Ac-
[25] J. Arevalo, F. A. González, R. Ramos-Pollán, J. L. Oliveira, and ademic Radiology, vol. 19, no. 2, pp. 236–248, 2012.
M. A. Guevara Lopez, “Representation learning for mam- [41] J. Suckling, J. Parker, D. Dance et al., “Mammographic image
mography mass lesion classification with convolutional neural analysis society digital mammogram database exerpta
networks,” Computer Methods and Programs in Biomedicine, medica,” in Proceedings of the International Congress Series,
vol. 127, pp. 248–257, 2016. vol. 1069, pp. 375–378, Kyoto, Japan, March 1994.
[26] Z. Wang, M. Li, H. Wang et al., “Breast cancer detection using [42] S. M. Pizer, E. P. Amburn, J. D. Austin et al., “Adaptive
extreme learning machine based on feature fusion with CNN histogram equalization and its variations,” Computer Vision,
deep features,” Institute of Electrical and Electronics Engineers Graphics, and Image Processing, vol. 39, no. 3, pp. 355–368,
Access, vol. 7, pp. 105146–105158, 2019. 1987.
[27] Z. He, W. Lyu, G. Qin et al., “A feasibility study of building up [43] Q. Dai, “A competitive ensemble pruning approach based on
deep learning classification model based on breast digital cross-validation technique,” Knowledge-Based Systems,
breast tomosynthesis image texture feature extraction of the vol. 37, pp. 394–414, 2013.
simple mass lesions,” Chinese Journal of Radiology, vol. 52, [44] M. Hussain, “Effective extraction of gabor features for false
no. 9, pp. 668–672, 2018. positive reduction and mass classification in mammography,”
[28] T. Leung and J. Malik, “Representing and recognizing the Applied Mathematics & Information Sciences, vol. 8, no. 1,
visual appearance of materials using three-dimensional tex- pp. 397–412, 2014.
tons,” International Journal of Computer Vision, vol. 43, no. 1, [45] L. Fangyi, S. Changjing, L. Ying et al., “Interpretable mam-
pp. 29–44, 2001. mographic mass classification with fuzzy interpolative rea-
[29] S. Lazebnik, C. Schmid, and J. Ponce, “A sparse texture soning,” Knowledge-Based Systems, vol. 191, Article ID
representation using local affine regions,” Institute of Elec- 105279, 2019.
trical and Electronics Engineers Transactions on Pattern [46] N. Dhungel, G. Carneiro, and A. P. Bradley, “A deep learning
Analysis and Machine Intelligence, vol. 27, no. 8, pp. 1265– approach for the analysis of masses in mammograms with
1278, 2005. minimal user intervention,” Medical Image Analysis, vol. 37,
[30] U. R. Acharya, W. L. Ng, K. Rahmat et al., “Shear wave pp. 114–128, 2017.
elastography for characterization of breast lesions: shearlet [47] C. Yuanqin, Z. Qian, W. Yaping et al., “Fine-tuning ResNet
transform and local binary pattern histogram techniques,” for breast cancer classification from mammography,” in
Proceedings of 2nd International Conference on Healthcare
Computers in Biology and Medicine, vol. 91, pp. 13–20, 2017.
Science and Engineering, pp. 83–96, Guilin, China, May 2019.
[31] C. Sanderson and K. K. Paliwal, “Identity verification using
speech and face information,” Digital Signal Processing,
vol. 14, no. 5, pp. 449–480, 2004.
[32] A. J. Ma, P. C. Yuen, and J.-H. Jian-Huang Lai, “Linear de-
pendency modeling for classifier fusion and feature combi-
nation,” Institute of Electrical and Electronics Engineers
Transactions on Pattern Analysis and Machine Intelligence,
vol. 35, no. 5, pp. 1135–1148, 2013.
[33] D. C. Luvizon, H. Tabia, and D. Picard, “Learning features
combination for human action recognition from skeleton
sequences,” Pattern Recognition Letters, vol. 99, pp. 13–20,
2017.
[34] C. Strobl, A.-L. Boulesteix, T. Kneib et al., “Conditional
variable importance for random forests,” Bmc Bioinformatics,
vol. 9, 2008.
[35] D. M. Reif, A. A. Motsinger, B. A. McKinney et al., “Feature
selection using a random forests classifier for the integrated