FPN-IAIA-BL: A Multi-Scale Interpretable Deep Learning Model for Classification of Mass Margins in Digital Mammography

Abstract

Digital mammography is essential to breast cancer detection, and deep learning offers promising tools for faster and more accurate mammogram analysis. In radiology and other high-stakes environments, uninterpretable (“black box”) deep learning models are unsuitable and there is a call in these fields to make interpretable models. Recent work in interpretable computer vision provides transparency to these formerly black boxes by utilizing prototypes for case-based explanations, achieving high accuracy in applications including mammography. However, these models struggle with precise feature localization, reasoning on large portions of an image when only a small part is relevant. This paper addresses this gap by proposing a novel multi-scale interpretable deep learning model for mammographic mass margin classification. Our contribution not only offers an interpretable model with reasoning aligned with radiologist practices, but also provides a general architecture for computer vision with user-configurable prototypes from coarse- to fine-grained prototypes.

1 Introduction

Figure 1: Activation maps for FPN-IAIA-BL in comparison to IAIA-BL. FPN-IAIA-BL can learn human interpretable prototypes at any scale, including fine-grained details most salient to mass margin classification.

Digital mammography plays a crucial role in detecting and diagnosing breast cancer, a pervasive health concern worldwide. Advancements in deep learning and computer vision have increased the speed and accuracy of lesion classifications for mammography. However, when used for high-stakes tasks like medical diagnoses, deep learning models should be inherently interpretable so that, among other advantages, models can be “fact checked” [18].

Recent work has shown that interpretable, case-based machine learning models can provide accurate, human understandable explanations for their predictions while performing on par with other state-of-the-art models [13, 6]. These prototype-based deep learning models have also been applied to digital mammography by Barnett et al. [4], who developed the Interpretable AI Algorithm for Breast Lesions (IAIA-BL) model, an interpretable model for mass margin classification. They focused on classification on margins, a descriptor of the edges around the mass, because it is a key factor in identifying cancerous lesions under the Breast Imaging Reporting and Data System (BI-RADS). IAIA-BL successfully classified margins using prototypes, as shown in the third column of Figure 1. However, the prototypes often identified more than just the margin or even the entire lesion, leaving any detailed analysis of the margin to the user.

To address this gap, we develop FPN-IAIA-BL, a multi-scale interpretable deep learning model for mammographic mass margin classification. It can be configured to provide prototypes at various levels of granularity, with multiple scales within the same model. We build the model’s architecture using both the Feature Pyramid Network (FPN) and IAIA-BL model. We developed a new training schedule and objective function, as the training methods and loss terms used by these predecessors were insufficient to train the combined architecture. The main contributions of this work are that:

• We develop an inherently interpretable deep learning architecture that learns prototypes at multiple scales.

• We train FPN-IAIA-BL, which provides specific prototype activations for mass margin classification.

2 Related Work

Interpretability of deep learning models is critical for high-stakes applications like breast cancer detection and diagnosis. In recent years, inherently interpretable deep neural networks have grown in popularity. As compared to posthoc explanation techniques such as saliency visualizations [1, 21, 22, 23, 24, 29], activation maximization [9, 17, 25, 27, 28], and image perturbation methods [10, 11] which approximate model reasoning after training, inherently interpretable techniques such as [15, 19, 26, 6, 8, 2, 16, 13, 3] provide explanations guaranteed to be faithful to the model’s underlying decision-making process.

FPN-IAIA-BL uses inherently interpretable case-based reasoning with prototypes by building upon IAIA-BL [4], a case-based model for mass margin classification. IAIA-BL was limited to learning prototypes at only one scale, with prototypes often identifying more of the image than is relevant for margin classification. In contrast, FPN-IAIA-BL learns prototypes at various scales including highly-localized, fine-grained prototypes that select small details, as shown in Figure 1. This is possible because FPN-IAIA-BL incorporates features at various scales.

Typically, a key challenge in mammogram analysis is capturing information at various scales, since traditional CNN architectures focus on a single image resolution. Multi-scale approaches like [7] and [14] address this challenge by incorporating features extracted at different scales within the network. A foundational architecture for multi-scale predictions is the Feature Pyramid Network (FPN) [14] which introduces a bottom-up and top-down pyramidal architecture that produces multiple feature maps from fine-grained to coarse. As a result, FPN’s are able to localize to objects of multiple scales for object detection.

Our FPN-IAIA-BL architecture leverages this bottom-up and top-down pyramidal architecture to learn prototypes at multiple scales by augmenting IAIA-BL’s VGG-16 backbone with a similar structure, detailed in Section 3. Furthermore, our model also provides visual, human interpretable, case-based reasoning for each classification.

3 FPN-IAIA-BL Architecture

Figure 2: FPN-IAIA-BL Architectre. The input image $\mathbf{x}$ passes through convolutional layers $f$ consisting of an FPN with a VGG-16 backbone, which creates an pyramid of feature maps $f(\mathbf{x})$. Each patch of each level of the feature pyramid (referred to as FPN level) is then compared to each prototype of the same FPN level using a cosine distance to produce an activation map. The activation map is then used to calculate an overall similarity score $s_{j}$ between the input image and the prototype for each prototype. Finally, a set of fully connected last layer produces logits $y_{\text{margin}}$ for each margin class.

Inspired by the Feature Pyramid Network (FPN), the FPN-IAIA-BL model adds lateral and top-down connections to the original VGG-16 convolutional layers used in the IAIA-BL architecture as its foundation. Figure 2 illustrates this architecture. The model consists first of an FPN that extracts useful feature maps at multiple scales, allowing for more varied representation than single-scale IAIA-BL. The FPN is followed by the prototype layer $g$ in which the input image’s feature maps are compared to learned prototypes to produce similarity scores. Fully connected layer $h$ then uses the similarity scores to produce margin class predictions.

3.1 Multi-Scale Feature Maps from Feature Pyramid Network

IAIA-BL [4] uses a CNN to create a single feature map $z$ which limits the network to prototypes at the scale of that output feature map. In contrast, FPN-IAIA-BL uses the latent feature maps from multiple layers in the CNN, which have different spatial and semantic scales. Thus, the output of the set of convolutional layers $f$ in FPN-IAIA-BL is a set of feature maps of varying spatial scale, which we refer to as the feature pyramid $f(x)=\mathbf{Z}=\{\mathbf{z}^{(2)},\mathbf{z}^{(3)},\mathbf{z}^{(4)},\mathbf{z}^{(5)}\}$. For our implementation, the coarsest feature maps were 14 by 14, and finest were 56 by 56.

For the VGG-16 backbone, we use the output from each block’s max-pooling layer to form the intermediate feature map levels in the bottom-up pathway (left column of backbone in Figure 2). We also include the final output of the convolutional layers as a feature map at the top. We denote these bottom-up feature maps as $\mathbf{C}=\{\mathbf{c}^{(2)},\mathbf{c}^{(3)},\mathbf{c}^{(4)},\mathbf{c}^{(5)}\}$ where $\mathbf{c}^{(2)}$ is the base of the bottom-up pyramid, and $\mathbf{c}^{(5)}$ is the top.

As in FPN [14], the top-down pathway produces a second feature pyramid. For each level, an upsampled feature map with spatially coarser information is combined with a corresponding laterally connected feature maps from the bottom-up pyramid. Then, each combined feature map is passed through a $3\times 3$ convolution to reduce the aliasing effect of upsampling and output the feature map $\mathbf{z}^{(l)}$.

$$\begin{gathered}\mathbf{z}^{(5)}=\text{Conv1x1}(\mathbf{c}^{(5)})\\ \mathbf{z}^{(l)}=\text{Conv3x3}\Big{(}\text{Up}(\mathbf{z}^{(l+1)})+\text{Conv1x1}(\mathbf{c}^{(l)})\Big{)};l\in\{2,3,4\}\end{gathered}$$ (1)

3.2 Prototype Layer

In the prototype layer $g$, we have $m$ prototypes where each prototype can be configured to represent a specific class $c$ and FPN level $l$. For $m$ prototypes, let $S=\{(c_{j},l_{j},j)\}_{j=1}^{m}$ represent our prototype configuration, and denote our prototypes as $\mathbf{P}=\{\mathbf{p}^{(c,l,j)}\}_{S}$ where the $j$-th prototype is from class $c$ with FPN level $l$. Each prototype is $1\times 1\times d$ so that each prototype has the same feature dimension $d$ as the convolutional feature pyramid. As in IAIA-BL [4], the prototypes can be interpreted as a characteristic pattern representing a specific class. It can be visually understood by examining a segment of the training image where this pattern was derived.

Once we have computed each feature map in the convolutional feature pyramid $f(x),$ we compute the similarity between each prototype in prototype layer $g$ and the corresponding feature map. The FPN-IAIA-BL similarity score $s_{j}$ differs from that of IAIA-BL in three ways.

First, because the prototypes are assigned to specific FPN levels $l$, similarities for a set of prototypes $\mathbf{p}^{(\cdot,l,\cdot)}$ are computed only using the feature map from the same FPN level $\mathbf{z}^{(l)}$. Second, instead of using inverted $L_{2}$ distance based similarity, we use a cosine similarity as described in [8, 26]. The cosine similarity is calculated between a prototype and each $1\times 1\times d$ patch within the corresponding feature map. We denote the patch in a feature map of size $\eta_{l}\times\eta_{l}\times d$ as $n\in\{(1,1),\dots,(1,\eta_{l}),(2,1),\dots,(\eta_{l},\eta_{l})\}$. Thus, the cosine similarity for a single patch is:

$$s^{(l)}_{j,n}=\frac{\textbf{z}_{n}^{(l)}}{||\textbf{z}_{n}^{(l)}||}\cdot\frac{\textbf{p}^{(c,l,j)}}{||\textbf{p}^{(c,l,j)}||}$$ (2)

Third, in order to focus activation on the most salient features in each image, we use focal similarity as introduced in ProtoPool [19]. Retaining the top-k average pooling from Kalchbrenner et al. [12] and IAIA-BL [4], focal cosine similarity is computed as:

$$g(l,j)=\frac{1}{k}\sum\text{top}_{k} (\{s^{(l)}_{j,n}\}_{n=(1,1)}^{(\eta_{l},\eta_{l})})\;-$$ (3)

$$\frac{1}{\eta_{l}^{2}}\sum_{n=(1,1)}^{(\eta_{l},\eta_{l})}(\{s^{(l)}_{j,n}\}_{n=(1,1)}^{(\eta_{l},\eta_{l})})$$

The last stage of FPN-IAIA-BL is a fully connected layer $h$ which weights the similaritie scores and applies a softmax to predict probabilities for each mass-margin class.

4 Data and Training

Figure 3: Case-based explanation generated by FPN-IAIA-BL. This circumscribed (circ.) lesion is correctly classified as circumscribed. a, Test images. b, Activation of prototype on test images. c, Most relevant part of prototype. d, Learned prototypical lesion. e, Prototype self-activation. f, Contribution to class score. This visualization format for this figure matches that of [4].

The dataset, previously studied in [4], includes 2D digital breast x-rays from patients at the Duke University Health System taken between 2008 and 2018. Data collection was approved by Duke Health IRB and labeled by a fellowship-trained breast imaging radiologist. While IAIA-BL used only the subset of the images that contained a lesion, we also introduce a negative class which consists of images of tissue without lesions. Supplement Section C details how the data for this class were generated.

The training of FPN-IAIA-BL consists of three stages: (A) a warmup stage, (B) a projection of prototypes, and (C) full network fine-tuning. Because we use a trained VGG-16 backbone from IAIA-BL to construct our FPN, we first freeze the VGG-16 backbone in Stage A to warm up all the other layers. Stage B projects the learned prototype vectors onto a patch from any input image’s corresponding feature map in the same fashion as in [4, 6]. Stage C continues these two stages and unfreezes the VGG-16 backbone to allow for fine-tuning of the full network.

For stages A and C, we minimize the loss function:

$$\ell=\text{CE}+\lambda_{1}\ell_{clust}+\lambda_{2}\ell_{sep}+\lambda_{3}\ell_{ortho}+\lambda_{4}\ell_{fine}$$ (4)

where cross entropy (CE) penalizes misclassification and $\lambda_{1},\lambda_{2},\lambda_{3},\lambda_{4}$ are coefficients chosen empirically to balance the cluster ($\ell_{clust}$), separation ($\ell_{sep}$), and orthogonality ($\ell_{ortho}$) losses as defined in [8] and fine-annotation loss ($\ell_{fine}$) modified from [4]. The modifications to the fine-annotation loss introduce user-configurable coefficients which encourage and penalize the model for activating inside and outside the fine annotations differently for each class pair. Supplement Section B details the fine-annotation coefficients.

These loss terms have not previously been combined.

Figure 4: FPN-IAIA-BL in comparison to other saliency methods (adapted from [4]). We compare explanations from FPN-IAIA-BL with GradCAM [20], GradCAM++ [5], ProtoPNet [6], and IAIA-BL [4]. GradCam and GradCAM++ are two popular saliency explanation methods, and ProtoPNet and IAIA-BL are case-based explanation methods. The explanations from FPN-IAIA-BL highlight the most important parts of the lesion margin.

(a) Circumscribed

(b) Spiculated

Figure 5: Learned prototypes at different FPN-levels. FPN-level 2 prototypes are more localized because they are learned from the base of the feature pyramid which is a finer-grained feature map while FPN-level 5 prototypes are learned from the top of the feature pyramid, a coarser-grained feature map.

5 Experiments and Results

In our experiments, we find that FPN-IAIA-BL is able to learn localized prototypes that achieve acceptable performance. An interpretable visual result of the FPN-IAIA-BL is shown in Figure 3 and is compared to baselines in Figure 4. The best performing FPN-IAIA-BL model was able to achieve an average AUROC of 0.88 with one-vs-rest AUROC’s of 0.865 for circumscribed, for indistinct, and 0.908 for spiculated margin classes. A further comparison of the performance with IAIA-BL and an uninterpretable baseline (VGG16) is presented in Table 1. The confusion matrix of this model is shown in Supplement Section A.

	Avg. AUROC	Circ.	Ind.	Spic
FPN-IAIA-BL	0.88	0.87	0.86	0.91
IAIA-BL	0.95	0.97	0.93	0.96
VGG16	0.95	0.95	0.94	0.95

Table 1: AUROC metrics for FPN-IAIA-BL as compared to IAIA-BL and the uninterpretable baseline (VGG16).

As shown in Figure 5, prototypes from each FPN level represent relevant features from multiple scales. FPN-level 2 localizes to the most fine-grained features, and FPN-level 5 activations cover large swaths of the image. The model successfully learned prototypes at each FPN-level that captured information of different scales. In our application for mass margin classification, FPN-level 3 provided prototypes that activated on the most specific and salient parts of the margin. In other applications, the FPN-level of each prototype can be configured such that the prototypes capture the most relevant scale of information for the application. Figure 4 compares the activation maps provided by FPN-IAIA-BL, IAIA-BL, ProtoPNet, GradCAM and GradCAM++. The explanations from FPN-IAIA-BL highlight the most important parts of the lesion margin.

5.1 Limitations

While FPN-IAIA-BL consistently produces prototypes for circumscribed and spiculated lesion that our radiology team finds compelling, the prototypes for indistinct margins often activate outside of the lesion. This could be because an indistinct margin is defined as a faded, soft boundary between the lesion and normal tissue, and soft boundaries can occur in healthy breast tissue. Additionally, the AUROC for FPN-IAIA-BL is lower than that of IAIA-BL (0.951 overall) and the uninterpretable baseline (0.947 overall). This is because FPN-IAIA-BL architecture is larger and harder to train than IAIA-BL and the baseline.

6 Conclusion

We presented FPN-IAIA-BL, a novel neural network architecture for multi-scale case-based reasoning. We showed its effectiveness for the task of breast lesion margin classification, creating a model that can articulate more detailed reasoning behind its predictions, improving interpretability.

Supplementary Material

Appendix A Confusion Matrix for FPN-IAIA-BL

Figure 6 contains the confusion matrix for FPN-IAIA-BL on the test set. It has the highest specificity and lowest sensitivity for the circumscribed class.

Figure 6: Confusion matrix for predictions on the test dataset.

Appendix B Fine Annotation Coefficients

FPN-IAIA-BL introduces fine-annotation coefficients ${\lambda_{\text{in}}^{(y(i),c)},\lambda_{\text{out}}^{(y(i),c)}}$ which are used in the fine-annotation loss to encourage and penalize the model for activating inside and outside the fine annotations. For example, it is considered “worse” for a spiculated prototype to activate on a circumscribed lesion than for a circumscribed prototype to activate on a spiculated lesion. The fine-annotation coefficients designed by board-certified radiologist F.S. are as follow in tables 2 and 3.

		Prototype Class
		Circ.	Ind.	Spic.	Neg.
Sample’s Class	Circ.	1	1	1	1
	Ind.	1	1	1	1
	Spic.	1	1	1	1
	Neg.	0	0	0	0

Table 2: Fine annotation coefficients penalizing the prototypes from class $c_{proto}$ from activating outside fine annotations for a sample from class $c_{i}$

		Prototype Class
		Circ.	Ind.	Spic.	Neg.
Sample’s Class	Circ.	0	0	0	1
	Ind.	0	0	0	1
	Spic.	1	1	0	1
	Neg.	0	0	0	0

Table 3: Fine annotation coefficients penalizing the prototypes from class $c_{proto}$ from activating inside the fine annotations for a sample from class $c_{i}$.

Incorporating the fine-annotation coefficients, the fine-annotation loss is now defined as:

$$\ell_{\text{fine}}=\sum_{i\in D^{\prime}}\sum_{\mathbf{p}^{(c,l,j)}}\Big{(}||\lambda_{\text{in}}^{(y(i),c)}\mathbf{m}_{i} \odot\text{PAM}_{i,j}+$$ (5)

$$\lambda_{\text{full}}^{(y(i),c)}\text{PAM}_{i,j}||_{2}\Big{)}$$

where the prototype activation map $\text{PAM}_{i,j}$ is computed by bilinearly upsampling the similarity map $[s_{j,n}]_{n=(1,1)}^{(\eta_{l},\eta_{l})}$ for prototype $\mathbf{p}^{(c,l,j)}$ and image $\mathbf{x}_{i}$ such that it has the same dimensions as the fine-annotation mask $\mathbf{m}_{i}$.

Appendix C Negative Class Data

As discussed in Section 4, we include a negative class during training to discourage “classification by elimination.” The negative class data consist of 5,000 image-mask pairs. The negative class images were created by sampling the full-size mammogram images and cropping to a section without any of the lesion region of interest for each image. We pair each image with a fully negative mask where no region of interest is identified in the mask. For training, we randomly select a subset of 200 negative samples.