About the generalizability of deep learning based image quality assessment in mammography

The EUREF guidelines recommend CDMAM phantoms to assess the image quality of a mammography unit [4, 5]. The CDMAM phantom comprises a 1 mm thick aluminium plate and a polymethyl methacrylate (PMMA) with grid structure integrated in the PMMA. The total, thickness of the phantom is 5 mm. One image of a CDMAM phantom (version 3.4) is displayed in figure 1. It consists of 205 cells and each cell contains two gold cylinders. The gold cylinders in different cells vary in their diameters and thicknesses. The thickness of the gold cylinders differs on a logarithmic scale. In total, the CDMAM phantom contains gold cylinders of 16 different diameters [4, 5]. In each cell, one of the gold cylinders is placed in the centre and the other one is located randomly in one of the four corners of the cell. The detection limit of the mammography unit is evaluated by comparing the pixel intensities in each of the four corners of the cell, for details see [4, 5]. To locate the disks at each cell, a template is created. The average pixel value under each template is calculated and the corner in which the average has the highest value is chosen. For each diameter, the CDC shows the minimum thickness of a gold cylinder such that it can still be detected. The lower the CDC, the better the image quality of a mammography unit is therefore judged, and the image quality is viewed as being sufficiently good when the CDC lies below a pre-defined limiting CDC.

Zoom In Zoom Out Reset image size

To obtain reliable CDCs, the guidelines recommend using at least 16 images of a CDMAM phantom. These images are processed by an automated readout which computes the threshold for 12 of the 16 available diameters. To make this readout comparable with the human-readout method, these values are rescaled by an experimentally found relation [13, 14]. The automated readout can be carried out by the CDMAM Analyser software. If the automated readout fails, it is always possible to perform a human readout to determine the CDC.

We have simplified the determination of a CDC via a regression task solved by a NN, which only needs one single image as had already been suggested in [10]. We extended and generalized the results obtained in [10] by using a scale-independent objective function and a larger database to train the NN. To this end, we have tested our approach on real images of 31 mammography units. Table 3 in the appendix provides a summary of the radiographic factors of the images and mammography units. This method of evaluating the quality of the synthetic data by testing the performance of the NN on real data is not uncommon [15].

In mammography, synthetic data in the context of IQA have been used before [16, 17]. To train our NN, we used synthetic data generated by the data generator from [12]. This simulator uses a simulated phantom as an accumulation of voxels with certain features associated with each voxel that mimic physical properties of the real phantom.

In the following we describe the important steps, such as the generation of the data, our objective function and the preprocessing methods in detail.

2.2.1. Generating synthetic data

2.2.2. Training the NN

We trained the NN to predict for a single image the same CDC as the CDMAM Analyser software which uses at least 16 images. We have therefore taken the CDC obtained by the CDMAM Analyser software as our ground truth ³ .

In general, the mean squared error (MSE) is a suitable objective for training statistical models for regression problems. It minimizes the squared differences between the prediction $\hat{y}$ and the true value y, but this objective led to a systematic bias for some scenarios in our experiments. This might be related to the scale dependency of the MSE. To account for the different magnitudes of the diameters, we conclude that the mean squared log error (MSLE)

$$\begin{equation} \begin{split} \mathrm{MSLE}\left(y, \hat{y}\right) = \frac{1}{n} \sum_{i = 1}^n \sum_{j = 1}^C \left(\log \hat{y}_i^j - \log y_i^j \right)^2 = \frac{1}{n} \sum_{i = 1}^n \sum_{j = 1}^C \left(\log \frac{\hat{y}_i^j}{y_i^j} \right)^2 \end{split} \end{equation} \tag{ 1 }$$

is a better suited objective to perform optimization in our case. The MSLE takes the mean over all n samples while accumulating the differences of all C classes. For a CDC we have C = 12 representing the 12 different diameters that are used by the CDMAM Analyser software.

Let us mention some important properties of (1):

• 1.  
  It is well-defined only if the quotient $\hat{y}_i^{\,j} /y_i^{\,j}$ is positive.
• 2.  
  It is a positive function and the minimum is obtained if and only if the prediction coincides with the true value: $\forall i,j: \hat{y}_i^{\,j} = y_i^{\,j}$.

The true values are always positive but the predictions can be close to zero or even negative. Especially at the beginning of the training the NN can produce negative predictions. Therefore, we cannot use (1) as an optimization objective. We shall approximate (1) such that it is well-defined for all values ⁴ . To this end, we propose

$$\begin{equation} \begin{split} \mathcal{L}_{\beta}\left(y, \hat{y}\right) & = \frac{1}{n} \sum_{i = 1}^n \sum_{j = 1}^C \left(\ln \left( h_{\beta}\left(\frac{\hat{y}_i^{\,j}}{y_i^{\,j}}\right) + \epsilon \right) \right)^2 \quad \textrm{with} \\ h_{\beta}\left(x\right) & = g_{\beta}\left(x + f\left(\beta\right) \right) \in \mathbb{R}_{>0}, \\ g_{\beta}\left(x\right) & = \frac{1}{\beta} \ln\left(1 + \mathrm e^{\beta x}\right), \\ f\left(\beta\right) & = \frac{\ln\left(\mathrm e^{\beta} - 1\right)}{\beta} - 1 \end{split} \end{equation} \tag{ 2 }$$

as an approximation of (1). The softplus function $g_{\beta}(x)$ is strictly positive and approximates the identity function for large values of x which can be seen by a Taylor expansion. This implies that (2) is always well-defined. For (2) to be a proper approximation of the objective (1), we have to ensure that the minima of $\ln(x)^2$ and $\ln (h_{\beta} (x))^2$ coincide. $f(\beta)$ shifts the position of the minimum of $\left(\ln g_{\beta}(x)\right)^2$ such that

$$\begin{align*} \mathop{\mathrm{argmin}}_{x \in \mathbb{R}} \left(\ln h_{\beta}\left(x\right)\right)^2 = 1 = \mathop{\mathrm{argmin}}_{x \in \mathbb{R}_+} \left(\ln x\right)^2. \end{align*}$$

In addition we have added a regularizer ε > 0 for numerical stability, because the logarithm might diverge for arguments close to zero. The constant ε guarantees finite results of the objective function while training. For our analysis we fixed

$$\begin{equation*} \begin{split} \beta & = 3 \quad \textrm{and} \quad \epsilon = 10^{-20}, \end{split} \end{equation*}$$

but the qualitative form of (2) is independent of these values as can be seen in figure 2.

Zoom In Zoom Out Reset image size

2.2.3. Preprocessing: reducing the image size

In this section we present the prediction of the CDCs on real images obtained for (ResNet18, 500) for all the resizing methods. The ground truth CDCs (black, solid) are obtained by the CDMAM Analyser software from at least 16 images for each unit, but the NN predicts each CDC (colored, dashed) based on a single image alone. We plotted the CDCs for 18 out of 31 different mammography units. To give a good impression of the performance of the NN, we have shown the nine best CDCs (in figure 3) as well as the 9 worst CDCs (in figure 4). We selected these CDCs based on the visual (dis)agreement of the ground truth and the predictions. Studying the CDCs, it can be observed that the predictions do not look smooth in contrast to the true values. The reason for this is that we have not trained the NN to predict the functional form

$$\begin{equation} \begin{split} f\left(x\right) = \sum_{i = 0}^3 a_i x^{-i} \quad \textrm{with }a_i \unicode{x2A7E} 0 \end{split} \end{equation} \tag{ 3 }$$

of the CDCs, which is estimated by the CDMAM Analyser software. Instead, we trained it to predict each diameter independently [4, 5]. It is thus not unexpected that these predictions do not reproduce this functional form exactly. In fact, some of the CDCs predicted by the trained NNs slightly violate monotony. Enforcing monotony could be achieved, for example, by training a NN to predict the coefficients in (3). However, we chose to predict the CDC for each diameter independently in order to not restrict the approach to a specific functional form.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

In figure 4, it can be noted that predictions of some units have a bias—their predictions are either above or below the true value. Even though there are some mammography units on which the performance of our NN is limited, we see in figure 3 that many units are fitted very well. Furthermore, these plots suggest that an ensemble of NNs with different resizing methods could enhance the quality of the prediction. For a quantitative assessment we refer to the tables 1 and 2 below.

To obtain an overview of the image-wise performance, we display in figure 5 the image-wise correlation between the predicted and true minimal thickness of the 12 relevant gold cylinders for cubic resizing. This plot gives complementary information because it shows the different predictions of the NN for differently selected images; each point represents the minimal detected thickness for one diameter value of one image of the CDMAM phantom for a specific mammography unit. All points with the same color belong to the same mammography unit. We see that the deviation from the diagonal appears to be small. This indicates that the NN makes meaningful predictions. In addition, points of the same color look clustered in this scatterplot. This implies that the predictions have small variations with respect to different inputs for the same mammography unit. Hence, the NN is able to predict the CDC based on a single image. For all scenarios, the prediction quality and its variation are made quantitative in tables 1 and 2.

Zoom In Zoom Out Reset image size

Our experiments show that the NN makes good predictions under the chosen scenario for almost all units. In table 1 we compare the different scenarios. First, we computed the average value of the approximated root MSLE (ARMSLE) per unit and diameter. To obtain a sensible dimensionless quantity, we related it to $r_j = \log_{10} l_{j;\max} - \log_{10} l_{j;\min}$, where $l_{j;\max}$ and $l_{j;\min}$ denote the maximum and minimum of the thicknesses for each diameter j with respect to the CDMAM phantom (version 3.4). Then we averaged this quantity over all C diameters. We call this quantity normalized ARMSLE (NARMSLE):

$$\begin{equation} \begin{split} \mathcal{L}_{\mathrm{device}}\left(d\right) = \frac{1}{C}\sum_{j = 1}^C \frac{1}{r_j} \sqrt{\frac{1}{m_d} \sum_{i = 1}^{m_d} \left( \log_{10} \left( h_{10;\beta} \left( \frac{\hat{y}_{d,i}^{\,j}}{y_{d,i}^{\,j}}\right) + \epsilon \right) \right)^2} \end{split} \end{equation} \tag{ 4 }$$

where m_d is the number of images per unit d. In contrast to (2), we used the base 10 instead of e for this evaluation because the CDCs are plotted on a logarithmic scale of base 10. This change of base is indicated by an additional subscript. The overall performance has been assessed by the median and the rescaled median absolute value of NARMSLE of all the units. The median absolute value has been rescaled by the inverse of 0.675 (rMAD) ⁶ . The median and rMAD

$$\begin{equation*} \begin{split} \mathcal{L}_{\mathrm{Median}} & = \mathrm{Median} \left\{\mathcal{L}_{\mathrm{device}}\left(1\right), \mathcal{L}_{\mathrm{device}}\left(2\right), \ldots, \mathcal{L}_{\mathrm{device}}\left(31\right)\right\}, \\ \mathrm{rMAD} & = \frac{1}{0.675} \mathrm{Median} \left\{\lvert \mathcal{L}_{\mathrm{device}}\left(1\right) - \mathcal{L}_{\mathrm{Median}} \rvert, \ldots, \lvert \mathcal{L}_{\mathrm{device}}\left(31\right) - \mathcal{L}_{\mathrm{Median}} \rvert \right\} \end{split} \end{equation*}$$

of NARMSLE for all of these units are shown in table 1. These results demonstrate that our method depends only slightly on the architecture and preprocessing method. We highlighted all the scenarios in bold whose $\mathcal{L}_{\mathrm{Median}} + \mathrm{rMAD} \unicode{x2A7D} \tau$ to show which of them performed particularly well. We chose τ = 0.075. However, we observed that the NN had had difficulties in fitting device 8, device 14 or device 18 in some scenarios. We elaborate on this issue in the appendix. To improve the performance for a specific scenario, the optimization process should however be fine-tuned to that scenario. We have not done any scenario-specific fine-tuning to show the generalizability of our method. In this analysis the error of the ground truth has not been considered. This means that it has been assumed that the CDMAM Analyser software gives the same prediction for every subset of at least 16 images.

Furthermore, the predictions of the NN hardly depend on the chosen image. Table 2 displays the variation of the predictions. A small value indicates that the prediction of the NN is insensitive to the chosen image recorded by the device. To evaluate this variation for a unit d, the prediction of each of its images was compared to its mean prediction $\bar{y}^{\,j}_d = \frac{1}{m_d} \sum_i y^{\,j}_{d,i}$ by formula (4) (and replacing the ground truth y with the mean of the predictions $\bar{y}$). Table 2 shows the median and rMAD with respect to these quantities. For most scenarios the variations are less than 1%.

These results suggest that the prediction of the NN does not vary significantly when the number of images is increased to obtain a prediction. Indeed, we checked for several units that this is the case. The prediction of the NN is rather stable across the number of images to predict the CDC.

Table 1 shows that our approach works for almost all mammography units since the median and the rMAD of the chosen quality metric of the results are smaller or equal to τ = 0.075. However, we observed that in some scenarios the NN could not predict a few units well. A detailed analysis of this issue reveals that there are three problematic units: device 8, device 14 and device 18.

In figure 6 we give an example of this phenomenon. In this scatterplot the image-wise predictions by (EfficientNetS, Nearest, 250) are visualized. It can be seen that most units are fitted better by (EfficientNetS, Nearest, 250) than by (ResNet18, Cubic, 500), cf figure 5, but the three units: device 8 (light pink), device 14 (cyan) and device 18 (neon-green) perform worse. Furthermore, for these problematic units the NN shows a high variance as the predictions vary more strongly. Thus there is a relationship between prediction quality and variance of the prediction.

Zoom In Zoom Out Reset image size

In addition, in our experiments we could not trace back the occasionally insufficient performance of these devices to any scenario. Hence, we conclude that this phenomenon is related to the training data itself. To improve upon this, different data augmentations to cover the idiosyncrasies of these images, a different simulator or the inclusion of real data could be applied.

In this study we tested our approach on 31 real mammography units. In this section we summarize some key aspects of these units to characterize their variety and provide the specifications of the tested mammography units in table 3.

We considered four types of image receptors: 9 Selenium detector, 19 CsI–Si detector, 2 computed radiography system and 1 photon counting system. These units cover the following anodes and filters: tungsten (W), rhodium (Rh), molybdenum (Mo) and aluminum (Al) where the anode/filter configuration Rh/Rh is most common; this configuration is used by 18 units with CsI–Si detector. For these 18 units we varied between three different tube current products of 32 mAs, 63 mAs and 125 mAs and six different versions of technical phantoms which are characterized by their serial number. Overall, the applied tube voltage (TV) ranges from 27 kV to 31 kV and the current time product (CTP) varies between 17 mAs and 140 mAs.