Deep-learning-based Detection and Segmentation-assisted Management on

Brain Metastases

Downloaded from https://academic.oup.com/neuro-oncology/advance-article-abstract/doi/10.1093/neuonc/noz234/5684915 by University of Toronto user on 25 December 2019

Corresponding author: Ying Mao, Yingchao Liu, Dengwang Li

t
Funding: This work was supported in part by the National Natural Science Foundation of China (No.

ip
61802234, 61876101, 61971271), Natural Science Foundation of Shandong Province (No.

cr
ZR2019QF007), China Postdoctoral Project (No. 2017M612339), Key Technologies R & D Program of

us
Shandong Province (2017CXCG1209), and the Taishan Scholars Program (No. tsqn20161070).
an
Disclosures of Conflicts of Interest: All the authors declare no relevant relationship with any funding

agencies or commercial institutes.

M

Authorship Jie Xue1#, Bao Wang2#, Yang Ming3, Xuejun Liu4, Zekun Jiang5, Chengwei Wang6, Xiyu Liu1,

Ligang Chen3, Jianhua Qu1, Shangchen Xu7,8, Xuqun Tang9, Ying Mao9*, Yingchao Liu7,8*, Dengwang Li5*
ed

1
School of Business, Shandong Normal University, Jinan, 250014, China
pt

2
Department of Radiology, Qilu Hospital of Shandong University, Jinan, 250012, China
ce

3
Department of Neurosurgery, The Affiliated Hospital of Southwest Medical University, Luzhou,
Ac

646000, China

4
Department of Radiology, the Affiliated Hospital of Qingdao University Medical College, Qingdao,

266003, China

5
Shandong Key Laboratory of Medical Physics and Image Processing, School of Physics and

Electronics, Shandong Normal University, Jinan, 250014, China

6
Department of Neurosurgery, the Second Hospital of Shandong University, Jinan, 250033, China

© The Author(s) 2019. Published by Oxford University Press on behalf of the Society for Neuro-Oncology. All rights
reserved. For permissions, please e-mail: [email protected]
 N-O-D-19-758R1

7
Department of Neurosurgery, Shandong Provincial Hospital Affiliated to Shandong First Medical

University, Jinan, 250021, China

Downloaded from https://academic.oup.com/neuro-oncology/advance-article-abstract/doi/10.1093/neuonc/noz234/5684915 by University of Toronto user on 25 December 2019

8
Department of Neurosurgery, Shandong Provincial Hospital Affiliated to Shandong University,

Jinan, 250021, China

9
Department of Neurosurgery, Huashan Hospital Affiliated to Fudan University, Shanghai, 200040,

t
China

ip
﹟These authors contributed equally to this work.

cr
us
Authorship statement

Jie Xue, Bao Wang, Ying Mao, Dengwang Li and Yingchao Liu conceived and designed the study. Jie
an
Xue performed the experiments. Bao Wang and Yingchao Liu provided data analysis. Jie Xue and Bao

Wang wrote the paper. Yang Ming, Xuejun Liu, Zekun Jiang, Chengwei Wang, Xiyu Liu, Ligang Chen,
M

Jianhua Qu, Shangchen Xu, Ying Mao, Yingchao Liu and Dengwang Li reviewed and edited the
ed

manuscript. All authors read and approved the manuscript.

pt
ce
Ac

2
 N-O-D-19-758R1

Abstract

Downloaded from https://academic.oup.com/neuro-oncology/advance-article-abstract/doi/10.1093/neuonc/noz234/5684915 by University of Toronto user on 25 December 2019

Background: Three-dimensional T1-magnetization-prepared rapid gradient-echo (3D-T1-MPRAGE) is

preferred in detecting brain metastases (BMs) among MRI. We developed an automatic deep-

learning-based detection and segmentation method for BMs (named BMDS net) on 3D-T1-MPRAGE

images and evaluated its performance.

t
Methods: The BMDS net is a cascaded 3D fully convolution network (FCN) to automatically detect

ip
and segment BMs. In total, 1,652 patients with 3D-T1-MPRAGE images from three hospitals (1,201,

cr
231 and 220, respectively) were retrospectively included. Manual segmentations are obtained by a

us
neuroradiologist and a radiation oncologist in a consensus reading in 3D-T1-MPRAGE images.

Sensitivity, specificity and dice ratio of the segmentation were evaluated. Specificity and sensitivity
an
measure the fractions of relevant segmented voxels. Dice ratio was used to quantitatively measure

the overlap between automatic and manual segmentation results. Paired samples t tests and
M

analysis of variance were employed for statistical analysis.

ed

Results: The BMDS net can detect all BMs, providing a detection result with an accuracy of 100%.

Automatic segmentations correlated strongly with manual segmentations through 4-fold cross
pt

validation of the dataset with 1,201 patients: the sensitivity was 0.96±0.03 (range, 0.84-0.99), the
ce

specificity was 0.99±0.0002 (range, 0.99-1.00), and the dice ratio was 0.85 ± 0.08 (range, 0.62-0.95)

for total tumor volume. Similar performance on the other two datasets also demonstrate the
Ac

robustness of BMDS net in correctly detecting and segmenting BMs in various settings.

Conclusions: The BMDS net yields the accurate detection and segmentation of BMs automatically

and could assist stereotactic radiotherapy management for the diagnosis, therapy planning and

follow up.

Key words: deep learning, fully convolution network, brain metastases, stereotactic radiotherapy,

MRI

3
 N-O-D-19-758R1

Key points

Downloaded from https://academic.oup.com/neuro-oncology/advance-article-abstract/doi/10.1093/neuonc/noz234/5684915 by University of Toronto user on 25 December 2019

1. A 3D deep learning model was developed for detection and segmentation of brain metastases.

2. Experimental results showed a good consistency with manual delineations by human experts.

3. BMDS net can facilitate the stereotactic radiotherapy management on brain metastases.

t
ip
Importance of the Study

cr
Detection and segmentation are essential for the clinical management of brain metastases, including

us
the diagnosis, therapy planning and follow up. Nevertheless, the manual delineations are time
an
consuming and subject to operators. With the recent advances, deep learning methods have

obtained superior performance in medicine. We developed a deep-learning-based method for the

M

automatic detection and segmentation of brain metastases. Our method agrees with those of

human experts, resulting in a fast differential diagnosis, tumor delineation and tumor burden
ed

assessment with less subjective bias, and the performance was validated in an external dataset,

facilitating the clinical management of brain metastases by assisting clinicians.

pt
ce
Ac

4
 N-O-D-19-758R1

Introduction

Downloaded from https://academic.oup.com/neuro-oncology/advance-article-abstract/doi/10.1093/neuonc/noz234/5684915 by University of Toronto user on 25 December 2019

Brain metastases (BMs) is the most common malignant tumor in the central nervous system.

Patients with BMs show a poor outcome without efficient treatment 1. Both an accurate diagnosis

and therapeutic strategy have a strong influence on the clinical outcomes of BMs 2. Benefiting from

efficient local control, stereotactic radiotherapy (SRT) is becoming popular in the treatment of BMs

t
ip
3-5
even in cases with more than 10 lesions and is often used instead of whole-brain radiotherapy

cr
(WBRT), yielding >80%–90% durable local control.

us
Detection and segmentation are the necessary steps in the clinical management of SRT of BMs.

However, manual delineation is time consuming and often irreproducible 6. Although time taken for
an
the BMs is associated with their size, a median value of 2.8 minutes was still recorded in finishing the

manual segmentation of per lesion in our clinical practice, while a deep learning method could
M

perform the work within 25 seconds. Moreover, omitting small and even large BMs is incidentally
ed

seen during the planning of gamma knife radiosurgery in clinical practice, because of carelessness,

exhaustion and other human factors 7. Additionally, interobserver variability in target volume
pt

8,9
delineation is another important contributor to uncertainty in SRT . Therefore, there is a calling
ce

need to develop an efficient computer-aided detection and segmentation method to help radiation

oncologists identify and cover BMs, especially under the condition of multiple brain metastases 10.
Ac

Various methods have been used to detect brain tumors. Pereira et al. 11 proposed the automatic

segmentation method based on convolutional neural network for the segmentation of glioma.

Symmetrically driven FCN 12 completed segmentation of glioma using multimodal data. Havaei et al.

13
proposed a new segmentation framework for heterogeneous multimodal MRI, which allows

segmentation of data that lacks certain modalities. Hu et al. 14 extracted multi-scale features through

multi-level convolutional neural networks, and used CRF to combine spatial context information to

15
segment glioma. A priori-driven FC-CRF combined with prior knowledge of the tumor core was

5
 N-O-D-19-758R1

used to segment the tumor core. Feature recombination and spatial adaptive recalibration blocks

were proposed, and the feature maps were recalibrated to improve the segmentation result of

Downloaded from https://academic.oup.com/neuro-oncology/advance-article-abstract/doi/10.1093/neuonc/noz234/5684915 by University of Toronto user on 25 December 2019

16 17
glioma . Chen et al. proposed the dual-force training strategy that attaches the auxiliary loss

function to the network to explicitly encourage the learning of multi-level features. Interactive

18,19
methods were also used for segmentation of glioma. However, few works studied in brain

20
metastases. Farjam et al. used a set of unevenly spaced 3D spherical shell templates to localize

t
21

ip
BMs using T1-weight MRI. Robert et al. proposed spherical tumor appearance models to match

the expected geometry of BMs. Krafft et al. 22 utilized soft independent modeling of class analogies

cr
to cluster brain metastases.

us
To our best knowledge, no study has investigated the fully automatic segmentation of BMs
an
because of the following reasons. 1) many very small BMs exist in the brain; 2) niduses show various

distributions among either individual participants or patient groups; 3) the intensities of voxels in
M

niduses would be very similar to those of the neighboring tissues if patients were administrated with

some specific targeted drugs . With the recent advances, deep learning methods have been
23
ed

regarded as a very promising approach for medical image processing, including brain tumor

24,25
segmentation . Davy et al. divided 3D MR images into 2D images and trained a convolutional
pt

24 26 27
neural network (CNN) to predict their center pixel class . Urban et al. and Zikic et al.
ce

implemented a fairly common CNN, comprising a series of convolutional layers, a nonlinear

activation function between each layer and a softmax output layer. However, brain metastases are
Ac

uneven and sometimes very small in size. For such tumors, deep learning-based segmentation

methods can be disrupted by the nontarget region, which often occupies a large fraction in

postcontrast T1-weighted MR images. Furthermore, the methods above divided the 3D MR images

24,27 26
into 2D or 3D patches to train a CNN, unavoidably missing some spatial information across

slices. These deficiencies often resulted in inaccurate segmentation results compared to experts’

segmentations, impeding their further applications in clinical practice.

6
 N-O-D-19-758R1

Aimed to overcome the abovementioned challenges and compare with the state of art method,

we proposed and evaluated a cascaded 3D fully convolution network for the detection and

Downloaded from https://academic.oup.com/neuro-oncology/advance-article-abstract/doi/10.1093/neuonc/noz234/5684915 by University of Toronto user on 25 December 2019

segmentation of BMs (named BMDS net) using whole 3D MR images. Our hypothesis is that the

proposed BMDS net will provide accurate detection and segmentation of BMs and outperform the

classical FCN. In this approach, the first FCN focuses on fast location of the region of the brain

metastases, and segmentation by the second FCN is implemented on the detected region, thereby

t
ip
reducing the influence of similar or neighboring tissues to assist the precise management of BMs

with SRT.

cr
Materials and Methods

us
Study populations
an
This study was approved by the Institutional Review Board of the three hospitals, and all
M
experiments were performed in compliance with the Declaration of Helsinki. Written informed

consent was acquired from each patient or next of kin.

ed

From October 2016 to May 2019, 1,201 cases of patients with BM from the Shandong Provincial

Hospital were retrospectively collected as Dataset 1. Two hundred thirty-one cases had been
pt

retrospectively collected from the Affiliated Hospital of Qingdao University Medical College as
ce

Dataset 2, and 220 cases were collected from the Second Hospital of Shandong University as Dataset

3. The detailed distribution of all datasets and inclusion and exclusion criteria are provided in
Ac

Supplemental Content 1. The summary results of the three datasets are provided in Supplemental

Table S1.

MRI acquisitions

All the patients from the three different institutes were examined using 3.0 T MRI machines but

with different manufacturers and imaging parameters. Detailed information about the MR machines

and imaging parameters of the three hospitals are provided in Supplemental Content 2.

7
 N-O-D-19-758R1

Manual Segmentation

Downloaded from https://academic.oup.com/neuro-oncology/advance-article-abstract/doi/10.1093/neuonc/noz234/5684915 by University of Toronto user on 25 December 2019

Manual segmentation of the BMs on postcontrast 3D T1 MPRAGE was performed by one

neuroradiologist and one radiation oncologist (with 10 years and 15 years of experience,

respectively) together using the noncommercial software ITK-SNAP (Version 3.6.0,

http://www.itksnap.org), and a consensus was reached after a careful discussion if divergence

existed. Only the whole BMs (including both enhancing area and necrotic area) were covered for

t
ip
segmentation, avoiding adjacent vessels. Lesions that were only visible in one axial slice were not

cr
considered as a lesion for segmentation. Any lesions that occupied over two axial slices would be

segmented whatever the sizes are. Time taken of manual segmentations, and the mean value of two

us
performers’ time on one lesion was considered as the time taken of this lesion.
an
BMDS Net Architecture
M
Deep learning models can learn a hierarchy of features by building high-level features from low-

level features. The FCN 28 is a special case of a convolutional network, in which the training is an end-
ed

to-end process and embodies fewer network parameters. The FCN comprises multiple convolution

and pooling layers that reduce the number of network parameters to a large extent, without
pt

incorporating any fully connected layers. The FCN can take inputs of arbitrary size and generate
ce

same-size outputs, through efficient inference and learning. It generates dense pixel outputs by

interpolating the coarse output via deconvolution layers. Recently, the 3D FCN was developed and
Ac

applied to deal with 3D imaging data 29. The proposed BMDS net employed and further extended the

3D FCNs, involving two stages—detection and segmentation. In the detection stage, we designed a

3D FCN to rapidly localize the region of brain metastases in raw 3D T1-weighted MR images, which

will propose bounding boxes for segmentation. This can be considered as coarse segmentation of

brain metastases. The FCN is devised for fast segment brain metastases from the downsampled MR

image. Specifically, for accurate detection, the original MR image is downsampled to ¼ of its original

resolution. After inference with the proposed FCN, we upsampled the coarse segmentation results

8
 N-O-D-19-758R1

to the original resolution. Next, the bounding box of brain metastases is obtained by morphological

operation on this coarse segmentation. The intermediate point of the bounding box is selected as

Downloaded from https://academic.oup.com/neuro-oncology/advance-article-abstract/doi/10.1093/neuonc/noz234/5684915 by University of Toronto user on 25 December 2019

the centroid to crop the region from the raw MR images. The region size is ensured to cover whole

sizes of brain metastases. In the segmentation stage, a second 3D FCN utilizes the proposed

bounding boxes to predict segmentation masks for the brain metastases. An illustration of the

framework is shown in Fig. 1. The BMDS net iteratively compares the output masks with reference

t
ip
masks. The reference masks ensure that the network learns the relationship between MR image

contexts and these reference masks. Given new MRI data, the trained BMDS net can output the

cr
location of the brain metastases and corresponding segmented masks.

us
BMDS Net Implementation Details
an
The training data for BMDS net consist of 3D T1-weighted MR images as input and manual

segmentations as the reference masks. The size of the image was 256×256×120. For preprocessing,
M

all the intensity values were first saturated into [0, 1000]. Next, we normalized intensities into the
ed

range of [0, 1] by dividing the maximum intensity value. Particularly, BMDS net does not need to

correct deviation, remove skull or register heterogeneous images on the training data, a procedure
pt

that is less time consuming.

ce

BMDS net was trained via the standard stochastic gradient descent (SGD) algorithm using a

stepped learning rate initialized at 0.002 and decayed over the training iterations at a rate of 0.1 until
Ac

the training reached 500 epochs. A momentum of 0.9 was used to make a tradeoff between the

previously observed image and newly observed image. All the network parameters were initialized by

Xavier’s 30 method.
31
BMDS net was implemented based on the widely used TensorFlow open-source framework on

an NVIDIA Tesla V100 GPU with 32 GB of memory. Our experiments were conducted by splitting the

dataset into four folds. In each round of the 4-fold cross-validation (CV-4) procedure, three data

9
 N-O-D-19-758R1

folds were employed as training cases and the remaining fold was used for testing. Ten percent of

the training cases were randomly selected for validation.

Downloaded from https://academic.oup.com/neuro-oncology/advance-article-abstract/doi/10.1093/neuonc/noz234/5684915 by University of Toronto user on 25 December 2019

Statistical Analysis

Statistical analyses were conducted using python language (V3.7, USA). Quantitative results are

displayed as mean (± standard deviation). Paired samples t tests were used in pairwise comparison

t
between BMDS net and its baseline model (i.e., one-FCN). To evaluate the performance of automatic

ip
32
segmentation, results were compared on the level of brain metastases volume. Dice ratio

cr
computed the overlap of ground truth segmentation Vg and automatic segmentation Vs:

us
2* || Vs  Vg ||
diceratio 
|| Vs  Vg ||
(1)
an
Specificity (or the true negative rate) and sensitivity (or the true positive rate) measure the
M

fractions of relevant segmented voxels. The definitions of these metrics are given below:

TN
ed

specificity 
TN  FP (2)
pt

TP
sensitivity 
TP  FN (3)
ce

The true positive (TP) score reflects the number of tumor voxels correctly identified as tumor
Ac

voxels. The false positive (FP) score reflects the number of non-tumor voxels incorrectly identified as

tumor voxels. The true negative (TN) score reflects the number of background voxels correctly

identified as background voxels. Finally, the false negative (FN) score reflects the number of non-

background voxels incorrectly identified as background voxels.

Results

BMDS net required approximately 23 hours to train, whereas labeling a single input image using the

trained model required approximately 24 s (4 s for detection and 20 s for segmentation). The

10
 N-O-D-19-758R1

median time taken for manual segmentation of one lesion was 2.8 minutes (interquartile range [2.1,

4.2]). For all 1,201 cases, the first, second and third quartiles of the tumor number (range from 1 to

Downloaded from https://academic.oup.com/neuro-oncology/advance-article-abstract/doi/10.1093/neuonc/noz234/5684915 by University of Toronto user on 25 December 2019

17) were 1, 2, and 4, respectively. The BMDS net could detect the presence of all brain metastases

(i.e., the detection accuracy was 100%), ensuring the identification of all target lesions without

omission. 3D visualization of the ground-truth segmentations and our detections are shown in Fig. 2.

Detection performance of BMDS Net

t
ip
In clinical practice, a higher positive predictive rate (ideally, 100%) is more preferable than a

cr
higher negative predictive rate because false segmentation can be excluded during the process of

us
planning. The BMDS net can detect all BMs, providing a detection result with an accuracy of 100%.

The minimal diameter of the smallest BM was 2 mm. The BMDS net could yield both high positive
an
predictive rate and negative predictive rate. The true positive rate (i.e., sensitivity) and true negative

rate (i.e., specificity) were 0.96±0.03 (range, 0.84-0.99) and 0.99±0.0002 (range, 0.99-1.00),
M

respectively. The diagnostic performance demonstrates the ability of BMDS net to assist the
ed

stereotactic radiotherapy management of BMs.

Segmentation performance of BMDS net on various volumes

pt

Because a lethal radiation dose of gamma knife radiotherapy could be performed on the small
ce

lesions with a diameter <= 5 mm, resulting an excellent local control. Under this condition,
Ac

interobserver variability in target volume delineation could have very limited influences on the

results of these very small lesions. In addition, Response Assessment in Neuro-Oncology Brain

Metastases (RANO-BM) suggests the target lesions for evaluation be 10 mm, but they also

supplemented that a smaller target lesion could be selected if the imaging slice thickness decreased.

Recently, Neil K, et al 33 chose to lower the minimum size limit of measurable disease to 5 mm using

a imaging slice thickness less than 1.5mm, which is gradually accepted by more institutions.

Therefore, we calculated the dice ratio of lesions with a diameter over 5 mm.

11
 N-O-D-19-758R1

For the Dataset 1 of 1,201 cases, the first, second and third quartiles of the mean brain metastasis

volume (MBMV) (which is per metastasis volume, range from 0.07 cm3 to 23.81 cm3) were 0.79 cm3,

Downloaded from https://academic.oup.com/neuro-oncology/advance-article-abstract/doi/10.1093/neuonc/noz234/5684915 by University of Toronto user on 25 December 2019

2.22 cm3, and 5.42 cm3, respectively, and the mean value was 4.01 cm3. The total brain metastasis

volumes (TBMVs) (which is total metastases in entire brain volume, range from 0.08 cm3 to 71.44

cm3) as the union of brain metastasis volumes from manual segmentations in the 3D T1-MPRAGE

images were 1.47 cm3, 3.86 cm3, and 10.45 cm3, respectively. The dice ratios were 0.81 ± 0.08

t
(range, 0.62-0.93) and 0.88 ± 0.07 (range, 0.71-0.95) for tumors of MBMV<4.01 cm3 and MBMV≥4.01

ip
cm3, respectively. Therefore, the proposed method works well on both small (MBMV<4.01 cm3) and

cr
large tumors (MBMV≥4.01 cm3) as shown in Fig. 3 (a).

us
Segmentation performance of BMDS net on various area sizes
an
The first, second and third quartiles of the diameter to enhance the BM area (range from 6 mm to

45 mm) were 10.50 mm, 15.75 mm, and 22.50 mm, respectively. The mean diameter of the
M

enhancing BM area was 17.58 mm. The dice ratios were 0.83 ± 0.09 (range, 0.62-0.95) and 0.89 ±
ed

0.07 (range, 0.71-0.95) for the diameters ranging from 6 to 18 mm and from 18 to 45 mm,

respectively. The results verify the effectiveness of BMDS net for BMs with different diameters, as
pt

shown in red in Fig. 3 (a).

ce

Robustness of BMDS net for different datasets

Ac

We trained the networks for Dataset 2 (231 cases) and Dataset 3 (220 cases) using the same

training settings as those for the 1,201 cases. The mean sensitivity, specificity and dice ratio (with

standard deviation) were 0.95±0.02, 0.99±0.0001, 0.84±0.07 and 0.96±0.04, 0.99±0.0001, 0.83±0.06,

respectively. As shown in Fig. 3 (b and c), the segmentations generated by our proposed method are

highly consistent with the ground-truth segmentations for all the three datasets, indicating potential

feasibility in real clinical applications.

Comparison with the classical FCN

12
 N-O-D-19-758R1

The FCN is a special case of convolutional networks that is widely used in medical image

processing. We compared the segmentation performance of the proposed method with the classical

Downloaded from https://academic.oup.com/neuro-oncology/advance-article-abstract/doi/10.1093/neuonc/noz234/5684915 by University of Toronto user on 25 December 2019

FCN(i.e., U-net), using the mean sensitivity, specificity and dice ratio (with standard deviation). The

three indices over 1,201 samples of the classical FCN were 0.83±0.08, 0.99±0.0004 and 0.69±0.15,

respectively, which were all lower than those of the proposed method. In addition, we also

compared our results with the classical FCN through a paired-sample t-test. Collectively, our

t
ip
proposed method showed statistically significant improvement (p<0.001) compared with the

classical FCN. The summary of the results of the classical FCN and BMDS net is shown in Fig. 3(c and

cr
d).

us
Discussion
an
We developed a cascaded 3D fully convolution network for the detection and segmentation of

BMs (named BMDS net) using whole 3D MR images that not only would improve the accuracy of
M

localizing and segmenting BMs but also would reduce the labor intensity of SRT planning and follow-
ed

up assessment. BMDS net could detect all the BMs, ensuring the identification of all target lesions

without omission. Automated segmentation correlated well with manual segmentations, and
pt

reliable segmentation results were shown in different stages of the BM managements.

ce

BMDS net could help separating BM from high-grade gliomas

Ac

BM and high-grade gliomas (HGGs) have different strategies of treatments, while the differential

diagnosis between the two entities is not always effortless, especially in solitary BM and HGGs 34.

Because the radiological appearance of solitary intracranial metastasis and HGGs are often similar,

this will result in misdiagnosis or delayed treatment, further affecting the prognosis. In clinical

practice, multiple lesions are considered as one of the most important imaging characteristics in

separating BMs from HGGs 35, and identifying the very small BMs in the brain would improve the

confidence of neuroradiologists when diagnosing BMs. Therefore, the detection and segmentation

13
 N-O-D-19-758R1

of small BMs is also important to identify solitary lesions. Our results reveal that BMDS net could

detect the very small nidus accurately (Fig. 4) to avoid misdiagnosis.

Downloaded from https://academic.oup.com/neuro-oncology/advance-article-abstract/doi/10.1093/neuonc/noz234/5684915 by University of Toronto user on 25 December 2019

BMDS net simplifies treatment planning

On the one hand, gamma knife radiotherapy, which is the main technology of SRT, is usually a

one-day outpatient procedure that requires fast segmentation for a rapid clinical workflow 35. On the

t
other hand, omitting small and even large BMs is incidentally seen during the planning of gamma

ip
knife radiosurgery in clinical practice because of carelessness, exhaustion and other human factors 7.

cr
To solve this clinical problem, an automatic method of detection and segmentation is urgently

us
needed. Our BMDS net has a strong potential to help radiation oncologists accelerate the workflow

without omission of lesions. An example of quick segmentation of all the multiple small BMs by
an
BMDS net is shown in Fig. 5.
M
Although the proposed method still omitted some slices of tumors, this result may not discourage

radiation oncologists because almost all the omitting slices are the begin and ending slices of BMs.
ed

The radiation oncologists could make amends in the process of planning; particularly, the beginning

and ending slices should not always be included (partial volume effect).
pt

BMDS net facilitates the follow up of BM

ce

Because assessment of the treatment response is mostly based on the change in the lesion
Ac

volume, a quick and visualized spatial distribution of all lesions would liberate the neuro-oncologists

from measuring the volumes repeatedly. Moreover, conditions of new lesions appearing or old

lesions disappearing would coexist during the follow up. Therefore, the identification of spatial

distribution and volumetric changes is crucial to assess the treatment response of each lesion and

help treatment decision making. BMDS net could display the spatial distribution and volumetric

changes of BM accurately and simplify the treatment response evaluation. An example of the

segmentations in the pretreatment and follow up of one patient is shown in Fig. 6. Regarding the

14
 N-O-D-19-758R1

Response Assessment in Neuro-Oncology Brain Metastases (RANO-BM) 36, lesions with a diameter

greater than 1 cm should be considered as target lesions for treatment response assessment. Our

Downloaded from https://academic.oup.com/neuro-oncology/advance-article-abstract/doi/10.1093/neuonc/noz234/5684915 by University of Toronto user on 25 December 2019

results showed that BMDS net could produce a very high dice ratio in lesions over 1 cm,

guaranteeing its value in the treatment response assessment.

Vascular epidermal growth factor (VEGF) receptor inhibitor , as the first-line treatment modality

for brain metastases 37, have always caused a dilemma concerning evaluation during the follow-up

t
ip
assessment of BMs because these agents could tighten the vascular endothelial space and further

cr
decrease the permeability of the ruptured blood brain barrier, preventing contrast agent leakage

into the extravascular and extracellular space and resulting in light or no enhancement in the

us
postcontrast T1WI images of patients with BMs. Therefore, identifying all the lesions would be more
an
challenging. Remarkably, BMDS net performed well in detecting and segmenting nonenhancing BMs.

This is an important progress for the clinical management of SRT of BMs. An example of segmenting
M

nonenhancing lesions after VEGF receptor inhibitor treatment by BMDS net is shown in

Supplemental Fig. S1.

ed

There are several limitations in this study. First, only 3D T1-MPRAGE images (similar to 3D BRAVO
pt

T1WI) were included in the study. Other 3D imaging sequences are also used in clinical practice—for

example, 3D fat-saturated T1 sampling perfection with application-optimized contrast using

ce

different flip angle evolutions (3D T1-SPACE); the diagnostic performance of BMDS net in these
Ac

imaging sequences should be investigated in a future study. Second, in the first stage of BMDS net,

we could not afford any missing detected region of a case; thus, we chose a fixed size of the located

nidus region. Additionally, we know that an adaptive method will be more flexible. In future work,

we will explore more stable region proposal methods based on the predicted region size to alleviate

the usage of prior knowledge.

Conclusion

15
 N-O-D-19-758R1

In this paper, we have proposed a cascaded 3D FCN to automatically detect and segment brain

metastases, in which the first FCN focuses on fast location and the second on segmentations.

Downloaded from https://academic.oup.com/neuro-oncology/advance-article-abstract/doi/10.1093/neuonc/noz234/5684915 by University of Toronto user on 25 December 2019

Experimental results across different institutions indicated that our BMDS net could not only detect

all the BMs, ensuring the identification of all target lesions without omission, but also segment

lesions with high accuracy and robustness. The BMDS net shows great potential in help separating

BM from HGGs, simplifying treatment planning and facilitating the follow up of BMs. The deep-

t
ip
learning-based method could be a useful tool to reduce the labor intensity of clinicians in the clinical

management of BMs.

cr
Acknowledgements

us
We would like to thank American Journal Expert for English language editing.
an
M
ed
pt
ce
Ac

16
 N-O-D-19-758R1

References

1. Takei H, Rouah E, Ishida Y. Brain metastasis: clinical characteristics, pathological findings

Downloaded from https://academic.oup.com/neuro-oncology/advance-article-abstract/doi/10.1093/neuonc/noz234/5684915 by University of Toronto user on 25 December 2019

and molecular subtyping for therapeutic implications. Brain tumor pathology. 2016;33(1):1-
12.
2. Achrol AS, Rennert RC, Anders C, et al. Brain metastases. Nature Reviews Disease Primers.
2019;5(1):1-26.
3. Aoyama H, Shirato H, Tago M, et al. Stereotactic radiosurgery plus whole-brain radiation
therapy vs stereotactic radiosurgery alone for treatment of brain metastases: a randomized
controlled trial. Jama. 2006;295(21):2483-2491.
4. Patel TR, Knisely JP, Chiang VL. Management of brain metastases: surgery, radiation, or
both? Hematology/Oncology Clinics. 2012;26(4):933-947.

t
5. Pinkham MB, Whitfield GA, Brada M. New developments in intracranial stereotactic

ip
radiotherapy for metastases. Clinical Oncology. 2015;27(5):316-323.
6. Pérez‐Ramírez Ú, Arana E, Moratal D. Brain metastases detection on MR by means of three‐

cr
dimensional tumor‐appearance template matching. Journal of Magnetic Resonance Imaging.
2016;44(3):642-652.
7. Higuchi Y, Yamamoto M, Serizawa T, Aiyama H, Sato Y, Barfod BE. Modern management

us
for brain metastasis patients using stereotactic radiosurgery: literature review and the authors’
gamma knife treatment experiences. Cancer management and research. 2018;10:1889.
8. Growcott S, Dembrey T, Patel R, Eaton D, Cameron A. Inter-Observer Variability in Target
an
Volume Delineations of Benign and Metastatic Brain Tumours for Stereotactic Radiosurgery:
Results of a National Quality Assurance Programme. Clinical Oncology. 2019.
9. Chang ATY, Tan LT, Duke S, Ng W-T. Challenges for quality assurance of target volume
delineation in clinical trials. Frontiers in Oncology. 2017;7:221.
M
10. Rudie JD, Rauschecker AM, Bryan RN, Davatzikos C, Mohan S. Emerging applications of
artificial intelligence in neuro-oncology. Radiology. 2019;290(3):607-618.
11. Pereira S, Pinto A, Alves V, Silva CA. Brain tumor segmentation using convolutional neural
networks in MRI images. IEEE transactions on medical imaging. 2016;35(5):1240-1251.
ed

12. Shen H, Zhang J, Zheng W. Efficient symmetry-driven fully convolutional network for
multimodal brain tumor segmentation. Paper presented at: 2017 IEEE International
Conference on Image Processing (ICIP)2017.
13. Havaei M, Guizard N, Chapados N, Bengio Y. Hemis: Hetero-modal image segmentation.
pt

Paper presented at: International Conference on Medical Image Computing and Computer-
Assisted Intervention2016.
ce

14. Hu K, Gan Q, Zhang Y, et al. Brain Tumor Segmentation Using Multi-Cascaded

Convolutional Neural Networks and Conditional Random Field. IEEE Access. 2019;7:92615-
92629.
15. Shen H, Zhang J. Fully connected CRF with data-driven prior for multi-class brain tumor
Ac

segmentation. Paper presented at: 2017 IEEE International Conference on Image Processing
(ICIP)2017.
16. Pereira S, Pinto A, Amorim J, Ribeiro A, Alves V, Silva CA. Adaptive feature recombination
and recalibration for semantic segmentation with Fully Convolutional Networks. IEEE
transactions on medical imaging. 2019.
17. Chen S, Ding C, Liu M. Dual-force convolutional neural networks for accurate brain tumor
segmentation. Pattern Recognition. 2019;88:90-100.
18. Wang G, Li W, Zuluaga MA, et al. Interactive medical image segmentation using deep
learning with image-specific fine tuning. IEEE transactions on medical imaging.
2018;37(7):1562-1573.
19. Wang G, Zuluaga MA, Li W, et al. DeepIGeoS: a deep interactive geodesic framework for
medical image segmentation. IEEE transactions on pattern analysis and machine intelligence.
2018;41(7):1559-1572.

17
 N-O-D-19-758R1

20. Farjam R, Parmar HA, Noll DC, Tsien CI, Cao Y. An approach for computer-aided detection
of brain metastases in post-Gd T1-W MRI. Magnetic resonance imaging. 2012;30(6):824-
836.

Downloaded from https://academic.oup.com/neuro-oncology/advance-article-abstract/doi/10.1093/neuonc/noz234/5684915 by University of Toronto user on 25 December 2019

21. Ambrosini RD, Wang P, O'Dell WG. Computer‐aided detection of metastatic brain tumors
using automated three‐dimensional template matching. Journal of Magnetic Resonance
Imaging. 2010;31(1):85-93.
22. Krafft C, Shapoval L, Sobottka SB, Geiger KD, Schackert G, Salzer R. Identification of
primary tumors of brain metastases by SIMCA classification of IR spectroscopic images.
Biochimica et Biophysica Acta (BBA)-Biomembranes. 2006;1758(7):883-891.
23. Minniti G, Romano A, Scaringi C, Bozzao A. Imaging in Neuro-Oncology. In:
Neurorehabilitation in Neuro-Oncology. Springer; 2019:53-68.
24. Havaei M, Davy A, Warde-Farley D, et al. Brain tumor segmentation with deep neural

t
networks. Medical image analysis. 2017;35:18-31.

ip
25. Shen D, Wu G, Suk H-I. Deep learning in medical image analysis. Annual review of
biomedical engineering. 2017;19:221-248.
26. Urban G, Bendszus M, Hamprecht F, Kleesiek J. Multi-modal brain tumor segmentation

cr
using deep convolutional neural networks. MICCAI BraTS (Brain Tumor Segmentation)
Challenge Proceedings, winning contribution. 2014:31-35.
27. Zikic D, Ioannou Y, Brown M, Criminisi A. Segmentation of brain tumor tissues with

us
convolutional neural networks. Proceedings MICCAI-BRATS. 2014:36-39.
28. Sun W, Wang R. Fully convolutional networks for semantic segmentation of very high
resolution remotely sensed images combined with DSM. IEEE Geoscience and Remote
an
Sensing Letters. 2018;15(3):474-478.
29. Nie D, Wang L, Adeli E, Lao C, Lin W, Shen D. 3-D fully convolutional networks for
multimodal isointense infant brain image segmentation. IEEE transactions on cybernetics.
2018;49(3):1123-1136.
M

30. Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural
networks. 2010. Paper presented at: International Conference on Artificial Intelligence and
Statistics2018.
31. Tensorflow. https://www.tensorflow.org/.
ed

32. Dice LR. Measures of the amount of ecologic association between species. Ecology.
1945;26(3):297-302.
33. Taunk NK, Oh JH, Shukla-Dave A, et al. Early posttreatment assessment of MRI perfusion
pt

biomarkers can predict long-term response of lung cancer brain metastases to stereotactic
radiosurgery. Neuro-oncology. 2017;20(4):567-575.
34. Jung BC, Arevalo‐Perez J, Lyo JK, et al. Comparison of glioblastomas and brain metastases
ce

using dynamic contrast‐enhanced perfusion MRI. Journal of Neuroimaging. 2016;26(2):240-

246.
35. Barajas RJaSC. Metastasis in Adult Brain Tumors. Neuroimaging Clin N Am.
2016;26(4):601-620.
Ac

36. Lin NU, Lee EQ, Aoyama H, et al. Response assessment criteria for brain metastases:
proposal from the RANO group. The lancet oncology. 2015;16(6):e270-e278.
37. Roca E, Pozzari M, Vermi W, et al. Outcome of EGFR-mutated adenocarcinoma NSCLC
patients with changed phenotype to squamous cell carcinoma after tyrosine kinase inhibitors:
A pooled analysis with an additional case. Lung Cancer. 2018.

18
 N-O-D-19-758R1

Figures Legends

Figure 1. Network architecture of BMDS Net.

Downloaded from https://academic.oup.com/neuro-oncology/advance-article-abstract/doi/10.1093/neuonc/noz234/5684915 by University of Toronto user on 25 December 2019

Figure 2. 3D visualization of the ground-truth segmentations and our detections. The first row

represents the ground-truth segmentations (up), and the second row (down) represents our

detections by BMDS net.

t
Figure 3. Comparison between various volumes and size of the enhancing area for each slice (a);

ip
comparison among different datasets (b and c); comparison between BMDS net and the classic

cr
FCN (d). BMDS net achieved a better dice ratio among BMs with a MBMV greater than 4.01 cm3

us
and a diameter greater than 18 mm (a); a high sensitivity, specificity and dice ratio were reached using

BMDS net among the three different datasets (b); the receiver operating characteristic curve of BMDS
an
for three different datasets and the classic FCN (C); BMDS net showed higher sensitivity, specificity

and dice ratio than the classic FCN (d).

M

Figure 4. Example of segmentation of small enhancing lesions to avoid the misdiagnosis of BMs. A

58-year-old female patients without a history of primary cancer complained of dizziness and
ed

memory deterioration. An inhomogeneous enhancing lesion was obviously seen in the right

thalamus (A). However, a very small nidus was also identified in the left cerebellum (B-C), suggesting
pt

a high possibility of the diagnosis of brain metastases. (D) Manual segmentation. The predicted
ce

segmentations (E) showed good concordance with manual labels. The specificity, sensitivity and dice

ratio were 0.99, 0.97 and 0.89, respectively.

Ac

Figure 5. Example of segmentation of multiple brain metastases by BMDS net. A 67-year-old male

patient with a history of lung cancer had a chief complaint of headache. Contrast-enhanced 3D T1-

MPRAGE images showed multiple brain metastases in the brain (A). The brain metastases were

separately segmented manually using various colored labels. (C) Manual segmentation in a spatial

view. The predicted segmentations (D) showed strong concordance with the manual labels. The

specificity, sensitivity and dice ratio were 0.99, 0.93 and 0.86, respectively.

19
 N-O-D-19-758R1

Figure 6. Example of tumor burden changes between pretreatment (a) and the follow up (b) of one

patient. A 49-year-old male patient with multiple BMs was treated by stereotactic radiotherapy.

Downloaded from https://academic.oup.com/neuro-oncology/advance-article-abstract/doi/10.1093/neuonc/noz234/5684915 by University of Toronto user on 25 December 2019

Pretreatment images (A and E) clearly showed solid lesions, and then segmentations were manually

performed (B and E). Posttreatment images (C and G) showed that the lesions decreased.

Additionally, a new metastatic lesion was discovered (white G). The lesions were also manually

segmented (D and H). Manual pretreatment (I) and posttreatment (K) segmentations showed strong

t
ip
concordance with the predicted pretreatment (K) and posttreatment (L) segmentations by BMDS

net. The specificity, sensitivity and dice ratio of pretreatment segmentations were 0.99, 0.97 and

cr
0.85, respectively, while those of posttreatment segmentations were 0.99, 0.97 and 0.80,

us
independently. an
M
ed
pt
ce
Ac

20
 Downloaded from https://academic.oup.com/neuro-oncology/advance-article-abstract/doi/10.1093/neuonc/noz234/5684915 by University of Toronto user on 25 December 2019
t
ip
cr
us
an
N-O-D-19-758R1

Figure 1

21
M
ed
pt
ce
Ac
 Downloaded from https://academic.oup.com/neuro-oncology/advance-article-abstract/doi/10.1093/neuonc/noz234/5684915 by University of Toronto user on 25 December 2019
t
ip
cr
us
an
N-O-D-19-758R1

Figure 2

22
M
ed
pt
ce
Ac
 Downloaded from https://academic.oup.com/neuro-oncology/advance-article-abstract/doi/10.1093/neuonc/noz234/5684915 by University of Toronto user on 25 December 2019
t
ip
cr
us
an
N-O-D-19-758R1

Figure 3

23
M
ed
pt
ce
Ac
 Downloaded from https://academic.oup.com/neuro-oncology/advance-article-abstract/doi/10.1093/neuonc/noz234/5684915 by University of Toronto user on 25 December 2019
t
ip
cr
us
an
N-O-D-19-758R1

Figure 4

24
M
ed
pt
ce
Ac
 Downloaded from https://academic.oup.com/neuro-oncology/advance-article-abstract/doi/10.1093/neuonc/noz234/5684915 by University of Toronto user on 25 December 2019
t
ip
cr
us
an
N-O-D-19-758R1

Figure 5

25
M
ed
pt
ce
Ac
 Downloaded from https://academic.oup.com/neuro-oncology/advance-article-abstract/doi/10.1093/neuonc/noz234/5684915 by University of Toronto user on 25 December 2019
t
ip
cr
us
an
N-O-D-19-758R1

Figure 6

26
M
ed
pt
ce
Ac