See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/329206678

An overview of deep learning in medical imaging focusing on MRI

Preprint · November 2018

CITATIONS READS
0 712

2 authors:

Alexander Lundervold Arvid Lundervold

National Institute for Research in Computer Science and Control University of Bergen
15 PUBLICATIONS   920 CITATIONS    8 PUBLICATIONS   712 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

MR Image Processing View project

All content following this page was uploaded by Arvid Lundervold on 13 May 2020.

The user has requested enhancement of the downloaded file.

 An overview of deep learning in medical imaging
focusing on MRI

Alexander Selvikvåg Lundervolda,b,∗, Arvid Lundervolda,c,d

a
Mohn Medical Imaging and Visualization Centre (MMIV), Haukeland University Hospital, Norway
b
Department of Computing, Mathematics and Physics, Western Norway University of Applied Sciences, Norway
c
Neuroinformatics and Image Analysis Laboratory, Department of Biomedicine, University of Bergen, Norway
d
Department of Health and Functioning, Western Norway University of Applied Sciences, Norway
arXiv:1811.10052v2 [cs.CV] 16 Dec 2018

Abstract
What has happened in machine learning lately, and what does it mean for the future of medical
image analysis? Machine learning has witnessed a tremendous amount of attention over the last
few years. The current boom started around 2009 when so-called deep artificial neural networks
began outperforming other established models on a number of important benchmarks. Deep neural
networks are now the state-of-the-art machine learning models across a variety of areas, from image
analysis to natural language processing, and widely deployed in academia and industry. These
developments have a huge potential for medical imaging technology, medical data analysis, medical
diagnostics and healthcare in general, slowly being realized. We provide a short overview of recent
advances and some associated challenges in machine learning applied to medical image processing
and image analysis. As this has become a very broad and fast expanding field we will not survey
the entire landscape of applications, but put particular focus on deep learning in MRI.
Our aim is threefold: (i) give a brief introduction to deep learning with pointers to core refer-
ences; (ii) indicate how deep learning has been applied to the entire MRI processing chain, from
acquisition to image retrieval, from segmentation to disease prediction; (iii) provide a starting point
for people interested in experimenting and perhaps contributing to the field of machine learning for
medical imaging by pointing out good educational resources, state-of-the-art open-source code, and
interesting sources of data and problems related medical imaging.
Keywords: Machine learning, Deep learning, Medical imaging, MRI

1. Introduction
Machine learning has seen some dramatic developments recently, leading to a lot of interest from
industry, academia and popular culture. These are driven by breakthroughs in artificial neural
networks, often termed deep learning, a set of techniques and algorithms that enable computers
to discover complicated patterns in large data sets. Feeding the breakthroughs is the increased
access to data (“big data”), user-friendly software frameworks, and an explosion of the available
compute power, enabling the use of neural networks that are deeper than ever before. These
models nowadays form the state-of-the-art approach to a wide variety of problems in computer
vision, language modeling and robotics.
Deep learning rose to its prominent position in computer vision when neural networks started
outperforming other methods on several high-profile image analysis benchmarks. Most famously

∗
Corresponding author
Email addresses: [email protected] (Alexander Selvikvåg Lundervold), [email protected] (Arvid Lundervold)

Preprint submitted to Zeitschrift für Medizinische Physik December 18, 2018

on the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC)1 in 2012 [1] when a deep
learning model (a convolutional neural network ) halved the second best error rate on the image
classification task. Enabling computers to recognize objects in natural images was until recently
thought to be a very difficult task, but by now convolutional neural networks have surpassed even
human performance on the ILSVRC, and reached a level where the ILSVRC classification task is
essentially solved (i.e. with error rate close to the Bayes rate). The last ILSVRC competition
was held in 2017, and computer vision research has moved on to other more difficult benchmark
challenges. For example the Common Objects in Context Challenge (COCO) [2].
Deep learning techniques have become the de facto standard for a wide variety of computer vision
problems. They are, however, not limited to image processing and analysis but are outperforming
other approaches in areas like natural language processing [3, 4, 5], speech recognition and synthesis
[6, 7]2 , and in the analysis of unstructured, tabular-type data using entity embeddings [8, 9].3
The sudden progress and wide scope of deep learning, and the resulting surge of attention and
multi-billion dollar investment, has led to a virtuous cycle of improvements and investments in
the entire field of machine learning. It is now one of the hottest areas of study world-wide [15],
and people with competence in machine learning are highly sought-after by both industry and
academia4 .
Healthcare providers generate and capture enormous amounts of data containing extremely
valuable signals and information, at a pace far surpassing what “traditional” methods of analysis
can process. Machine learning therefore quickly enters the picture, as it is one of the best ways
to integrate, analyze and make predictions based on large, heterogeneous data sets (cf. health
informatics [16]). Healthcare applications of deep learning range from one-dimensional biosignal
analysis [17] and the prediction of medical events, e.g. seizures [18] and cardiac arrests [19], to
computer-aided detection [20] and diagnosis [21] supporting clinical decision making and survival
analysis [22], to drug discovery [23] and as an aid in therapy selection and pharmacogenomics [24],
to increased operational efficiency [25], stratified care delivery [26], and analysis of electronic health
records [27, 28].
The use of machine learning in general and deep learning in particular within healthcare is still in
its infancy, but there are several strong initiatives across academia, and multiple large companies are
pursuing healthcare projects based on machine learning. Not only medical technology companies,
but also for example Google Brain [29, 30, 31]5 , DeepMind [32]6 , Microsoft [33, 34]7 and IBM [35]8 .
There is also a plethora of small and medium-sized businesses in the field9 .

1
Colloquially known as the ImageNet challenge
2
Try it out here: https://deepmind.com/blog/wavenet-generative-model-raw-audio
3
As a perhaps unsurprising side-note, these modern deep learning methods have also entered the field of physics.
Among other things, they are tasked with learning physics from raw data when no good mathematical models are
available. For example in the analysis of gravitational waves where deep learning has been used for classification
[10], anomaly detection [11] and denoising [12], using methods that are highly transferable across domains (think
EEG and fMRI). They are also part of mathematical model and machine learning hybrids [13, 14], formed to reduce
computational costs by having the mathematical model train a machine learning model to perform its job, or to
improve the fit with observations in settings where the mathematical model can’t incorporate all details (think noise).
4
See e.g. https://economicgraph.linkedin.com/research/LinkedIns-2017-US-Emerging-Jobs-Report for a
study focused on the US job market
5
https://ai.google/research/teams/brain/healthcare-biosciences
6
https://deepmind.com/applied/deepmind-health/
7
https://www.microsoft.com/en-us/research/research-area/medical-health-genomics
8
https://www.research.ibm.com/healthcare-and-life-sciences
9
Aidoc, Arterys, Ayasdi, Babylon Healthcare Services, BenevolentAI, Enlitic, EnvoiAI, H2O, IDx, MaxQ AI,
Mirada Medical, Viz.ai, Zebra Medical Vision, and many more.

2
2. Machine learning, artificial neural networks, deep learning
In machine learning one develops and studies methods that give computers the ability to solve
problems by learning from experiences. The goal is to create mathematical models that can be
trained to produce useful outputs when fed input data. Machine learning models are provided ex-
periences in the form of training data, and are tuned to produce accurate predictions for the training
data by an optimization algorithm. The main goal of the models are to be able to generalize their
learned expertise, and deliver correct predictions for new, unseen data. A model’s generalization
ability is typically estimated during training using a separate data set, the validation set, and used
as feedback for further tuning of the model. After several iterations of training and tuning, the final
model is evaluated on a test set, used to simulate how the model will perform when faced with new,
unseen data.
There are several kinds of machine learning, loosely categorized according to how the models
utilize its input data during training. In reinforcement learning one constructs agents that learn
from their environments through trial and error while optimizing some objective function. A famous
recent application of reinforcement learning is AlphaGo and AlphaZero [36], the Go-playing machine
learning systems developed by DeepMind. In unsupervised learning the computer is tasked with
uncovering patterns in the data without our guidance. Clustering is a prime example. Most of
today’s machine learning systems belong to the class of supervised learning. Here, the computer
is given a set of already labeled or annotated data, and asked to produce correct labels on new,
previously unseen data sets based on the rules discovered in the labeled data set. From a set of
input-output examples, the whole model is trained to perform specific data-processing tasks. Image
annotation using human-labeled data, e.g. classifying skin lesions according to malignancy [37] or
discovering cardiovascular risk factors from retinal fundus photographs [38], are two examples of
the multitude of medical imaging related problems attacked using supervised learning.
Machine learning has a long history and is split into many sub-fields, of which deep learning is
the one currently receiving the bulk of attention.
There are many excellent, openly available overviews and surveys of deep learning. For short
general introductions to deep learning, see [39, 40]. For an in-depth coverage, consult the freely
available book [41]10 . For a broad overview of deep learning applied to medical imaging, see [42].
We will only mention some bare essentials of the field, hoping that these will serve as useful pointers
to the areas that are currently the most influential in medical imaging.

2.1. Artificial neural networks

Artificial neural networks (ANNs) is one of the most famous machine learning models, introduced
already in the 1950s, and actively studied since [41, Chapter 1.2].11
Roughly, a neural network consists of a number of connected computational units, called neurons,
arranged in layers. There’s an input layer where data enters the network, followed by one or more
hidden layers transforming the data as it flows through, before ending at an output layer that
produces the neural network’s predictions. The network is trained to output useful predictions by
identifying patterns in a set of labeled training data, fed through the network while the outputs are
compared with the actual labels by an objective function. During training the network’s parameters–
the strength of each neuron–is tuned until the patterns identified by the network result in good

10
https://www.deeplearningbook.org/
11
The loose connection between artificial neural networks and neural networks in the brain is often mentioned, but
quite over-blown considering the complexity of biological neural networks. However, there is some interesting recent
work connecting neuroscience and artificial neural networks, indicating an increase in the cross-fertilization between
the two fields [43, 44, 45].

3
predictions for the training data. Once the patterns are learned, the network can be used to make
predictions on new, unseen data, i.e. generalize to new data.
It has long been known that ANNs are very flexible, able to model and solve complicated
problems, but also that they are difficult and very computationally expensive to train.12 This has
lowered their practical utility and led people to, until recently, focus on other machine learning
models. But by now, artificial neural networks form one of the dominant methods in machine
learning, and the most intensively studied. This change is thanks to the growth of big data, powerful
processors for parallel computations (in particular, GPUs), some important tweaks to the algorithms
used to construct and train the networks, and the development of easy-to-use software frameworks.
The surge of interest in ANNs leads to an incredible pace of developments, which also drives other
parts of machine learning with it.
The freely available books [41, 50] are two of the many excellent sources to learn more about
artificial neural networks. We’ll only give a brief indication of how they are constructed and trained.
The basic form of artificial neural networks13 , the feedforward neural networks, are parametrized
mathematical functions y = f (x; θ) that maps an input x to an output y by feeding it through a
number of nonlinear transformations: f (x) = (fn ◦ · · · ◦ f1 )(x). Here each component fk , called a
network layer, consists of a simple linear transformation of the previous component’s output, fol-
lowed by a nonlinear function: fk = σk (θkT fk−1 ). The nonlinear functions σk are typically sigmoid
functions or ReLUs, as discussed below, and the θk are matrices of numbers, called the model’s
weights. During the training phase, the network is fed training data and tasked with making pre-
dictions at the output layer that match the known labels, each component of the network producing
an expedient representation of its input. It has to learn how to best utilize the intermediate repre-
sentations to form a complex hierarchical representation of the data, ending in correct predictions
at the output layer. Training a neural network means changing its weights to optimize the outputs
of the network. This is done using an optimization algorithm, called gradient descent, on a function
measuring the correctness of the outputs, called a cost function or loss function. The basic ideas
behind training neural networks are simple: as training data is fed through the network, compute
the gradient of the loss function with respect to every weight using the chain rule, and reduce the
loss by changing these weights using gradient descent. But one quickly meets huge computational
challenges when faced with complicated networks with thousands or millions of parameters and an
exponential number of paths between the nodes and the network output. The techniques designed
to overcome these challenges gets quite complicated. See [41, Chapter 8] and [51, Chapter 3 and 4]
for detailed descriptions of the techniques and practical issues involved in training neural networks.
Artificial neural networks are often depicted as a network of nodes, as in Figure 1.14

12
According to the famous universal approximation theorem for artificial neural networks [46, 47, 48, 49], ANNs are
mathematically able to approximate any continuous function defined on compact subspaces of Rn , using finitely many
neurons. There are some restrictions on the activation functions, but these can be relaxed (allowing for ReLUs for
example) by restricting the function space. This is an existence theorem and successfully training a neural network
to approximate a given function is another matter entirely. However, the theorem does suggests that neural networks
are reasonable to study and develop further, at least as an engineering endeavour aimed at realizing their theoretical
powers.
13
These are basic when compared to for example recurrent neural networks, whose architectures are more involved
14
As we shall see, modern architectures are often significantly more complicated than captured by the illustration
and equations above, with connections between non-consecutive layers, input fed in also at later layers, multiple
outputs, and much more.

4
Figure 1: Artificial neural networks are built from simple linear functions followed by nonlinearities. One of the
simplest class of neural network is the multilayer perceptron, or feedforward neural network, originating from the work
of Rosenblatt in the 1950s [52]. It’s based on simple computational units, called neurons, organized in layers. Writing
 T
(i) (i)
i for the i-th layer and j for the j-th unit of that layer, the output of the j-th unit at the i-th layer is zj = θj x.
Here x consists of the outputs from the previous layer after they are fed through a simple nonlinear function called an
activation function, typically a sigmoid function σ(z) = 1/(1 + e−z ) or a rectified linear unit ReLU(z) = max(0, z) or
small variations thereof. Each layer therefore computes a weighted sum of the all the outputs from the neurons in the
previous layers, followed by a nonlinearity. These are called the layer activations. Each layer activation is fed to the
next layer in the network, which performs the same calculation, until you reach the output layer, where the network’s
predictions are produced. In the end, you obtain a hierarchical representation of the input data, where the earlier
features tend to be very general, getting increasingly specific towards the output. By feeding the network training data,
propagated through the layers, the network is trained to perform useful tasks. A training data point (or, typically,
a small batch of training points) is fed to the network, the outputs and local derivatives at each node are recorded,
and the difference between the output prediction and the true label is measured by an objective function, such as
mean absolute error (L1), mean squared error (L2), cross-entropy loss, or Dice loss, depending on the application.
The derivative of the objective function with respect to the output is calculated, and used as a feedback signal. The
discrepancy is propagated backwards through the network and all the weights are updated to reduce the error. This is
achieved using backward propagation [53, 54, 55], which calculates the gradient of the objective function with respect
to the weights in each node using the chain rule together with dynamic programming, and gradient descent [56], an
optimization algorithm tasked with improving the weights.

2.2. Deep learning

Traditionally, machine learning models are trained to perform useful tasks based on manually
designed features extracted from the raw data, or features learned by other simple machine learning
models. In deep learning, the computers learn useful representations and features automatically,
directly from the raw data, bypassing this manual and difficult step. By far the most common models
in deep learning are various variants of artificial neural networks, but there are others. The main
common characteristic of deep learning methods is their focus on feature learning: automatically
learning representations of data. This is the primary difference between deep learning approaches

5
and more “classical” machine learning. Discovering features and performing a task is merged into
one problem, and therefore both improved during the same training process. See [39] and [41] for
general overviews of the field.
In medical imaging the interest in deep learning is mostly triggered by convolutional neural
networks (CNNs) [57]15 , a powerful way to learn useful representations of images and other struc-
tured data. Before it became possible to use CNNs efficiently, these features typically had to be
engineered by hand, or created by less powerful machine learning models. Once it became possible
to use features learned directly from the data, many of the handcrafted image features were typi-
cally left by the wayside as they turned out to be almost worthless compared to feature detectors
found by CNNs.16 There are some strong preferences embedded in CNNs based on how they are
constructed, which helps us understand why they are so powerful. Let us therefore take a look at
the building blocks of CNNs.

Figure 2: Building blocks of a typical CNN. A slight modification of a figure in [59], courtesy of the author.

2.3. Building blocks of convolutional neural networks

When applying neural networks to images one can in principle use the simple feedforward neural
networks discussed above. However, having connections from all nodes of one layer to all nodes in the
next is extremely inefficient. A careful pruning of the connections based on domain knowledge, i.e.
the structure of images, leads to much better performance. A CNN is a particular kind of artificial
neural network aimed at preserving spatial relationships in the data, with very few connections
between the layers. The input to a CNN is arranged in a grid structure and then fed through
layers that preserve these relationships, each layer operation operating on a small region of the
previous layer (Fig. 2). CNNs are able to form highly efficient representation of the input data17 ,

15
Interestingly, CNNs was applied in medical image analysis already in the early 90s, e.g. [58], but with limited
success.
16
However, combining hand-engineered features with CNN features is a very reasonable approach when low amounts
of training data makes it difficult to learn good features automatically
17
It’s interesting to compare this with the biological vision systems and their receptive fields of variable size (volumes
in visual space) of neurons at different hierarchical levels

6
well-suited for image-oriented tasks. A CNN has multiple layers of convolutions and activations,
often interspersed with pooling layers, and is trained using backpropagation and gradient descent
as for standard artificial neural networks. See Section 2.1. In addition, CNNs typically have fully-
connected layers at the end, which compute the final outputs.18

i) Convolutional layers: In the convolutional layers the activations from the previous layers
are convolved with a set of small parameterized filters, frequently of size 3 × 3, collected in a
tensor W (j,i) , where j is the filter number and i is the layer number. By having each filter share
the exact same weights across the whole input domain, i.e. translational equivariance at each
layer, one achieves a drastic reduction in the number of weights that need to be learned. The
motivation for this weight-sharing is that features appearing in one part of the image likely
also appear in other parts. If you have a filter capable of detecting horizontal lines, say, then
it can be used to detect them wherever they appear. Applying all the convolutional filters at
all locations of the input to a convolutional layer produces a tensor of feature maps.

ii) Activation layer: The feature maps from a convolutional layer are fed through nonlinear
activation functions. This makes it possible for the entire neural network to approximate
almost any nonlinear function [48, 49]19 The activation functions are generally the very simple
rectified linear units, or ReLUs, defined as ReLU(z) = max(0, z), or variants like leaky ReLUs
or parametric ReLUs.20 See [60, 61] for more information about these and other activation
functions. Feeding the feature maps through an activation function produces new tensors,
typically also called feature maps.

iii) Pooling: Each feature map produced by feeding the data through one or more convolutional
layer is then typically pooled in a pooling layer. Pooling operations take small grid regions as
input and produce single numbers for each region. The number is usually computed by using
the max function (max-pooling) or the average function (average pooling). Since a small shift
of the input image results in small changes in the activation maps, the pooling layers gives the
CNN some translational invariance.
A different way of getting the downsampling effect of pooling is to use convolutions with in-
creased stride lengths. Removing the pooling layers simplifies the network architecture without
necessarily sacrificing performance [62].

Other common elements in many modern CNNs include

iv) Dropout regularization: A simple idea that gave a huge boost in the performance of CNNs.
By averaging several models in an ensemble one tend to get better performance than when
using single models. Dropout [63] is an averaging technique based on stochastic sampling of
neural networks.21 By randomly removing neurons during training one ends up using slightly

18
Lately, so-called fully-convolution CNNs have become popular, in which average pooling across the whole input
after the final activation layer replaces the fully-connected layers, significantly reducing the total number of weights
in the network.
19
A neural network with only linear activations would only be able to perform linear approximation. Adding further
layers wouldn’t improve its expressiveness.
20
Other options include exponential linear units (ELUs), and the now rarely used sigmoid or tanh activation
functions.
21
The idea of dropout is also used for other machine learning models, as in the DART technique for regression trees
[64]

7
 different networks for each batch of training data, and the weights of the trained network are
tuned based on optimization of multiple variations of the network.22

iiv) Batch normalization: These layers are typically placed after activation layers, producing
normalized activation maps by subtracting the mean and dividing by the standard deviation
for each training batch. Including batch normalization layers forces the network to periodically
change its activations to zero mean and unit standard deviation as the training batch hits
these layers, which works as a regularizer for the network, speeds up training, and makes it
less dependent on careful parameter initialization [67].

In the design of new and improved CNN architectures, these components are combined in increas-
ingly complicated and interconnected ways, or even replaced by other more convenient operations.
When architecting a CNN for a particular task there are multiple factors to consider, including
understanding the task to be solved and the requirements to be met, figuring out how to best feed
the data to the network, and optimally utilizing one’s budget for computation and memory con-
sumption. In the early days of modern deep learning one tended to use very simple combinations
of the building blocks, as in Lenet [57] and AlexNet [1]. Later network architectures are much more
complex, each generation building on ideas and insights from previous architectures, resulting in
updates to the state-of-the-art. Table 1 contains a short list of some famous CNN architectures,
illustrating how the building blocks can be combined and how the field moves along.

Table 1: A far from exhaustive, non-chronological, list of CNN architectures and some high-level descriptions

AlexNet [1] The network that launched the current deep learning boom by winning the
2012 ILSVRC competition by a huge margin. Notable features include the
use of RELUs, dropout regularization, splitting the computations on multiple
GPUs, and using data augmentation during training. ZFNet [68], a relatively
minor modification of AlexNet, won the 2013 ILSVRC competition.

VGG [69] Popularized the idea of using smaller filter kernels and therefore deeper net-
works (up to 19 layers for VGG19, compared to 7 for AlexNet and ZFNet),
and training the deeper networks using pre-training on shallower versions.

GoogLeNet [70] Promoted the idea of stacking the layers in CNNs more creatively, as networks
in networks, building on the idea of [71]. Inside a relatively standard archi-
tecture (called the stem), GoogLeNet contains multiple inception modules, in
which multiple different filter sizes are applied to the input and their results
concatenated. This multi-scale processing allows the module to extract fea-
tures at different levels of detail simultaneously. GoogLeNet also popularized
the idea of not using fully-connected layers at the end, but rather global av-
erage pooling, significantly reducing the number of model parameters. It won
the 2014 ILSVRC competition.

22
In addition to increased model performance, dropout can also be used to produce robust uncertainty measures
in neural networks. By leaving dropout turned on also during inference one effectively performs variational inference
[65, 59, 66]. This relates standard deep neural networks to Bayesian neural networks, synthesized in the field of
Bayesian deep learning.

8
 ResNet [72] Introduced skip connections, which makes it possible to train much deeper
networks. A 152 layer deep ResNet won the 2015 ILSVRC competition, and
the authors also successfully trained a version with 1001 layers. Having skip
connections in addition to the standard pathway gives the network the option
to simply copy the activations from layer to layer (more precisely, from ResNet
block to ResNet block), preserving information as data goes through the layers.
Some features are best constructed in shallow networks, while others require
more depth. The skip connections facilitate both at the same time, increasing
the network’s flexibility when fed input data. As the skip connections make
the network learn residuals, ResNets perform a kind of boosting.

Highway nets [73] Another way to increase depth based on gating units, an idea from Long Short
Term Memory (LSTM) recurrent networks, enabling optimization of the skip
connections in the network. The gates can be trained to find useful combina-
tions of the identity function (as in ResNets) and the standard nonlinearity
through which to feed its input.

DenseNet [74] Builds on the ideas of ResNet, but instead of adding the activations produced
by one layer to later layers, they are simply concatenated together. The origi-
nal inputs in addition to the activations from previous layers are therefore kept
at each layer (again, more precisely, between blocks of layers), preserving some
kind of global state. This encourages feature reuse and lowers the number of
parameters for a given depth. DenseNets are therefore particularly well-suited
for smaller data sets (outperforming others on e.g. Cifar-10 and Cifar-100).

ResNext [75] Builds on ResNet and GoogLeNet by using inception modules between skip
connections.

SENets [76] Squeeze-and-Excitation Networks, which won the ILSVRC 2017 competition,
builds on ResNext but adds trainable parameters that the network can use to
weigh each feature map, where earlier networks simply added them up. These
SE-blocks allows the network to model the channel and spatial information
separately, increasing the model capacity. SE-blocks can easily be added to
any CNN model, with negligible increase in computational costs.

NASNet [77] A CNN architecture designed by a neural network, beating all the previous
human-designed networks at the ILSVRC competition. It was created using
AutoML23 , Google Brain’s reinforcement learning approach to architecture
design [78]. A controller network (a recurrent neural network) proposes archi-
tectures aimed to perform at a specific level for a particular task, and by trial
and error learns to propose better and better models. NASNet was based on
Cifar-10, and has relatively modest computational demands, but still outper-
formed the previous state-of-the-art on ILSVRC data.

23
https://cloud.google.com/automl

9
 YOLO [79] Introduced a new, simplified way to do simultaneous object detection and clas-
sification in images. It uses a single CNN operating directly on the image and
outputting bounding boxes and class probabilities. It incorporates several el-
ements from the above networks, including inception modules and pretraining
a smaller version of the network. It’s fast enough to enable real-time pro-
cessing24 . YOLO makes it easy to trade accuracy for speed by reducing the
model size. YOLOv3-tiny was able to process images at over 200 frames per
second on a standard benchmark data set, while still producing reasonable
predictions.

GANs [80] A generative adversarial network consists of two neural networks pitted against
each other. The generative network G is tasked with creating samples that the
discriminative network D is supposed to classify as coming from the generative
network or the training data. The networks are trained simultaneously, where
G aims to maximize the probability that D makes a mistake while D aims for
high classification accuracy.

Siamese nets [81] An old idea (e.g. [82]) that’s recently been shown to enable one-shot learning,
i.e. learning from a single example. A siamese network consists of two identical
neural networks, both the architecture and the weights, attached at the end.
They are trained together to differentiate pairs of inputs. Once trained, the
features of the networks can be used to perform one-shot learning without
retraining.

U-net [83] A very popular and successful network for segmentation in 2D images. When
fed an input image, it is first downsampled through a “traditional” CNN, be-
fore being upsampled using transpose convolutions until it reaches its original
size. In addition, based on the ideas of ResNet, there are skip connections
that concatenates features from the downsampling to the upsampling paths.
It is a fully-convolutional network, using the ideas first introduced in [84].

V-net [85] A three-dimensional version of U-net with volumetric convolutions and skip-
connections as in ResNet.

These neural networks are typically implemented in one or more of a small number of software
frameworks that dominates machine learning research, all built on top of NVIDIA’s CUDA plat-
form and the cuDNN library. Today’s deep learning methods are almost exclusively implemented
in either TensorFlow, a framework originating from Google Research, Keras, a deep learning li-
brary originally built by François Chollet and recently incorporated in TensorFlow, or Pytorch, a
framework associated with Facebook Research. There are very few exceptions (YOLO built using
the Darknet framework [86] is one of the rare ones). All the main frameworks are open source and
under active development.

3. Deep learning, medical imaging and MRI

Deep learning methods are increasingly used to improve clinical practice, and the list of examples
is long, growing daily. We will not attempt a comprehensive overview of deep learning in medical
imaging, but merely sketch some of the landscape before going into a more systematic exposition
of deep learning in MRI.

24
You can watch YOLO in action here https://youtu.be/VOC3huqHrss

10
 Convolutional neural networks can be used for efficiency improvement in radiology practices
through protocol determination based on short-text classification [87]. They can also be used to
reduce the gadolinium dose in contrast-enhanced brain MRI by an order of magnitude [88] without
significant reduction in image quality. Deep learning is applied in radiotherapy [89], in PET-MRI
attenuation correction [90, 91], in radiomics [92, 93] (see [94] for a review of radiomics related
to radiooncology and medical physics), and for theranostics in neurosurgical imaging, combining
confocal laser endomicroscopy with deep learning models for automatic detection of intraoperative
CLE images on-the-fly [95].
Another important application area is advanced deformable image registration, enabling quan-
titative analysis across different physical imaging modalities and across time.25 . For example elastic
registration between 3D MRI and transrectal ultrasound for guiding targeted prostate biopsy [96];
deformable registration for brain MRI where a “cue-aware deep regression network” learns from
a given set of training images the displacement vector associated with a pair of reference-subject
patches [97]; fast deformable image registration of brain MR image pairs by patch-wise prediction
of the Large Deformation Diffeomorphic Metric Mapping model [98]26 ; unsupervised convolutional
neural network-based algorithm for deformable image registration of cone-beam CT to CT using
a deep convolutional inverse graphics network [99]; deep learning-based 2D/3D registration frame-
work for registration of preoperative 3D data and intraoperative 2D X-ray images in image-guided
therapy [100]; real-time prostate segmentation during targeted prostate biopsy, utilizing temporal
information in the series of ultrasound images [101].
This is just a tiny sliver of the many applications of deep learning to central problems in medical
imaging. There are several thorough reviews and overviews of the field to consult for more informa-
tion, across modalities and organs, and with different points of view and level of technical details. For
example the comprehensive review [102]27 , covering both medicine and biology and spanning from
imaging applications in healthcare to protein-protein interaction and uncertainty quantification; key
concepts of deep learning for clinical radiologists [103, 104, 105, 106, 107, 108, 109, 110, 111, 112],
including radiomics and imaging genomics (radiogenomics) [113], and toolkits and libraries for deep
learning [114]; deep learning in neuroimaging and neuroradiology [115]; brain segmentation [116];
stroke imaging [117, 118]; neuropsychiatric disorders [119]; breast cancer [120, 121]; chest imaging
[122]; imaging in oncology [123, 124, 125]; medical ultrasound [126, 127]; and more technical sur-
veys of deep learning in medical image analysis [42, 128, 129, 130]. Finally, for those who like to
be hands-on, there are many instructive introductory deep learning tutorials available online. For
example [131], with accompanying code available at https://github.com/paras42/Hello_World_
Deep_Learning, where you’ll be guided through the construction of a system that can differentiate
a chest X-ray from an abdominal X-ray using the Keras/TensorFlow framework through a Jupyter
Notebook. Other nice tutorials are http://bit.ly/adltktutorial, based on the Deep Learning
Toolkit (DLTK) [132], and https://github.com/usuyama/pydata-medical-image, based on the
Microsoft Cognitive Toolkit (CNTK).

Let’s now turn to the field of MRI, in which deep learning has seen applications at each step
of entire workflows. From acquisition to image retrieval, from segmentation to disease prediction.
We divide this into two parts: (i) the signal processing chain close to the physics of MRI, including
image restoration and multimodal image registration (Fig. 3), and (ii) the use of deep learning in

25
e.g. test-retest examinations, or motion correction in dynamic imaging
26
available at https://github.com/rkwitt/quicksilver
27
A continuous collaborative manuscript (https://greenelab.github.io/deep-review) with >500 references.

11
MR image segmentation, disease detection, disease prediction and systems based on images and
text data (reports), addressing a few selected organs such as the brain, the kidney, the prostate and
the spine (Fig. 4).

3.1. From image acquisition to image registration

Deep learning in MRI has typically been focused on segmentation and classification of recon-
structed magnitude images. Its penetration into the lower levels of MRI measurement techniques
is more recent, but already impressive. From MR image acquisition and signal processing in MR
fingerprinting, to denoising and super-resolution, and into image synthesis.

IMAGE ACQUISITION IMAGE RECONSTRUCTION IMAGE RESTORATION IMAGE REGISTRATION

y Magnitude

sMRI

z
w
RF
Re dMRI
Phase
FFT-1

Multiparametric
k-space
x MRI
Im
fMRI

Figure 3: Deep learning in the MR signal processing chain, from image acquisition (in complex-valued k-space) and
image reconstruction, to image restoration (e.g. denoising) and image registration. The rightmost column illustrates
coregistration of multimodal brain MRI. sMRI = structural 3D T1-weighted MRI, dMRI = diffusion weighted MRI
(stack of slices in blue superimposed on sMRI), fMRI = functional BOLD MRI (in red).

3.1.1. Data acquisition and image reconstruction

Research on CNN and RNN-based image reconstruction methods is rapidly increasing, pioneered
by Yang et al. [133] at NIPS 2016 and Wang et al. [134] at ISBI 2016. Recent applications ad-
dresses e.g. convolutional recurrent neural networks for dynamic MR image reconstruction [135],
reconstructing good quality cardiac MR images from highly undersampled complex-valued k-space
data by learning spatio-temporal dependencies, outperforming 3D CNN approaches and compressed
sensing-based dynamic MRI reconstruction algorithms in computational complexity, reconstruction
accuracy and speed for different undersampling rates. Schlemper et.al. [136] created a deep cascade
of concatenated CNNs for dynamic MR image reconstruction, making use of data augmentation,
both rigid and elastic deformations, to increase the variation of the examples seen by the network
and reduce overfitting28 . Using variational networks for single-shot fast spin-echo MRI with variable
density sampling, Chen et.al. [137] enabled real-time (200 ms per section) image reconstruction,
outperforming conventional parallel imaging and compressed sensing reconstruction. In [138], the
authors explored the potential for transfer learning (pretrained models) and assessed the gener-
alization of learned image reconstruction regarding image contrast, SNR, sampling pattern and
image content, using a variational network and true measurement k-space data from patient knee
MRI recordings and synthetic k-space data generated from images in the Berkeley Segmentation
Data Set and Benchmarks. Employing least-squares generative adversarial networks (GANs) that

28
Code available at https://github.com/js3611/Deep-MRI-Reconstruction

12
learns texture details and suppresses high-frequency noise, [139] created a novel compressed sensing
framework that can produce diagnostic quality reconstructions “on the fly” (30 ms)29 . A unified
framework for image reconstruction [140], called automated transform by manifold approximation
(AUTOMAP) consisting of a feedforward deep neural network with fully connected layers followed
by a sparse convolutional autoencoder, formulate image reconstruction generically as a data-driven
supervised learning task that generates a mapping between the sensor and the image domain based
on an appropriate collection of training data (e.g. MRI examinations collected from the Human
Connectome Project, transformed to the k-space sensor domain).

There are also other approaches and reports on deep learning in MR image reconstruction, e.g.
[141, 142, 143, 144], a fundamental field rapidly progressing.

3.1.2. Quantitative parameters - QSM and MR fingerprinting

Another area that is developing within deep learning for MRI is the estimation of quantitative
tissue parameters from recorded complex-valued data. For example within quantitative susceptibility
mapping, and in the exciting field of magnetic resonance fingerprinting.
Quantitative susceptibility mapping (QSM) is a growing field of research in MRI, aiming to
noninvasively estimate the magnetic susceptibility of biological tissue [145, 146]. The technique is
based on solving the difficult, ill-posed inverse problem of determining the magnetic susceptibility
from local magnetic fields. Recently Yoon et al. [147] constructed a three-dimensional CNN, named
QSMnet and based on the U-Net architecture, able to generate high quality susceptibility source
maps from single orientation data. The authors generated training data by using the gold-standard
for QSM: the so-called COSMOS method [148]. The data was based on 60 scans from 12 healthy
volunteers. The resulting model both simplified and improved the state-of-the-art for QSM. Ras-
mussen and coworkers [149] took a different approach. They also used a U-Net-based convolutional
neural network to perform field-to-source inversion, called DeepQSM, but it was trained on syn-
thetically generated data containing simple geometric shapes such as cubes, rectangles and spheres.
After training their model on synthetic data it was able to generalize to real-world clinical brain
MRI data, computing susceptibility maps within seconds end-to-end. The authors conclude that
their method, combined with fast imaging sequences, could make QSM feasible in standard clinical
practice.
Magnetic resonance fingerprinting (MRF) was introduced a little more than five years ago [150],
and has been called “a promising new approach to obtain standardized imaging biomarkers from
MRI” by the European Society of Radiology [151]. It uses a pseudo-randomized acquisition that
causes the signals from different tissues to have a unique signal evolution (“fingerprint”) that is a
function of the multiple material properties being investigated. Mapping the signals back to known
tissue parameters (T1, T2 and proton density) is then a rather difficult inverse problem. MRF is
closely related to the idea of compressed sensing [152] in MRI [153] in that MRF undersamples
data in k-space producing aliasing artifacts in the reconstructed images that can be suppressed by
compressed sensing.30 It can be regarded as a quantitative multiparametric MRI analysis, and with
recent acquisition schemes using a single-shot spiral trajectory with undersampling, whole-brain
coverage of T1 , T2 and proton density maps can be acquired at 1.2 × 1.2 × 3 mm3 voxel resolution

29
In their GAN setting, a generator network is used to map undersampled data to a realistic-looking image with
high measurement fidelity, while a discriminator network is trained jointly to score the quality of the reconstructed
image.
30
See [154, 155, 156, 157, 158] for recent perspectives and developments connecting deep learning-based reconstruc-
tion methods to the more general research field of inverse problems.

13
in less than 5 min [159].
The processing of MRF after acquisition usually involves using various pattern recognition algo-
rithms that try to match the fingerprints to a predefined dictionary of predicted signal evolutions31 ,
created using the Bloch equations [150, 164].
Recently, deep learning methodology has been applied to MR fingerprinting. Cohen et al.
[165] reformulated the MRF reconstruction problem as learning an optimal function that maps
the recorded signal magnitudes to the corresponding tissue parameter values, trained on a sparse
set of dictionary entries. To achieve this they fed voxel-wise MRI data acquired with an MRF
sequence (MRF-EPI, 25 frames in ∼3 s; or MRF-FISP, 600 frames in ∼7.5 s) to a four-layer neural
network consisting of two hidden layers with 300 × 300 fully connected nodes and two nodes in
the output layer, considering only T1 and T2 parametric maps. The network, called MRF Deep
RecOnstruction NEtwork (DRONE), was trained by an adaptive moment estimation stochastic
gradient descent algorithm with a mean squared error loss function. Their dictionary consisted of
∼70000 entries (product of discretized T1 and T2 values) and training the network to convergence
with this dictionary (∼10 MB for MRF-EPI and ∼300 MB for MRF-FISP) required 10 to 70 min
using an NVIDIA K80 GPU with 2 GB memory. They found their reconstruction time (10 to 70 ms
per slice) to be 300 to 5000 times faster than conventional dictionary-matching techniques, using
both well-characterized calibrated ISMRM/NIST phantoms and in vivo human brains.
A similar deep learning approach to predict quantitative parameter values (T1 and T2 ) from
MRF time series was taken by Hoppe et al. [166]. In their experiments they used 2D MRF-
FISP data with variable TR (12-15 ms), flip angles (5◦ -74◦ ) and 3000 repetitions, recorded on a
MAGNETOM 3T Skyra. A high resolution dictionary was simulated to generate a large collection
of training and testing data, using tissues T1 and T2 relaxation time ranges as present in normal
brain at 3T (e.g. [167]) resulting in ∼ 1.2 × 105 time series. In contrast to [165], their deep neural
network architecture was inspired from the domain of speech recognition due to the similarity of
the two tasks. The architecture with the smallest average error for validation data was a standard
convolutional neural network consisting of an input layer of 3000 nodes (number of samples in
the recorded time series), four hidden layers, and an output layers with two nodes (T1 and T2 ).
Matching one time series was about 100 times faster than the conventional [150] matching method
and with very small mean absolute deviations from ground truth values.
In the same context, Fang et al. [168] used a deep learning method to extract tissue properties
from highly undersampled 2D MRF-FISP data in brain imaging, where 2300 time points were
acquired from each measurement and each time point consisted of data from one spiral readout
only. The real and imaginary parts of the complex signal were separated into two channels. They
used MRF signal from a patch of 32 × 32 pixels to incorporate correlated information between
neighboring pixels. In their work they designed a standard three-layer CNN with T1 and T2 as
output.
Virtue et.al. [169] investigated a different approach to MRF. By generating 100.000 synthetic
MRI signals using a Bloch equation simulator they were able to train feedforward deep neural net-
works to map new MRI signals to the tissue parameters directly, producing approximate solutions
to the inverse mapping problem of MRF. In their work they designed a new complex activation
function, the complex cardioid, that was used to construct a complex-valued feedforward neural
network. This three-layer network outperformed both the standard MRF techniques based on dic-

31
A dictionary of time series for every possible combination of parameters like (discretized) T1 and T2 relaxation
times, spin-density (M0 ), B0 , off-resonance (∆f ), and also voxel-wise cerebral blood volume (CBV), mean vessel
radius (R), blood oxygen saturation (SO2 ) and T∗2 [160, 161, 162], and more, e.g. MFR-ASL [163].

14
tionary matching, and also the analogous real neural network operating on the real and imaginary
components separately. This suggested that complex-valued networks are better suited at uncover-
ing information in complex data.32

3.1.3. Image restoration (denoising, artifact detection)

Estimation of noise and image denoising in MRI has been an important field of research for many
years [172, 173], employing a plethora of methods. For example Bayesian Markov random field
models [174], rough set theory [175], higher-order singular value decomposition [176], wavelets [177],
independent component analysis [178], or higher order PDEs [179].
Recently, deep learning approaches have been introduced to denoising. In their work on learning
implicit brain MRI manifolds using deep neural networks, Bermudez et al. [180] implemented an
autoencoder with skip connections for image denoising, testing their approach with adding various
levels of Gaussian noise to more than 500 T1-weighted brain MR images from healthy controls in the
Baltimore Longitudinal Study of Aging. Their autoencoder network outperformed the current FSL
SUSAN denoising software according to peak signal-to-noise ratios. Benou et al. [181] addressed
spatio-temporal denoising of dynamic contrast-enhanced MRI of the brain with bolus injection of
contrast agent (CA), proposing a novel approach using ensembles of deep neural networks for noise
reduction. Each DNN was trained on a different range of SNRs and types of CA concentration
time curves (denoted “pathology experts”, “healthy experts”, “vessel experts”) to generate a re-
construction hypothesis from noisy input by using a classification DNN to select the most likely
hypothesis and provide a “clean output” curve. Training data was generated synthetically using a
three-parameter Tofts pharmacokinetic (PK) model and noise realizations. To improve this model,
accounting for spatial dependencies of PK pharmacokinetics, they used concatenated noisy time
curves from first-order neighbourhood pixels in their expert DNNs and ensemble hypothesis DNN,
collecting neighboring reconstructions before a boosting procedure produced the final clean out-
put for the pixel of interest. They tested their trained ensemble model on 33 patients from two
different DCE-MRI databases with either stroke or recurrent glioblastoma (RIDER NEURO33 ),
acquired at different sites, with different imaging protocols, and with different scanner vendors and
field strengths. The qualitative and quantitative (MSE) denoising results were better than spatio-
temporal Beltrami, moving average, the dynamic Non Local Means method [182], and stacked
denoising autoencoders [183]. The run-time comparisons were also in favor of the proposed sDNN.
In this context of DCE-MRI, it’s tempting to speculate whether deep neural network approaches
could be used for direct estimation of tracer-kinetic parameter maps from highly undersampled
(k, t)-space data in dynamic recordings [184, 185], a powerful way to by-pass 4D DCE-MRI recon-
struction altogether and map sensor data directly to spatially resolved pharmacokinetic parameters,
e.g. Ktrans , vp , ve in the extended Tofts model or parameters in other classic models [186]. A related
approach in the domain of diffusion MRI, by-passing the model-fitting steps and computing voxel-
wise scalar tissue properties (e.g. radial kurtosis, fiber orientation dispersion index) directly from
the subsampled DWIs was taken by Golkov et al. [187] in their proposed “q-space deep learning”
family of methods.
Deep learning methods has also been applied to MR artifact detection, e.g. poor quality spectra
in MRSI [188]; detection and removal of ghosting artifacts in MR spectroscopy [189]; and automated
reference-free detection of patient motion artifacts in MRI [190].

32
Complex-valued deep learning is also getting some attention in a broader community of researchers, and has been
shown to lead to improved models. See e.g. [170, 171] and the references therein.
33
https://wiki.cancerimagingarchive.net/display/Public/RIDER+NEURO+MRI

15
3.1.4. Image super-resolution
Image super-resolution, reconstructing a higher-resolution image or image sequence from the ob-
served low-resolution image [191], is an exciting application of deep learning methods34 .
Super-resolution for MRI have been around for almost 10 years [192, 193] and can be used to
improve the trade-off between resolution, SNR, and acquisition time [194], generate 7T-like MR
images on 3T MRI scanners [195], or obtain super-resolution T1 maps from a set of low resolution
T1 weighted images [196]. Recently deep learning approaches has been introduced, e.g. generating
super-resolution single (no reference information) and multi-contrast (applying a high-resolution
image of another modality as reference) brain MR images using CNNs [197]; constructing super-
resolution brain MRI by a CNN stacked by multi-scale fusion units [198]; and super-resolution
musculoskeletal MRI (“DeepResolve”) [199]. In DeepResolve thin (0.7 mm) slices in knee images
(DESS) from 124 patients included in the Osteoarthritis Initiative were used for training and 17
patients for testing, with a 10s inference time per 3D (344 × 344 × 160) volume. The resulting im-
ages were evaluated both quantitatively (MSE, PSNR, and the perceptual window-based structural
similarity SSIM35 index) and qualitatively by expert radiologists.

3.1.5. Image synthesis

Image synthesis in MRI have traditionally been seen as a method to derive new parametric images
or new tissue contrast from a collection of MR acquisition performed at the same imaging session,
i.e. “an intensity transformation applied to a given set of input images to generate a new image with
a specific tissue contrast” [200]. Another avenue of MRI synthesis is related to quantitative imaging
and the development and use of physical phantoms, imaging calibration/standard test objects with
specific material properties. This is done in order to assess the performance of an MRI scanner or
to assess imaging biomarkers reliably with application-specific phantoms such as a structural brain
imaging phantom, DCE-MRI perfusion phantom, diffusion phantom, flow phantom, breast phantom
or a proton-density fat fraction phantom [201]. The in silico modeling of MR images with certain
underlying properties, e.g. [202, 203], or model-based generation of large databases of (cardiac)
images from real healthy cases [204] is also part of this endeavour. In this context, deep learning
approaches have accelerated research and the amount of costly training data.
The last couple of years have seen impressive results for photo-realistic image synthesis using deep
learning techniques, especially generative adversarial networks (GANs, introduced by Goodfellow et
al. in 2014 [80]), e.g. [205, 206, 207]. These can also be used for biological image synthesis [208, 209]
and text-to-image synthesis [210, 211, 212].36 Recently, a group of researchers from NVIDIA,
MGH & BWH Center for Clinical Data Science in Boston, and the Mayo Clinic in Rochester
[213] designed a clever approach to generate synthetic abnormal MRI images with brain tumors by
training a GAN based on pix2pix37 using two publicly available data sets of brain MRI (ADNI
and the BRATS’15 Challenge, and later also the Ischemic Stroke Lesion Segmentation ISLES’2018
Challenge). This approach is highly interesting as medical imaging datasets are often imbalanced,
with few pathological findings, limiting the training of deep learning models. Such generative
models for image synthesis serve as a form of data augmentation, and also as an anonymization
tool. The authors achieved comparable tumor segmentation results when trained on the synthetic

34
See http://course.fast.ai/lessons/lesson14.html for an instructive introduction to super-resolution
35
http://www.cns.nyu.edu/~lcv/ssim
36
See here https://github.com/xinario/awesome-gan-for-medical-imaging for a list of interesting applications
of GAN in medical imaging
37
https://phillipi.github.io/pix2pix

16
data rather than on real patient data. A related approach to brain tumor segmentation using coarse-
to-fine GANs was taken by Mok & Chung [214]. Guibas et al. [215] used a two-stage pipeline for
generating synthetic medical images from a pair of GANs, addressing retinal fundus images, and
provided an online repository (SynthMed) for synthetic medical images. Kitchen & Seah [216] used
GANs to synthetize realistic prostate lesions in T2 , ADC, Ktrans resembling the SPIE-AAPM-NCI
ProstateX Challenge 201638 training data.
Other applications are unsupervised synthesis of T1-weighted brain MRI using a GAN [180];
image synthesis with context-aware GANs [217]; synthesis of patient-specific transmission image
for PET attenuation correction in PET/MR imaging of the brain using a CNN [218]; pseudo-CT
synthesis for pelvis PET/MR attenuation correction using a Dixon-VIBE Deep Learning (DIVIDE)
network [219]; image synthesis with GANs for tissue recognition [220]; synthetic data augmentation
using a GAN for improved liver lesion classification [221]; and deep MR to CT synthesis using
unpaired data [222].

3.1.6. Image registration

Image registration39 is an increasingly important field within MR image processing and analysis as
more and more complementary and multiparametric tissue information are collected in space and
time within shorter acquisition times, at higher spatial (and temporal) resolutions, often longitudi-
nally, and across patient groups, larger cohorts, or atlases. Traditionally one has divided the tasks
of image registration into dichotomies: intra vs. inter-modality, intra vs. inter-subject, rigid vs. de-
formable, geometry-based vs. intensity-based, and prospective vs. retrospective image registration.
Mathematically, registration is a challenging mix of geometry (spatial transformations), analysis
(similarity measures), optimization strategies, and numerical schemes. In prospective motion cor-
rection, real-time MR physics is also an important part of the picture [224, 225]. A wide range of
methodological approaches have been developed and tested for various organs and applications40
[229, 230, 231, 232, 233, 234, 235, 236, 237, 238], including “previous generation” artificial neural
networks [239].
Recently, deep learning methods have been applied to image registration in order to improve
accuracy and speed (e.g. Section 3.4 in [42]). For example: deformable image registration [240,
98]; model-to-image registration [241, 242]; MRI-based attenuation correction for PET [243, 244];
PET/MRI dose calculation [245]; unsupervised end-to-end learning for deformable registration of
2D CT/MR images [246]; an unsupervised learning model for deformable, pairwise 3D medical
image registration by Balakrishnan et al. [247]41 ; and a deep learning framework for unsupervised
affine and deformable image registration [248].

3.2. From image segmentation to diagnosis and prediction

We leave the lower-level applications of deep learning in MRI to consider higher-level (down-
stream) applications such as fast and accurate image segmentation, disease prediction in selected
organs (brain, kidney, prostate, and spine) and content-based image retrieval, typically applied

38
https://www.aapm.org/GrandChallenge/PROSTATEx-2
39
Image registration can be defined as “the determination of a one-to-one mapping between the coordinates in one
space and those in another, such that points in the two spaces that correspond to the same anatomical point are
mapped to each other” (C.R Maurer [223], 1993).
40
and different hardware e.g. GPUs [226, 227, 228] as image registration is often computationally time consuming.
41
with code available at https://github.com/voxelmorph/voxelmorph

17
to reconstructed magnitude images. We have chosen to focus our overview on deep learning ap-
plications close to the MR physics and will be brief in the present section, even if the following
applications are very interesting and clinically important.

BRAIN KIDNEY PROSTATE SPINE

Figure 4: Deep learning for MR image analysis in selected organs, partly from ongoing work at MMIV.

3.2.1. Image segmentation

Image segmentation, the holy grail of quantitative image analysis, is the process of partitioning
an image into multiple regions that share similar attributes, enabling localization and quantifi-
cation.42 It has an almost 50 years long history, and has become the biggest target for deep
learning approaches in medical imaging. The multispectral tissue classification report by Van-
nier et al. in 1985 [249], using statistical pattern recognition techniques (and satellite image pro-
cessing software from NASA), represented one of the most seminal works leading up to today’s
machine learning in medical imaging segmentation. In this early era, we also had the opportu-
nity to contribute with supervised and unsupervised machine learning approaches for MR image
segmentation and tissue classification [250, 251, 252, 253]. An impressive range of segmentation
methods and approaches have been reported (especially for brain segmentation) and reviewed, e.g.
[254, 255, 256, 257, 258, 259, 260, 261, 262]. MR image segmentation using deep learning ap-
proaches, typically CNNs, are now penetrating the whole field of applications. For example acute
ischemic lesion segmentation in DWI [263]; brain tumor segmentation [264]; segmentation of the
striatum [265]; segmentation of organs-at-risks in head and neck CT images [266]; and fully auto-
mated segmentation of polycystic kidneys [267]; deformable segmentation of the prostate [268]; and
spine segmentation with 3D multiscale CNNs [269].
See [42] and [102] for more comprehensive lists.

3.2.2. Diagnosis and prediction

A presumably complete list of papers up to 2017 using deep learning techniques for brain image
analysis is provided as Table 1 in Litjens at al. [42]. In the following we add some more recent
work on organ-specific deep learning using MRI, restricting ourselves to brain, kidney, prostate and
spine.

Table 2: A short list of deep learning applications per organ, task, reference and description.

BRAIN

42
Segmentation is also crucial for functional imaging, enabling tissue physiology quantification with preservation of
anatomical specificity.

18
Brain extraction [270] A 3D CNN for skull stripping

Functional connectomes [271] Transfer learning approach to enhance deep neural network classifica-
tion of brain functional connectomes

[272] Multisite diagnostic classification of schizophrenia using discriminant

deep learning with functional connectivity MRI

Structural connectomes [273] A convolutional neural network-based approach (https://github.com/

MIC-DKFZ/TractSeg) that directly segments tracts in the field of fiber
orientation distribution function (fODF) peaks without using tractog-
raphy, image registration or parcellation. Tested on 105 subjects from
the Human Connectome Project

Brain age [274] Chronological age prediction from raw brain T1-MRI data, also testing
the heritability of brain-predicted age using a sample of 62 monozygotic
and dizygotic twins

Alzheimer’s disease [275] Landmark-based deep multi-instance learning evaluated on 1526 sub-
jects from three public datasets (ADNI-1, ADNI-2, MIRIAD)

[276] Identify different stages of AD

[277] Multimodal and multiscale deep neural networks for the early diagnosis
of AD using structural MR and FDG-PET images

Vascular lesions [278] Evaluation of a deep learning approach for the segmentation of brain
tissues and white matter hyperintensities of presumed vascular origin
in MRI

Identification of MRI [279] Using deep learning algorithms to automatically identify the brain MRI
contrast contrast, with implications for managing large databases

Meningioma [280] Fully automated detection and segmentation of meningiomas using

deep learning on routine multiparametric MRI

Glioma [281] Glioblastoma segmentation using heterogeneous MRI data from clinical
routine

[282] Deep learning for segmentation of brain tumors and impact of cross-
institutional training and testing

[283] Automatic semantic segmentation of brain gliomas from MRI using a

deep cascaded neural network

[284] AdaptAhead optimization algorithm for learning deep CNN applied to

MRI segmentation of glioblastomas (BRATS)

Multiple sclerosis [285] Deep learning of joint myelin and T1w MRI features in normal-
appearing brain tissue to distinguish between multiple sclerosis patients
and healthy controls

KIDNEY

Abdominal organs [286] CNNs to improve abdominal organ segmentation, including left kidney,
right kidney, liver, spleen, and stomach in T2 -weighted MR images

19
Cyst segmentation [267] An artificial multi-observer deep neural network for fully automated
segmentation of polycystic kidneys

Renal transplant [287] A deep-learning-based classifier with stacked non-negative constrained

autoencoders to distinguish between rejected and non-rejected renal
transplants in DWI recordings

PROSTATE

Cancer (PCa) [288] Proposed a method for end-to-end prostate segmentation by integrating
holistically (image-to-image) nested edge detection with fully convolu-
tional networks. their nested networks automatically learn a hierar-
chical representation that can improve prostate boundary detection.
Obtained very good results (Dice coefficient, 5-fold cross validation) on
MRI scans from 250 patients

[289] Computer-aided diagnosis with a CNN, deciding ‘cancer’ ‘no cancer’

trained on data from 301 patients with a prostate-specific antigen level
of < 20 ng/mL who underwent MRI and extended systematic prostate
biopsy with or without MRI-targeted biopsy

[290] Automatic approach based on deep CNN, inspired from VGG, to clas-
sify PCa and noncancerous tissues with multiparametric MRI using
data from the PROSTATEx database

[291] Deep CNN and a non-deep learning using feature detection (the scale-
invariant feature transform and the bag-of-words model, a representa-
tive method for image recognition and analysis) were used to distinguish
pathologically confirmed PCa patients from prostate benign conditions
patients with prostatitis or prostate benign hyperplasia in a collection
of 172 patients with more than 2500 morphologic 2D T2 -w MR images

[292] Designed a system which can concurrently identify the presence of PCa
in an image and localize lesions based on deep CNN features (co-trained
CNNs consisting of two parallel convolutional networks for ADC and
T2 -w images respectively) and a single-stage SVM classifier for au-
tomated detection of PCa in multiparametric MRI. Evaluated on a
dataset of 160 patients

[293] Designed and tested multimodel CNNs, using clinical data from 364
patients with a total of 463 PCa lesions and 450 identified noncancer-
ous image patches. Carefully investigated three critical factors which
could greatly affect the performance of their multimodal CNNs but
had not been carefully studied previously: (1) Given limited training
data, how can these be augmented in sufficient numbers and variety for
fine-tuning deep CNN networks for PCa diagnosis? (2) How can mul-
timodal mp-MRI information be effectively combined in CNNs? (3)
What is the impact of different CNN architectures on the accuracy of
PCa diagnosis?

SPINE

Vertebrae labeling [294] Designed a CNN for detection and labeling of vertebrae in MR images
with clinical annotations as training data

Intervertebral disc local- [269] 3D multi-scale fully connected CNNs with random modality voxel
ization dropout learning for intervertebral disc localization and segmentation
from multi-modality MR images

20
 Disc-level labeling, [295] CNN model denoted DeepSPINE, having a U-Net architecture com-
spinal stenosis grading bined with a spine-curve fitting method for automated lumbar verte-
bral segmentation, disc-level designation, and spinal stenosis grading
with a natural language processing scheme

Lumbal neural forminal [296] Addressed the challenge of automated pathogenesis-based diagnosis, si-
stenosis (LNFS) multaneously localizing and grading multiple spinal structures (neural
foramina, vertebrae, intervertebral discs) for diagnosing LNFS and dis-
cover pathogenic factors. Proposed a deep multiscale multitask learning
network integrating a multiscale multi-output learning and a multi-
task regression learning into a fully convolutional network where (i) a
DMML-Net merges semantic representations to reinforce the salience
of numerous target organs (ii) a DMML-Net extends multiscale convo-
lutional layers as multiple output layers to boost the scale-invariance
for various organs, and (iii) a DMML-Net joins the multitask regression
module and the multitask loss module to combine the mutual benefit
between tasks

Spondylitis vs tuberculo- [297] CNN model for differentiating between tuberculous and pyogenic
sis spondylitis in MR images. Compared their CNN performance with
that of three skilled radiologists using spine MRIs from 80 patients

Metastasis [291] A multi-resolution approach for spinal metastasis detection using deep
Siamese neural networks comprising three identical subnetworks for
multi-resolution analysis and detection. Detection performance was
evaluated on a set of 26 cases using a free-response receiver operat-
ing characteristic analysis (observer is free to mark and rate as many
suspicious regions as are considered clinically reportable)

3.3. Content-based image retrieval

The objective of content-based image retrieval (CBIR) in radiology is to provide medical cases
similar to a given image in order to assist radiologists in the decision-making process. It typically
involves large case databases, clever image representations and lesion annotations, and algorithms
that are able to quickly and reliably match and retrieve the most similar images and their anno-
tations in the case database. CBIR has been an active area of research in medical imaging for
many years, addressing a wide range of applications, imaging modalities, organs, and method-
ological approaches, e.g. [298, 299, 300, 301, 302, 303, 304], and at a larger scale outside the
medical field using deep learning techniques, e.g. at Microsoft, Apple, Facebook, and Google
(reverse image search43 ), and others. See e.g. [305, 306, 307, 308, 309] and the code reposito-
ries https://github.com/topics/image-retrieval. One of the first application of deep learning for
CBIR in the medical domain came in 2015 when Sklan et al. [310] trained a CNN to perform CBIR
with more than one million random MR and CT images, with disappointing results (true positive
rate of 20%) on their independent test set of 2100 labeled images. Medical CBIR is now, however,
dominated by deep learning algorithms [311, 312, 313]. As an example, by retrieving medical cases
similar to a given image, Pizarro et al. [279] developed a CNN for automatically inferring the
contrast of MRI scans based on the image intensity of multiple slices.

43
See “search by image” https://images.google.com, https://developers.google.com/custom-search, and also
https://tineye.com, indexing more than 30 billion images

21
Recently, deep learning methods have also been used for automated generation of radiology reports,
typically incorporating long-short-term-memory (LSTM) network models to generate the textual
paragraphs [314, 315, 316, 317], and also to identify findings in radiology reports [318, 319, 320].

4. Open science and reproducible research in machine learning for medical imaging

Machine learning is moving at a breakneck speed, too fast for the standard peer-review process
to keep up. Many of the most celebrated and impactful papers in machine learning over the past
few years are only available as preprints, or published in conference proceedings long after their
results are well-known and incorporated in the research of others. Bypassing peer-review has some
downsides, of course, but these are somewhat mitigated by researchers’ willingness to share code
and data.44
Most of the main new ideas and methods are posted to the arXiv preprint server45 , and the
accompanying code shared on the GitHub platform46 . The data sets used are often openly available
through various repositories. This, in addition to the many excellent online educational resources47 ,
makes it easy to get started in the field. Select a problem you find interesting based on openly
available data, a method described in a preprint, and an implementation uploaded to GitHub. This
forms a good starting point for an interesting machine learning project.
Another interesting aspect about modern machine learning and data science is the prevalence of
competitions, with the now annual ImageNet Large Scale Visual Recognition Challenge (ILSVRC)
competition as the main driver of progress in deep learning for computer vision since 2012. Each
competition typically draws large number of participants, and the top results often push the state-
of-the art to a new level. In addition to inspiring new ideas, competitions also provide natural entry
points to modern machine learning. It is interesting to note how deep learning-based models are
completely dominating the leaderboards of essentially all image-based competitions. Other machine
learning models, or non-machine learning-based techniques, have largely been outclassed.
What’s true about the openness of machine learning in general is increasingly true also for the
sub-field of machine learning for medical image analysis. We’ve listed a few examples of openly
available implementations, data sets and challenges in tables 3, 4 and 5 below.

Table 3: A short list of openly available code for ML in medical imaging

Summary Reference Implementation

NiftyNet. An open source convolutional [321, 322] http://niftynet.io

neural networks platform for medical im-
age analysis and image-guided therapy

DLTK. State of the art reference imple- [132] https://github.com/DLTK/DLTK

mentations for deep learning on medical
images

44
In the spirit of sharing and open science, we’ve created a GitHub repository to accompany our article, available
at https://github.com/MMIV-ML/DLMI2018.
45
http://arxiv.org
46
https://github.com
47
For example http://www.fast.ai, https://www.deeplearning.ai, http://cs231n.stanford.edu, https://
developers.google.com/machine-learning/crash-course

22
 DeepMedic [323] https://github.com/Kamnitsask/
deepmedic

U-Net: Convolutional Networks for [324] https://lmb.informatik.uni-freiburg.

Biomedical Image Segmentation de/people/ronneber/u-net

V-net [85] https://github.com/faustomilletari/

VNet

SegNet: A Deep Convolutional Encoder- [325] https://mi.eng.cam.ac.uk/projects/

Decoder Architecture for Robust Semantic segnet
Pixel-Wise Labelling

Brain lesion synthesis using GANs [213] https://github.com/khcs/

brain-synthesis-lesion-segmentation

GANCS: Compressed Sensing MRI based [326] https://github.com/gongenhao/GANCS

on Deep Generative Adversarial Network

Deep MRI Reconstruction [136] https://github.com/js3611/

Deep-MRI-Reconstruction

Graph Convolutional Networks for brain [327] https://github.com/parisots/

analysis in populations, combining imag- population-gcn
ing and non-imaging data

Table 4: A short list of medical imaging data sets and repositories

Name Summary Link

OpenNeuro An open platform for sharing neuroimag- https://openneuro.org48

ing data under the public domain license.
Contains brain images from 168 studies
(4,718 participants) with various imaging
modalities and acquisition protocols.

UK Biobank Health data from half a million partici- http://www.ukbiobank.ac.uk/

pants. Contains MRI images from 15.000
participants, aiming to reach 100.000.

TCIA The cancer imaging archive hosts a large http://www.cancerimagingarchive.net

archive of medical images of cancer ac-
cessible for public download. Currently
contains images from from 14.355 patients
across 77 collections.

ABIDE The autism brain imaging data exchange. http://fcon_1000.projects.nitrc.org/

Contains 1114 datasets from 521 individ- indi/abide
uals with Autism Spectrum Disorder and
593 controls.

48
Data can be downloaded from the AWS S3 Bucket https://registry.opendata.aws/openneuro.

23
ADNI The Alzheimer’s disease neuroimaging ini- http://adni.loni.usc.edu/
tiative. Contains image data from almost
2000 participants (controls, early MCI,
MCI, late MCI, AD)

Table 5: A short list of medical imaging competitions

Name Summary Link

Grand-Challenges Grand challenges in biomedi- https://grand-challenge.org/

cal image analysis. Hosts and
lists a large number of compe-
titions

RSNA Pneumonia Detec- Automatically locate lung https://www.kaggle.com/c/

tion Challenge opacities on chest radiographs rsna-pneumonia-detection-challenge

HVSMR 2016 Segment the blood pool and http://segchd.csail.mit.edu/

myocardium from a 3D cardio-
vascular magnetic resonance
image

ISLES 2018 Ischemic Stroke Lesion Seg- http://www.isles-challenge.org/

mentation 2018. The goal is
to segment stroke lesions based
on acute CT perfusion data.

BraTS 2018 Multimodal Brain Tumor Seg- http://www.med.upenn.edu/sbia/

mentation. The goal is to seg- brats2018.html
ment brain tumors in multi-
modal MRI scans.

CAMELYON17 The goal is to develop al- https://camelyon17.

gorithms for automated de- grand-challenge.org/Home
tection and classification of
breast cancer metastases in
whole-slide images of histolog-
ical lymph node sections.

ISIC 2018 Skin Lesion Analysis Towards https://challenge2018.

Melanoma Detection isic-archive.com/

Kaggle’s 2018 Data Science Spot Nuclei. Speed Cures. https://www.kaggle.com/c/

Bowl data-science-bowl-2018

Kaggle’s 2017 Data Science Turning Machine Intelligence https://www.kaggle.com/c/

Bowl Against Lung Cancer data-science-bowl-2017

Kaggle’s 2016 Data Science Transforming How We Diag- https://www.kaggle.com/c/

Bowl nose Heart Disease second-annual-data-science-bowl

MURA Determine whether a bone X- https://stanfordmlgroup.github.io/

ray is normal or abnormal competitions/mura/

24
5. Challenges, limitations and future perspectives

It is clear that deep neural networks are very useful when one is tasked with producing accurate
decisions based on complicated data sets. But they come with some significant challenges and
limitations that you either have to accept or try to overcome. Some are general: from technical
challenges related to the lack of mathematical and theoretical underpinnings of many central deep
learning models and techniques, and the resulting difficulty in deciding exactly what it is that makes
one model better than another, to societal challenges related to maximization and spread of the
technological benefits [328, 329] and the problems related to the tremendous amounts of hype and
excitement49 . Others are more domain-specific.
In deep learning for standard computer vision tasks, like object recognition and localization,
powerful models and a set of best practices have been developed over the last few years. The pace
of development is still incredibly high, but certain things seem to be settled, at least momentarily.
Using the basic building blocks described above, placed according to the ideas behind, say, ResNet
and SENet, will easily result in close to state-of-the-art performance on two-dimensional object
detection, image classification and segmentation tasks.
However, the story for deep learning in medical imaging is not quite as settled. One issue is that
medical images are often three-dimensional, and three-dimensional convolutional neural networks
are as well-developed as their 2D counterparts. One quickly meet challenges associated to memory
and compute consumption when using CNNs with higher-dimensional image data, challenges that
researchers are trying various approaches to deal with (treating 3D as stacks of 2Ds, patch- or
segment-based training and inference, downscaling, etc). It is clear that the ideas behind state-
of-the-art two-dimensional CNNs can be lifted to three dimensions, but also that adding a third
spatial dimension results in additional constraints. Other important challenges are related to data,
trust, interpretability, workflow integration, and regulations, as discussed below.

5.1. Data
This is a crucially important obstacle for deep neural networks, especially in medical data
analysis. When deploying deep neural networks, or any other machine learning model, one is
instantly faced with challenges related to data access, privacy issues, data protection, and more.
As privacy and data protection is often a requirement when dealing with medical data, new
techniques for training models without exposing the underlying training data to the user of the
model are necessary. It is not enough to merely restrict access to the training set used to construct
the model, as it is easy to use the model itself to discover details about the training set [330]. Even
hiding the model and only exposing a prediction interface would still leave it open to attack, for
example in the form of model-inversion [331] and membership attacks [332]. Most current work on
deep learning for medical data analysis use either open, anonymized data sets (as those in Table
4), or locally obtained anonymized research data, making these issues less relevant. However, the
general deep learning community are focusing a lot of attention on the issue of privacy, and new
techniques and frameworks for federated learning [333]50 , split learning [334, 335] and differential
privacy [336, 337, 338] are rapidly improving. See [339] for a recent survey. There are a few
examples of these ideas entering the medical machine learning community, as in [340] where the
distribution of deep learning models among several medical institutions was investigated, but then

49
Lipton: Machine Learning: The Opportunity and the Opportunists https://www.technologyreview.com/video/
612109, Jordan: Artificial Intelligence – The Revolution Hasn’t Happened Yet https://medium.com/@mijordan3/
artificial-intelligence-the-revolution-hasnt-happened-yet-5e1d5812e1e7
50
See for example https://ai.googleblog.com/2017/04/federated-learning-collaborative.html

25
without considering the above privacy issues. As machine learning systems in medicine grows to
larger scales, perhaps even including computations and learning on the “edge”, federated learning
and differential privacy will likely become the focus of much research in our community.
If you are able to surmount these obstacles, you will be confronted with deep neural networks’
insatiable appetite for training data. These are very inefficient models, requiring large number
of training samples before they can produce anything remotely useful, and labeled training data
is typically both expensive and difficult to produce. In addition, the training data has to be
representative of the data the network will meet in the future. If the training samples are from
a data distribution that is very different from the one met in the real world, then the network’s
generalization performance will be lower than expected. See [341] for a recent exploration of this
issue. Considering the large difference between the high-quality images one typically work with
when doing research and the messiness of the real, clinical world, this can be a major obstacle when
putting deep learning systems into production.
Luckily there are ways to alleviate these problems somewhat. A widely used technique is transfer
learning, also called fine-tuning or pre-training: first you train a network to perform a task where
there is an abundance of data, and then you copy weights from this network to a network designed
for the task at hand. For two-dimensional images one will almost always use a network that has
been pre-trained on the ImageNet data set. The basic features in the earlier layers of the neural
network found from this data set typically retain their usefulness in any other image-related task
(or are at least form a better starting point than random initialization of the weights, which is
the alternative). Starting from weights tuned on a larger training data set can also make the
network more robust. Focusing the weight updates during training on later layers requires less
data than having to do significant updates throughout the entire network. One can also do inter-
organ transfer learning in 3D, an idea we have used for kidney segmentation, where pre-training
a network to do brain segmentation decreased the number of annotated kidneys needed to achieve
good segmentation performance [342]. The idea of pre-training networks is not restricted to images.
Pre-training entire models has recently been demonstrated to greatly impact the performance of
natural language processing systems [3, 4, 5].
Another widely used technique is augmenting the training data set by applying various trans-
formations that preserves the labels, as in rotations, scalings and intensity shifts of images, or more
advanced data augmentation techniques like anatomically sound deformations, or other data set
specific operations (for example in our work on kidney segmentation from DCE-MRI, where we
used image registration to propagate labels through a time course of images [343]). Data synthesis,
as in [213], is another interesting approach.
In short, as expert annotators are expensive, or simply not available, spending large computa-
tional resources to expand your labeled training data set, e.g. indirectly through transfer learning
or directly through data augmentation, is typically worthwhile. But whatever you do, the way cur-
rent deep neural networks are constructed and trained results in significant data size requirements.
There are new ways of constructing more data-efficient deep neural networks on the horizon, for ex-
ample by encoding more domain-specific elements in the neural network structure as in the capsule
systems of [344, 345], which adds viewpoint invariance. It is also possible to add attention mecha-
nisms to neural networks [346, 347], enabling them to focus their resources on the most informative
components of each layer input.
However, the networks that are most frequently used, and with the best raw performance, remain
the data-hungry standard deep neural networks.

26
5.2. Interpretability, trust and safety
As deep neural networks relies on complicated interconnected hierarchical representations of the
training data to produce its predictions, interpreting these predictions becomes very difficult. This
is the “black box” problem of deep neural networks [348]. They are capable of producing extremely
accurate predictions, but how can you trust predictions based on features you cannot understand?
Considerable effort goes into developing new ways to deal with this problem, including DARPA
launching a whole program “Explainable AI ”51 dedicated to this issue, and lots of research going
into enhancing interpretability [349, 350], and finding new ways to measure sensitivity and visualize
features [68, 351, 352, 353? ].
Another way to increase their trustworthiness is to make them produce robust uncertainty
estimates in addition to predictions. The field of Bayesian Deep Learning aims to combine deep
learning and Bayesian approaches to uncertainty. The ideas date back to the early 90s [354, 355, 356],
but the field has recently seen renewed interest from the machine learning community at large, as
new ways of computing uncertainty estimates from state of the art deep learning models have been
developed [59, 65, 357]. In addition to producing valuable measures that function as uncertainty
measures [358, 359, 66], these techniques can also lessen deep neural networks susceptibility to
adversarial attacks [357, 360].

5.3. Workflow integration, regulations

Another stumbling block for successful incorporation of deep learning methods is workflow in-
tegration. It is possible to end up developing clever machine learning system for clinical use that
turn out to be practically useless for actual clinicians. Attempting to augment already established
procedures necessitates knowledge of the entire workflow. Involving the end-user in the process of
creating and evaluating systems can make this a little less of an issue, and can also increase the end
users’ trust in the systems52 , as you can establish a feedback loop during the development process.
But still, even if there is interest on the “ground floor” and one is able to get prototype systems
into the hands of clinicians, there are many higher-ups to convince and regulatory, ethical and legal
hurdles to overcome.

5.4. Perspectives and future expectations

Deep learning in medical data analysis is here to stay. Even though there are many challenges
associated to the introduction of deep learning in clinical settings, the methods produce results
that are too valuable to discard. This is illustrated by the tremendous amounts of high-impact
publications in top-journals dealing with deep learning in medical imaging (for example [17, 21, 30,
32, 40, 90, 162, 137, 140, 273, 285], all published in 2018). As machine learning researchers and
practitioners gain more experience, it will become easier to classify problems according to what
solution approach is the most reasonable: (i) best approached using deep learning techniques end-
to-end, (ii) best tackled by a combination of deep learning with other techniques, or (iii) no deep
learning component at all.
Beyond the application of machine learning in medical imaging, we believe that the attention
in the medical community can also be leveraged to strengthen the general computational mindset
among medical researchers and practitioners, mainstreaming the field of computational medicine 53 .
Once there are enough high-impact software-systems based on mathematics, computer science,

51
https://www.darpa.mil/program/explainable-artificial-intelligence
52
The approach we have taken at our MMIV center https://mmiv.no, located inside the Department of Radiology
53
In-line with the ideas of the convergence of disciplines and the “future of health”, as described in [361]

27
physics and engineering entering the daily workflow in the clinic, the acceptance for other such
systems will likely grow. The access to bio-sensors and (edge) computing on wearable devices for
monitoring disease or lifestyle, plus an ecosystem of machine learning and other computational
medicine-based technologies, will then likely facilitate the transition to a new medical paradigm
that is predictive, preventive, personalized, and participatory - P4 medicine [362]54 .

Acknowledgements

We thank Renate Grüner for useful discussions. The anonymous reviewers gave us excellent
constructive feedback that led to several improvements throughout the article. Our work was
financially supported by the Bergen Research Foundation through the project “Computational
medical imaging and machine learning – methods, infrastructure and applications”.

References

References [7] A. van den Oord, S. Dieleman, H. Zen, K. Si-

monyan, O. Vinyals, A. Graves, N. Kalchbren-
[1] A. Krizhevsky, I. Sutskever, G. E. Hinton, Ima- ner, A. Senior, K. Kavukcuoglu, WaveNet: A
geNet classification with deep convolutional neural generative model for raw audio, arXiv preprint
networks, in: F. Pereira, C. J. C. Burges, L. Bottou, arXiv:1609.03499v2 (2016).
K. Q. Weinberger (Eds.), Advances in Neural Infor-
mation Processing Systems 25, Curran Associates,
Inc., 2012, pp. 1097–1105. [8] C. Guo, F. Berkhahn, Entity embeddings of cate-
gorical variables, arXiv preprint arXiv:1604.06737
[2] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Per- (2016).
ona, D. Ramanan, P. Dollár, C. L. Zitnick, Mi-
crosoft COCO: Common objects in context, in: Eu- [9] A. De Brébisson, É. Simon, A. Auvolat, P. Vin-
ropean Conference on Computer Vision, Springer, cent, Y. Bengio, Artificial neural networks ap-
pp. 740–755. plied to taxi destination prediction, arXiv preprint
arXiv:1508.00021 (2015).
[3] M. Peters, M. Neumann, M. Iyyer, M. Gardner,
C. Clark, K. Lee, L. Zettlemoyer, Deep contextu-
alized word representations, in: Proceedings of the [10] D. George, E. Huerta, Deep learning for real-time
2018 Conference of the North American Chapter of gravitational wave detection and parameter estima-
the Association for Computational Linguistics: Hu- tion: Results with advanced LIGO data, Physics
man Language Technologies, Volume 1 (Long Pa- Letters B 778 (2018) 64–70.
pers), volume 1, pp. 2227–2237.

[4] J. Howard, S. Ruder, Universal language model fine- [11] D. George, H. Shen, E. Huerta, Classification and
tuning for text classification, in: Proceedings of the unsupervised clustering of LIGO data with deep
56th Annual Meeting of the Association for Compu- transfer learning, Physical Review D 97 (2018)
tational Linguistics (Volume 1: Long Papers), vol- 101501.
ume 1, pp. 328–339.
[12] H. Shen, D. George, E. Huerta, Z. Zhao, Denois-
[5] A. Radford, K. Narasimhan, T. Salimans, ing gravitational waves using deep learning with
I. Sutskever, Improving language understand- recurrent denoising autoencoders, arXiv preprint
ing by generative pre-training (2018). arXiv:1711.09919 (2017).
[6] W. Xiong, L. Wu, F. Alleva, J. Droppo, X. Huang,
A. Stolcke, The Microsoft 2017 Conversational [13] M. Raissi, G. E. Karniadakis, Hidden physics mod-
speech recognition system, in: Proc. Speech and els: Machine learning of nonlinear partial differen-
Signal Processing (ICASSP) 2018 IEEE Int. Conf. tial equations, Journal of Computational Physics
Acoustics, pp. 5934–5938. 357 (2018) 125–141.

54
http://p4mi.org

28
[14] A. Karpatne, G. Atluri, J. H. Faghmous, M. Stein- [24] A. A. Kalinin, G. A. Higgins, N. Reamaroon,
bach, A. Banerjee, A. Ganguly, S. Shekhar, N. Sam- S. Soroushmehr, A. Allyn-Feuer, I. D. Dinov, K. Na-
atova, V. Kumar, Theory-guided data science: A jarian, B. D. Athey, Deep learning in pharmacoge-
new paradigm for scientific discovery from data, nomics: from gene regulation to patient stratifica-
IEEE Transactions on Knowledge and Data Engi- tion, Pharmacogenomics 19 (2018) 629–650.
neering 29 (2017) 2318–2331.
[25] S. Jiang, K.-S. Chin, K. L. Tsui, A universal deep
[15] Gartner, Top Strategic Technology Trends for 2018, learning approach for modeling the flow of patients
2018. under different severities, Computer methods and
programs in biomedicine 154 (2018) 191–203.
[16] D. Ravi, C. Wong, F. Deligianni, M. Berthelot,
J. Andreu-Perez, B. Lo, G.-Z. Yang, Deep learning [26] K. C. Vranas, J. K. Jopling, T. E. Sweeney, M. C.
for health informatics., IEEE journal of biomedical Ramsey, A. S. Milstein, C. G. Slatore, G. J. Esco-
and health informatics 21 (2017) 4–21. bar, V. X. Liu, Identifying distinct subgroups of icu
patients: A machine learning approach., Critical
[17] N. Ganapathy, R. Swaminathan, T. M. Deserno, care medicine 45 (2017) 1607–1615.
Deep learning on 1-D biosignals: a taxonomy-based
survey, Yearbook of medical informatics 27 (2018) [27] A. Rajkomar, E. Oren, K. Chen, A. M. Dai, N. Ha-
98–109. jaj, M. Hardt, P. J. Liu, X. Liu, J. Marcus, M. Sun,
et al., Scalable and accurate deep learning with elec-
[18] L. Kuhlmann, K. Lehnertz, M. P. Richardson, tronic health records, npj Digital Medicine 1 (2018)
B. Schelter, H. P. Zaveri, Seizure prediction - ready 18.
for a new era, Nature reviews. Neurology (2018).
[28] B. Shickel, P. J. Tighe, A. Bihorac, P. Rashidi, Deep
EHR: A Survey of Recent Advances in Deep Learn-
[19] J.-M. Kwon, Y. Lee, Y. Lee, S. Lee, J. Park, An
ing Techniques for Electronic Health Record (EHR)
algorithm based on deep learning for predicting in-
Analysis, IEEE Journal of Biomedical and Health
hospital cardiac arrest, Journal of the American
Informatics (2017).
Heart Association 7 (2018).
[29] V. Gulshan, L. Peng, M. Coram, M. C. Stumpe,
[20] H.-C. Shin, H. R. Roth, M. Gao, L. Lu, Z. Xu, D. Wu, A. Narayanaswamy, S. Venugopalan,
I. Nogues, J. Yao, D. Mollura, R. M. Summers, K. Widner, T. Madams, J. Cuadros, et al., Develop-
Deep convolutional neural networks for computer- ment and validation of a deep learning algorithm for
aided detection: Cnn architectures, dataset charac- detection of diabetic retinopathy in retinal fundus
teristics and transfer learning., IEEE transactions photographs, Jama 316 (2016) 2402–2410.
on medical imaging 35 (2016) 1285–1298.
[30] R. Poplin, A. V. Varadarajan, K. Blumer, Y. Liu,
[21] D. S. Kermany, M. Goldbaum, W. Cai, C. C. S. M. McConnell, G. Corrado, L. Peng, D. Webster,
Valentim, H. Liang, S. L. Baxter, A. McKeown, Predicting Cardiovascular Risk Factors in Retinal
G. Yang, X. Wu, F. Yan, J. Dong, M. K. Prasadha, Fundus Photographs using Deep Learning, Nature
J. Pei, M. Y. L. Ting, J. Zhu, C. Li, S. Hewett, Biomedical Engineering (2018).
J. Dong, I. Ziyar, A. Shi, R. Zhang, L. Zheng,
R. Hou, W. Shi, X. Fu, Y. Duan, V. A. N. Huu, [31] R. Poplin, P.-C. Chang, D. Alexander, S. Schwartz,
C. Wen, E. D. Zhang, C. L. Zhang, O. Li, X. Wang, T. Colthurst, A. Ku, D. Newburger, J. Dijamco,
M. A. Singer, X. Sun, J. Xu, A. Tafreshi, M. A. N. Nguyen, P. T. Afshar, S. S. Gross, L. Dorfman,
Lewis, H. Xia, K. Zhang, Identifying medical di- C. Y. McLean, M. A. DePristo, A universal SNP
agnoses and treatable diseases by image-based deep and small-indel variant caller using deep neural net-
learning, Cell 172 (2018) 1122–1131.e9. works, Nature Biotechnology (2018).

[22] J. L. Katzman, U. Shaham, A. Cloninger, J. Bates, [32] J. De Fauw, J. R. Ledsam, B. Romera-Paredes,

T. Jiang, Y. Kluger, DeepSurv: personalized treat- S. Nikolov, N. Tomasev, S. Blackwell, H. Askham,
ment recommender system using a Cox proportional X. Glorot, B. O’Donoghue, D. Visentin, et al., Clin-
hazards deep neural network, BMC medical re- ically applicable deep learning for diagnosis and re-
search methodology 18 (2018) 24. ferral in retinal disease, Nature medicine 24 (2018)
1342.
[23] J. Jiménez, M. Škalič, G. Martı́nez-Rosell, G. De
Fabritiis, KDEEP: Protein-Ligand absolute bind- [33] Y. Qin, K. Kamnitsas, S. Ancha, J. Nanavati,
ing affinity prediction via 3D-Convolutional Neu- G. Cottrell, A. Criminisi, A. Nori, Autofocus
ral Networks, Journal of Chemical Information and Layer for Semantic Segmentation, arXiv preprint
Modeling 58 (2018) 287–296. arXiv:1805.08403 (2018).

29
[34] K. Kamnitsas, C. Baumgartner, C. Ledig, V. New- [46] G. Cybenko, Approximation by superpositions of a
combe, J. Simpson, A. Kane, D. Menon, A. Nori, sigmoidal function, Mathematics of control, signals
A. Criminisi, D. Rueckert, et al., Unsupervised do- and systems 2 (1989) 303–314.
main adaptation in brain lesion segmentation with
adversarial networks, in: International Confer- [47] K. Hornik, M. Stinchcombe, H. White, Multilayer
ence on Information Processing in Medical Imaging, feedforward networks are universal approximators,
Springer, pp. 597–609. Neural networks 2 (1989) 359–366.

[35] C. Xiao, E. Choi, J. Sun, Opportunities and chal- [48] M. Leshno, V. Y. Lin, A. Pinkus, S. Schocken, Mul-
lenges in developing deep learning models using tilayer feedforward networks with a nonpolynomial
electronic health records data: a systematic review, activation function can approximate any function,
Journal of the American Medical Informatics Asso- Neural networks 6 (1993) 861–867.
ciation (2018).
[49] S. Sonoda, N. Murata, Neural network with un-
[36] D. Silver, J. Schrittwieser, K. Simonyan, bounded activation functions is universal approxi-
I. Antonoglou, A. Huang, A. Guez, T. Hu- mator, Applied and Computational Harmonic Anal-
bert, L. Baker, M. Lai, A. Bolton, et al., Mastering ysis 43 (2017) 233–268.
the game of Go without human knowledge, Nature
550 (2017) 354. [50] M. A. Nielsen, Neural networks and deep learning,
Determination Press, 2015.
[37] A. Esteva, B. Kuprel, R. A. Novoa, J. Ko, S. M.
Swetter, H. M. Blau, S. Thrun, Dermatologist-level [51] C. C. Aggarwal, Neural networks and deep learning,
classification of skin cancer with deep neural net- Springer, 2018.
works, Nature 542 (2017) 115–118.
[52] F. Rosenblatt, The perceptron: a probabilistic
[38] R. Poplin, A. V. Varadarajan, K. Blumer, Y. Liu, model for information storage and organization in
M. V. McConnell, G. S. Corrado, L. Peng, D. R. the brain., Psychological review 65 (1958) 386.
Webster, Prediction of cardiovascular risk factors
from retinal fundus photographs via deep learning, [53] S. Linnainmaa, The representation of the cumu-
Nature Biomedical Engineering 2 (2018) 158. lative rounding error of an algorithm as a taylor
expansion of the local rounding errors, Master’s
[39] Y. LeCun, Y. Bengio, G. Hinton, Deep learning, Thesis (in Finnish), Univ. Helsinki (1970) 6–7.
nature 521 (2015) 436.
[54] P. Werbos, Beyond regression: New tools for pre-
[40] G. Hinton, Deep Learning A Technology With the diction and analysis in the behavioral sciences, Ph.
Potential to Transform Health Care (2018) 1–2. D. dissertation, Harvard University (1974).

[41] I. Goodfellow, Y. Bengio, A. Courville, Deep Learn- [55] D. E. Rumelhart, G. E. Hinton, R. J. Williams,
ing, MIT Press, 2016. www.deeplearningbook.org. Learning representations by back-propagating er-
rors, nature 323 (1986) 533.
[42] G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Se-
tio, F. Ciompi, M. Ghafoorian, J. A. W. M. van der [56] A. Cauchy, Méthode générale pour la résolution des
Laak, B. van Ginneken, C. I. Snchez, A survey on systemes déquations simultanées, Comp. Rend. Sci.
deep learning in medical image analysis., Medical Paris 25 (1847) 536–538.
image analysis 42 (2017) 60–88.
[57] Y. LeCun, L. Bottou, Y. Bengio, P. Haffner,
[43] A. H. Marblestone, G. Wayne, K. P. Kording, To- Gradient-based learning applied to document recog-
ward an Integration of Deep Learning and Neuro- nition, Proceedings of the IEEE 86 (1998) 2278–
science, Frontiers in Computational Neuroscience 2324.
10 (2016) 94.
[58] S. C. Lo, M. T. Freedman, J. S. Lin, S. K.
[44] D. Hassabis, D. Kumaran, C. Summerfield, Mun, Automatic lung nodule detection using pro-
M. Botvinick, Neuroscience-Inspired Artificial In- file matching and back-propagation neural network
telligence, Neuron 95 (2017) 245–258. techniques., Journal of digital imaging 6 (1993) 48–
54.
[45] A. Banino, C. Barry, B. Uria, C. Blundell, T. Lil-
licrap, P. Mirowski, A. Pritzel, M. J. Chadwick, [59] S. Murray, An exploratory analysis of multi-class
T. Degris, J. Modayil, et al., Vector-based nav- uncertainty approximation in Bayesian convolu-
igation using grid-like representations in artificial tion neural networks, Master’s thesis, University of
agents, Nature 557 (2018) 429. Bergen, 2018.

30
[60] D.-A. Clevert, T. Unterthiner, S. Hochreiter, Fast [73] R. K. Srivastava, K. Greff, J. Schmidhuber, Train-
and accurate deep network learning by exponential ing very deep networks, in: Advances in neural
linear units (elus), arXiv preprint arXiv:1511.07289 information processing systems, pp. 2377–2385.
(2015).
[74] G. Huang, Z. Liu, L. Van Der Maaten, K. Q. Wein-
[61] K. He, X. Zhang, S. Ren, J. Sun, Delving deep berger, Densely connected convolutional networks,
into rectifiers: Surpassing human-level performance in: CVPR, volume 1, p. 3.
on imagenet classification, in: Proceedings of the
IEEE international conference on computer vision, [75] S. Xie, R. Girshick, P. Dollár, Z. Tu, K. He, Ag-
pp. 1026–1034. gregated residual transformations for deep neural
networks, in: Computer Vision and Pattern Recog-
[62] J. T. Springenberg, A. Dosovitskiy, T. Brox, nition (CVPR), 2017 IEEE Conference on, IEEE,
M. Riedmiller, Striving for simplicity: The all pp. 5987–5995.
convolutional net, arXiv preprint arXiv:1412.6806
(2014). [76] J. Hu, L. Shen, G. Sun, Squeeze-and-excitation net-
works, arXiv preprint arXiv:1709.01507 7 (2017).
[63] N. Srivastava, G. Hinton, A. Krizhevsky,
I. Sutskever, R. Salakhutdinov, Dropout: a [77] B. Zoph, V. Vasudevan, J. Shlens, Q. V. Le, Learn-
simple way to prevent neural networks from over- ing transferable architectures for scalable image
fitting, The Journal of Machine Learning Research recognition, arXiv preprint arXiv:1707.07012 2
15 (2014) 1929–1958. (2017).

[64] K. Rashmi, R. Gilad-Bachrach, Dart: Dropouts [78] I. Bello, B. Zoph, V. Vasudevan, Q. V. Le, Neural
meet multiple additive regression trees, in: Inter- optimizer search with reinforcement learning, in:
national Conference on Artificial Intelligence and D. Precup, Y. W. Teh (Eds.), Proceedings of the
Statistics, pp. 489–497. 34th International Conference on Machine Learn-
ing, volume 70 of Proceedings of Machine Learning
[65] Y. Gal, Uncertainty in deep learning, Ph.D. thesis, Research, PMLR, International Convention Centre,
University of Cambridge, 2016. Sydney, Australia, 2017, pp. 459–468.
[66] K. Wickstrøm, M. Kampffmeyer, R. Jenssen, Un- [79] J. Redmon, S. Divvala, R. Girshick, A. Farhadi, You
certainty Modeling and Interpretability in Convolu- only look once: Unified, real-time object detection,
tional Neural Networks for Polyp Segmentation, in: in: Proceedings of the IEEE conference on com-
2018 IEEE 28th International Workshop on Machine puter vision and pattern recognition, pp. 779–788.
Learning for Signal Processing (MLSP), IEEE, pp.
1–6. [80] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu,
D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio,
[67] S. Ioffe, C. Szegedy, Batch Normalization: Acceler-
Generative Adversarial Nets, in: Z. Ghahramani,
ating Deep Network Training by Reducing Internal
M. Welling, C. Cortes, N. D. Lawrence, K. Q. Wein-
Covariate Shift, in: International Conference on
berger (Eds.), Advances in Neural Information Pro-
Machine Learning, pp. 448–456.
cessing Systems 27, Curran Associates, Inc., 2014,
[68] M. D. Zeiler, R. Fergus, Visualizing and under- pp. 2672–2680.
standing convolutional networks, in: European con-
[81] G. Koch, R. Zemel, R. Salakhutdinov, Siamese neu-
ference on computer vision, Springer, pp. 818–833.
ral networks for one-shot image recognition, in:
[69] K. Simonyan, A. Zisserman, Very deep convolu- ICML Deep Learning Workshop, volume 2.
tional networks for large-scale image recognition,
[82] J. Bromley, I. Guyon, Y. LeCun, E. Säckinger,
arXiv preprint arXiv:1409.1556 (2014).
R. Shah, Signature verification using a “siamese”
[70] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, time delay neural network, in: Advances in neural
D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabi- information processing systems, pp. 737–744.
novich, Going deeper with convolutions, in: Pro-
ceedings of the IEEE conference on computer vision [83] O. Ronneberger, P. Fischer, T. Brox, U-net: Con-
and pattern recognition, pp. 1–9. volutional networks for biomedical image segmenta-
tion, in: International Conference on Medical im-
[71] M. Lin, Q. Chen, S. Yan, Network in network, arXiv age computing and computer-assisted intervention,
preprint arXiv:1312.4400 (2013). Springer, pp. 234–241.

[72] K. He, X. Zhang, S. Ren, J. Sun, Deep residual [84] J. Long, E. Shelhamer, T. Darrell, Fully convolu-
learning for image recognition, in: Proceedings of tional networks for semantic segmentation, in: Pro-
the IEEE conference on computer vision and pat- ceedings of the IEEE conference on computer vision
tern recognition, pp. 770–778. and pattern recognition, pp. 3431–3440.

31
[85] F. Milletari, N. Navab, S.-A. Ahmadi, V-net: Fully [97] X. Cao, J. Yang, J. Zhang, Q. Wang, P.-T. Yap,
convolutional neural networks for volumetric medi- D. Shen, Deformable image registration using a cue-
cal image segmentation, in: 3D Vision (3DV), 2016 aware deep regression network, IEEE transactions
Fourth International Conference on, IEEE, pp. 565– on bio-medical engineering 65 (2018) 1900–1911.
571.
[98] X. Yang, R. Kwitt, M. Styner, M. Niethammer,
[86] J. Redmon, Darknet: Open source neural networks Quicksilver: Fast predictive image registration - a
in C, http://pjreddie.com/darknet/, 2013–2016. deep learning approach., NeuroImage 158 (2017)
378–396.
[87] Y. H. Lee, Efficiency improvement in a busy ra-
diology practice: Determination of musculoskeletal [99] V. P. Kearney, S. Haaf, A. Sudhyadhom, G. Valdes,
magnetic resonance imaging protocol using deep- T. D. Solberg, An unsupervised convolutional
learning convolutional neural networks., Journal of neural network-based algorithm for deformable im-
digital imaging (2018). age registration, Physics in medicine and biology
[88] E. Gong, J. M. Pauly, M. Wintermark, G. Za- (2018).
harchuk, Deep learning enables reduced gadolinium
[100] J. Zheng, S. Miao, Z. Jane Wang, R. Liao, Pairwise
dose for contrast-enhanced brain MRI, Journal of
domain adaptation module for CNN-based 2-D/3-D
magnetic resonance imaging 48 (2018) 330–340.
registration., Journal of medical imaging (Belling-
[89] P. Meyer, V. Noblet, C. Mazzara, A. Lallement, ham, Wash.) 5 (2018) 021204.
Survey on deep learning for radiotherapy., Com-
puters in biology and medicine 98 (2018) 126–146. [101] E. M. A. Anas, P. Mousavi, P. Abolmaesumi, A
deep learning approach for real time prostate seg-
[90] F. Liu, H. Jang, R. Kijowski, T. Bradshaw, A. B. mentation in freehand ultrasound guided biopsy,
McMillan, Deep learning MR imaging-based atten- Medical image analysis 48 (2018) 107–116.
uation correction for PET/MR imaging, Radiology
286 (2018) 676–684. [102] T. Ching, D. S. Himmelstein, B. K. Beaulieu-Jones,
A. A. Kalinin, B. T. Do, G. P. Way, E. Ferrero,
[91] A. Mehranian, H. Arabi, H. Zaidi, Vision 20/20: P.-M. Agapow, M. Zietz, M. M. Hoffman, W. Xie,
Magnetic resonance imaging-guided attenuation G. L. Rosen, B. J. Lengerich, J. Israeli, J. Lan-
correction in pet/mri: Challenges, solutions, and chantin, S. Woloszynek, A. E. Carpenter, A. Shriku-
opportunities., Medical physics 43 (2016) 1130– mar, J. Xu, E. M. Cofer, C. A. Lavender, S. C.
1155. Turaga, A. M. Alexandari, Z. Lu, D. J. Harris,
D. DeCaprio, Y. Qi, A. Kundaje, Y. Peng, L. K.
[92] J. Lao, Y. Chen, Z.-C. Li, Q. Li, J. Zhang, J. Liu, Wiley, M. H. S. Segler, S. M. Boca, S. J. Swami-
G. Zhai, A deep learning-based radiomics model for dass, A. Huang, A. Gitter, C. S. Greene, Opportu-
prediction of survival in glioblastoma multiforme., nities and obstacles for deep learning in biology and
Scientific reports 7 (2017) 10353. medicine, Journal of the Royal Society, Interface 15
[93] L. Oakden-Rayner, G. Carneiro, T. Bessen, J. C. (2018).
Nascimento, A. P. Bradley, L. J. Palmer, Precision
[103] J.-G. Lee, S. Jun, Y.-W. Cho, H. Lee, G. B. Kim,
radiology: Predicting longevity using feature engi-
J. B. Seo, N. Kim, Deep learning in medical imag-
neering and deep learning methods in a radiomics
ing: General overview., Korean journal of radiology
framework., Scientific reports 7 (2017) 1648.
18 (2017) 570–584.
[94] J. C. Peeken, M. Bernhofer, B. Wiestler, T. Gold-
berg, D. Cremers, B. Rost, J. J. Wilkens, S. E. [104] D. Rueckert, B. Glocker, B. Kainz, Learning clini-
Combs, F. Nsslin, Radiomics in radiooncology - cally useful information from images: Past, present
challenging the medical physicist., Physica medica and future., Medical image analysis 33 (2016) 13–
48 (2018) 27–36. 18.

[95] M. Izadyyazdanabadi, E. Belykh, M. A. Mooney, [105] G. Chartrand, P. M. Cheng, E. Vorontsov,

J. M. Eschbacher, P. Nakaji, Y. Yang, M. C. Preul, M. Drozdzal, S. Turcotte, C. J. Pal, S. Kadoury,
Prospects for theranostics in neurosurgical imag- A. Tang, Deep learning: A primer for radiologists,
ing: Empowering confocal laser endomicroscopy di- Radiographics : a review publication of the Radi-
agnostics via deep learning., Frontiers in oncology ological Society of North America, Inc 37 (2017)
8 (2018) 240. 2113–2131.

[96] G. Haskins, J. Kruecker, U. Kruger, S. Xu, P. A. [106] B. J. Erickson, P. Korfiatis, Z. Akkus, T. L. Kline,
Pinto, B. J. Wood, P. Yan, Learning deep simi- Machine learning for medical imaging, Radiograph-
larity metric for 3D MR-TRUS registration, arXiv ics : a review publication of the Radiological Society
preprint arXiv:1806.04548v1 (2018). of North America, Inc 37 (2017) 505–515.

32
[107] M. A. Mazurowski, M. Buda, A. Saha, M. R. Bashir, [120] J. R. Burt, N. Torosdagli, N. Khosravan,
Deep learning in radiology: an overview of the con- H. RaviPrakash, A. Mortazi, F. Tissavirasingham,
cepts and a survey of the state of the art, arXiv S. Hussein, U. Bagci, Deep learning beyond cats and
preprint arXiv:1802.08717v1 (2018). dogs: recent advances in diagnosing breast cancer
with deep neural networks, The British journal of
[108] M. P. McBee, O. A. Awan, A. T. Colucci, C. W. radiology 91 (2018) 20170545.
Ghobadi, N. Kadom, A. P. Kansagra, S. Tridanda-
pani, W. F. Auffermann, Deep learning in radiol- [121] R. K. Samala, H.-P. Chan, L. M. Hadjiiski, M. A.
ogy., Academic radiology (2018). Helvie, K. H. Cha, C. D. Richter, Multi-task trans-
fer learning deep convolutional neural network: ap-
[109] P. Savadjiev, J. Chong, A. Dohan, plication to computer-aided diagnosis of breast can-
M. Vakalopoulou, C. Reinhold, N. Paragios, cer on mammograms., Physics in medicine and bi-
B. Gallix, Demystification of AI-driven medical ology 62 (2017) 8894–8908.
image interpretation: past, present and future.,
European radiology (2018). [122] B. van Ginneken, Fifty years of computer analy-
sis in chest imaging: rule-based, machine learning,
[110] J. H. Thrall, X. Li, Q. Li, C. Cruz, S. Do, K. Dreyer, deep learning, Radiological physics and technology
J. Brink, Artificial intelligence and machine learn- 10 (2017) 23–32.
ing in radiology: Opportunities, challenges, pitfalls,
and criteria for success., Journal of the American [123] O. Morin, M. Vallires, A. Jochems, H. C. Woodruff,
College of Radiology : JACR 15 (2018) 504–508. G. Valdes, S. E. Braunstein, J. E. Wildberger, J. E.
Villanueva-Meyer, V. Kearney, S. S. Yom, T. D.
[111] R. Yamashita, M. Nishio, R. K. G. Do, K. Togashi, Solberg, P. Lambin, A deep look into the future
Convolutional neural networks: an overview and ap- of quantitative imaging in oncology: A statement
plication in radiology., Insights into imaging (2018). of working principles and proposal for change., In-
[112] K. Yasaka, H. Akai, A. Kunimatsu, S. Kiryu, ternational journal of radiation oncology, biology,
O. Abe, Deep learning with convolutional neural physics (2018).
network in radiology., Japanese journal of radiol-
[124] C. Parmar, J. D. Barry, A. Hosny, J. Quackenbush,
ogy 36 (2018) 257–272.
H. J. W. L. Aerts, Data analysis strategies in med-
[113] M. L. Giger, Machine learning in medical imag- ical imaging., Clinical cancer research : an official
ing, Journal of the American College of Radiology journal of the American Association for Cancer Re-
: JACR 15 (2018) 512–520. search 24 (2018) 3492–3499.

[114] B. J. Erickson, P. Korfiatis, Z. Akkus, T. Kline, [125] Y. Xue, S. Chen, J. Qin, Y. Liu, B. Huang, H. Chen,
K. Philbrick, Toolkits and libraries for deep learn- Application of deep learning in automated analysis
ing, Journal of digital imaging 30 (2017) 400–405. of molecular images in cancer: A survey., Contrast
media & molecular imaging 2017 (2017) 9512370.
[115] G. Zaharchuk, E. Gong, M. Wintermark, D. Rubin,
C. P. Langlotz, Deep learning in neuroradiology., [126] L. J. Brattain, B. A. Telfer, M. Dhyani, J. R. Grajo,
AJNR. American journal of neuroradiology (2018). A. E. Samir, Machine learning for medical ultra-
sound: status, methods, and future opportunities,
[116] Z. Akkus, A. Galimzianova, A. Hoogi, D. L. Rubin, Abdominal radiology 43 (2018) 786–799.
B. J. Erickson, Deep learning for brain MRI seg-
mentation: State of the art and future directions., [127] Q. Huang, F. Zhang, X. Li, Machine learning in
Journal of digital imaging 30 (2017) 449–459. ultrasound computer-aided diagnostic systems: A
survey, BioMed research international 2018 (2018)
[117] E.-J. Lee, Y.-H. Kim, N. Kim, D.-W. Kang, Deep 5137904.
into the brain: Artificial intelligence in stroke imag-
ing., Journal of stroke 19 (2017) 277–285. [128] D. Shen, G. Wu, H.-I. Suk, Deep learning in med-
ical image analysis., Annual review of biomedical
[118] R. Feng, M. Badgeley, J. Mocco, E. K. Oermann, engineering 19 (2017) 221–248.
Deep learning guided stroke management: a review
of clinical applications, Journal of neurointerven- [129] K. Suzuki, Overview of deep learning in medical
tional surgery 10 (2018) 358–362. imaging., Radiological physics and technology 10
(2017) 257–273.
[119] S. Vieira, W. H. L. Pinaya, A. Mechelli, Using deep
learning to investigate the neuroimaging correlates [130] C. Cao, F. Liu, H. Tan, D. Song, W. Shu, W. Li,
of psychiatric and neurological disorders: Methods Y. Zhou, X. Bo, Z. Xie, Deep learning and its ap-
and applications., Neuroscience and biobehavioral plications in biomedicine, Genomics, proteomics &
reviews 74 (2017) 58–75. bioinformatics 16 (2018) 17–32.

33
[131] P. Lakhani, D. L. Gray, C. R. Pett, P. Nagy, [142] Y. Han, J. Yoo, H. H. Kim, H. J. Shin, K. Sung,
G. Shih, Hello world deep learning in medical imag- J. C. Ye, Deep learning with domain adaptation
ing., Journal of digital imaging (2018). for accelerated projection-reconstruction MR, Mag-
netic resonance in medicine 80 (2018) 1189–1205.
[132] N. Pawlowski, S. I. Ktena, M. C. Lee, B. Kainz,
D. Rueckert, B. Glocker, M. Rajchl, DLTK: State of [143] J. Shi, Q. Liu, C. Wang, Q. Zhang, S. Ying, H. Xu,
the art reference implementations for deep learning Super-resolution reconstruction of mr image with a
on medical images, arXiv preprint arXiv:1711.06853 novel residual learning network algorithm., Physics
(2017). in medicine and biology 63 (2018) 085011.
[133] Y. Yang, J. Sun, H. Li, Z. Xu, Deep ADMM- [144] G. Yang, S. Yu, H. Dong, G. Slabaugh, P. L.
Net for compressive sensing MRI, in: D. D. Lee, Dragotti, X. Ye, F. Liu, S. Arridge, J. Keegan,
M. Sugiyama, U. V. Luxburg, I. Guyon, R. Garnett Y. Guo, D. Firmin, J. Keegan, G. Slabaugh, S. Ar-
(Eds.), Advances in Neural Information Processing ridge, X. Ye, Y. Guo, S. Yu, F. Liu, D. Firmin,
Systems 29, Curran Associates, Inc., 2016, pp. 10– P. L. Dragotti, G. Yang, H. Dong, DAGAN: deep
18. de-aliasing generative adversarial networks for fast
[134] S. Wang, Z. Su, L. Ying, X. Peng, S. Zhu, F. Liang, compressed sensing MRI reconstruction., IEEE
D. Feng, D. Liang, Accelerating magnetic resonance transactions on medical imaging 37 (2018) 1310–
imaging via deep learning, in: Biomedical Imaging 1321.
(ISBI), 2016 IEEE 13th International Symposium
[145] A. Deistung, A. Schäfer, F. Schweser, U. Bieder-
on, IEEE, pp. 514–517.
mann, R. Turner, J. R. Reichenbach, Toward in vivo
[135] C. Qin, J. V. Hajnal, D. Rueckert, J. Schlemper, histology: a comparison of quantitative susceptibil-
J. Caballero, A. N. Price, Convolutional recurrent ity mapping (QSM) with magnitude-, phase-, and
neural networks for dynamic mr image reconstruc- R2*-imaging at ultra-high magnetic field strength,
tion., IEEE transactions on medical imaging (2018). Neuroimage 65 (2013) 299–314.

[136] J. Schlemper, J. Caballero, J. V. Hajnal, A. N. [146] A. Deistung, F. Schweser, J. R. Reichenbach,

Price, D. Rueckert, A deep cascade of convolutional Overview of quantitative susceptibility mapping,
neural networks for dynamic MR image reconstruc- NMR in Biomedicine 30 (2017).
tion., IEEE transactions on medical imaging 37
(2018) 491–503. [147] J. Yoon, E. Gong, I. Chatnuntawech, B. Bilgic,
J. Lee, W. Jung, J. Ko, H. Jung, K. Setsompop,
[137] F. Chen, V. Taviani, I. Malkiel, J. Y. Cheng, J. I. G. Zaharchuk, E. Y. Kim, J. Pauly, J. Lee, Quanti-
Tamir, J. Shaikh, S. T. Chang, C. J. Hardy, J. M. tative susceptibility mapping using deep neural net-
Pauly, S. S. Vasanawala, Variable-density single- work: QSMnet., NeuroImage 179 (2018) 199–206.
shot fast Spin-Echo MRI with deep learning recon-
struction by using variational networks, Radiology [148] T. Liu, P. Spincemaille, L. De Rochefort,
(2018) 180445. B. Kressler, Y. Wang, Calculation of susceptibil-
ity through multiple orientation sampling (COS-
[138] F. Knoll, K. Hammernik, E. Kobler, T. Pock, M. P.
MOS): a method for conditioning the inverse prob-
Recht, D. K. Sodickson, Assessment of the gener-
lem from measured magnetic field map to suscepti-
alization of learned image reconstruction and the
bility source image in MRI, Magnetic Resonance in
potential for transfer learning, Magnetic resonance
Medicine 61 (2009) 196–204.
in medicine (2018).

[139] M. Mardani, E. Gong, J. Y. Cheng, S. S. [149] K. G. B. Rasmussen, M. J. Kristensen, R. G.

Vasanawala, G. Zaharchuk, L. Xing, J. M. Pauly, Blendal, L. R. Ostergaard, M. Plocharski,
Deep generative adversarial neural networks for K. O’Brien, C. Langkammer, A. Janke, M. Barth,
compressive sensing (GANCS) MRI., IEEE trans- S. Bollmann, DeepQSM-Using Deep Learning to
actions on medical imaging (2018). Solve the Dipole Inversion for MRI Susceptibility
Mapping, Biorxiv (2018) 278036.
[140] B. Zhu, J. Z. Liu, S. F. Cauley, B. R. Rosen, M. S.
Rosen, Image reconstruction by domain-transform [150] D. Ma, V. Gulani, N. Seiberlich, K. Liu, J. L. Sun-
manifold learning., Nature 555 (2018) 487–492. shine, J. L. Duerk, M. A. Griswold, Magnetic reso-
nance fingerprinting, Nature 495 (2013) 187–192.
[141] T. Eo, Y. Jun, T. Kim, J. Jang, H.-J. Lee,
D. Hwang, KIKI-net: cross-domain convolutional [151] E. S. of Radiology (ESR), Magnetic resonance fin-
neural networks for reconstructing undersampled gerprinting - a promising new approach to obtain
magnetic resonance images, Magnetic resonance in standardized imaging biomarkers from mri., In-
medicine 80 (2018) 2188–2201. sights into imaging 6 (2015) 163–165.

34
[152] D. L. Donoho, Compressed sensing, IEEE Transac- [164] A. Panda, B. B. Mehta, S. Coppo, Y. Jiang, D. Ma,
tions on Information Theory 52 (2006) 1289–1306. N. Seiberlich, M. A. Griswold, V. Gulani, Mag-
netic resonance fingerprinting-an overview., Cur-
[153] M. Lustig, D. Donoho, J. M. Pauly, Sparse mri: rent opinion in biomedical engineering 3 (2017) 56–
The application of compressed sensing for rapid mr 66.
imaging., Magnetic resonance in medicine 58 (2007)
1182–1195. [165] O. Cohen, B. Zhu, M. S. Rosen, MR fingerprinting
deep reconstruction network (DRONE), Magnetic
[154] M. T. McCann, K. H. Jin, M. Unser, Convolutional resonance in medicine 80 (2018) 885–894.
neural networks for inverse problems in imaging: A
review, IEEE Signal Processing Magazine 34 (2017) [166] E. Hoppe, G. Krzdrfer, T. Wrfl, J. Wetzl, F. Lu-
85–95. gauer, J. Pfeuffer, A. Maier, Deep learning for mag-
netic resonance fingerprinting: A new approach for
[155] V. Shah, C. Hegde, Solving Linear Inverse Prob-
predicting quantitative parameter values from time
lems Using GAN Priors: An Algorithm with Prov-
series, Studies in health technology and informatics
able Guarantees, arXiv preprint arXiv:1802.08406
243 (2017) 202–206.
(2018).
[167] J. Z. Bojorquez, S. Bricq, C. Acquitter, F. Brunotte,
[156] A. Lucas, M. Iliadis, R. Molina, A. K. Katsaggelos,
P. M. Walker, A. Lalande, What are normal relax-
Using deep neural networks for inverse problems in
ation times of tissues at 3 T?, Magnetic resonance
imaging: beyond analytical methods, IEEE Signal
imaging 35 (2017) 69–80.
Processing Magazine 35 (2018) 20–36.

[157] H. K. Aggarwal, M. P. Mani, M. Jacob, MoDL: [168] Z. Fang, Y. Chen, W. Lin, D. Shen, Quantification
Model Based Deep Learning Architecture for In- of relaxation times in MR fingerprinting using deep
verse Problems, IEEE transactions on medical learning., Proceedings of the International Society
imaging (2018). for Magnetic Resonance in Medicine ... Scientific
Meeting and Exhibition. International Society for
[158] H. Li, J. Schwab, S. Antholzer, M. Haltmeier, Magnetic Resonance in Medicine. Scientific Meet-
NETT: Solving Inverse Problems with Deep Neural ing and Exhibition 25 (2017).
Networks, arXiv preprint arXiv:1803.00092 (2018).
[169] P. Virtue, S. X. Yu, M. Lustig, Better than real:
[159] D. Ma, Y. Jiang, Y. Chen, D. McGivney, B. Mehta, Complex-valued neural nets for MRI fingerprinting,
V. Gulani, M. Griswold, Fast 3D magnetic res- in: Proc. IEEE Int. Conf. Image Processing (ICIP),
onance fingerprinting for a whole-brain coverage, pp. 3953–3957.
Magnetic resonance in medicine 79 (2018) 2190–
2197. [170] M. Tygert, J. Bruna, S. Chintala, Y. LeCun, S. Pi-
antino, A. Szlam, A mathematical motivation
[160] T. Christen, N. A. Pannetier, W. W. Ni, D. Qiu, for complex-valued convolutional networks, Neural
M. E. Moseley, N. Schuff, G. Zaharchuk, MR vas- computation 28 (2016) 815–825.
cular fingerprinting: A new approach to compute
cerebral blood volume, mean vessel radius, and oxy- [171] C. Trabelsi, O. Bilaniuk, Y. Zhang, D. Serdyuk,
genation maps in the human brain, Neuroimage 89 S. Subramanian, J. F. Santos, S. Mehri, N. Ros-
(2014) 262–270. tamzadeh, Y. Bengio, C. J. Pal, Deep complex net-
works, arXiv preprint arXiv:1705.09792 (2017).
[161] B. Lemasson, N. Pannetier, N. Coquery, L. S. B.
Boisserand, N. Collomb, N. Schuff, M. Moseley, [172] J. Sijbers, A. J. den Dekker, J. Van Audekerke,
G. Zaharchuk, E. L. Barbier, T. Christen, Mr vascu- M. Verhoye, D. Van Dyck, Estimation of the noise in
lar fingerprinting in stroke and brain tumors mod- magnitude MR images., Magnetic resonance imag-
els., Scientific reports 6 (2016) 37071. ing 16 (1998) 87–90.

[162] B. Rieger, M. Akakaya, J. C. Pariente, S. Llufriu, [173] E. R. McVeigh, R. M. Henkelman, M. J. Bronskill,

E. Martinez-Heras, S. Weingrtner, L. R. Schad, Noise and filtration in magnetic resonance imaging.,
Time efficient whole-brain coverage with MR Medical physics 12 (1985) 586–591.
fingerprinting using slice-interleaved echo-planar-
imaging, Scientific reports 8 (2018) 6667. [174] F. Baselice, G. Ferraioli, V. Pascazio, A. Sorriso,
Bayesian mri denoising in complex domain, Mag-
[163] K. L. Wright, Y. Jiang, D. Ma, D. C. Noll, M. A. netic resonance imaging 38 (2017) 112–122.
Griswold, V. Gulani, L. Hernandez-Garcia, Estima-
tion of perfusion properties with mr fingerprinting [175] A. Phophalia, S. K. Mitra, 3d mr image denoising
arterial spin labeling., Magnetic resonance imaging using rough set and kernel pca method., Magnetic
50 (2018) 68–77. resonance imaging 36 (2017) 135–145.

35
[176] X. Zhang, Z. Xu, N. Jia, W. Yang, Q. Feng, [187] V. Golkov, A. Dosovitskiy, J. I. Sperl, M. I. Men-
W. Chen, Y. Feng, Denoising of 3D magnetic res- zel, M. Czisch, P. Samann, T. Brox, D. Cremers, q-
onance images by using higher-order singular value space deep learning: Twelve-fold shorter and model-
decomposition., Medical image analysis 19 (2015) free diffusion MRI scans, IEEE transactions on
75–86. medical imaging 35 (2016) 1344–1351.

[177] D. Van De Ville, M. L. Seghier, F. Lazeyras, T. Blu, [188] S. S. Gurbani, E. Schreibmann, A. A. Maudsley,
M. Unser, Wspm: wavelet-based statistical para- J. S. Cordova, B. J. Soher, H. Poptani, G. Verma,
metric mapping., NeuroImage 37 (2007) 1205–1217. P. B. Barker, H. Shim, L. A. D. Cooper, A con-
volutional neural network to filter artifacts in spec-
[178] G. Salimi-Khorshidi, G. Douaud, C. F. Beckmann, troscopic MRI, Magnetic resonance in medicine 80
M. F. Glasser, L. Griffanti, S. M. Smith, Automatic (2018) 1765–1775.
denoising of functional MRI data: combining inde-
pendent component analysis and hierarchical fusion [189] S. P. Kyathanahally, A. Dring, R. Kreis, Deep learn-
of classifiers, Neuroimage 90 (2014) 449–468. ing approaches for detection and removal of ghost-
ing artifacts in MR spectroscopy, Magnetic reso-
[179] M. Lysaker, A. Lundervold, X.-C. Tai, Noise re- nance in medicine 80 (2018) 851–863.
moval using fourth-order partial differential equa-
tion with applications to medical magnetic reso- [190] T. Küstner, A. Liebgott, L. Mauch, P. Martirosian,
nance images in space and time, IEEE transactions F. Bamberg, K. Nikolaou, B. Yang, F. Schick, S. Ga-
on image processing 12 (2003) 1579–1590. tidis, Automated reference-free detection of motion
artifacts in magnetic resonance images, MAGMA
[180] C. Bermudez, A. J. Plassard, T. L. Davis, A. T. 31 (2018) 243–256.
Newton, S. M. Resnick, B. A. Landman, Learning
[191] L. Yue, H. Shen, J. Li, Q. Yuan, H. Zhang,
implicit brain MRI manifolds with deep learning,
L. Zhang, Image super-resolution: The techniques,
Proceedings of SPIE–the International Society for
applications, and future, Signal Processing 128
Optical Engineering 10574 (2018).
(2016) 389–408.
[181] A. Benou, R. Veksler, A. Friedman, T. Riklin Raviv, [192] R. Z. Shilling, T. Q. Robbie, T. Bailloeul, K. Mewes,
Ensemble of expert deep neural networks for spatio- R. M. Mersereau, M. E. Brummer, A super-
temporal denoising of contrast-enhanced MRI se- resolution framework for 3-d high-resolution and
quences, Medical image analysis 42 (2017) 145–159. high-contrast imaging using 2-d multislice mri.,
IEEE transactions on medical imaging 28 (2009)
[182] Y. Gal, A. J. H. Mehnert, A. P. Bradley, K. McMa-
633–644.
hon, D. Kennedy, S. Crozier, Denoising of dynamic
contrast-enhanced MR images using dynamic non- [193] S. Ropele, F. Ebner, F. Fazekas, G. Reishofer,
local means, IEEE transactions on medical imaging Super-resolution mri using microscopic spatial mod-
29 (2010) 302–310. ulation of magnetization., Magnetic resonance in
medicine 64 (2010) 1671–1675.
[183] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio,
P.-A. Manzagol, Stacked denoising autoencoders: [194] E. Plenge, D. H. J. Poot, M. Bernsen, G. Kotek,
learning useful representations in a deep network G. Houston, P. Wielopolski, L. van der Weerd, W. J.
with a local denoising criterion, Journal of Machine Niessen, E. Meijering, Super-resolution methods in
Learning Research (JMLR) 11 (2010) 3371–3408. mri: can they improve the trade-off between reso-
lution, signal-to-noise ratio, and acquisition time?,
[184] N. Dikaios, S. Arridge, V. Hamy, S. Punwani, Magnetic resonance in medicine 68 (2012) 1983–
D. Atkinson, Direct parametric reconstruction from 1993.
undersampled (k,t)-space data in dynamic contrast
enhanced MRI, Medical image analysis 18 (2014) [195] K. Bahrami, F. Shi, I. Rekik, Y. Gao, D. Shen, 7T-
989–1001. guided super-resolution of 3T MRI, Medical physics
44 (2017) 1661–1677.
[185] Y. Guo, S. G. Lingala, Y. Zhu, R. M. Lebel, K. S.
Nayak, Direct estimation of tracer-kinetic parame- [196] G. Van Steenkiste, D. H. J. Poot, B. Jeuris-
ter maps from highly undersampled brain dynamic sen, A. J. den Dekker, F. Vanhevel, P. M.
contrast enhanced MRI, Magnetic resonance in Parizel, J. Sijbers, Super-resolution t,
medicine 78 (2017) 1566–1578. javax.xml.bind.jaxbelement@1458c115, es-
timation: Quantitative high resolution
[186] S. P. Sourbron, D. L. Buckley, Classic mod- t, javax.xml.bind.jaxbelement@62365375,
els for dynamic contrast-enhanced mri., NMR in mapping from a set of low resolution t,
biomedicine 26 (2013) 1004–1027. javax.xml.bind.jaxbelement@20656587, -weighted

36
 images with different slice orientations., Magnetic [206] Y. Hong, U. Hwang, J. Yoo, S. Yoon, How
resonance in medicine 77 (2017) 1818–1830. generative adversarial networks and their variants
work: An overview of GAN, arXiv preprint
[197] K. Zeng, H. Zheng, C. Cai, Y. Yang, K. Zhang, arXiv:1711.05914v7 (2017).
Z. Chen, Simultaneous single- and multi-contrast
super-resolution for brain MRI images based on a [207] H. Huang, P. S. Yu, C. Wang, An introduction
convolutional neural network., Computers in biol- to image synthesis with generative adversarial nets,
ogy and medicine 99 (2018) 133–141. arXiv preprint arXiv:1803.04469v1 (2018).

[198] C. Liu, X. Wu, X. Yu, Y. Tang, J. Zhang, J. Zhou, [208] A. Osokin, A. Chessel, R. E. C. Salas, F. Vaggi,
Fusing multi-scale information in convolution net- Gans for biological image synthesis, in: Proc. IEEE
work for mr image super-resolution reconstruction., Int. Conf. Computer Vision (ICCV), pp. 2252–2261.
Biomedical engineering online 17 (2018) 114.
[209] G. Antipov, M. Baccouche, J. Dugelay, Face aging
[199] A. S. Chaudhari, Z. Fang, F. Kogan, J. Wood, K. J. with conditional generative adversarial networks,
Stevens, E. K. Gibbons, J. H. Lee, G. E. Gold, in: Proc. IEEE Int. Conf. Image Processing (ICIP),
B. A. Hargreaves, Super-resolution musculoskele- pp. 2089–2093.
tal MRI using deep learning, Magnetic resonance
in medicine 80 (2018) 2139–2154. [210] C. Bodnar, Text to image synthesis using gen-
erative adversarial networks, arXiv preprent
[200] A. Jog, A. Carass, S. Roy, D. L. Pham, J. L. Prince, arXiv:1805.00676v1 (2018).
Random forest regression for magnetic resonance
image synthesis, Medical image analysis 35 (2017) [211] H. Dong, S. Yu, C. Wu, Y. Guo, Semantic image
475–488. synthesis via adversarial learning, arXiv preprint
arXiv:1707.06873v1 (2017).
[201] K. E. Keenan, M. Ainslie, A. J. Barker, M. A. Boss,
K. M. Cecil, C. Charles, T. L. Chenevert, L. Clarke, [212] S. Reed, Z. Akata, X. Yan, L. Logeswaran,
J. L. Evelhoch, P. Finn, D. Gembris, J. L. Gunter, B. Schiele, H. Lee, Generative adversarial text to
D. L. G. Hill, C. R. Jack, E. F. Jackson, G. Liu, image synthesis, arXiv preprint arXiv:1605.05396v2
S. E. Russek, S. D. Sharma, M. Steckner, K. F. (2016).
Stupic, J. D. Trzasko, C. Yuan, J. Zheng, Quan-
titative magnetic resonance imaging phantoms: A [213] H.-C. Shin, N. A. Tenenholtz, J. K. Rogers, C. G.
review and the need for a system phantom, Mag- Schwarz, M. L. Senjem, J. L. Gunter, K. P. Andri-
netic resonance in medicine 79 (2018) 48–61. ole, M. Michalski, Medical Image Synthesis for Data
Augmentation and Anonymization Using Genera-
[202] K. Jurczuk, M. Kretowski, P.-A. Eliat, H. Saint- tive Adversarial Networks, in: International Work-
Jalmes, J. Bezy-Wendling, In silico modeling of shop on Simulation and Synthesis in Medical Imag-
magnetic resonance flow imaging in complex vascu- ing, Springer, pp. 1–11.
lar networks, IEEE transactions on medical imaging
33 (2014) 2191–2209. [214] T. C. W. Mok, A. C. S. Chung, Learning
data augmentation for brain tumor segmentation
[203] Y. Zhou, S. Giffard-Roisin, M. De Craene, with coarse-to-fine generative adversarial networks,
S. Camarasu-Pop, J. D’Hooge, M. Alessandrini, arXiv 1805.11291 (2018).
D. Friboulet, M. Sermesant, O. Bernard, A frame-
work for the generation of realistic synthetic car- [215] J. T. Guibas, T. S. Virdi, P. S. Li, Synthetic medical
diac ultrasound and magnetic resonance imaging images from dual generative adversarial networks,
sequences from the same virtual patients., IEEE arXiv preprint arXiv 1709.01872 (2017).
transactions on medical imaging 37 (2018) 741–754.
[216] A. Kitchen, J. Seah, Deep generative adversarial
[204] N. Duchateau, M. Sermesant, H. Delingette, N. Ay- neural networks for realistic prostate lesion MRI
ache, Model-based generation of large databases of synthesis, arXiv preprint arXiv:1708.00129 (2017).
cardiac images: Synthesis of pathological cine MR
sequences from real healthy cases, IEEE transac- [217] D. Nie, R. Trullo, J. Lian, C. Petitjean, S. Ruan,
tions on medical imaging 37 (2018) 755–766. Q. Wang, D. Shen, Medical image synthesis
with context-aware generative adversarial networks,
[205] A. Creswell, T. White, V. Dumoulin, K. Arulku- Medical image computing and computer-assisted
maran, B. Sengupta, A. A. Bharath, Generative intervention : MICCAI ... International Confer-
adversarial networks: An overview, IEEE Signal ence on Medical Image Computing and Computer-
Processing Magazine 35 (2018) 53–65. Assisted Intervention 10435 (2017) 417–425.

37
[218] K. D. Spuhler, J. Gardus, Y. Gao, C. DeLorenzo, [230] B. Glocker, A. Sotiras, N. Komodakis, N. Paragios,
R. Parsey, C. Huang, Synthesis of patient-specific Deformable medical image registration: setting the
transmission image for PET attenuation correction state of the art with discrete methods., Annual re-
for PET/MR imaging of the brain using a convolu- view of biomedical engineering 13 (2011) 219–244.
tional neural network, Journal of nuclear medicine
(2018). [231] A. Sotiras, C. Davatzikos, N. Paragios, Deformable
medical image registration: a survey., IEEE trans-
[219] A. Torrado-Carvajal, J. Vera-Olmos, D. Izquierdo- actions on medical imaging 32 (2013) 1153–1190.
Garcia, O. A. Catalano, M. A. Morales, J. Margolin,
A. Soricelli, M. Salvatore, N. Malpica, C. Catana, [232] F. P. M. Oliveira, J. M. R. S. Tavares, Medical im-
Dixon-VIBE deep learning (DIVIDE) pseudo-CT age registration: a review., Computer methods in
synthesis for pelvis PET/MR attenuation correc- biomechanics and biomedical engineering 17 (2014)
tion, Journal of nuclear medicine (2018). 73–93.

[233] P. K. Saha, R. Strand, G. Borgefors, Digital topol-

[220] Q. Zhang, H. Wang, H. Lu, D. Won, S. W. Yoon,
ogy and geometry in medical imaging: A survey,
Medical image synthesis with generative adversarial
IEEE transactions on medical imaging 34 (2015)
networks for tissue recognition, in: Proc. IEEE Int.
1940–1964.
Conf. Healthcare Informatics (ICHI), pp. 199–207.
[234] M. A. Viergever, J. B. A. Maintz, S. Klein, K. Mur-
[221] M. Frid-Adar, E. Klang, M. Amitai, J. Goldberger, phy, M. Staring, J. P. W. Pluim, A survey of medi-
H. Greenspan, Synthetic data augmentation using cal image registration - under review., Medical im-
GAN for improved liver lesion classification, in: age analysis 33 (2016) 140–144.
Proc. IEEE 15th Int. Symp. Biomedical Imaging
(ISBI 2018), pp. 289–293. [235] G. Song, J. Han, Y. Zhao, Z. Wang, H. Du, A re-
view on medical image registration as an optimiza-
[222] J. M. Wolterink, A. M. Dinkla, M. H. F. Savenije, tion problem., Current medical imaging reviews 13
P. R. Seevinck, C. A. T. van den Berg, I. Isgum, (2017) 274–283.
Deep MR to CT synthesis using unpaired data,
arXiv:1708.01155v1 (2017). [236] E. Ferrante, N. Paragios, Slice-to-volume medical
image registration: A survey., Medical image anal-
[223] J. M. F. Calvin R. Maurer, Jr., A review of medical ysis 39 (2017) 101–123.
image registration, 1993.
[237] A. P. Keszei, B. Berkels, T. M. Deserno, Survey of
[224] J. Maclaren, M. Herbst, O. Speck, M. Zaitsev, non-rigid registration tools in medicine., Journal of
Prospective motion correction in brain imaging: a digital imaging 30 (2017) 102–116.
review, Magnetic resonance in medicine 69 (2013)
[238] S. Nag, Image registration techniques: A survey,
621–636.
arXiv preprint arXiv:1712.07540v1 (2017).
[225] M. Zaitsev, B. Akin, P. LeVan, B. R. Knowles,
[239] J. Jiang, P. Trundle, J. Ren, Medical image anal-
Prospective motion correction in functional MRI,
ysis with artificial neural networks., Computerized
Neuroimage 154 (2017) 33–42.
medical imaging and graphics : the official journal
of the Computerized Medical Imaging Society 34
[226] O. Fluck, C. Vetter, W. Wein, A. Kamen, B. Preim,
(2010) 617–631.
R. Westermann, A survey of medical image regis-
tration on graphics hardware., Computer methods [240] G. Wu, M. Kim, Q. Wang, B. C. Munsell,
and programs in biomedicine 104 (2011) e45–e57. D. Shen, Scalable high-performance image registra-
tion framework by unsupervised deep feature rep-
[227] L. Shi, W. Liu, H. Zhang, Y. Xie, D. Wang, A sur- resentations learning., IEEE transactions on bio-
vey of GPU-based medical image computing tech- medical engineering 63 (2016) 1505–1516.
niques., Quantitative imaging in medicine and
surgery 2 (2012) 188–206. [241] S. S. M. Salehi, S. Khan, D. Erdogmus,
A. Gholipour, Real-time deep pose estimation with
[228] A. Eklund, P. Dufort, D. Forsberg, S. M. LaConte, geodesic loss for image-to-template rigid registra-
Medical image processing on the gpu - past, present tion., IEEE transactions on medical imaging (2018).
and future., Medical image analysis 17 (2013) 1073–
1094. [242] D. Toth, S. Miao, T. Kurzendorfer, C. A. Rinaldi,
R. Liao, T. Mansi, K. Rhode, P. Mountney, 3D/2D
[229] J. B. Maintz, M. A. Viergever, A survey of med- model-to-image registration by imitation learning
ical image registration., Medical image analysis 2 for cardiac procedures., International journal of
(1998) 1–36. computer assisted radiology and surgery (2018).

38
[243] X. Han, Mr-based synthetic CT generation using a [255] D. Garca-Lorenzo, S. Francis, S. Narayanan, D. L.
deep convolutional neural network method, Medical Arnold, D. L. Collins, Review of automatic segmen-
physics 44 (2017) 1408–1419. tation methods of multiple sclerosis white matter le-
sions on conventional magnetic resonance imaging,
[244] M. Liu, D. Cheng, K. Wang, Y. Wang, A. D. N. Medical image analysis 17 (2013) 1–18.
Initiative, Multi-modality cascaded convolutional
neural networks for alzheimer’s disease diagnosis., [256] E. Smistad, T. L. Falch, M. Bozorgi, A. C. Elster,
Neuroinformatics 16 (2018) 295–308. F. Lindseth, Medical image segmentation on GPUs–
a comprehensive review., Medical image analysis 20
[245] L. Xiang, Y. Qiao, D. Nie, L. An, Q. Wang, D. Shen, (2015) 1–18.
Deep auto-context convolutional neural networks
for standard-dose PET image estimation from low- [257] J. Bernal, K. Kushibar, D. S. Asfaw, S. Valverde,
dose PET/MRI., Neurocomputing 267 (2017) 406– A. Oliver, R. Mart, X. Llad, Deep convolutional
416. neural networks for brain image analysis on mag-
netic resonance imaging: a review, arXiv preprint
[246] S. Shan, W. Yan, X. Guo, E. I.-C. Chang, Y. Fan, arXiv:1712.03747v3 (2017).
Y. Xu, Unsupervised end-to-end learning for de-
formable medical image registration, arXiv preprint [258] L. Dora, S. Agrawal, R. Panda, A. Abraham, State-
arXiv:1711.08608v2 (2017). of-the-art methods for brain tissue segmentation: A
review, IEEE Reviews in Biomedical Engineering 10
[247] G. Balakrishnan, A. Zhao, M. R. Sabuncu, J. Gut- (2017) 235–249.
tag, A. V. Dalca, An unsupervised learning model
for deformable medical image registration, arXiv [259] H. R. Torres, S. Queiros, P. Morais, B. Oliveira,
preprint arXiv:1802.02604v3 (2018). J. C. Fonseca, J. L. Vilaa, Kidney segmentation in
ultrasound, magnetic resonance and computed to-
[248] B. D. de Vos, F. F. Berendsen, M. A. Viergever, mography images: A systematic review, Computer
H. Sokooti, M. Staring, I. Isgum, A deep methods and programs in biomedicine 157 (2018)
learning framework for unsupervised affine and 49–67.
deformable image registration, arXiv preprint
arXiv:1809.06130v1 (2018). [260] J. Bernal, K. Kushibar, D. S. Asfaw, S. Valverde,
A. Oliver, R. Mart, X. Llad, Deep convolutional
[249] M. W. Vannier, R. L. Butterfield, D. Jordan, W. A. neural networks for brain image analysis on mag-
Murphy, R. G. Levitt, M. Gado, Multispectral anal- netic resonance imaging: a review, Artificial intel-
ysis of magnetic resonance images., Radiology 154 ligence in medicine (2018).
(1985) 221–224.
[261] S. Moccia, E. De Momi, S. El Hadji, L. S. Mattos,
[250] A. Lundervold, K. Moen, T. Taxt, Automatic recog- Blood vessel segmentation algorithms - review of
nition of normal and pathological tissue types in methods, datasets and evaluation metrics, Com-
MR images, in: Proc. of the NOBIM Conference, puter methods and programs in biomedicine 158
Oslo, Norway, 1988. (2018) 71–91.

[251] T. Taxt, A. Lundervold, B. Fuglaas, H. Lien, [262] A. Makropoulos, S. J. Counsell, D. Rueckert, A

V. Abeler, Multispectral analysis of uterine corpus review on automatic fetal and neonatal brain MRI
tumors in magnetic resonance imaging., Magnetic segmentation., NeuroImage 170 (2018) 231–248.
resonance in medicine 23 (1992) 55–76.
[263] L. Chen, P. Bentley, D. Rueckert, Fully automatic
[252] T. Taxt, A. Lundervold, Multispectral analysis of acute ischemic lesion segmentation in DWI using
the brain using magnetic resonance imaging., IEEE convolutional neural networks, NeuroImage. Clini-
transactions on medical imaging 13 (1994) 470–481. cal 15 (2017) 633–643.

[253] A. Lundervold, G. Storvik, Segmentation of brain [264] M. Havaei, A. Davy, D. Warde-Farley, A. Biard,
parenchyma and cerebrospinal fluid in multispectral A. Courville, Y. Bengio, C. Pal, P.-M. Jodoin,
magnetic resonance images, IEEE Transactions on H. Larochelle, Brain tumor segmentation with deep
Medical Imaging 14 (1995) 339–349. neural networks, Medical image analysis 35 (2017)
18–31.
[254] M. Cabezas, A. Oliver, X. Llad, J. Freixenet, M. B.
Cuadra, A review of atlas-based segmentation for [265] H. Choi, K. H. Jin, Fast and robust segmentation
magnetic resonance brain images, Computer meth- of the striatum using deep convolutional neural net-
ods and programs in biomedicine 104 (2011) e158– works, Journal of Neuroscience Methods 274 (2016)
e177. 146–153.

39
[266] B. Ibragimov, L. Xing, Segmentation of organs-at- [277] D. Lu, K. Popuri, G. W. Ding, R. Balachandar, Beg,
risks in head and neck CT images using convolu- M. Faisal, Multimodal and multiscale deep neural
tional neural networks, Medical physics 44 (2017) networks for the early diagnosis of alzheimer’s dis-
547–557. ease using structural MR and FDG-PET images,
Scientific reports 8 (2018) 5697.
[267] T. L. Kline, P. Korfiatis, M. E. Edwards, J. D. Blais,
F. S. Czerwiec, P. C. Harris, B. F. King, V. E. Tor- [278] P. Moeskops, J. de Bresser, H. J. Kuijf, A. M. Men-
res, B. J. Erickson, Performance of an artificial drik, G. J. Biessels, J. P. W. Pluim, I. Igum, Evalu-
multi-observer deep neural network for fully auto- ation of a deep learning approach for the segmenta-
mated segmentation of polycystic kidneys., Journal tion of brain tissues and white matter hyperinten-
of digital imaging 30 (2017) 442–448. sities of presumed vascular origin inmri., NeuroIm-
age. Clinical 17 (2018) 251–262.
[268] Y. Guo, Y. Gao, D. Shen, Deformable mr prostate
segmentation via deep feature learning and sparse [279] R. Pizarro, H.-E. Assemlal, D. De Nigris, C. El-
patch matching., IEEE transactions on medical liott, S. Antel, D. Arnold, A. Shmuel, Using deep
imaging 35 (2016) 1077–1089. learning algorithms to automatically identify the
[269] X. Li, Q. Dou, H. Chen, C.-W. Fu, X. Qi, D. L. brain mri contrast: Implications for managing large
Belav, G. Armbrecht, D. Felsenberg, G. Zheng, P.- databases., Neuroinformatics (2018).
A. Heng, 3D multi-scale FCN with random modal-
[280] K. R. Laukamp, F. Thiele, G. Shakirin, D. Zopfs,
ity voxel dropout learning for intervertebral disc
A. Faymonville, M. Timmer, D. Maintz,
localization and segmentation from multi-modality
M. Perkuhn, J. Borggrefe, Fully automated
MR images, Medical image analysis 45 (2018) 41–
detection and segmentation of meningiomas using
54.
deep learning on routine multiparametric MRI,
[270] J. Kleesiek, G. Urban, A. Hubert, D. Schwarz, European radiology (2018).
K. Maier-Hein, M. Bendszus, A. Biller, Deep MRI
brain extraction: A 3D convolutional neural net- [281] M. Perkuhn, P. Stavrinou, F. Thiele, G. Shakirin,
work for skull stripping, Neuroimage 129 (2016) M. Mohan, D. Garmpis, C. Kabbasch, J. Borggrefe,
460–469. Clinical evaluation of a multiparametric deep learn-
ing model for glioblastoma segmentation using het-
[271] H. Li, N. A. Parikh, L. He, A novel transfer learning erogeneous magnetic resonance imaging data from
approach to enhance deep neural network classifica- clinical routine., Investigative radiology (2018).
tion of brain functional connectomes., Frontiers in
neuroscience 12 (2018) 491. [282] E. A. AlBadawy, A. Saha, M. A. Mazurowski, Deep
learning for segmentation of brain tumors: Impact
[272] L.-L. Zeng, H. Wang, P. Hu, B. Yang, W. Pu, of cross-institutional training and testing., Medical
H. Shen, X. Chen, Z. Liu, H. Yin, Q. Tan, physics 45 (2018) 1150–1158.
K. Wang, D. Hu, Multi-site diagnostic classification
of schizophrenia using discriminant deep learning [283] S. Cui, L. Mao, J. Jiang, C. Liu, S. Xiong, Au-
with functional connectivity MRI., EBioMedicine tomatic semantic segmentation of brain gliomas
30 (2018) 74–85. from mri images using a deep cascaded neural
network., Journal of healthcare engineering 2018
[273] J. Wasserthal, P. Neher, K. H. Maier-Hein, Tract-
(2018) 4940593.
Seg - fast and accurate white matter tract segmen-
tation, Neuroimage 183 (2018) 239–253. [284] F. Hoseini, A. Shahbahrami, P. Bayat, Adaptahead
[274] J. H. Cole, R. P. K. Poudel, D. Tsagkrasoulis, optimization algorithm for learning deep cnn ap-
M. W. A. Caan, C. Steves, T. D. Spector, G. Mon- plied to mri segmentation., Journal of digital imag-
tana, Predicting brain age with deep learning from ing (2018).
raw imaging data results in a reliable and heritable
[285] Y. Yoo, L. Y. W. Tang, T. Brosch, D. K. B. Li,
biomarker, Neuroimage 163 (2017) 115–124.
S. Kolind, I. Vavasour, A. Rauscher, A. L. MacKay,
[275] M. Liu, J. Zhang, E. Adeli, D. Shen, Landmark- A. Traboulsee, R. C. Tam, Deep learning of joint
based deep multi-instance learning for brain disease myelin and T1w MRI features in normal-appearing
diagnosis., Medical image analysis 43 (2018) 157– brain tissue to distinguish between multiplesclero-
168. sispatients and healthy controls., NeuroImage. Clin-
ical 17 (2018) 169–178.
[276] J. Islam, Y. Zhang, Brain mri analysis for
Alzheimer’s disease diagnosis using an ensemble sys- [286] M. F. Bobo, S. Bao, Y. Huo, Y. Yao, J. Virostko,
tem of deep convolutional neural networks, Brain A. J. Plassard, I. Lyu, A. Assad, R. G. Abramson,
informatics 5 (2018) 2. M. A. Hilmes, B. A. Landman, Fully convolutional

40
 neural networks improve abdominal organ segmen- stenosis grading using deep learning, arXiv preprint
tation., Proceedings of SPIE–the International So- arXiv:1807.10215v1 (2018).
ciety for Optical Engineering 10574 (2018).
[296] Z. Han, B. Wei, S. Leung, I. B. Nachum, D. Lai-
[287] M. Shehata, F. Khalifa, A. Soliman, M. Ghazal, dley, S. Li, Automated pathogenesis-based diag-
F. Taher, M. Abou El-Ghar, A. Dwyer, nosis of lumbar neural foraminal stenosis via deep
G. Gimel’farb, R. Keynton, A. El-Baz, Computer- multiscale multitask learning., Neuroinformatics 16
aided diagnostic system for early detection of acute (2018) 325–337.
renal transplant rejection using diffusion-weighted
[297] K. H. Kim, W.-J. Do, S.-H. Park, Improving res-
MRI, IEEE transactions on bio-medical engineering
olution of MR images with an adversarial network
(2018).
incorporating images with different contrast, Med-
[288] R. Cheng, H. R. Roth, N. Lay, L. Lu, B. Turk- ical physics 45 (2018) 3120–3131.
bey, W. Gandler, E. S. McCreedy, T. Pohida, P. A. [298] A. H. Pilevar, CBMIR: content-based image re-
Pinto, P. Choyke, M. J. McAuliffe, R. M. Summers, trieval algorithm for medical image databases, Jour-
Automatic magnetic resonance prostate segmenta- nal of medical signals and sensors 1 (2011) 12–18.
tion by deep learning with holistically nested net-
works, Journal of medical imaging 4 (2017) 041302. [299] A. Kumar, J. Kim, W. Cai, M. Fulham, D. Feng,
Content-based medical image retrieval: a survey of
[289] J. Ishioka, Y. Matsuoka, S. Uehara, Y. Yasuda, applications to multidimensional and multimodal-
T. Kijima, S. Yoshida, M. Yokoyama, K. Saito, ity dat., Journal of digital imaging 26 (2013) 1025–
K. Kihara, N. Numao, T. Kimura, K. Kudo, I. Ku- 1039.
mazawa, Y. Fujii, Computer-aided diagnosis of
prostate cancer on magnetic resonance imaging us- [300] A. V. Faria, K. Oishi, S. Yoshida, A. Hillis,
ing a convolutional neural network algorithm, BJU M. I. Miller, S. Mori, Content-based image re-
international (2018). trieval for brain mri: an image-searching engine
and population-based analysis to utilize past clin-
[290] Y. Song, Y.-D. Zhang, X. Yan, H. Liu, M. Zhou, ical data for future diagnosis, NeuroImage. Clinical
B. Hu, G. Yang, Computer-aided diagnosis of 7 (2015) 367–376.
prostate cancer using a deep convolutional neural
network from multiparametric MRI, Journal of [301] A. Kumar, F. Nette, K. Klein, M. Fulham, J. Kim,
magnetic resonance imaging : JMRI (2018). A visual analytics approach using the exploration of
multidimensional feature spaces for content-based
[291] X. Wang, W. Yang, J. Weinreb, J. Han, Q. Li, medical image retrieval., IEEE journal of biomedi-
X. Kong, Y. Yan, Z. Ke, B. Luo, T. Liu, L. Wang, cal and health informatics 19 (2015) 1734–1746.
Searching for prostate cancer by fully automated
magnetic resonance imaging classification: deep [302] M. V. N. Bedo, D. Pereira Dos Santos, M. Ponciano-
learning versus non-deep learning., Scientific re- Silva, P. M. de Azevedo-Marques, A. P. d. L. Fer-
ports 7 (2017) 15415. reira de Carvalho, C. Traina, Endowing a content-
based medical image retrieval system with percep-
[292] X. Yang, C. Liu, Z. Wang, J. Yang, H. L. Min, tual similarity using ensemble strategy, Journal of
L. Wang, K.-T. T. Cheng, Co-trained convolu- digital imaging 29 (2016) 22–37.
tional neural networks for automated detection of
[303] C. Muramatsu, Overview on subjective similarity
prostate cancer in multi-parametric MRI, Medical
of images for content-based medical image retrieval,
image analysis 42 (2017) 212–227.
Radiological physics and technology (2018).
[293] M. H. Le, J. Chen, L. Wang, Z. Wang, W. Liu, [304] A. B. Spanier, N. Caplan, J. Sosna, B. Acar,
K.-T. T. Cheng, X. Yang, Automated diagnosis of L. Joskowicz, A fully automatic end-to-end method
prostate cancer in multi-parametric MRI based on for content-based image retrieval of CT scans with
multimodal convolutional neural networks, Physics similar liver lesion annotations, International jour-
in medicine and biology 62 (2017) 6497–6514. nal of computer assisted radiology and surgery 13
(2018) 165–174.
[294] D. Forsberg, E. Sjöblom, J. L. Sunshine, Detection
and labeling of vertebrae in MR images using deep [305] A. Gordo, J. Almazan, J. Revaud, D. Larlus, End-
learning with clinical annotations as training data, to-end learning of deep visual representations for
Journal of digital imaging 30 (2017) 406–412. image retrieval (????).

[295] J.-T. Lu, S. Pedemonte, B. Bizzo, S. Doyle, K. P. [306] P. Liu, J. Guo, C. Wu, D. Cai, Fusion of deep learn-
Andriole, M. H. Michalski, R. G. Gonzalez, S. R. ing and compressed domain features for content-
Pomerantz, DeepSPINE: automated lumbar verte- based image retrieval, IEEE Transactions on Image
bral segmentation, disc-level designation, and spinal Processing 26 (2017) 5706–5717.

41
[307] J. Han, D. Zhang, G. Cheng, N. Liu, D. Xu, [320] D. J. Goff, T. W. Loehfelm, Automated radiology
Advanced deep-learning techniques for salient and report summarization using an open-source natu-
category-specific object detection: A survey, IEEE ral language processing pipeline, Journal of digital
Signal Processing Magazine 35 (2018) 84–100. imaging 31 (2018) 185–192.

[308] T. Piplani, D. Bamman, Deepseek: Content [321] E. Gibson, W. Li, C. Sudre, L. Fidon, D. I. Shakir,
based image search & retrieval, arXiv preprint G. Wang, Z. Eaton-Rosen, R. Gray, T. Doel, Y. Hu,
arXiv:1801.03406v2 (2018). T. Whyntie, P. Nachev, M. Modat, D. C. Bar-
ratt, S. Ourselin, M. J. Cardoso, T. Vercauteren,
[309] J. Yang, J. Liang, H. Shen, K. Wang, P. L. Rosin, NiftyNet: a deep-learning platform for medical
M. Yang, Dynamic match kernel with deep convo- imaging, Computer methods and programs in
lutional features for image retrieval, IEEE Transac- biomedicine 158 (2018) 113–122.
tions on Image Processing 27 (2018) 5288–5302.
[322] W. Li, G. Wang, L. Fidon, S. Ourselin, M. J. Car-
[310] J. E. S. Sklan, A. J. Plassard, D. Fabbri, B. A. doso, T. Vercauteren, On the compactness, effi-
Landman, Toward content based image retrieval ciency, and representation of 3d convolutional net-
with deep convolutional neural networks, Proceed- works: Brain parcellation as a pretext task, in: In-
ings of SPIE–the International Society for Optical ternational Conference on Information Processing
Engineering 9417 (2015). in Medical Imaging (IPMI).
[311] R. S. Bressan, D. H. A. Alves, L. M. Valerio, P. H.
[323] K. Kamnitsas, C. Ledig, V. F. Newcombe, J. P.
Bugatti, P. T. M. Saito, DOCToR: the role of deep
Simpson, A. D. Kane, D. K. Menon, D. Rueckert,
features in content-based mammographic image re-
B. Glocker, Efficient multi-scale 3D CNN with fully
trieval, in: Proc. IEEE 31st Int. Symp. Computer-
connected CRF for accurate brain lesion segmenta-
Based Medical Systems (CBMS), pp. 158–163.
tion, Medical image analysis 36 (2017) 61–78.
[312] A. Qayyum, S. M. Anwar, M. Awais, M. Majid,
[324] O. Ronneberger, P.Fischer, T. Brox, U-net: Con-
Medical image retrieval using deep convolutional
volutional networks for biomedical image segmenta-
neural network, arXiv preprint arXiv:1703.08472v1
tion, in: Medical Image Computing and Computer-
(2017).
Assisted Intervention (MICCAI), volume 9351 of
[313] Y.-A. Chung, W.-H. Weng, Learning deep represen- LNCS, Springer, 2015, pp. 234–241. (available on
tations of medical images using siamese CNNs with arXiv:1505.04597 [cs.CV]).
application to content-based image retrieval, arXiv
[325] V. Badrinarayanan, A. Kendall, R. Cipolla, Seg-
preprint arXiv:1711.08490v2 (2017).
Net: A Deep Convolutional Encoder-Decoder Ar-
[314] B. Jing, P. Xie, E. Xing, On the automatic gen- chitecture for Image Segmentation, IEEE Transac-
eration of medical imaging reports, arXiv preprint tions on Pattern Analysis and Machine Intelligence
arXiv:1711.08195v3 (2017). (2017).

[315] C. Y. Li, X. Liang, Z. Hu, E. P. Xing, Hy- [326] M. Mardani, E. Gong, J. Y. Cheng, S. Vasanawala,
brid retrieval-generation reinforced agent for med- G. Zaharchuk, M. Alley, N. Thakur, S. Han,
ical image report generation, arXiv preprint W. Dally, J. M. Pauly, et al., Deep generative adver-
arXiv:1805.08298v1 (2018). sarial networks for compressed sensing automates
mri, arXiv preprint arXiv:1706.00051 (2017).
[316] M. Moradi, A. Madani, Y. Gur, Y. Guo, T. Syeda-
Mahmood, Bimodal network architectures for au- [327] S. Parisot, S. I. Ktena, E. Ferrante, M. Lee, R. G.
tomatic generation of image annotation from text, Moreno, B. Glocker, D. Rueckert, Spectral graph
arXiv preprint arXiv:1809.01610v1 (2018). convolutions for population-based disease predic-
tion, in: International Conference on Medical Im-
[317] Y. Zhang, D. Y. Ding, T. Qian, C. D. Manning, age Computing and Computer-Assisted Interven-
C. P. Langlotz, Learning to summarize radiology tion, Springer, pp. 177–185.
findings, arXiv preprint arXiv:1809.04698v1 (????).
[328] G. Marcus, Deep learning: A critical appraisal,
[318] E. Pons, L. M. M. Braun, M. G. M. Hunink, J. A. arXiv preprint arXiv:1801.00631 (2018).
Kors, Natural language processing in radiology: A
systematic review, Radiology 279 (2016) 329–343. [329] Z. C. Lipton, J. Steinhardt, Troubling Trends in
Machine Learning Scholarship (2018).
[319] J. Zech, M. Pain, J. Titano, M. Badgeley, J. Schef-
flein, A. Su, A. Costa, J. Bederson, J. Lehar, E. K. [330] C. Zhang, S. Bengio, M. Hardt, B. Recht,
Oermann, Natural language-based machine learn- O. Vinyals, Understanding deep learning re-
ing models for the annotation of clinical radiology quires rethinking generalization, arXiv preprint
reports, Radiology 287 (2018) 570–580. arXiv:1611.03530 (2016).

42
[331] M. Fredrikson, S. Jha, T. Ristenpart, Model in- [342] A. Lundervold, A. Lundervold, J. Rørvik, Fast semi-
version attacks that exploit confidence information supervised segmentation of the kidneys in DCE-
and basic countermeasures, in: Proceedings of the MRI using convolutional neural networks and trans-
22nd ACM SIGSAC Conference on Computer and fer learning, 2017.
Communications Security, ACM, pp. 1322–1333.
[343] A. Lundervold, K. Sprawka, A. Lundervold, Fast
[332] R. Shokri, M. Stronati, C. Song, V. Shmatikov, estimation of kidney volumes and time courses
Membership inference attacks against machine in DCE-MRI using convolutional neural networks,
learning models, in: Security and Privacy (SP), 2018.
2017 IEEE Symposium on, IEEE, pp. 3–18.
[344] G. E. Hinton, A. Krizhevsky, S. D. Wang, Trans-
[333] B. McMahan, E. Moore, D. Ramage, S. Hampson, forming auto-encoders, in: International Confer-
B. A. y Arcas, Communication-Efficient Learning ence on Artificial Neural Networks, Springer, pp.
of Deep Networks from Decentralized Data, in: 44–51.
A. Singh, J. Zhu (Eds.), Proceedings of the 20th [345] S. Sabour, N. Frosst, G. E. Hinton, Dynamic rout-
International Conference on Artificial Intelligence ing between capsules, in: Advances in Neural Infor-
and Statistics, volume 54 of Proceedings of Machine mation Processing Systems, pp. 3856–3866.
Learning Research, PMLR, Fort Lauderdale, FL,
USA, 2017, pp. 1273–1282. [346] V. Mnih, N. Heess, A. Graves, et al., Recurrent
models of visual attention, in: Advances in neural
[334] O. Gupta, R. Raskar, Distributed learning of deep information processing systems, pp. 2204–2212.
neural network over multiple agents, Journal of Net-
work and Computer Applications 116 (2018) 1–8. [347] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville,
R. Salakhudinov, R. Zemel, Y. Bengio, Show, at-
[335] P. Vepakomma, O. Gupta, T. Swedish, R. Raskar, tend and tell: Neural image caption generation with
Split learning for health: Distributed deep learning visual attention, in: International conference on
without sharing raw patient data, arXiv preprint machine learning, pp. 2048–2057.
arXiv:1812.00564 (2018).
[348] D. Castelvecchi, Can we open the black box of AI?,
[336] N. Papernot, M. Abadi, U. Erlingsson, I. Goodfel- Nature News 538 (2016) 20.
low, K. Talwar, Semi-supervised knowledge transfer
[349] C. Olah, A. Satyanarayan, I. Johnson, S. Carter,
for deep learning from private training data, arXiv
L. Schubert, K. Ye, A. Mordvintsev, The building
preprint arXiv:1610.05755 (2016).
blocks of interpretability, Distill 3 (2018).
[337] N. Papernot, S. Song, I. Mironov, A. Raghunathan, [350] G. Montavon, W. Samek, K.-R. Müller, Methods
K. Talwar, Ú. Erlingsson, Scalable Private Learn- for interpreting and understanding deep neural net-
ing with PATE, arXiv preprint arXiv:1802.08908 works, Digital Signal Processing (2017).
(2018).
[351] J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, H. Lip-
[338] H. B. McMahan, D. Ramage, K. Talwar, L. Zhang, son, Understanding neural networks through deep
Learning Differentially Private Recurrent Language visualization, in: Deep Learning Workshop, 31st In-
Models, in: International Conference on Learning ternational Conference on Machine Learning, 2015.
Representations.
[352] C. Olah, A. Mordvintsev, L. Schubert, Feature vi-
[339] P. Vepakomma, T. Swedish, R. Raskar, O. Gupta, sualization, Distill 2 (2017).
A. Dubey, No Peek: A Survey of private distributed
deep learning, arXiv preprint arXiv:1812.03288 [353] F. M. Hohman, M. Kahng, R. Pienta, D. H. Chau,
(2018). Visual Analytics in Deep Learning: An Interroga-
tive Survey for the Next Frontiers, IEEE Trans-
[340] K. Chang, N. Balachandar, C. Lam, D. Yi, actions on Visualization and Computer Graphics
J. Brown, A. Beers, B. Rosen, D. L. Rubin, (2018).
J. Kalpathy-Cramer, Distributed deep learning net-
[354] R. M. Neal, Bayesian learning for neural networks,
works among institutions for medical imaging, Jour-
Ph.D. thesis, University of Toronto, 1995.
nal of the American Medical Informatics Associa-
tion : JAMIA 25 (2018) 945–954. [355] D. J. MacKay, A practical Bayesian framework for
backpropagation networks, Neural computation 4
[341] J. R. Zech, M. A. Badgeley, M. Liu, A. B. Costa, (1992) 448–472.
J. J. Titano, E. K. Oermann, Variable generaliza-
tion performance of a deep learning model to detect [356] P. Dayan, G. E. Hinton, R. M. Neal, R. S. Zemel,
pneumonia in chest radiographs: A cross-sectional The Helmholtz machine, Neural computation 7
study, PLoS medicine 15 (2018). (1995) 889–904.

43
 [357] Y. Li, Y. Gal, Dropout Inference in Bayesian Neural [360] R. Feinman, R. R. Curtin, S. Shintre, A. B. Gard-
Networks with Alpha-divergences, in: International ner, Detecting adversarial samples from artifacts,
Conference on Machine Learning, pp. 2052–2061. arXiv preprint arXiv:1703.00410 (2017).

[358] C. Leibig, V. Allken, M. S. Ayhan, P. Berens,

S. Wahl, Leveraging uncertainty information from [361] P. Sharp, S. Hockfield, Convergence: The future of
deep neural networks for disease detection, Scien- health, Science 355 (2017) 589.
tific reports 7 (2017) 17816.

[359] A. Kendall, V. Badrinarayanan, R. Cipolla, [362] L. Hood, M. Flores, A personal view on sys-
Bayesian segnet: Model uncertainty in deep con- tems medicine and the emergence of proactive P4
volutional encoder-decoder architectures for scene medicine: predictive, preventive, personalized and
understanding, arXiv preprint arXiv:1511.02680 participatory, New biotechnology 29 (2012) 613–
(2015). 624.

44

View publication stats