Physica Medica 83 (2021) 79–87

Contents lists available at ScienceDirect

Physica Medica
journal homepage: www.elsevier.com/locate/ejmp

Review paper

Machine learning in Magnetic Resonance Imaging: Image reconstruction

Javier Montalt-Tordera a, Vivek Muthurangu a, Andreas Hauptmann b, c,
Jennifer Anne Steeden a, *
a
UCL Centre for Cardiovascular Imaging, University College London, London WC1N 1EH, United Kingdom
b
University of Oulu, Research Unit of Mathematical Sciences, Oulu, Finland
c
Department of Computer Science, University College London, London WC1E 6BT, United Kingdom

A R T I C L E I N F O A B S T R A C T

Keywords: Magnetic Resonance Imaging (MRI) plays a vital role in diagnosis, management and monitoring of many dis­
Machine learning eases. However, it is an inherently slow imaging technique. Over the last 20 years, parallel imaging, temporal
Artificial intelligence encoding and compressed sensing have enabled substantial speed-ups in the acquisition of MRI data, by accu­
Magnetic Resonance Imaging
rately recovering missing lines of k-space data. However, clinical uptake of vastly accelerated acquisitions has
Image reconstruction
been limited, in particular in compressed sensing, due to the time-consuming nature of the reconstructions and
unnatural looking images. Following the success of machine learning in a wide range of imaging tasks, there has
been a recent explosion in the use of machine learning in the field of MRI image reconstruction.
A wide range of approaches have been proposed, which can be applied in k-space and/or image-space.
Promising results have been demonstrated from a range of methods, enabling natural looking images and rapid
computation.
In this review article we summarize the current machine learning approaches used in MRI reconstruction,
discuss their drawbacks, clinical applications, and current trends.

1. Introduction if the sampling rate is reduced below that required by the Nyquist cri­
terion, aliasing artefacts will appear in the image.
1.1. The image reconstruction problem In general terms, the image reconstruction can be formulated as the
following inverse problem:
Magnetic Resonance Imaging (MRI) is extensively employed in
y = Ax + ∊ (1)
medical diagnosis and is a reference standard in many applications.
However, it has a significant drawback: the inherently slow nature of where y is the measured k-space data, A is the system matrix, x is the
data acquisition. The MRI signal is generated by the nuclei of hydrogen image and ∊ is a random noise term. When k-space data is undersampled
atoms as they interact with external electromagnetic fields. However, an and noise corrupted, the inverse problem in Eq. (1) is ill-posed: a solu­
MRI scanner cannot measure spatially dependent signals (i.e. images) tion might not exist, infinite solutions might exist, and it may be un­
directly. Rather, the spatial dependence is encoded into the frequency stable with respect to measurement errors. As a result, direct inversion of
and phase of the MRI signal. This encoding process is inherently A is generally not possible. Instead, an optimal solution in the least-
sequential, which leads to long acquisition times. Ultimately, a spatial squares sense may be obtained by recasting the problem as the
frequency map is obtained, which is referred to as k-space. In the simple following minimization:
case, the inverse Fourier transform (iFT) can then be used to reconstruct
1
the k-space data into clinically interpretable images. ̂x = argmin ‖Ax − y‖22 (2)
2
Due to the sequential nature of MRI scanning, acquisition time is x

roughly proportional to the number of k-space samples collected. Much research effort has been devoted to image reconstruction from
Therefore, it is desirable to collect as few samples as possible. However, an undersampled k-space over the last few decades. Two broad

* Corresponding author at: UCL Centre for Cardiovascular Imaging, Institute of Cardiovascular Science, 30 Guildford Street, London WC1N 1EH, United Kingdom.
E-mail addresses: [email protected] (J. Montalt-Tordera), [email protected] (V. Muthurangu), [email protected] (A. Hauptmann),
[email protected] (J.A. Steeden).

https://doi.org/10.1016/j.ejmp.2021.02.020
Received 30 November 2020; Accepted 23 February 2021
Available online 13 March 2021
1120-1797/© 2021 Associazione Italiana di Fisica Medica. Published by Elsevier Ltd. All rights reserved.
J. Montalt-Tordera et al. Physica Medica 83 (2021) 79–87

technologies stand out for their importance and deserve a brief overview output, which is then passed to the next layer. These layers have two
here, namely parallel imaging and compressed sensing. These enable crucial properties. First, the majority of these layers perform non-linear
substantial reductions in acquisition time while preserving image operations (often a linear transformation followed by a non-linearity),
quality. which when combined can represent very complex functions. Second,
Parallel imaging techniques exploit multi-channel receiver arrays to the layers have trainable parameters, i.e. parameters which are not fixed
compensate for the undersampling of k-space. This is enabled by the fact or designed, but optimized during a training process. During the training
that receiver coils exhibit spatially varying responses, which can be process, the parameters are iteratively adjusted by an optimization al­
leveraged to unfold aliased images or estimate missing k-space samples. gorithm in order to minimize a loss function for a given set of training
Parallel imaging techniques, such as SENSE (Sensitivity Encoding) [1] data. As it learns, the network approximates the mapping from inputs to
and GRAPPA (Generalized Autocalibrating Partial Parallel Acquisition) outputs.
[2], enjoy tremendous success and are routinely used in the clinical Deep neural networks are artificial neural networks which have
environment. However, increasing acceleration factors lead to signal-to- multiple layers, where in general, the deeper a neural network is, the
noise ratio (SNR) losses, which in practice limits the achievable higher its representational power. The use of deep neural networks in
acceleration. order to discover mappings or representations is referred to as deep
Compressed sensing (CS) [3] enables the reconstruction of sub­ learning. Some special types of neural networks deserve mention for
sampled signals provided that the signal to be reconstructed is sparse in their importance in MRI. Convolutional neural networks (CNN’s) are
some domain. A signal is sparse if it contains few non-zero elements those primarily based on shift-invariant convolutional layers, where the
compared to its size. MRI images are not typically sparse. However, like trainable parameters are a set of convolutional kernels which are
most natural images, they contain many redundancies, and have sparse translated along the image dimensions in a sliding window fashion. They
representations in other domains such as the finite difference or wavelet have several properties which make them ideally suited to image pro­
domain. The expectation that the solution be sparse in some domain can cessing, including the ability to encode local relationships, and that they
be incorporated into the optimization problem in Eq. (2) as a regulari­ are agnostic to image size [4]. Furthermore, a commonly used principle
zation term: is based on recurrent neural networks (RNN), which are designed to
process sequences of inputs. These maintain a hidden state, which acts
1
x = argmin ‖Ax − y‖22 + λ‖Dx‖1
̂ (3) like a memory about previous inputs in the sequence.
2
It is essential that the network contains non-linearities, also called
x

where D is the sparsifying transform, mapping a redundant image to activations. The most commonly used non-linearity is the rectified linear
its sparse representation,‖.‖1 is the l 1 -norm and λ is a regularization unit, or ReLU, which is a piecewise linear function that returns the input
parameter. The first term in this equation serves to enforce that the value for positive inputs and zero for negative inputs. This function has
solution is consistent with the measured data, while the second term become the default choice in many deep learning models because it
favors solutions that are sparse in the transform domain. The parameter often outperforms more complex activations and leads to models which
λ balances both terms and can be tuned to optimize image quality. are easier to train, due to its beneficial gradient properties. Nevertheless,
Compressed sensing is not only based on the expectation that the other activations such as sigmoid or extensions such as leaky ReLU
correct solution of Eq. (3) is sparse in the transform domain, but also that (including negative values) are sometimes used.
the aliased solutions are not sparse. This translates into another In general, machine learning problems can be formulated in a su­
requirement: that the aliasing artefact is incoherent, i.e. that it resembles pervised or unsupervised manner. Supervised learning uses known
noise. This can be satisfied in MRI by using non-regular or pseudo- ground-truth data to learn a mapping between data pairs, whereas un­
random sampling patterns, within the hardware constraints. supervised learning infers structures within the sample without labelled
Compressed sensing, which can be readily combined with parallel outputs. Currently, most applications in MRI reconstruction use super­
imaging, has enabled high acceleration factors. However, clinical vised learning, however unsupervised techniques remain an area of
translation has been complicated by several factors. First, the non-linear active research. There have been many approaches to machine learning
iterative reconstruction often takes too long or requires computational MRI reconstruction (both in terms of supervised and unsupervised
resources not currently available in most clinical services. Second, im­ techniques), including methods which work in image-space, those which
ages have been reported as looking unnatural and blocky. Finally, tuning work in k-space, those which operate in different domains, those that
of the regularization term is an empirical process that often depends on learn the direct mapping from k-space to image-space, and unrolled
the specific application and may be different for each patient. optimization methods (Fig. 1).
In the general case, aliasing and signal degradation cannot be avoi­
ded when sampling below the Nyquist rate, as per the Nyquist-Shannon 2. Supervised machine learning
theorem. Reconstruction methods for accelerated MRI rely on some form
of prior information or additional constraints on the reconstructed In supervised learning the ML algorithm learns a function that maps
signal. However, the priors used in parallel imaging and compressed the input to an output from a training data set, consisting of paired input
sensing are often crude, and more representative priors have the po­ and output images. This requires a gold-standard fully sampled data set
tential to improve current reconstruction techniques. However, (the desired output), with paired undersampled data (the input). These
designing such priors by hand is difficult. Artificial intelligence, on the approaches require a qualitative metric, or loss function, which is used to
other hand, excels at discovering patterns in data. Therefore, it is the evaluate how close the current output of the network is to the target
ideal tool to inject the knowledge provided by historic MRI data into the image. The most commonly used loss functions are pixel-wise Mean
image reconstruction. Squared Error (MSE, l 2 -loss) and Mean Absolute Error (MAE, l 1 -loss).
The following section is a very brief overview of deep learning, the However, these metrics do not reflect a radiologists’ perspective well
subset of artificial intelligence most relevant to MRI. [5], and are generally not good at representing small structures.
Development of new loss functions, including feature losses, remains an
1.2. Deep learning area of active research [6–8].

Artificial neural networks are a class of machine learning algorithm 2.1. Image restoration methods
that apply a series of cascaded layers (where each layer consists of a
series of connected nodes, or neurons), mapping inputs to outputs. Each Image restoration techniques are those that operate in the image
of these layers receives inputs, performs an operation and returns an domain only (see Fig. 1-a). These methods relate closely to general

80
J. Montalt-Tordera et al. Physica Medica 83 (2021) 79–87

Fig. 1. Image reconstruction methods can be roughly classified into five categories: a) image restoration, b) k-space completion, c) direct mapping, d) cross-domain
enhancement, and e) unrolled optimization, depending on how neural networks are used. A is the image formation model, and A* is the adjoint operator.

image problems in non-medical contexts, including image processing. As rather than the corrected image, because the residual has lower topo­
a result, they can directly benefit from and contribute to the rich body of logical complexity [11]. This has been expanded to 2.5D (where time is
literature on CNN-based image enhancement, including de-noising and included in the channel dimension) using a 2D Poisson disk sampling
super-resolution. The first applications of machine learning to MRI mask [12], as well as to golden-angle radial sampling using a 2D CNN
reconstruction were based on image restoration methods [9]. A popular with spatio-temporal slices [13], and using a 3D U-Net (2D plus time)
network in these methods is the convolutional encoder-decoder archi­ [14]. It has also been used on complex data, to de-alias phase contrast
tecture with skip connections, also known as U-Net [10]. It consists of an MRI images [15].
encoder path, with multiple down-sampling steps with increasing Another group of de-aliasing methods uses generative adversarial
number of channels, followed by a decoder path, with multiple up- networks (GAN’s) [16]. GAN’s consist of two subnetworks: a generator,
sampling layers with decreasing number of channels (see Fig. 2). In which produces images based on some input; and a discriminator, which
addition, skip connections are added between encoding and decoding attempts to distinguish the generator output from ground-truth images.
steps operating at the same scale, so that each decoding step receives the During training, the generator learns to produce realistic images so as to
concatenation of the previous decoding step and the corresponding deceive the discriminator, by minimizing an adversarial loss. In MRI
encoding step as its inputs. reconstruction, this adversarial loss is typically combined with a pixel-
Undersampling of k-space results in aliasing artefacts in the recon­ wise distance loss (such as the l 1 - or l 2 -norm) to stabilize training
structed images, which are dependent on the trajectory and under­ and ensure consistency with the ground-truth image. Discriminator
sampling pattern. Where the undersampling is performed in a non- networks are typically vanilla CNN classifiers; however, there is more
uniform manner, the resultant artefacts are incoherent and noise-like. variability in the types of generator networks used. Some examples
Therefore, it is possible to train a machine learning network to include; Deep De-Aliasing Generative Adversarial Networks (DAGAN)
remove such artefacts in a similar manner to image de-noising. It has [17] uses a U-Net architecture for the generator network. RefineGAN
been shown that it is possible to perform de-aliasing from data acquired [18] uses a cascade of two U-Nets, with the first performing the recon­
using a random undersampling scheme in the phase direction of 2D struction and the second refining this result. GANCS (GAN for
images [11]. In these applications it has been shown that there is a compressive sensing) [19] uses a deep residual network (ResNet) [20] as
benefit to training a CNN to learn the residual (i.e. the aliasing artefact) the generator, and also includes an affine projection operator for data

Fig. 2. A possible variant of the commonly

used U-Net architecture (encoder-decoder
with skip connections). The height of the
blocks represents changes in spatial resolu­
tion while the width represents the number
of channels. Example values are shown for
the sake of clarity. Downsampling (DS) is
often done using max-pooling layers.
Upsampling (US) can be achieved using up-
sampling or transpose convolution layers.
The activation is often a ReLU. Batch
normalization (BN) layers are sometimes
added to stabilize training. A special 1 × 1
convolution, also called bottleneck layer, is
often used at the end to reduce the channel
dimension. Concatenation operations relay
the features at each scale of the encoder path
to the corresponding scale in the decoder
path. The number of scales may vary.

81
J. Montalt-Tordera et al. Physica Medica 83 (2021) 79–87

consistency. transform between the undersampled k-space data and the uncorrupted
An alternative method to speed up MRI imaging is to acquire lower images (see Fig. 1-c). These end-to-end reconstructions have the po­
resolution data, using a smaller base matrix. It is then possible to apply tential to mitigate against errors caused by field inhomogeneity, eddy
the image enhancement method; super-resolution (SR), which attempts current effects, phase distortions, and re-gridding.
to predict high-frequency details from low-resolution images. Because AUTOMAP (automated transform by manifold approximation) [41]
SR can be simply applied as a post-processing step, there have been was trained using a large database of paired synthetic undersampled k-
many applications of super-resolution in MRI reconstruction. Simple space data (input), and reconstructed images (desired output). The
network structures include Super-Resolution Convolutional Neural network architecture consists of a feedforward deep neural network
Networks (SRCNN) [21] which learn end-to-end mapping. This has been consisting of fully connected layers with hyperbolic tangent activations
applied to 2D brain MRI images [22], and extended to 3D brain images (which learns the transform), followed by convolutional layers with
[23], as well as dynamic cardiac MRI data [24]. This has further been rectifier nonlinearity activations that form a convolutional autoencoder
improved through the use of 3D densely-connected blocks (DCSRN) [25] (which performs image domain refinement). Unfortunately, this results
and dense connections with deconvolution layers (DDSR) [26], as well in a large number of parameters, which grows quadratically with the
as residual U-Net structures [27]. Most studies demonstrate good results number of image pixels, which limited the use of AUTOMAP to small
with two or three-fold down sampling. A full review of the use of ma­ images (up to 128 × 128).
chine learning super-resolution in medical imaging can be found [28]. To reduce the parameter complexity of AUTOMAP, it is possible to
decompose the two-dimensional inverse Fourier Transform into two
2.2. k-space methods one-dimensional iFTs; dAUTOMAP (decompose AUTOMAP) [42]. Here
the model parameter complexity only increases linearly with the num­
Machine learning networks have been trained to perform k-space ber of image pixels. A similar approach reduces the complexity using a
enhancement (see Fig. 1-b), in a supervised manner, similarly to multi-layer perceptron network to learn the one-dimensional iFT in a
GRAPPA. Some approaches use large training databases without the line-by-line approach, rather than the whole image [43,44]. Alterna­
need for explicit coil-sensitivity information, whereas others learn the tively it is possible to replace the fully connected layers of AUTOMAP, by
relationship between coil elements from a small amount of fully sampled a bidirectional RNN: ETER-net (End to End MR Image Reconstruction
reference data (the auto-calibration signal, ACS). Using Recurrent Neural Network) [45]. ETER-net also decomposes the
DeepSPIRiT [29] uses CNN’s to interpolate undersampled multi-coil two-dimensional iFT, using two sequential recurrent neural networks.
k-space data. It is based on the SPIRiT (iterative self-consistent parallel These methods showed a reduced number of training parameters, which
imaging reconstruction) algorithm [30], which is a generalizable coil- allows its use in reconstruction of higher resolution images.
by-coil reconstruction based on self-consistency with the acquisition
data. To enable DeepSPIRiT to be used with different hardware config­ 2.4. Cross-domain methods
urations and different numbers/types of coils, the data is first normal­
ized using coil compression with principal component analysis (PCA) Cross-domain methods are hybrid methods that operate in both the
[31]. This places the dominant virtual sensitivity map in the first image domain and the frequency domain. They are based on the idea
channel, and the second dominant in the second channel, etc. Different that CNN’s operating on k-space and images exhibit different properties;
regions of k-space are trained separately in a multi-resolution approach, therefore, a combination of them might outperform them separately.
using a large database without the need for explicit coil sensitivity maps Typically, frequency domain subnetworks attempt to estimate the
or reference data. Where multiple contiguous slices are available, missing k-space samples, while image domain subnetworks attempt to
spatially adjacent slices can be used as multi-channel input to improve remove residual artefacts.
the accuracy; adaptive convolutional neural networks for k-space data Some cross-domain methods apply a single k-space completion step,
interpolation (ACNN-k-Space) [32]. followed by an image restoration step. This is the case of W-Net [46],
Alternative methods exploit the low-rank of the MRI signal, similarly which applies a frequency domain U-Net followed by an image domain
to ALOHA (annihilating filter based low-rank Hankel matrix) [33]. It has U-Net. Another example is the multi-domain CNN (MD-CNN) [47],
been shown that it is possible to train a U-Net using a large database, which uses a ResNet architecture for the k-space subnetwork and a U-
which exploits the efficient signal representation in k-space [34,35]. Net for the image subnetwork in a dynamic imaging context.
Other machine learning approaches are more closely related to the Other hybrid methods use a cascading approach. In KIKI-Net [48],
parallel imaging technique, GRAPPA. RAKI (Scan-specific robust artifi­ alternating k-space and image deep CNN’s are applied, separated by the
cial-neural-networks for k-space interpolation) [36] is trained on the Fourier transform (the network architecture operates on k-space, image-
ACS data to learn the non-linear relationship between coil elements. space, k-space, and then image-space sequentially). Another proposal,
Therefore, RAKI does not require a large training database, but instead the hybrid cascade [49], is based on a deep cascade of CNN’s (DC-CNN)
the neural networks are trained using the ACS data from the scan itself. [47,50]. However, unlike the original DC-CNN, it uses both k-space and
This means that the network must be trained for each scan. The resulting image CNN’s. The W-Net method was extended to WW-Net [51] by
RAKI networks have been shown to lead to a reduction in noise ampli­ cascading more U-Net networks. This work also suggests that dual-
fication compared to GRAPPA. The use of arbitrary sampling patterns domain networks may be most advantageous in multi-channel set­
are possible with the use of self-consistent RAKI (sRAKI) [37]. Other tings, where the k-space correlations between coils can be efficiently
advances includes residual RAKI (rRAKI) [38], which uses a residual exploited by k-space domain networks.
CNN to simultaneously approximate a linear convolutional operator and A different approach is that of the dual-domain deep lattice network
a non-linear component that compensates for noise amplification arte­ (DD-DLN) [52]. This method employs two DC-CNN’s, one for each
facts. Furthermore, RAKI has been combined with LORAKS (Low-rank domain, which run in parallel rather than sequentially. In order to share
modelling of local k-space neighborhoods) [39], in a method called information between both subnetworks, at the end of each block the
LORAKI [40]. LORAKI uses an auto-calibrated scan-specific convolu­ outputs are concatenated (after transforming to the relevant domain)
tional RNN, which simultaneously incorporates support, phase, and and fed into the next block in the cascade.
parallel imaging constraints. Another proposal is the Dual-Encoder-Unet [53], which unlike other
methods is not based on single-domain subnetworks. Instead, a modified
2.3. Direct mapping U-Net operates simultaneously on both domains, which is achieved by
adding a second encoder path. One is fed the measured k-space, while
A few studies have shown the possibility of directly learning the the other is fed the zero-filled reconstructed images. The features from

82
J. Montalt-Tordera et al. Physica Medica 83 (2021) 79–87

both paths are combined via concatenation and fed into a single decoder struction methods define data consistency in the least-squares sense,
path, which produces the reconstructed image. assuming that measurement noise is normally distributed:
Finally, hybrid methods may operate on domains other than the
1
image and the k-space domain. This is the case of IKWI-Net [54], which f (Ax, y) = ‖Ax − y‖22 (5)
2
also includes a subnetwork in the wavelet domain (sequentially utilizing
CNN’s in the image domain, k-space, wavelet domain and image There exists more variability in the use of regularization functions.
domain). Some of the earliest approaches consider a regularization term of the
form g(x) = R (Dx), which contains an explicit sparsifying transform D
and a sparsity-promoting function R . This formulation is similar to
2.5. Unrolled optimization compressed sensing, where D might be the wavelet transform or the
finite difference operator, and R would typically be the l 1 -norm. In
Unrolled optimization methods are inspired by iterative optimiza­ unrolled optimization, these terms can be learned rather than manually
tion algorithms used in compressed sensing MRI. The idea is to unroll the designed. ADMM-Net [63,65], VarNet (Variational Network) [55] and
iterations of such an algorithm to an end-to-end neural network, map­ TVINet (Total Variation Inspired Network) [64], which unroll ADMM,
ping the measured k-space to the corresponding reconstructed image. GD and PDHG, respectively, use this formulation. All three explicitly
Then image transforms, sparsity-promoting functions, regularization learn linear sparsifying transforms D, parameterized by convolutional
parameters and update rates can be treated as either explicitly or layers, and non-linear sparsity-promoting functions R . The latter are not
implicitly trainable and fitted to a training dataset using back- learned directly, but rather implicitly through their proximal operators.
propagation. This has three advantages with respect to classic optimi­ Different parameterizations are used: ADMM-Net uses piecewise linear
zation. First, learned parameters may be better adapted to image char­ functions, VarNet uses radial basis functions, and TVINet uses a CNN.
acteristics than hand-engineered ones. Second, it avoids the need for Another method, ISTA-Net (Iterative Shrinkage-Thresholding Algo­
manual tuning, which is not a trivial process. Finally, reconstruction is rithm) [58], uses the regularizer λ‖D(x)‖1 , where D is a non-linear
faster, because such learned iterative schemes are trained to produce sparsifying transform (two convolutional layers separated by a ReLU
results with fewer iterations. activation). In this case, the sparsity-promoting function is not learned,
Several optimization algorithms have so far been successfully but fixed to be the l 1 -norm.
unrolled into neural networks. These include gradient descent (GD) Another class of methods, inspired by image restoration approaches
[55], proximal gradient descent (PGD) [50,56,57], the iterative (see section 2.1), use the regularizer g(x) = ‖x − C (x)‖22 , designed to
shrinkage-thresholding algorithm (ISTA) [58], the alternating minimi­ formulate an explicit image de-noising problem. Here C is an operator
zation algorithm (AMA) [59–61], the alternating direction method of that removes noise and aliasing artefact from an image. As a result, the
multipliers (ADMM) [62,63], and the primal dual hybrid gradient overall term g(x) is a noise estimator. Naturally, the operator C is
(PDHG) [64]. All unrolled methods solve some form of the following complex and unknown, but can be learned by a CNN. Methods using this
optimization problem: approach include DC-CNN (a Deep Cascade of CNN’s) [50] and MoDL
(Model-Based Deep Learning) [59].
x = argminf (Ax, y) + g(x)
̂
x
(4) Some work has combined several regularizers in the same network. It
is the case of MoDL-SToRM (MoDL with SmooThness regularization on
where f(Ax, y) is a generic data consistency term, which ensures that the manifolds) [66], which combines a learned noise estimator with a fixed
solution x agrees with the observations y, and g(x) is a generic regula­ SToRM regularizer. The first operates as has just been discussed, while
rization term which incorporates prior information. The definitions of f the latter ensures that the reconstructed dynamic sequence lies in a
and g, together with the optimization strategy, determine the funda­ smooth low-dimensional manifold.
mental structure of the resulting neural network. Several approaches are Finally, some methods do not constrain the formulation of the reg­
outlined hereafter. A summary of the techniques described is presented ularizer. Instead, they consider a generic term R (x), and use a CNN to
in Table 1, which the reader is encouraged to use for reference. estimate its proximal mapping directly. This is the case of R-GANCS
Like their compressed sensing counterparts, most unrolled recon­

Table 1
Summary of unrolled optimization methods. A selection of unrolled optimization methods and their fundamental characteristics: optimization algorithm, data con­
sistency term, regularization term and learned parameters. Regularization parameters λ, as well penalty parameters, update rates, step sizes, etc., are learned too, but
omitted from the learned parameters column for conciseness. ADMM: alternating direction method of multipliers; GD: gradient descent; PGD: proximal gradient
descent; ISTA: iterative shrinkage-thresholding algorithm; AMA: alternating minimization algorithm; PDHG: primal dual hybrid gradient method; Conv: convolutional
layer; ReLU: rectified linear unit; GAN: generative adversarial network; CNN: convolutional neural network; CRNN: convolutional recurrent neural network. In the
regularization term for MoDL-SToRM, tr denotes the trace operator and L denotes the graph Laplacian operator.
Ref. Name Algorithm f g Learned parameters

1 ∑L
[63,65] ADMM-Net ADMM l=1 λl R (Dl x)
Dl (Conv), R (implicit, proximal operator, piecewise linear function)
‖Ax − y‖22
[55] VarNet GD 2 Dl (Conv), g (implicit, first order derivative, radial basis functions)
[58] ISTA-Net PGD (ISTA) λ‖D(x)‖1 D(Conv-ReLU-Conv)
[57] R-GANCS PGD R (x) R (implicit, proximal operator, GAN).
[56] HC-PGD PGD R (x) R (implicit, proximal operator, CNN).
[50] DC-CNN PGD λ‖x − C (x)‖22 C (CNN)
[59] MoDL AMA C (CNN)
( )
[66] MoDL-SToRM AMA λ1 ‖x − C (x)‖22 + λ2 tr xT Lx C (CNN)
[60] VS-Net AMA R (x) R (implicit, proximal operator, CNN).
[61] CRNN-MRI AMA R (x) R (implicit, proximal operator, CRNN).
[64] TVINet PDHG R (Dx) D(conv), R (CNN)
[67] PDHG-CSNet PDHG R (x) R (implicit, proximal operator, CNN).
[67] CP-Net, PD-Net PDHG F (Ax, y) R (x) F , R (implicit, proximal operators, CNN’s)

83
J. Montalt-Tordera et al. Physica Medica 83 (2021) 79–87

[57], CRNN-MRI (convolutional recurrent neural network) [61], VS-Net as along the dynamic dimension. In HC-PGD [56], dense connections are
(Variable Splitting Network) [60], HC-PGD (history cognizant PGD) added across steps in order to accelerate convergence and improve
[56] and PDHG-CSNet (primal dual hybrid gradient, compressive overall performance.
sensing) [67]. As a result of all this variability, unrolled methods may rely on
Although most methods use the l 2 -norm as the data consistency training data to different extents. Model-driven approaches might use a
function, other approaches have been proposed. In CP-Net (Chambolle- more constrained formulation and shallower priors, and have a smaller
Pock Net) [67] the data consistency term is relaxed to a generic form number of parameters [55,58]. Such methods may be easier to interpret
f(x) = F (Ax, y), where F is learned. Similar relaxations have been and validate, and may require less training data. As constraints are
proposed for ADMM and ISTA-based unrolled networks [67,68]. This relaxed and deeper priors are used [59,67], methods become more data-
may increase the generality of the models, at the cost of looser data driven and may have more parameters. Such methods have a looser
consistency guarantees. connection to the physics-driven model, but given enough training data
Some methods [67–69], such as PD-Net (Primal Dual Net), also they might outperform more model-driven approaches on a particular
suggest relaxing the update rules, which are otherwise determined by task, due to increased representational power.
the optimization algorithm, to further increase the generality of the
model. A related approach is that taken in recurrent inference machines 3. Unsupervised machine learning
(RIMs) [70–72]. These parameterize the optimization process as a
recurrent neural network, where each “time step” is an iteration of the In unsupervised learning the ML algorithm looks to find patterns in
optimizer. RIMs learn the optimizer itself along with the prior; therefore, data without the need for any ground-truth data or user guidance. This is
unlike other approaches in this section, they are not based in any particularly challenging in the field of MRI reconstruction. It has been
particular optimization algorithm. The underlying idea is that a shown that state-of-the-art unsupervised learning techniques are
specialized data-driven optimizer might outperform hand-designed currently unable to achieve as good image quality as supervised learning
ones. Unrolled optimization methods can incorporate parallel imaging techniques [73,74]. However, in applications where ground-truth fully
by making coil sensitivity operators a part of the image formation model sampled datasets are unavailable and difficult or impossible to acquire
A. Some of the approaches outlined here have done so [56,59,60,63], (e.g. 4D flow), unsupervised learning techniques provide a promising
while others have worked in a single-coil context [50,58,61,65,67] alternative.
The different formulations and optimization algorithms lead to sig­ Unsupervised learning has been used to train image restoration
nificant variability in the resulting network architectures, which cannot methods to remove noise from MRI images (see section 2.1) using only
be covered in detail in this review. However, some structures are found noisy training data; examples include Noise2Noise [75] and regulari­
often. For example, proximal gradient methods [50,56,57] and alter­ zation by artifact-removal (RARE) [76]. Additionally, unsupervised
nating minimization methods with quadratic penalty splitting [59,61] learning is used in DeepResolve [77], in which a 3D cascade of con­
map naturally to alternating blocks in the resulting neural network: a volutional filters is trained to perform super-resolution (see Section 2.1).
model-driven data consistency block and a learned prior block (see Other unsupervised approaches which have shown promise, are al­
Fig. 3). Augmented Lagrangian methods, such as ADMM-Net [62], gorithms which exploit image sparsity, similarly to compressive sensing.
exhibit in addition an update block for the Lagrange multiplier. These simultaneously reconstruct the image and learn dictionaries or
All unrolled methods are ultimately deep neural networks, with the sparsifying transforms for image patches (also called blind compressed
important difference, with respect to other deep learning approaches, sensing) [78,79]. A further extension to this is Deep Basis Pursuit (DBP),
that their architecture is informed by a physics-driven model. There are which uses known noise statistics for each data set. This unrolled opti­
also important differences with respect to traditional optimization mization alternates between auto-encoder CNN layers and data consis­
methods, besides the obvious data-driven design. For example, they are tency constraint of basis pursuit de-noising [74]. Actual data consistency
often truncated to a fixed number of steps (iterations), and trained in an has also been used by cross-validation [80].
end-to-end fashion, with backpropagation across steps. They may share Generative adversarial networks have been used to enforce data
weights across steps [59,61], or each step may have its own weights consistency in unsupervised learning. Here, a conditional GAN is used to
[50,55], thus imparting different behavior to different steps. There may directly learn the mapping from k-space to image domain [73], where
be additional components which do not have an immediate optimization the generator network outputs an image (from undersampled k-space
equivalent, often borrowed from the rich body of deep learning litera­ data), and the discriminator network tries to differentiate between the
ture. For example, in R-GANCS [57], GAN’s are used to enhance the original k-space and a randomly undersampled k-space created from the
perceptual quality of the reconstructed images. In CRNN-MRI [61], generated image. GAN’s have also been used to learn the probability
recurrent units are used to exploit redundancies across iterations as well distribution of uncorrupted MRI data in an unsupervised manor, and

Fig. 3. Example of an unrolled optimization deep

neural network, with an alternating structure con­
taining data consistency (DC) blocks and prior (P)
blocks. DC blocks implement a gradient descent or
proximal mapping step to minimize the data con­
sistency term. They use the system matrix A and the
original k-space measurements, and may use coil
sensitivities in multi-channel settings. P blocks
implement the proximal mapping of the regulariza­
tion term and are learned by a CNN. The exact CNN
architectures vary between methods. Note that this is
not an accurate representation of all unrolled net­
works, but it shows commonly found features and is
the basic backbone of several of the methods
presented.

84
J. Montalt-Tordera et al. Physica Medica 83 (2021) 79–87

provide implicit priors for iterative reconstruction approaches [81]. Unfortunately, comparisons were made against a CS reconstruction,
Superior image quality may be achieved by also allowing the generating rather than a gold-standard technique, but showed clinically acceptable
network to learn its range space with respect to the measured data [82]. visual scores, with no significant difference in terms of mean vessel di­
ameters for six out of seven standardized locations in the thoracic aorta.
4. Clinical implications In another study, coronary artery length has been measured from ML
reconstructed 3D angiographic data (using a multi-scale variational
Despite the number of publications showing technical advances in neural network, see section 2.5) [85]. They showed negligible differ­
machine learning for MRI reconstruction, many publications do not ences in terms of quantitative vessel sharpness and coronary length,
demonstrate clinical utility. Instead performance of the resulting compared to a fully-sampled scan in 8 healthy subjects.
network is often evaluated using quantitative metrics generated from Myocardial scar quantification has been performed for 3D late gad­
synthetic data, including MSE, MAE, Root Mean-Squared Error (RMSE), olinium enhancement (LGE) MRI data reconstructed with a ML de-
Peak signal-to-noise ratio (pSNR) and Structural Similarity Index aliasing reconstruction (see Section 2.1) [86]. Unfortunately, the study
(SSIM). However, these metrics do not agree well with expert radiolo­ compared the ML reconstructed data against a CS reconstruction, rather
gists in ascertaining image quality, and ultimately diagnostic confidence than a gold-standard technique, however an excellent correlation in scar
[5]. In addition, real data may not perform as well as synthetic data, extent was observed (with a per-patient scar percentage error was 0.17
therefore demonstration in prospective data sets is essential. In order to ± 1.49%).
move towards clinical translation, it is necessary to evaluate qualitative Flow quantification has been calculated from 2D phase contrast data
image quality, diagnostic scoring and measurement of quantitative in 14 subjects, with a k-space interpolation ML reconstruction (see
clinical metrics (against reference standard imaging techniques) from section 2.2) [15]. Unfortunately, the data was retrospectively under­
prospectively acquired data reconstructed using ML. sampled, however the flow waveforms and flow volumes were seen to
There have been a small number of clinical validation studies of ML agree well with fully-sampled data, although the acceleration rates were
reconstructions, in particular within cardiovascular MRI. In one study, low (x2, x3 and x5). Another study extended this to 4D flow using a deep
real-time acquisition of 2D cine data was achieved using a radially 13x variational neural network to perform an unrolled reconstruction (see
undersampled acquisition, with a ML de-aliasing reconstruction (see Section 2.5) [87]. The resultant network was tested on prospectively on
section 2.1) [14]. After training of the network, prospective data was 7 healthy subjects, and compared to a gold-standard technique, with
acquired in 10 patients with Congenital Heart Disease (CHD) and good agreement in terms of peak-velocities and peak-flow estimates.
reconstructed using the ML network. Qualitative image scoring
(myocardial delineation, motion fidelity, and artefact) and clinical 5. Current limitations
measures of left and right ventricular volumes were compared to those
from clinical gold-standard images. No statistically significantly differ­ Raw MRI data is complex-valued, however many machine learning
ences were found in qualitative image quality or left ventricular volumes frameworks do not use complex convolutions or complex activation
(EDV, end diastolic volume; ESV, end systolic volume; and EF, ejection functions. Some studies just use magnitude data (particularly in image
fraction), with a small underestimation of right ventricular end systolic restoration methods), whereas others train separate networks for the
volume (bias − 1.1 mL). This study demonstrated a reduction in total magnitude and phase data [88], or may separate the real and imaginary
scan time from ~279 s for gold-standard acquisition to just ~18 s, where parts into two separate channels [34,50,55]. These approaches do not
the ML reconstruction was >5× faster than a CS reconstruction of the necessarily maintain the phase information of the data. Development of
same data. complex-valued networks remains an area of active research [89–91].
Another study quantified left ventricular volumes in 20 healthy However, PyTorch has recently (year: 2020) introduced full complex
subjects and 15 patients with suspected cardiovascular disease, from a value support, which means that more studies may use complex-valued
3D CINE sequence with an unrolled ML network: CINENet [83] (which data in the future.
resembles a proximal gradient algorithm with sparsity-learning and data Many studies only consider single channel data, whereas raw data is
consistency steps). This also found good agreement in LV function ESV, normally acquired from multiple coils. Some studies handle multi-coil
EDV and EF compared to clinical gold-standard images, enabling 3D data without additional coil-sensitivity information or ACS lines [92],
CINE data to be acquired in less than 10 s scan with ~5 s reconstruction whereas others learn the coil weighting from ACS lines in training [36],
time. and some feed pre-calculated coil sensitivity maps into the network
In another clinical validation paper, vessel diameters, diagnostic [55].
accuracy and diagnostic confidence were assessed from 3D whole-heart There is a question about how specific a network needs to be. Even
images with a single volume super-resolution ML reconstruction (see where imaging is fixed to a specific anatomy, the image quality can be
section 2.1) [27]. Prospective data was acquired in 40 patients with variable. This may be due to different hardware (including field strength
CHD, and compared to results from clinical gold-standard images. and coils), the use of different protocols (including different imaging
Qualitative image scoring showed super-resolved images were similar to contrasts, acquisition trajectories, flip angles, bandwidth and pre-
high-resolution data (in terms of edge sharpness, residual artefacts and pulses), patient-specific variation (including different field-of-view,
image distortion), with significantly better quantitative edge sharpness matrix size, phase-encoding direction), and artefacts (e.g. from patient
and signal-to-noise ratio. Vessel diameters measurements showed no motion). There may also be great variability in the prescribed scan
significant differences and no bias was found in the super-resolution planes, as well as in the underlying anatomy across different diseases. As
measurements in any of the great vessels. However, a small but signif­ most articles report their results on private data sets, it is difficult to
icant for the underestimation was found in coronary artery diameter compare the methods and assess their robustness and generalizability.
measurements from super-resolution data. Diagnostic scoring showed Currently there are only a relatively small number of publicly
that although super-resolution did not improve accuracy of diagnosis available data sets, and these are often very specific. These include (but
compared to low-resolution data, it did improve diagnostic confidence. are not limited to) raw k-space data sets; mridata.org, NYU fastMRI [93]
This study demonstrated a ~3x speed-up in acquisition compared to and Calgary-Campinas-359 [94], as well as DICOM imaging data sets;
high-resolution data (173 s vs 488 s), where super-resolution recon­ UK Biobank [95], Hunan Connectome Project [96], The Montreal
struction took <1 s per volume. Neurological Institute’s Brain Images of Tumors for Evaluation (NTI
Vessel diameters have also been quantified from four-dimensional BITE) [97] and OASIS-3 [98]. The availability of these datasets enables
non-contrast MRI angiography data, with a ML de-aliasing reconstruc­ development of novel DL image reconstruction frameworks, as well as
tion (see section 2.1) in 14 patients with thoracic aortic disease [84]. making it possible to benchmark and compare networks in the same

85
J. Montalt-Tordera et al. Physica Medica 83 (2021) 79–87

setting [99]. [18] Quan TM, Nguyen-Duc T, Jeong W. Compressed sensing MRI reconstruction using
a generative adversarial network with a cyclic loss. IEEE Trans Med Imaging
One of the main limitations to successful use of machine learning
2018;37:1488–97.
reconstructions in MRI is the lack of integration into the clinical envi­ [19] Mardani M, Gong E, Cheng JY, Vasanawala SS, Zaharchuk G, Xing L, et al. Deep
ronment. This means that currently reconstructions are performed off- generative adversarial neural networks for compressive sensing MRI. IEEE Trans
line and are not available immediately to the clinician. Manufacturers Med Imaging 2019;38:167–79.
[20] He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition.
have been working to integrate machine learning frameworks into Proceedings of the IEEE conference on computer vision and pattern recognition,
standard clinical pipelines. In addition open source frameworks which 2016. p. 770–8.
may be integrated into the scanner, such as Gadgetron [100], may also [21] Dong C, Loy CC, He K, Tang X. Image super-resolution using deep convolutional
networks. IEEE Trans Pattern Anal Mach Intell 2015;38:295–307.
enable translation of these techniques into the clinical environment. [22] Cherukuri V, Guo T, Schiff SJ, Monga V. Deep MR image super-resolution using
This would also enable large multi-site validation studies to be per­ structural priors. 2018 25th IEEE International Conference on Image Processing
formed, which is essential in building confidence in these techniques. (ICIP): IEEE; 2018. p. 410–4.
[23] Pham C-H, Ducournau A, Fablet R, Rousseau F. Brain MRI super-resolution using
deep 3D convolutional networks. 2017 IEEE 14th International Symposium on
6. Conclusion Biomedical Imaging (ISBI 2017): IEEE; 2017. p. 197–200.
[24] Masutani EM, Bahrami N, Hsiao A. Deep learning single-frame and multiframe
super-resolution for cardiac MRI. Radiology 2020;295:552–61.
Deep learning approaches have been shown to provide a huge po­ [25] Chen Y, Xie Y, Zhou Z, Shi F, Christodoulou AG, Li D. Brain MRI super resolution
tential for the future of magnetic resonance image reconstruction. There using 3D deep densely connected neural networks. 2018 IEEE 15th International
has been an explosion of research in this field over the last five years, Symposium on Biomedical Imaging (ISBI 2018): IEEE; 2018. p. 739–42.
[26] Du J, Wang L, Gholipour A, He Z, Jia Y. Accelerated super-resolution MR image
across many different approaches. More robust testing and large-scale
reconstruction via a 3D densely connected deep convolutional neural network.
demonstration on prospectively acquired clinical data is required to 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM):
build confidence in these techniques. IEEE; 2018. p. 349–55.
[27] Steeden JA, Quail M, Gotschy A, Mortensen KH, Hauptmann A, Arridge S, et al.
Rapid whole-heart CMR with single volume super-resolution. J Cardiovasc Magn
Declaration of Competing Interest Reson 2020;22:56.
[28] Li Y, Sixou B, Peyrin F. A review of the deep learning methods for medical images
The authors declare that they have no known competing financial super resolution problems. IRBM 2020.
[29] Cheng JY, Mardani M, Alley MT, Pauly JM, Vasanawala SS. DeepSPIRiT:
interests or personal relationships that could have appeared to influence Generalized parallel imaging using deep convolutional neural networks. Annual
the work reported in this paper. Meeting of the International Society of Magnetic Resonance in Medicine, 2018.
[30] Lustig M, Pauly JM. SPIRiT: Iterative self-consistent parallel imaging
reconstruction from arbitrary k-space. Magn Reson Med 2010;64:457–71.
References [31] Huang F, Vijayakumar S, Li Y, Hertel S, Duensing GR. A software channel
compression technique for faster reconstruction with many channels. Magn Reson
[1] Pruessmann KP, Weiger M, Scheidegger MB, Boesiger P. SENSE: Sensitivity Imaging 2008;26:133–41.
encoding for fast MRI. MRM 1999;42:952–62. [32] Du T, Zhang H, Song HK, Fan Y. Adaptive convolutional neural networks for k-
[2] Griswold MA, Jakob PM, Heidemann RM, Nittka M, Jellus V, Wang J, et al. space data interpolation in fast magnetic resonance imaging. arXiv:200601385,
Generalized autocalibrating partially parallel acquisitions (GRAPPA). MRM 2002; 2020.
47:1202–10. [33] Jin KH, Lee D, Ye JC. A general framework for compressed sensing and parallel
[3] Lustig M, Donoho DL, Santos JM, Pauly JM. Compressed Sensing MRI. IEEE MRI using annihilating filter based low-rank Hankel matrix. IEEE Trans Comput
Signal Process Mag 2008;25:72–82. Imaging 2016;2:480–95.
[4] LeCun Y, Bengio Y. Convolutional networks for images, speech, and time series. [34] Han Y, Sunwoo L, Ye JC. k-Space Deep Learning for Accelerated MRI. IEEE Trans
The handbook of brain theory and neural networks: MIT Press; 1998. p. 255–8. Med Imaging 2019;39:377–86.
[5] Knoll F, Murrell T, Sriram A, Yakubova N, Zbontar J, Rabbat M, et al. Advancing [35] Cha E, Kim EY, Ye JC. k-space deep learning for parallel mri: Application to time-
machine learning for MR image reconstruction with an open competition: resolved mr angiography. arXiv:180600806, 2018.
Overview of the 2019 fastMRI challenge. Magn Reson Med 2020;84:3054–70. [36] Akçakaya M, Moeller S, Weingärtner S, Uğurbil K. Scan-specific robust artificial-
[6] Ghodrati V, Shao J, Bydder M, Zhou Z, Yin W, Nguyen K-L, et al. MR image neural-networks for k-space interpolation (RAKI) reconstruction: Database-free
reconstruction using deep learning: evaluation of network structure and loss deep learning for fast imaging. Magn Reson Med 2019;81:439–53.
functions. Quant Imaging Med Surg 2019;9:1516–27. [37] Hosseini SAH, Zhang C, Weingärtner S, Moeller S, Stuber M, Ugurbil K, et al.
[7] Zhao H, Gallo O, Frosio I, Kautz J. Loss functions for neural networks for image Accelerated coronary MRI with sRAKI: A database-free self-consistent neural
processing. arXiv preprint arXiv:151108861, 2015. network k-space reconstruction for arbitrary undersampling. PLoS ONE 2020;15:
[8] Zhang R, Isola P, Efros AA, Shechtman E, Wang O. The unreasonable effectiveness e0229418.
of deep features as a perceptual metric. Proceedings of the IEEE conference on [38] Zhang C, Moeller S, Weingärtner S, Uğurbil K, Akçakaya M. Accelerated MRI
computer vision and pattern recognition, 2018, p. 586–95. using residual RAKI: Scan-specific learning of reconstruction artifacts. Annual
[9] Wang S, Su Z, Ying L, Peng X, Zhu S, Liang F, et al. Accelerating magnetic Meeting of the International Society of Magnetic Resonance in Medicine, 2019.
resonance imaging via deep learning. In: 2016 IEEE 13th International [39] Haldar JP. Low-rank modeling of local k-space neighborhoods (LORAKS) for
Symposium on Biomedical Imaging (ISBI): IEEE; 2016. p. 514–7. constrained MRI. IEEE Trans Med Imaging 2013;33:668–81.
[10] Ronneberger O, Fischer P, U-net BT. Convolutional networks for biomedical [40] Kim TH, Garg P, Haldar JP. LORAKI: Autocalibrated recurrent neural networks
image segmentation. In: International Conference on Medical image computing for autoregressive MRI reconstruction in k-space. arXiv:190409390, 2019.
and computer-assisted intervention. Springer; 2015. p. 234–41. [41] Zhu B, Liu JZ, Cauley SF, Rosen BR, Rosen MS. Image reconstruction by domain-
[11] Lee D, Yoo J, Ye JC. Deep residual learning for compressed sensing MRI. 2017 transform manifold learning. Nature 2018;555:487–92.
IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017)2017. p. [42] Schlemper J, Oksuz I, Clough JR, Duan J, King AP, Schnabel JA, et al.
15–8. dAUTOMAP: Decomposing AUTOMAP to achieve scalability and enhance
[12] Sandino CM, Dixit N, Cheng JY, Vasanawala SS. Deep convolutional neural performance. arXiv:190910995, 2019.
networks for accelerated dynamic magnetic resonance imaging. preprint 2017. [43] Eo T, Shin H, Kim T, Jun Y, Hwang D. Translation of 1d inverse fourier transform
[13] Kofler A, Dewey M, Schaeffter T, Wald C, Kolbitsch C. Spatio-temporal deep of k-space to an image based on deep learning for accelerating magnetic
learning-based undersampling artefact reduction for 2D Radial Cine MRI with resonance imaging. International Conference on Medical Image Computing and
limited training data. IEEE Trans Med Imaging 2020;39:703–17. Computer-Assisted Intervention: Springer; 2018. p. 241–9.
[14] Hauptmann A, Arridge S, Lucka F, Muthurangu V, Steeden JA. Real-time [44] Eo T, Shin H, Jun Y, Kim T, Hwang D. Accelerating Cartesian MRI by domain-
cardiovascular MR with spatio-temporal artifact suppression using deep transform manifold learning in phase-encoding direction. Med Image Anal 2020;
learning–proof of concept in congenital heart disease. Magn Reson Med 2019;81: 101689.
1143–56. [45] Oh C, Kim D, Chung J-Y, Han Y, Park H. In: ETER-net: End to end MR image
[15] Nath R, Callahan S, Singam N, Stoddard M, Amini AA. Accelerated Phase Contrast reconstruction using recurrent neural network. Cham: Springer International
Magnetic Resonance Imaging via Deep Learning. 2020 IEEE 17th International Publishing; 2018. p. 12–20.
Symposium on Biomedical Imaging (ISBI), 2020, p. 834–8. [46] Souza R, Frayne R. A hybrid frequency-domain/image-domain deep network for
[16] Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, et al. magnetic resonance image reconstruction. In: 2019 32nd SIBGRAPI Conference
Generative adversarial networks. Adv Neural Inf Process Syst 2014;3. on Graphics, Patterns and Images (SIBGRAPI); 2019. p. 257–64.
[17] Yang G, Yu S, Dong H, Slabaugh G, Dragotti PL, Ye X, et al. DAGAN: Deep de- [47] El-Rewaidy H, Fahmy AS, Pashakhanloo F, Cai X, Kucukseymen S, Csecs I, et al.
aliasing generative adversarial networks for fast compressed sensing MRI Multi-domain convolutional neural network (MD-CNN) for radial reconstruction
reconstruction. IEEE Trans Med Imaging 2018;37:1310–21. of dynamic cardiac MRI. Magnetic Resonance in Medicine. n/a.

86
J. Montalt-Tordera et al. Physica Medica 83 (2021) 79–87

[48] Eo T, Jun Y, Kim T, Jang J, Lee H-J, Hwang D. KIKI-net: cross-domain [76] Liu J, Sun Y, Eldeniz C, Gan W, An H, Kamilov US. RARE: Image reconstruction
convolutional neural networks for reconstructing undersampled magnetic using deep priors learned without groundtruth. IEEE J Sel Top Signal Process
resonance images. Magn Reson Med 2018;80:2188–201. 2020;14:1088–99.
[49] Souza R, Lebel RM, Hybrid FRA. Dual domain, cascade of convolutional neural [77] Chaudhari AS, Fang Z, Kogan F, Wood J, Stevens KJ, Gibbons EK, et al. Super-
networks for magnetic resonance image reconstruction. In: Cardoso MJ, Aasa F, resolution musculoskeletal MRI using deep learning. Magn Reson Med 2018;80:
Ben G, Ender K, Ipek O, Gozde U, editors. Proceedings of The 2nd International 2139–54.
Conference on Medical Imaging with Deep Learning. Proceedings of Machine [78] Ravishankar S, Lahiri A, Blocker C, Fessler JA. Deep dictionary-transform
Learning Research: PMLR; 2019. p. 437–46. learning for image reconstruction. 2018 IEEE 15th International Symposium on
[50] Schlemper J, Caballero J, Hajnal JV, Price AN, Rueckert D. A Deep Cascade of Biomedical Imaging (ISBI 2018), 2018. p. 1208–12.
convolutional neural networks for dynamic MR image reconstruction. IEEE Trans [79] Ravishankar S, Bresler Y. Data-driven learning of a union of sparsifying
Med Imaging 2018;37:491–503. transforms model for blind compressed sensing. IEEE Trans Comput Imaging
[51] Souza R, Bento M, Nogovitsyn N, Chung KJ, Loos W, Lebel RM, et al. Dual-domain 2016;2:294–309.
cascade of U-nets for multi-channel magnetic resonance image reconstruction. [80] Yaman B, Hosseini SAH, Moeller S, Ellermann J, Uğurbil K, Akçakaya M. Self-
Magn Reson Imaging 2020;71:140–53. supervised physics-based deep learning MRI reconstruction without fully-sampled
[52] Sun L, Wu Y, Shu B, Ding X, Cai C, Huang Y, et al. A dual-domain deep lattice data. In: 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI):
network for rapid MRI reconstruction. Neurocomputing 2020;397:94–107. IEEE; 2020. p. 921–5.
[53] Jethi AK, Murugesan B, Ram K, Sivaprakasam M. Dual-Encoder-Unet For Fast Mri [81] Bora A, Jalal A, Price E, Dimakis AG. Compressed sensing using generative
Reconstruction. 2020 IEEE 17th International Symposium on Biomedical Imaging models. In: Doina P, Yee Whye T, editors. Proceedings of the 34th International
Workshops (ISBI Workshops)2020. p. 1–4. Conference on Machine Learning. Proceedings of Machine Learning Research:
[54] Wang Z, Jiang H, Du H, Xu J, Qiu B. IKWI-net: A cross-domain convolutional PMLR; 2017. p. 537–46.
neural network for undersampled magnetic resonance image reconstruction. [82] Narnhofer D, Hammernik K, Knoll F, Pock T. Inverse GANs for accelerated MRI
Magn Reson Imaging 2020;73:1–10. reconstruction. SPIE 2019.
[55] Hammernik K, Klatzer T, Kobler E, Recht MP, Sodickson DK, Pock T, et al. [83] Küstner T, Fuin N, Hammernik K, Bustin A, Qi H, Hajhosseiny R, et al. CINENet:
Learning a variational network for reconstruction of accelerated MRI data. Magn deep learning-based 3D cardiac CINE MRI reconstruction with multi-coil
Reson Med 2018;79:3055–71. complex-valued 4D spatio-temporal convolutions. Sci Rep 2020;10:13710.
[56] Hosseini SAH, Yaman B, Moeller S, Hong M, Akçakaya M. Dense recurrent neural [84] Haji-Valizadeh H, Shen D, Avery RJ, Serhal AM, Schiffers FA, Katsaggelos AK,
networks for accelerated MRI: History-cognizant unrolling of optimization et al. Rapid reconstruction of four-dimensional MR angiography of the thoracic
algorithms. IEEE J Sel Top Signal Process 2020;14:1280–91. aorta using a convolutional neural network. Radiology: Cardiothoracic Imaging.
[57] Mardani M, Monajemi H, Papyan V, Vasanawala S, Donoho D, Pauly J. Recurrent 2020;2:e190205.
generative adversarial networks for proximal learning and automated [85] Fuin N, Bustin A, Küstner T, Oksuz I, Clough J, King AP, et al. A multi-scale
compressive image recovery. arXiv preprint arXiv:171110046, 2017. variational neural network for accelerating motion-compensated whole-heart 3D
[58] Zhang J, Ghanem B. ISTA-Net: Interpretable optimization-inspired deep network coronary MR angiography. Magn Reson Imaging 2020;70:155–67.
for image compressive sensing. 2018 IEEE/CVF Conference on Computer Vision [86] El-Rewaidy H, Neisius U, Mancio J, Kucukseymen S, Rodriguez J, Paskavitz A,
and Pattern Recognition, 2018. p. 1828–37. et al. Deep complex convolutional network for fast reconstruction of 3D late
[59] Aggarwal HK, Mani MP, Jacob M. MoDL: Model-based deep learning architecture gadolinium enhancement cardiac MRI. NMR Biomed 2020;33:e4312.
for inverse problems. IEEE Trans Med Imaging 2019;38:394–405. [87] Vishnevskiy V, Walheim J, Kozerke S. Deep variational network for rapid 4D flow
[60] Duan J, Schlemper J, Qin C, Ouyang C, Bai W, Biffi C, et al. In: VS-Net: Variable MRI reconstruction. Nature Machine Intell 2020;2:228–35.
splitting network for accelerated parallel MRI reconstruction. Cham: Springer [88] Lee D, Yoo J, Tak S, Ye JC. Deep residual learning for accelerated MRI using
International Publishing; 2019. p. 713–22. magnitude and phase networks. IEEE Trans Biomed Eng 2018;65:1985–95.
[61] Qin C, Schlemper J, Caballero J, Price AN, Hajnal JV, Rueckert D. Convolutional [89] Virtue P, Yu SX, Lustig M. Better than real: Complex-valued neural nets for MRI
recurrent neural networks for dynamic MR image reconstruction. IEEE Trans Med fingerprinting. 2017 IEEE International Conference on Image Processing (ICIP),
Imaging 2019;38:280–90. 2017. p. 3953–7.
[62] Yang Y, Sun J, Li H, Xu Z. ADMM-Net: A deep learning approach for compressive [90] Trabelsi C, Bilaniuk O, Zhang Y, Serdyuk D, Subramanian S, Santos J, et al. Deep
sensing MRI. arXiv:170506869, 2017. Complex Networks. arXiv:170509792.
[63] Yang Y, Sun J, Li H, Xu Z. ADMM-CSNet: A deep learning approach for image [91] Cole E, Cheng J, Pauly J, Vasanawala S. Analysis of deep complex-valued
compressive sensing. IEEE Trans Pattern Anal Mach Intell 2020;42:521–38. convolutional neural networks for MRI reconstruction.
[64] Zhang X, Lian Q, Yang Y, Su Y. A deep unrolling network inspired by total [92] Kwon K, Kim D, Park H. A parallel MR imaging method using multilayer
variation for compressed sensing MRI. Digital Signal Process 2020;107:102856. perceptron. Med Phys 2017;44:6209–24.
[65] Sun J, Li H, Xu Z. Deep ADMM-net for compressive sensing MRI. Adv Neural Inf [93] Knoll F, Zbontar J, Sriram A, Muckley MJ, Bruno M, Defazio A, et al. fastMRI: A
Process Syst 2016:10–8. publicly available raw k-space and DICOM dataset of knee images for accelerated
[66] Biswas S, Aggarwal HK, Jacob M. Dynamic MRI using model-based deep learning MR image reconstruction using machine learning. radiology: Artificial
and SToRM priors: MoDL-SToRM. Magn Reson Med 2019;82:485–94. intelligence. 2020;2:e190007.
[67] Cheng J, Wang H, Ying L, Liang D. In: Model learning: Primal dual networks for [94] Souza R, Lucena O, Garrafa J, Gobbi D, Saluzzi M, Appenzeller S, et al. An open,
fast MR imaging. Springer; 2019. p. 21–9. multi-vendor, multi-field-strength brain MR dataset and analysis of publicly
[68] Cheng J, Wang H, Zhu Y, Liu Q, Zhang Q, Su T, et al. Model-based Deep Medical available skull stripping methods agreement. NeuroImage 2018;170:482–94.
Imaging: the roadmap of generalizing iterative reconstruction model using deep [95] Sudlow C, Gallacher J, Allen N, Beral V, Burton P, Danesh J, et al. UK biobank: an
learning. arXiv:190608143, 2019. open access resource for identifying the causes of a wide range of complex
[69] Adler J, Öktem O. Learned Primal-dual Reconstruction. arXiv:170706474, 2017. diseases of middle and old age. PloS Med 2015;12:e1001779.
[70] Putzky P, Welling M. Recurrent inference machines for solving inverse problems. [96] Van Essen DC, Smith SM, Barch DM, Behrens TEJ, Yacoub E, Ugurbil K. The WU-
arXiv preprint arXiv:170604008, 2017. Minn human connectome project: An overview. NeuroImage 2013;80:62–79.
[71] Lønning K, Putzky P, Sonke J-J, Reneman L, Caan MWA, Welling M. Recurrent [97] Mercier L, Del Maestro RF, Petrecca K, Araujo D, Haegelen C, Collins DL. Online
inference machines for reconstructing heterogeneous MRI data. Med Image Anal database of clinical MR and ultrasound images of brain tumors. Med Phys 2012;
2019;53:64–78. 39:3253–61.
[72] Putzky P, Welling M. Invert to learn to invert. Adv Neural Inf Process Syst 2019: [98] LaMontagne PJ, Benzinger TL, Morris JC, Keefe S, Hornbeck R, Xiong C, et al.
446–56. OASIS-3: Longitudinal neuroimaging, clinical, and cognitive dataset for normal
[73] Cole EK, Pauly JM, Vasanawala SS, Ong F. Unsupervised MRI Reconstruction with aging and Alzheimer disease. medRxiv. 2019:2019.12.13.19014902.
Generative Adversarial Networks. arXiv preprint arXiv:200813065, 2020. [99] Ramzi Z, Ciuciu P, Starck J-L. Benchmarking MRI reconstruction neural networks
[74] Tamir JI, Yu SX, Lustig M. Unsupervised Deep Basis Pursuit: Learning inverse on large public datasets. Appl Sci 2020;10:1816.
problems without ground-truth data. arXiv preprint arXiv:191013110, 2019. [100] Hansen MS, Sørensen TS. Gadgetron: An open source framework for medical
[75] Lehtinen J, Munkberg J, Hasselgren J, Laine S, Karras T, Aittala M, et al. image reconstruction. Magn Reson Med 2013;69:1768–76.
Noise2noise: Learning image restoration without clean data. arXiv preprint arXiv:
180304189, 2018.

87