2021 9th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO)

Amity University, Noida, India. Sep 3-4, 2021

A Review on Generative Adversarial Networks

2021 9th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO) | 978-1-6654-1703-7/21/$31.00 ©2021 IEEE | DOI: 10.1109/ICRITO51393.2021.9596487

used for Image Reconstruction in Medical imaging

Robin kumar Dr Rahul malik
Computer Science Engineering, Computer Science Engineering,
Lovely Professional University Lovely Professional University
Punjab, India Punjab, India
[email protected] [email protected]

Abstract— In computer vision, Because of its capacity to using deep learning in medical imaging. [11] Deep learning
generate data, generative adversarial networks have gotten has the potential of image interpretation, image classification,
much interest. Generative Adversarial Network (GANs) opens object detection, segmentation, registration. These are some
different ways to overcome difficulties in medical image applications in medical imaging such as neurological
investigations like de-noising of the captured image, pathology (brain diseases), retinal fundus pathology, chest
reconstruction of captured medical image, segmentation of the pathology, breast cancer, cardiac pathology, abdominal and
captured medical image, synthesis of the captured image, musculoskeletal pathologies. The massive amount of labelled
detection and classification of processed medical image. GANs
data set is the main obstacle in the path of deep learning to be
great potentials in image restoration or reconstruction for
the best in class in numerous tasks of medical imaging and
different anatomy open a new research area. However, image
reconstruction for specific anatomy is still facing the challenge.
standard computer image processing. In this article, the main
This article consists of a recent literature review on the focus is on Adversarial generative networks and their
applications of GAN for the reconstruction of medical images. application in the reconstruction of medical images.
Different reconstruction methods in the field of medical Compared to different methodology, Reconstruction of a
imaging were explored in this paper thoroughly. We have medical image using GAN produces the detailed image in the
studied the foremost relevant publications. indigenous form. In GAN, a pure training methodology
employed for
Keywords— Generative adversarial network, GAN model,
Medical imaging Anatomy, Reconstruction.
balancing generator, that gives a network to create
minimally damaged or corrupt image and differentiator
I. INTRODUCTION discovers prediction of legitimacy [3].
Development in machine learning leads to new trends and II. GENERATIVE ADVERSARIAL NETWORKS
provide a platform for researchers to deploy their interest.
Medical imaging is critical for disease diagnosis and therapy GANs was proposed by Ian Goodfellow in 2014. This
assistance. In medical imaging, the reconstruction process is concept of GAN is relatively new with continuous progress
the fundamental component. The primary goal of image [3].With the aid of deep learning approaches like CNNs,
reconstruction is to provide a clinical ready high-quality generative adversarial networks are a generative model for
image at low cost and less risk to the patient. Image creating new images. It is an unsupervised learning technique
restoration started in 1990. Earlier, the models are typically of machine learning. The inspiring force for GANs is the
mathematical and mostly handcrafted, designed by human game theory called the zero-sum game. It is a plan within
knowledge. Later the model is designed, which are both which two models are trained at the same time by the
handcrafted plus data-driven models. Around 1999 these adversarial method. It's a generative modal made up of both a
models are created. These models are learning-based generator and a discriminator. Figure 1 shows this model. In
handcrafted models. After handcrafted models, the next step GAN generator is made up of a neural network. The input to
is to derive knowledge from the properties of the data. Many the neural network is a random noise for basic GAN and
deep learning algorithms employ this principle, in which the transforms it into a sample from the model distribution [3].
network picks up information from the data set. In medical The discriminator is a neural network that distinguishes
imaging, the convolutional neural network is the most widely between output data point (fake) from the generator and
used model. With the help of filters, CNNs convert the input training data samples (real) [13]. The basic GAN models
to feature input for the next layer. CNN comes in lamplight consist of G the generator, D the discriminator, Pz(z)is the
after the advancement in computing technology. Around input noise, Pg(x) is generated distribution over the data set
2012, with the introduction of CNN, deep learning comes x.θd and θg are the parameters of discriminators and generator
into action. CNN's such as AlexNet, ResNet, and generative respectively. G(Z; θg) is the mapping between input random
adversarial networks have been introduced and used for noise to generative parameters, where G is a generator with
image reconstruction. These models are called deep learning multilayer perceptron. Whereas D(x; θd) is the binary output
models or data-driven models. With the introduction of new or we can say a classifier. The generator is minimizing
models, the community has noticed and transformed the function and D is the maximizing function.
existing handcrafted models into the new learning-based The training is frequently practiced by gradient-based
model. So for deep learning is the best choice in medical techniques. Unlike deep learning, Generative adversarial
imaging. The year 2012 marks the beginning of deep training is a new concept with significant recent research.
learning within machine vision. The vulnerability and This paper focuses on the usage of GANs in the medical
potential of deep learning, due to which researchers started industry for reconstructive work. We start with a quick

978-1-6654-1703-7/21/$31.00 ©2021 IEEE 1

Authorized licensed use limited to: KIIT University. Downloaded on June 07,2024 at 08:56:17 UTC from IEEE Xplore. Restrictions apply.
overview of GANs concepts. It is followed by work done generation method. In GAN, there's no management over
analysis in the medical image using GANs and also includes modes of the info to be generated [9]. The conditional GAN
work done for different pathologies. Summarize it with work changes that by adding the label as a further parameter to the
analysis on image reconstruction using GANs. generator and hopes that the corresponding image is
generated. The conditional GAN improves the generated
MinMax(D,G) = image but also helps in training stability. Generated image
Ex Pdata(x)[logD(x)] + Ez-Pz(z)[log(1 - D(G(z)))] features are preserved using conditional GAN. Some other
conditional GAN models are Markovian GAN proposed by
Markovian [13]. The VGG19 network serves as the
foundation for MGAN [22].The cGAN algorithm is often
used to convert one image to another. Proposed models like
Pix2Pix are used [17]. Pix2Pix model uses U-Net-based
architecture.

Fig. 1. vanila GAN

III. SOME GAN MODEL USED IN MEDICAL IMAGING

A. InfoGAN
InfoGAN was proposed in 2016 [21].In this case, the
input data is divided into two vectors, z and c. z are
incompressible, and c, known as the idle code and will focus
on the huge organized semantic highlights of the genuine
information distribution. The objective function of infoGAN
is minmaxVI (D, G) = V (D, G) -λLVI (c, Q) where lower
bound is LI and V (D, G) is function which called as an
objective function of GAN [21].

Fig. 3. cGAN

C. DCGAN
Deep Convolutional GAN (DCGAN) was proposed by
Radford in 2015 [23]. In DCGAN the generator and
discriminator both employ the deep convolutional network
and exploit hierarchic feature learning [23]. It's made up of a
convolution layer with fully connected layers with no max
pooling. This GAN architecture uses batch normalization and
leakyReLU activation function to improve training.
D. WGAN
Wasserstein-GAN was proposed by Arjovsky et al.2017.
They use Wasserstein distance as a divergence measure. It’s
an associate extension of GAN in which alternated training
method is used for better approximation. WGAN is
practically very easy to implement but has a problem of slow
optimization.
Fig. 2. InfoGAN E. LSGAN
In 2017 Mao proposed a new model called Least Squares
B. cGAN GAN which has similarities with WGAN but there is a
The Vanilla GAN has not had any management for the change in loss function to circumvent vanishing gradient.
particular image formation, Mirza and Osindero (2014)
planned a new model called conditional GAN to assimilate F. CycleGAN
extra data such as category marked data within the image In 2017 For image translation Zhihao and others
introduced the cycleGAN [18]. cycleGAN model provides

2
Authorized licensed use limited to: KIIT University. Downloaded on June 07,2024 at 08:56:17 UTC from IEEE Xplore. Restrictions apply.
the mapping between one domain to another domain. A. Reconstruction of medical image
Without paired example, cycleGAN can automatically train The constraints in medical imaging like radiation
from one image to another image conversion. The training hazardous to patients because x-rays can cause mutation in
process used in the particular model is unsupervised. genes and there is the tendency of causing carcinoma in
IV. APPLICATION OF GAN IN MEDICAL IMAGING relation to radiation dosage. Reduction in the radiation
dosage can amplify the noise and introduce the artifacts in
Using two aspects, we can apply GANs in medical the image to be reconstructed, which hampers the diagnostic
imaging. One is the generative aspect, where G generative details [11] [1].In order to obtain a high-resolution image,
learns and generates a new image. Second is discriminator D, MRI does have some limitations, and there is a need for
in which discriminator is thought to be learned previously extended capturing time [12]. Some other possible
from the traditional image and then employed as a detector limitations, such as small graphical coverage and movement
when exposed images with abnormality [1]. GANs can be artifacts, lead to lower image quality. Some other artifacts
applied in like foreign bodies present in the patients causing confusion
x classification to the examiner as they resemble the pathological findings.
These physical limitations deteriorate the quality of the
x detection clinical image that is to be interpreted due to
misinterpretation [12]. Table 1 shows the analysis of recent
x segmentation reconstruction work done on medical imaging. Fast MRI
x reconstruction medical Imaging without sacrificing the details is the main
problem. Fast MRI acquisition is highly desirable with
x image synthesis reduced motion artifacts. Reconstruction of the image is
possible using GAN because of its image generation
x registration
capability with realism in generated images. DAGAN
Figure 4 shows various GAN publications in the medical architecture was used earlier for MR reconstruction. In this
image field and figure 5 show the various applications of method, three-loss functions are used perceptual loss,
GAN in medical imaging. adversarial loss, pixel-wise loss. After DAGAN, 3D super-
resolution strategies with 3D convolutional layers are used.
Medical imaging several different modalities that allude
to a technique and procedures used to take pictures of
different sections of the human body for indicative and
treatment purposes, making it one of our most impressive
assets accessible to successful care about patients. Like we
have Magnetic resonance imaging, computed tomography,
Spine CT to MRI
Histopathology, RF Imaging, X-ray, Ultrasound sonography,
Dermatoscopy, Positron Emission Tomography,
Mammography. Figure 6 show GAN implementation in
different modalities. About 50 percent of work is done on
image synthesis that is cross-modality. Generation of the
image of one modality to another modality is the one that is
implemented the most like CT to MR. Synthesis of MR is
common because of the long acquisition time. Because of the
publicly accessible MR dataset, it is the most commonly
utilized medical imaging modality. Dataset of different MRI to CT
anatomy are available publicly but not for each anatomy
figure 7 shows work done according to different anatomy.
Figure 7 shows most of the work is done in brain imaging,
skin and eye because of publicly available datasets like
BRATS2017, ISIC2018, DEIVE.

Normal retinal fundus into Super resolution image

Fig. 4. GAN publications in Medical Image.

Skin lesion synthesis
Fig. 5. Applications of GAN in Medical Imaging.

3
Authorized licensed use limited to: KIIT University. Downloaded on June 07,2024 at 08:56:17 UTC from IEEE Xplore. Restrictions apply.
 V. DISCUSSION • CT may cause heath problem
in form of cancer.
In this paper, we have displayed various GAN-based Q.Yang et
• 2D fully-convolutional [20]
models and GAN ability in medical image reconstruction. 3. al.,2018
• Use perceptual loss function
Evidence suggests that GAN can be successfully used for • Used pre-trained deep CNN
different anatomy with different modalities. Most of the work like VGG
is done in the case of image synthesis. Some anatomy like • GAN based learning
skin, spine are less common because of the less availability architecture DA-GAN use U-
of dataset. In the case of reconstruction, images are NET architecture [5]
reconstructed because they include some artifacts and Yang et • Conditional model
distortion in the image. So the reconstruction of different 4.
al.,2018 • Used Content loss consist of
anatomy for MRI and CT is needed. In recent years, the pixel wise image domain MSE
application of GAN is Increasing significantly in various loss, fre-quency domin MSE
fields. We have various modalities in which we can apply loss and a percep- tual loss.
GAN but MRI is considered the one which is investigated the • 3D cGAN framework used [2]
most in GAN. The application of GAN in the MRI modality Wang et • U-NET architecture
5.
is the reduction in acquisition time. As we know that there is al.,2018 • Conditional input of PET
a need for a significant period spent on the acquisition [12]. medical image
So in minimizing the acquisition time, we can apply GAN for • Restoring HR image from
MRI modality. It will be beneficial to the patient as well as single LR [4].
for the operator. • Deep learning approach
Chen et mdcsrn using GAN
6.
al.,2018 • Divide a single densenet into
small shallow block called
multi level densely connected
structure
• A butterfly-formed system that
works on spine image [8]
Sekuboyina
• Method is on 2D with
7. A. Et
different rep- resentations.
al.,2018
• The btrfly network is consists
of an arm. we can say to view.
• Convert low dose CT image
into a reduced noise image.
• Data set is obtained from 28
Wolterink et
Fig. 6. GAN implementation in different modality.
patient’s for cardiac CT
8. al.,2017
• GAN with both the generator
and the discriminator is a CNN.
• Combine two-loss function
voxel and adversarial loss [1].
• Normal retinal fundus into
Super-resolution image
Mahapatra.et • Gans are used for image
9. al.,2017 super resolution using resnet.
• Loss function contain content
loss and generative loss and
balance by a factor [6].

VI. FUTURE CHALLENGE

Despite many positive applications of GAN in medical
Fig. 7. work done according to different anatomy. imaging, there is a certain challenging factor that requires
resolution. In the image reconstruction process and process
TABLE I. RECONSTRUCTION WORK DONE IN MEDICAL IMAGING of image synthesis, mostly traditional shallow reference
Analysis matrices like MAE, SSIM has been used for quantitative
Sno Publications Method/Anatomy/Modality assessment only but not for the visible quality of the
processed image. For example, during reconstruction, pixel-
• Lumbar spine CT to MRI [19] wise loss results in the creation of a blurry image. It looks
Lee et al.,
1. • 2D fully-convolutional
2020 tough to specify this range in horizontal differentiation of
• Preliminary study
Generative Adversarial Network-based work mostly. This
• Medgan is a framework [7]
• Image to image model
problem can be alleviated by using downstream work like
• Loss function adversarial loss classification and segmentation for enhancing the standard of
Armanious et the generated image.
percep- tual loss, style transfer
2. al.,
losses, content loss
2020 REFERENCES
• Casnet which is inspired
Resnets. [1] J. M. Wolterink, T. Leiner, M. A. Viergever and I. I sˇgum,
• U net based architecture ”Generative Adversarial Networks for Noise Reduction in Low-Dose

4
Authorized licensed use limited to: KIIT University. Downloaded on June 07,2024 at 08:56:17 UTC from IEEE Xplore. Restrictions apply.
 CT,” in IEEE Transactions on Medical Imaging, vol. 36, no. 12, pp. [12] Yi, Xin & Walia, Ekta & Babyn, Paul. (2019). Generative Adversarial
2536-2545, Dec. 2017, doi: 10.1109/TMI.2017.2708987. Network in Medical Imaging: A Review. Medical Image Analysis. 58.
[2] Wang Y, Yu B, Wang L, Zu C, Lalush DS, Lin W, Wu X, Zhou J, 101552. 10.1016/j.media.2019.101552.
Shen D, Zhou L. 3D conditional generative adversarial networks for [13] Singh S, Kumar D, Kumar S. Risk factors in cervical spondylosis. J
high-quality PET image estimation at low dose. Neuroimage. 2018 Jul Clin Orthop Trauma. 2014;5(4):221–226.
1;174:550-562. doi: 10.1016/j.neuroimage.2018.03.045. Epub 2018 doi:10.1016/j.jcot.2014.07.007
Mar 20. PMID: 29571715; PMCID: PMC6410574. [14] Wang, N., Huang, X., Rao, Y. et al. A Convenient Non-harm Cervical
[3] Goodfellow, Ian & Pouget-Abadie, Jean & Mirza, Mehdi & Xu, Bing Spondylosis Intelligent Identity method based on Machine Learning.
& Warde-Farley, David & Ozair, Sherjil & Courville, Aaron & Ben- Sci Rep 8, 17430 (2018). https://doi.org/10.1038/s41598-018-32377-3
gio, Y.. (2014). Generative Adversarial Networks. Advances in Neural [15] Hoggjohnson, S., Velde, G. V. D., Carroll, L. J., Holm, L. W. &
Information Processing Systems. 3. 10.1145/3422622. Cassidy, J. D. The burden and determinants of neck pain in the
[4] Chen, Yuhua & Shi, Feng & Christodoulou, Anthony & Zhou, general population: Results of the bone and joint decade 2000 2010
Zhengwei & Xie, Yibin & Li, Debiao. (2018). Efficient and Accurate task force on neck pain and its associated disorders. best evidence on
MRI Super Resolution Using a Generative Adversarial Network and the burden and determinants of neck. European Spine Journal 17,
3D Multi level Densely Connected Network. 39–51 (2008).
[5] G. Yang et al., ”DAGAN: Deep De-Aliasing Generative Adversarial [16] Li, C., & Wand, M. (2016). Precomputed Real-Time Texture Syn-
Networks for Fast Compressed Sensing MRI Reconstruction,” in thesis with Markovian Generative Adversarial Networks. ArXiv,
IEEE Transactions on Medical Imaging, vol. 37, no. 6, pp. 1310- abs/1604.04382.
1321, June 2018, doi: 10.1109/TMI.2017.2785879. [17] Isola, Phillip & Zhu, Jun-Yan & Zhou, Tinghui & Efros, Alexei.
[6] Mahapatra, Dwarikanath. (2017). Retinal Vasculature Segmentation (2016). Image-to-Image Translation with Conditional Adversarial
Us- ing Local Saliency Maps and Generative Adversarial Networks Networks.
For Image Super Resolution. [18] Tajbakhsh, Nima & Jeyaseelan, Laura & Li, Qian & Chiang, Jeffrey
[7] Armanious, Karim & Jiang, Chenming & Fischer, Marc & & Wu, Zhihao & Ding, Xiaowei. (2019). Embracing Imperfect
K¨ustner,Thomas & Hepp, Tobias & Nikolaou, Konstantin & Gatidis, Datasets: A Review of Deep Learning Solutions for Medical Image
Sergios & Yang, Bin. (2019). MedGAN: Medical image translation Segmentation.
using GANs. Computerized Medical Imaging and Graphics. 79. [19] Lee JH, Han IH, Kim DH, Yu S, Lee IS, Song YS, Joo S, Jin CB, Kim
101684.10.1016/j.compmedimag.2019.101684. H. Spine Computed Tomography to Magnetic Resonance Image
[8] Sekuboyina A. et al. (2018) Btrfly Net: Vertebrae Labelling with Synthesis Using Generative Adversarial Networks : A Preliminary
Energy- Based Adversarial Learning of Local Spine Prior. In: Frangi Study. J Korean Neurosurg Soc. 2020 May;63(3):386-396. doi:
A., Schnabel J., Davatzikos C., Alberola-López C., Fichtinger G. 10.3340/jkns.2019.0084. Epub 2020 Jan 14. PMID: 31931556;
(eds) Medical Image Computing and Computer Assisted Intervention PMCID: PMC7218205.
– MICCAI 2018. MICCAI 2018. Lecture Notes in Computer Science, [20] Q. Yang et al., ”Low-Dose CT Image Denoising Using a Generative
vol 11073. Springer, Cham Adversarial Network With Wasserstein Distance and Perceptual
[9] Kazeminia, Salome & Baur, Christoph & Kuijper, Arjan & Ginneken, Loss,” in IEEE Transactions on Medical Imaging, vol. 37, no. 6, pp.
Bram & Navab, Nassir & Albarqouni, Shadi & Mukhopadhyay, 1348-1357, June 2018, doi: 10.1109/TMI.2018.2827462.
Anirban. (2018). GANs for Medical Image Analysis. [21] Chen, Xi & Duan, Yan & Houthooft, Rein & Schulman, John &
[10] Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Sutskever, Ilya & Abbeel, Pieter. (2016). InfoGAN: Interpretable
Ghafoorian, M., . . . S´anchez, C. I. (2017). A survey on deep learning Repre- sentation Learning by Information Maximizing Generative
in medical image analysis. Medical Image Analysis, 42, 60–88. doi: Adversarial Nets.
10.1016/j.media.2017.07.005 [22] Simonyan, Karen & Zisserman, Andrew. (2014). Very Deep Convolu-
[11] Havsteen I, Ohlhues A, Madsen KH, Nybing JD, Christensen H, tional Networks for Large-Scale Image Recognition. arXiv
Chris- tensen A. Are Movement Artifacts in Magnetic Resonance 1409.1556.
Imaging a Real Problem?-A Narrative Review. Front Neurol. [23] Radford, Alec & Metz, Luke & Chintala, Soumith. (2015).
2017;8:232. Published 2017 May 30. doi:10.3389/fneur.2017.00232 Unsupervised Representation Learning with Deep Convolutional
Generative Adversar- ial Networks.

5
Authorized licensed use limited to: KIIT University. Downloaded on June 07,2024 at 08:56:17 UTC from IEEE Xplore. Restrictions apply.