Proceedings of 9th IOE Graduate Conference

Peer Reviewed
ISSN: 2350-8914 (Online), 2350-8906 (Print)
Year: 2021 Month: March Volume: 9

An Approach for Damaged Facial Image Restoration using GAN

Sangharsha Bidari a , Bal krishna Nyaupane b
a, b
Department of Electronic and Computer Engineering, Paschimanchal Campus, IOE, TU, Nepal
Corresponding Email: a [email protected], b [email protected]

Abstract
Deep learning has evolved as the fastest emerging machine learning technique in the field of Image processing.
The development of deep learning algorithms such CNN, RNN, GAN etc. have made possible in solving many
image related problems such as recovering damaged image, image implanting’s, increasing the Resolution
of image and so on. Aiming at achieving good restored facial image, a method of applying a known value
to distorted part of image from its surrounding rather than noise based on GAN can be viewed in this paper.
Firstly, a GAN model is trained with facial image data sets which include facial part of celebrity from all over
the world. Secondly, a masked different facial parts image is provided to check the ability of trained model to
generate the portion of masked part. The GAN network mainly consist of generator and discriminator working
together to carry out the task of filling the distorted part of image with some real one.
Keywords
GAN, CNN, image restoration, image repair

1. Introduction part from damaged image as input and compare the

resultant output with the symmetrically cropped image
Images have huge interaction on day to day lives from from the undamaged part to damaged part of the image.
sharing historical memories to recording current state
of day to day life events. Due to various reasons such
as fading, creasing, staining and tearing e.t.c images 3. Literature Review
gets damaged as it gets older with time. Image
restoration technique is one of the widely used Few years back, before the charm of deep learning the
computer vision task of recovering the damaged part task of image processing was tedious. The process of
of the image.From the past ten years the research for feature extraction, their inspection and manipulation,
restoring damaged image using machine learning analysis of image content has been made easy with
platform has been going on applying different advancement in deep learning. Deep learning has
techniques and methods. enhanced the possibility of things that was achievable
in the field of Digital Image Processing.[1]
Within this decade, image restoration research has
been conducted by utilizing the historical research Lehtinen [2] applied statistical reasoning to reconstruct
papers in solving restoration task. One of the most the signal by machine learning algorithm from which
widely researched techniques in this field is use of it was possible for algorithm to learn to restore image
Generative Adversarial Network (GAN) in restoring from the Damaged images or corrupted images. It can
or generating the missing parts of damaged image. be seen the approach has removed photographic noise
Different research are being conducted on restoring and reconstructed the under sampled MRI scans.
the facial part of image.This paper also deals with Tagare [3] has explored the difference between the
restoring facial parts such as eye, nose, leaps etc. nature of text and image with their effects on the design
of the medical image database trying to enable content
based indexing and retrieval.
2. Research Objective
Pathak [4] applied an unsupervised visual feature
The main objective of the research is to use a Deep learning technique enhanced by context-based pixel
learning method that self-generates recovered facial prediction method. The Context Encoders is a

Pages: 217 – 222

An Approach for Damaged Facial Image Restoration using GAN

convolutional neural network trained to generate the Wang et al. [9] have shown image impainting model
contents of an arbitrary image area conditioned on its DFG-GAN, which can effectively alleviate the artifacts
nearby features and in order to succeed the task, problem when the missing region area is too large.
context encoders used the content of the total image Unlike other image impainting models, this model can
and produce a plausible hypothesis for the missing transfer the image impainting task into a GAN task
parts. Also this paper found that a context encoder when the mask fills the total image. Apart from that,
learns a representation that captures not just it has also taken advantage of the extra class label
appearance but also the semantics of visual structures. information to tell what kind of the damage the image
have.
Liu [5] proposed a Deep Regulated Convolutional
Network (RC-Net), a deep network composed of Murugan [10] have suggested to develop an
regulated sub-network blocks cascaded by intelligence framework to recover the possible
skip-connections, to overcome bottleneck. It has information presented in the original scene of image.
applied both large and small convolution filters This paper provides a framework based on
balancing the effectiveness of prominent feature conditional-GAN to recover the information from the
extraction and the generalization ability of the model. heavily damaged images. Learning parameter of the
RC-Nets outperform state-of- the-art approaches with cGAN is optimized by multi-component loss function
large performance gains in various image restoration that includes improved Wasserstein loss with
tasks while demonstrating promising generalization regression loss function.
ability.
Peng [11] focuses a face de-morphing generative
Ulyanov [6] has shown that, the architecture of a adversarial net- work (FD-GAN) to restore the
generator network is enough to capture a great deal of accomplice’s facial image. It has utilized the
low-level image statistics prior to any learning also symmetric dual network architecture and two levels of
randomly started neural network can be used as a restoration losses to separate the identity feature of
handcrafted prior with excellent results in standard the morphing accomplice. It has great potential to be
inverse problems such as DE noising, super- applied for tracing the identity of face morphing
resolution, and impainting. This research also bridges attack’s accomplice in criminal investigation and
the gap between two very popular families of image judicial forensics.
restoration methods: learning-based methods using
Li [12] aims to repair damaged image by novel
deep convolution networks and learning-free methods
generative model-based approach, which consists of
based on handcrafted image priors such as
nested two Generative Adversarial Networks (GAN),
self-similarity.
the sub- confrontation GAN in generator and
According to Liu [7] image impainting has a good parent-confrontation GAN. The sub-confrontation
application value in image editing, however GAN is the image generator of parent-confrontation
traditional image impainting techniques cannot GAN that can find the location of missing area and
complete semantic repair in the case of insufficient reduce mode collapse as a prior constraint. The parent
sample resources. Deep learning neural network have confrontation GAN has an image generation part and
powerful learning capabilities and can extract a discrimination part.
high-level semantic features. These features can be
Jo [13] develops an image editing system that
used to semantically fill missing regions. Ideal image
generates images as the user provides free-form mask,
restoration needs to maintain structural consistency
sketch and color as an input. This paper has trained
and texture clarity.
network with additional style loss which made it
Javed [8] used an image processing technique to possible to generate realistic results, despite large
conceal identities of sensitive objects. It recovers the portions of the image being removed.
mosaiced parts in an image, especially focusing on
By analyzing the above stated review and research
facial parts. The paper have evaluated their method on
paper, it can be seen the advancement in research of
the CelebA dataset and achieved better results than
image restoration has been progressing Using GAN.
state-of-the-art image completion methods without
Most of paper are trying to improve result by changing
explicitly exploiting the location information of
the architecture of GAN model. This paper deals with
mosaiced parts.
improvement in restoration of facial image using GAN

218
 Proceedings of 9th IOE Graduate Conference

by replacing random noise with predefined noise from Also the task of data augmentation is also done before
the surrounding part of damaged image itself. dividing the data into testing and training dataset.

Training Damaged Image Dataset The training

4. Methodology datasets were used to train the learning model.The
model extracts features form these training dataset
Image restoration processing aims at filling the
from where the network learns the pattern of image.
damaged part of the image with the features that make
About 19,000 dataset were used to train the model.
the missing part look like a real part. This conversion
The damaged has been made by cropping the image at
can be achieved using a deep learning approach which
different level and portion.
is shown in figure one.
Testing Damaged Image Dataset About 1000
image dataset was separated in advance before
training of the model. After the model was trained
using training dataset, the unseen test dataset was used
to measure performance of the designed model. It
generally helps for performance evaluation of model.

U-net GAN model selection Recently, Generative

adversarial network are finding high emphasis in
supervised, semi supervised and unsupervised
learning vision tasks as the generative models
implicitly learn probability density of high
dimensional distributions of the data and generate
natural looking images.The U-net structure of
generator has been implemented which is the reason
for GAN model to be named as U-net GAN model.

Figure 1: Proposed Approach

Image Dataset The model has implemented 20,000

Large-scale CelebFaces Attributes (CelebA) Dataset.
The data set could be obtained from http://mmlab.
ie.cuhk.edu.hk/projects/CelebA.html

Figure 3: GAN Model

The generator and the discriminator in the GAN

network competing each other in zero sum game to
optimize the learning parameters. The schematic of
the GAN network is shown in the Figure three. The
generator generates images of natural looking data
Figure 2: Sample Images from celebA Dataset samples from noise input data to fake the
discriminator while the discriminator tends to
differentiate the generated samples from the real data.
Data Preprocessing Suitable training and testing Both the forger (Generator) and the expert
dataset is very essential for designing the model. The (Discriminator) learn simultaneously by minimize the
obtained image dataset was resized to 64 x 64 pixels. distance between the probability distribution of real

219
An Approach for Damaged Facial Image Restoration using GAN

and generated data. However, while the discriminator

has the access to the generated data and real data, the
generator has no access to the real data distribution.
The noise input data to the discriminator provide the
possible information about the ground truth to
distinguish between the synthetic generated data and
real data distribution. The same noise data distribution
is used for training the generator to produce natural
looking images close to the real data with superior Figure 6: Loss graph of Discriminator and Generator
quality. The generator and the discriminator
composed of deep convolutions layer and fully
6. Result and Discussion
connected dense layers. Since the necessity of direct
inevitable of the generator and the discriminator, the During the training of both generator and
both network modules has to be continuous and discriminator for fifty epoch , the loss value were
differential everywhere. calculated for both generator and discriminator using
The implemented architecture of generator and binary cross entropy(BCE) function.It is specially
discriminator shown in block structure connected with used to categories between real and fake image.The
each layer shown with arrow from starting to end model has used BCE as the cost function.
layer can be viewed in figure four and five. BCE function:

1 m  (i)
y log(a(i) )+(1−y(i) ) log(1−a(i) ) (1)

J=− ∑
m i=1

The term m1 represent average loss of the whole

batch.The term log(a(i) ) represents prediction made
Figure 4: Discriminator architecture
by model and the term y(i) is the label for different
examples i.e whether image is real or fake label. For
eg. real could be a label of 1 and fake could be a label
of 0. And y(i) are the features that are passed in
through the prediction. The obtained graph of
generator and discriminator loss can be viewed in
figure six. The graph shows losses of generator and
discriminator which is below over 10% till the end of
800 iteration or fifty epoch. Hence the graph shows
the trained model has low loss and can be used for the
objective of the work.

Figure 5: Generator architecture The output of Generator at different epoch were

captured from where the progress of model to restore
the damaged parts can be viewed.The some of output
5. Experimental setting of Generator at different epochs are as follows:

The model was trained on a machine having processor

of Intel core Pentium i5-8400 CPU with 2.80 GHZ
speed, 6 core(s), 8GB of RAM and 4GB graphics on
windows operating system. The coding was conducted
on python since the task of machine learning algorithm
can be easily programmed as python has all libraries
required for simulation. The time taken by model to
train from the supplied data nearly took fourteen days.
Figure 7: Generator output of damaged image

220
 Proceedings of 9th IOE Graduate Conference

image. The facial portion to be regenerated were

masked using mouse.Only the masked area were to be
regenerate . The distortion parts were small based on
different parts such as eye, nose and lips which were
restored as accordingly to their location.

Figure 8: Generator output of damaged image

Figure 9: Generator output of damaged image

Figure 10: Generator output of damaged image

Figure 12: Model output for given damaged image

Figure 11: Generator output of damaged image

From simplicity a known image were masked so that
model could generate only the area which were
Viewing above result of generator output image with masked.The masked area were increased in ratio from
increase in number of epoch it can be concluded that small to large area.The restored image shows that the
the model has learned much at each steps and at the model has somehow restored the facial part such as
end of fifty epoch the image generated are much closer eye, nose and lips.Thus the model could be
to the original image. implemented for recovery of small damaged portion
of image.
After the training session the testing image were
provided to model which produced restored part of The result were compared with Coarse-Refined

221
An Approach for Damaged Facial Image Restoration using GAN

structure network [13] who outputs were blurred at on Machine Learning, ICML 2018, 7(3):4620–4631,
different stages of model development also it required 2018.
huge amount of memories with the training time. This [3] Hemant D. Tagare, C. Carl Jaffe, and James Duncan.
paper approached method performs better with small Medical Image Databases: A Content-based Retrieval
Approach. Journal of the American Medical
damaged parts as shown in figure twelve. The Informatics Association, 4(3):184–198, 1997.
performance can be observed by viewing the obtained [4] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue,
output image from the model. Trevor Darrell, and Alexei A. Efros. Context
Encoders: Feature Learning by Inpainting.
Proceedings of the IEEE Computer Society
7. Conclusion Conference on Computer Vision and Pattern
Recognition, 2016-Decem:2536–2544, 2016.
GAN has become more advanced in image processing [5] Peng Liu, Xiaoxiao Zhou, Junyiyang Li, D. El Basha
technique such as image classification, image Mohammad, and Ruogu Fang. Image restoration
using deep regulated convolutional networks. arXiv,
impanting, and image restoration. With slight change 2019.
in the input to the model lead to better improvement [6] Dmitry Ulyanov, Andrea Vedaldi, and Victor
on the output of model by providing some information Lempitsky. Deep image prior. In Proceedings of
of image to generator rather than complete noise. As the IEEE conference on computer vision and pattern
from the output image it can be concluded that model recognition, pages 9446–9454, 2018.
has recognized the placing area of our facial part such [7] Huaming Liu, Guanming Lu, Xuehui Bi, Jingjie
Yan, and Weilan Wang. Image inpainting
as eye, lips and nose. based on generative adversarial networks. ICNC-
Further research can be extended to recover any body FSKD 2018 - 14th International Conference on
Natural Computation, Fuzzy Systems and Knowledge
part of human being and other animal also. Variety of Discovery, pages 373–378, 2018.
noise can be experimented so that by providing [8] Kamran Javed, Nizam Ud Din, Seho Bae, and Juneho
different nature of noise the output of the model can Yi. Image unmosaicing without location information
be changed in accordance. Also different architecture using stacked GAN. IET Computer Vision, 13(6):588–
of GAN can be tested for better improvement of 594, 2019.
generated facial image. Also the rotation of image can [9] Ziqiang Pei, Sheng Yang, and Guoyou Wang. Feature
guidance GAN for high quality image restoration.
make a huge impact on the output image so during (June):9, 2020.
testing the test images could be augmented.
[10] Pushparaja Murugan. Facial information recovery
from heavily damaged images using generative
adversarial network - Part 1. arXiv, pages 1–16, 2018.
References
[11] Fei Peng, Le Bing Zhang, and Min Long. FD-GAN:
[1] Niall O’Mahony, Sean Campbell, Anderson Carvalho, Face De-Morphing Generative Adversarial Network
Suman Harapanahalli, Gustavo Velasco Hernandez, for Restoring Accomplice’s Facial Image. IEEE
Lenka Krpalkova, Daniel Riordan, and Joseph Walsh. Access, 7, 2019.
Deep Learning vs. Traditional Computer Vision. [12] Zhijiang Li, Haonan Zhu, Liqin Cao, Lei Jiao, Yanfei
Advances in Intelligent Systems and Computing, Zhong, and Ailong Ma. Face Inpainting via Nested
943(Cv):128–144, 2020. Generative Adversarial Networks. IEEE Access,
[2] Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, 7:155462–155471, 2019.
Samuli Laine, Tero Karras, Miika Aittala, and Timo [13] Youngjoo Jo and Jongyoul Park. SC-FEGAN: Face
Aila. Noise2Noise: Learning image restoration editing generative adversarial network with user’s
without clean data. 35th International Conference sketch and color. arXiv, 2019.

222