Volume 6, Issue 5, May – 2021 International Journal of Innovative Science and Research Technology

ISSN No:-2456-2165

Image Captioning Using R-CNN & LSTM Deep

Learning Model
Aditya Kumar Yadav Prakash.J
UG Scholar Assistant Professor
PSG College of Technology Department of Computer Science & Engineering
PSG College of Technology

Abstract:- Image Captioning is the process of creating a on one of these works, which combines a variant of a neural
text description of an image. It uses both Natural network from Recurrent to a CNN. The main objective of is
Language Processing (NLP) and Computer Vision to to improve the model by making minor architectural
generate the captions. The image captioning task is done changes and using phrases as elementary units instead of
by combining the detection process when the words, which can lead to better semantic and syntactic
descriptions consist of a single word like cat, skateboard, subtitles.
etc. and Image Captioning when one predicted region
covers the full image, for example cat riding a II. IMAGE CAPTIONING APPROACHES
skateboard. To address the localization and description
task together, we propose a Fully Convolution 2.1 Retrieval Based Image Captioning
Localization Network that processes a picture with a Retrieval based image captioning technique is one of
single forward pass which can be consistently trained in the traditional ways of producing captions for the image. An
a single round of optimization. To process an image, first input image is given as input, retrieval based captioning
the input image is processed using CNN. Then methods generate a caption by retrieving one or many
Localization Layer proposes regions and includes a sentences from the pre-loaded pool of sentences. The
region detection network adopted from faster R-CNN caption that was generated can either be the exact same
and captioning network. The model directly combines sentence that was pre-loaded or a sentence that is framed
the faster R-CNN framework for region detection and from the retrieved ones.
long short-term memory (LSTM) for captioning.
The disadvantage of this approach is that a large pool
Keywords:- Image Caption, Recurrent Neural Network, of images is required to make captions from the allocated set
Long short-term memory, Convolution Neural Network, of sentences for those images. The output captions generated
Faster R-CNN, Natural Language Processing. from this image captioning technique does not have any
grammatical errors, but they are the sentences that were the
I. INTRODUCTION captions for other images. In some of the cases, the captions
may be irrelevant to the input image. The major
Creating captions is an important task that is relevant disadvantage of this method is that they have limitations on
to both computer vision and natural language processing . their capability to describe the images in their own unique
Limitation of human ability for writing pictures through a way.
machine is itself a remarkable step along the line of artificial
intelligence [1-2]. The motivation of our project is to capture 2.2 Template Based Image Captioning
how objects in the image are related to each other and Its different kind of methods which is early used and
express them in the English language (generate captions)[3- it’s based on templates. Based on the templates, the captions
4]. are generated by a syntactic and semantic restricted process.
Certain visual concepts must be recognized for generating a
Many image captioning techniques are used to description of an image using the template-based image
translate raw image files. These techniques do not translate captioning method. The visual concepts that were
the exact context of the image and therefore cannot be used recognized are then connected by a sentence. For the
to gain insight into the image files. A new system is creation of a sentence without any grammatical errors,
proposed for image captioning which includes usage of optimization algorithms are required.
contextual information and thereby accurately describe the
image [5-8]. The disadvantage of this method is that the generation
of descriptions is done under the template-based framework
This work aims to generate captions using neural which is strictly limited to recognized image content of
language models. The number of proposed models for the visual models, typically with a low number of visual models
image labeling task has increased significantly since the available. There are usually limitations on the coverage,
models of the neural language, ANN and the folding neural creativity, and complexity of the sentences generated.
networks (CNN) became popular [9-16]. This paper is based

IJISRT21MAY837 www.ijisrt.com 911

Volume 6, Issue 5, May – 2021 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
2.3 Image Captioning Methods based on Deep Learning display the most matched caption as the output.
2.3.1 Visual Space and Multimodal Space
Image-based subtitle methods based on learning can
Loading the Image
create subtitles from both the visual space and the image
multimodal space. In visual space-based image captioning
method, the features of the image and their captions are sent
Extracting Features
to the speech decoder, whereas in a multimodal space-based
image captioning method, the multimodal space is shared
where the device learns the image and generates captions.
This process also happens through the speech decoder. With
Data Processing And Tokenization
the multimodal representation, the speech decoder generates
the caption for the image.
Encoding
2.3.2 Supervised Learning and Deep Learning Methods
In supervised learning, the training data is given the
desired output called a label and in unsupervised learning, Preparing Model
the training data is given the desired output called the
unlabeled data. Unsupervised learning uses GAN -
Generative Adversarial Network. Reinforcement learning, Progressive Loading
which is the other type of learning along with the GAN
approach are used for produced captions for the image.
Train with Progressive Loading
The below-given steps are followed to produce
captions.
Model Evaluation
 Captions are generated using the CNN and RNN-based FIG 1. STAGES OF CAPTIONING THE IMAGE
combined networks. USING THE DEEP LEARNING MODEL
 To evaluate the captions and to send that response to the
first combined network, another CNN and RNN-based 3.1 Loading the Image
combined network is used. Loading of the image is done using keras. It provides a
function through which we load the images as a matrix, and
2.4 Dense Captioning Methods then the matrix is transformed to a NumPy array through
The image captioning techniques discussed earlier can keras and loads the images in a fixed dimension as specified
generate only a single caption for the given input image. in a VGG model.
Using different areas in the image, they get information
about different objects. Though they have information about 3.2 Extracting Features
many regions of the image, they do not generate a separate Next stage is to extract features from the loaded
caption for all the regions. The dense captioning method images. Pre-trained model is used to understand the key
generates captions for all the regions that were detected in elements of the photos. VGG model is used to obtain the
the image. required weights and Feature extraction takes place.
The following steps are used by the dense captioning 3.3 Data Processing and Tokenization
technique. Descriptions required be tokenizing and handling
 For different regions of the image, region suggestions are easily. Cleaning can be done by converting to lowercase,
created. removing punctuations, removing unnecessary words like
 To obtain information about the detected regions, CNN 'a', removing numbers. If vocabulary is small, then it model
is used. trains faster but it is not expressive. The image identifiers
 The result obtained from the step (2) is used by NLP to and descriptions are saved to a new file.
generate captions for each region.
3.4 Encoding
III. IMAGE CAPTIONING APPROACH For processing, the words need to be encoded, since
we are going to use probability, Keras is used to prepare the
Various stages of captioning the image using the deep descriptions. Firstly, we map the image identifiers to the
learning model is illustrated in the Fig.1 where the given present descriptions. At the end of this process, we get shape
input image is loaded and features are extracted from the of data for the model that is 8092 photos, 28 words (length
image then the caption text file is loaded and data pre- of description) and 4485 words(vocabulary dataset).
processing and tokenization is done. After tokenization,
mapping of the image with its caption is stored in a separate 3.4 Preparing Model
file. Then the image is encoded using a pre-trained model A simple model that generates the sentence word by
which is used for image classification (VGG model). After word is used. The input for the image is the image and the
encoding, train the model and then evaluate it. After this, just predicted word and model is called recursively it uses

IJISRT21MAY837 www.ijisrt.com 912

Volume 6, Issue 5, May – 2021 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
the already predicted words to generate the new words and it IV. CONCLUSION
uses input and output pairs and the new words are predicted
by probability. The caption for the given input image is
produced using deep learning model. Initially the images
3.5 Progressive Loading were preprocessed and the text in order to train a deep
If the computing facility doesn't have enough learning model. And designed and trained a deep learning
processing capability, the photos and descriptions can be image caption generation model. Then, evaluated the train
loaded in a progressive way. Using Keras, we can load in a caption generation model using which produced captions for
progressive way through a function. It is used to create new images that are given as input apart from the loaded
batches of samples for the training of model and the images.
generator gives an array in the form of input and output for
the model. And the input is the input images and encoded REFERENCES
word sequences which are in form of array. The hot encoded
words are the output. [1]. Jiuxiang Gu, Gang Wang, JianfeiCai, and Tsuhan
Chen. 2017. An empirical study of language cnn for
3.6 Train with Progressive Loading image captioning. In Proceedings of the International
Data generator along with a function on the model is Conference on Computer Vision (ICCV).
used to train it. Model is saved after each epoch (Epoch is [2]. Rajesh, R., & Mathivanan, B. (2017). Predicting Flight
one complete representation of data set), Models are created Delay using ANN with Multicore Map reduce
and the model with lowest loss is selected. framework, Communication and Power Engineering.
In Communication and power engineering. Walter de
3.7 Evaluation: Gruyter GmbH & Co KG
Once the model is created we evaluate the model. [3]. Kumar, Akshi. (2017). A survey of evolution of image
BLEU (Bilingual Evaluation Understudy Score) [8] is used captioning techniques. International Journal of Hybrid
for evaluating the model. It gives information about the Intelligent Systems. 14. 1-19. 10.3233/HIS 170246.
quality of text. It compares the generated sentence to the [4]. Yunchao Gong, Liwei Wang, Micah Hodosh, Julia
reference sentence. The Bilingual Evaluation Understudy Hockenmaier, and Svetlana Lazebnik. 2014.
Score of the model is determined and the model is resulting Improving image sentence embeddings using large
in better score. The BLEU score obtained is provided in weakly annotated photo collections. In European
Fig.2. Conference on Computer Vision. Springer.
[5]. DzmitryBahdanau, Kyunghyun Cho, and
Bilingual Evaluation Understudy Score YoshuaBengio. 2015. Neural machine translation by
jointly learning to align and translate. In International
BLEU-1 BLEU-2 BLEU-3 BLEU-4 Conference on Learning Representations (ICLR).
0.548790 0.296703 0.195520 0.082717 [6]. Vladimir Bychkovsky, Sylvain Paris, Eric Chan, and
Frédo Durand. 2011. Learning photographic global
FIG 2. BILINGUAL EVALUATION UNDERSTUDY tonal adjustment with a database of input/output image
SCORE FOR THE MODEL pairs. In Computer Vision and Pattern Recognition
(CVPR), 2011 IEEE Conference on. IEEE.
The sample test result is provided in Fig.3, where an
[7]. Chuang Gan, ZheGan, Xiaodong He, JianfengGao, and
input image of a dog walking on the water is provided to the Li Deng. 2017. Stylenet: Generating attractive visual
model and the model predicted the caption as dog is running captions with styles. In Proceedings of the IEEE
through the water. Conference on Computer Vision and Pattern
Recognition.
[8]. Abhaya Agarwal and AlonLavie. 2008. Meteor, m-
bleu and m-ter. In Proceedings of the Third Workshop
on Statistical Machine Translation. Association for
Dog is running through Computational Linguistics, 2008, 115–118.
the water [9]. R Sandhya, J Prakash, B Vinoth Kumar (2020).
Comparative Analysis of Clustering Techniques in
Anomaly Detection in Wind Turbine Data. Journal of
Xi'an University of Architecture & Technology, Vol.
Test Input Image Predicted Caption 12 No.3 2020, pp. 5684-5694.
FIG 3. PREDICTED CAPTION FOR THE GIVEN [10]. Prakash J., & Vinoth Kumar B. (2021). Exploration of
INPUT IMAGE computational intelligence insights and data analytics
to combat COVID-19. Advances in Data Mining and
Database Management, 373-383.
https://doi.org/10.4018/978-1-7998-3053-5.ch018

IJISRT21MAY837 www.ijisrt.com 913

Volume 6, Issue 5, May – 2021 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
[11]. Sivanandhini, P., & Prakash, J. (2020). Crop Yield
Prediction Analysis using Feed Forward and Recurrent
Neural Network. International Journal of Innovative
Science and Research Technology, 5(5), 1092–1096.
doi:10.38124/volume5issue5
[12]. Prakash J. (2018). Enhanced Mass Vehicle
Surveillance System, J4R, Volume 04, Issue 04 ,002,
5-9, June2018.
[13]. Prakash, J., & Joy, E. J. (2020). A comparison of
different surrogate models for delamination detection
in composite laminates using experimental modal
analysis. PROCEEDINGS OF ADVANCED
MATERIAL, ENGINEERING & TECHNOLOGY.
https://doi.org/10.1063/5.0019366
[14]. Sivanandhini, P., & Prakash, J. (2020). Comparative
Analysis of Machine Learning Techniques for Crop
Yield Prediction. International Journal of Advanced
Research in Computer and Communication
Engineering, 9(6), 289-293.
https://doi.org/10.17148/IJARCCE.2020.964
[15]. Prakash, J., Vinoth Kumar, B., & Shyam Ganesh, C.
R. (2020). A comparative analysis of deep learning
models to predict dermatological disorder. Journal of
Xi'an University of Architecture & Technology,
12(11), 630-639. https://www.xajzkjdx.cn/gallery/64-
nov2020.pdf
[16]. Prakash J., Kumar B.V. (2021) An Empirical Analysis
of Hierarchical and Partition-Based Clustering
Techniques in Optic Disc Segmentation. In: Sharma
H., Saraswat M., Kumar S., Bansal J.C. (eds)
Intelligent Learning for Computer Vision. CIS 2020.
Lecture Notes on Data Engineering and
Communications Technologies, vol 61. Springer,
Singapore. https://doi.org/10.1007/978-981-33-4582-
9_7

IJISRT21MAY837 www.ijisrt.com 914