ISSN 2347 - 3983

Volume
Urja Bahety et al., International Journal of Emerging 9. No.in8,Engineering
Trends August 2021Research, 9(8), August 2021, 1118 – 1123
International Journal of Emerging Trends in Engineering Research
Available Online at http://www.warse.org/IJETER/static/pdf/file/ijeter15982021.pdf
https://doi.org/10.30534/ijeter/2021/15982021

Overview on Image Captioning Techniques

Urja Bahety1, Surendra Gupta2
1
Research Scholar, Department of Computer Engineering, Shri Govindram Seksaria Institute of Technology and
Science, Indore, India, [email protected]
2
Professor, Department of Computer Engineering, Shri Govindram Seksaria Institute of Technology and Science,
Indore, India, [email protected]

 description but for machine it is difficult to emulate human

and machine needs description to get it. In an arena of deep
learning and artificial intelligence to make machine emulate
ABSTRACT human is a prolonged aim for researchers.
Though excessive progress have made in several tasks of
Image captioning is a process to assign a meaningful title for computer vision like attribute classification [1], [2], object
a given image with the help of Natural Language Processing recognition [3], [4], image classification [5] etc., but to let
(NLP) and Computer Vision techniques. Captioning of an machine to do a task of generation of human like caption
image first need to identify object, attribute and relationship when an image is given to it, is a fairly new task and it require
among these in image and second is to generate relevant both computer vision and natural language processing
description for the given image. So it require both NLP and researches.
Computer vision techniques to perform image captioning It is well known quest problem to caption the natural scene.
task. Due to complexity of finding relationship between the There are various application areas of image captioning, as
attribute of the object and its feature makes it a challenging for human whether it is spoken or written, our
task. Also for machine it is difficult to emulate human brain communication is depends on natural language and define
however researches have shown a prominent achievement in immense amount of details of our surroundings, allowing
this field and made it easy to solve such problems. The machine to depict visual world can lead immense number of
foremost aim of this survey paper is to describe several applications in this field like in information retrieval,
methods to achieve the same, the core involvement of this education for child [6], web searching [7], for visually
paper is to categorise different existing approaches for image disabled people [8], etc.
captioning, further discussed their subcategories of this Many social media site, google apps also using it to locate
method and classify them, also discussed some of their you, to analyse what you are doing, etc. also twitter and
strength and limitations. This survey paper gives theoretical facebook generate description like what are we wear, what we
analysis of image captioning methods and defines some doing there, where are we etc. To understand image it needs
earlier and newly approach for image captioning. This survey to identify the objects, their actions and relationship among
paper is basically a source of information for researchers in them. For machine it is no difficult task to identify the object
order to get idea of different approaches that were developed in an image but difficult to gather and identify the features or
so far in the field of image captioning. actions and their relationship in the image. Generated
description must be truthful in terms of syntactic and
Key words : Computer Vision, Deep Learning, Neural semantic manner. To generate good caption of an image must
Network, NLP, Image Captioning, Multimodal Learning. require understanding of natural scene in an image by
obtaining salient features from the image.
1. INTRODUCTION In the arena of Artificial Intelligence (AI), captioning the
image is very prevalent study field which deals with
Human beings are competent enough to certainly illustrate understanding of image and understanding language
the surrounding that we are in. At a quick glance to an image description for the image and needs to recognise object their
it is very easy and is usual ability for human to describe the properties and interactions and also understand location or
details when we have given an image. In everyday life we scene type in order to produce best captions.
encounter so many sources of images like news, internet, The chief goal of captioning of image is to produce
social media, advertisement and many more existing sources description of an image when image is given i.e. the
and viewers have to interpret the context behind the images description which is semantically and linguistically
and most of them do not have description with it but human ingenuous to the content of image. There are two main
have capability to understand them without any need of question are arrived first is linguistic processing and second is
visual understanding of an image.

1118
Urja Bahety et al., International Journal of Emerging Trends in Engineering Research, 9(8), August 2021, 1118 – 1123

To make sure that generated description of an image are Image captioning has enthralled more study in the area of AI
semantically and syntactically correct some techniques of and has played remarkable role in computer vision and
NLP and computer vision are adopted to deal the problem language processing. Captioning of image described objects
arrived from modality and integrity appropriately. Several attribute and their relationships. In order to understand image
approaches have been proposed to create description of an captioning and to produce captions for image this survey
image correctly. paper describes various methods to perform captioning task.
This survey paper presents theoretically analysis of several This survey provides the summarised understanding of
different methods some of which was used earlier and some method which has been achieved earlier and the latest
latest methods for image captioning. Below represent the methods which has been achieved so far. The methods to
overall categorization of image captioning method in caption an image which have surveyed are retrieval based,
structure form and detailed description of these methods deep neural network based along with their types. Detailed
describe in succeeding sections. discussion of these methods is discussed in remaining section
of the paper.
Remaining paper organisation is as - Segment 3 outlined
Retrieval based image captioning technique segment 4
outlined some of the deep neural network based techniques
and finally discussed concluding remarks.

3. RETRIEVAL BASED IMAGE CAPTIONING

METHOD

Earlier the kind of image captioning method which was very

ordinary was retrieval based image captioning. In this
technique given a query image, descriptions retrieved from
set of existing descriptions or from set of predefined caption
pool, from training dataset it firstly find visually alike images
with their captions, the caption that was generated may be
either a caption that has existed already or a caption collected
from retrieved ones. Now look over direction of research
Figure 1: Structure of Image Captioning Methods which openly used retrieved description as caption of image.
A. Farhadi et al. [9] established a (object, action, scene)
2. IMAGE CAPTIONING meaning space to associate sentences and images. “A query
image has been given and they map it into the meaning space
The foremost aim of captioning of image is to generate proper by solving Markov Random field and semantic distance
caption which is relevant to given image. Image captioning is between these images are determined by Lin Similarity
deals with understanding of given image and description for measure [10] and each existing sentence parsed by Curran
an image, needs to identify object, attributes, properties and and Clark Parser [11]. The caption which is closest to the
relationship among these and require both NLP and computer given image is considered as the caption of the query image”
[12].
vision techniques to give good description of a given image.
In order to caption an image V. Ordonez et al. [13] firstly used
When an image is given as input to the image captioning
global image descriptors to retrieve set of images from web
model, model should give relevant description to a given scale collection of captioned images. After that they use
image. Below is the image captioning diagram which shows semantic content of retrieve images to do re ranking and use
the caption generation when an image is given as input to the caption of top image as the description of the query.
model. Image captioning in accordance to M. Hodosh et al. [14],
frames captioning for image as a task of ranking. The author
gives the analysis of Kernal Canonical Correlation method
[15], [16] to project text and image to common space, where
training images and their captions are correlated greatly. To
choose top ranked captions to act as a description of the query
images, cosine similarity amongst captions and images are
calculated in new common space.
To lighten the effect of estimation of noise visual techniques
that are based on retrieval for image captioning, Charniak
and Mason [17] at first for query image use visual similarity
to retrieve set of captioned images. From the captions of
retrieved images they calculate a word probability density
Figure 2: Block Diagram of Image Captioning
1119
Urja Bahety et al., International Journal of Emerging Trends in Engineering Research, 9(8), August 2021, 1118 – 1123

conditioned on querying image. To retrieve the largest score Significantly by using deep neural network the execution of
as caption of the query the word probability density method is image captioning is enhanced. However using deep neural
used to rank recent caption. The prior principal determined network in template and retrieval based methods does not
that given a query image always there exist a sentence which overthrow their weaknesses so limits of caption produced by
is relevant to it. This hypothesis is barely accurate in practise these techniques are not taken out completely.
so in place of using selected sentences as description of query
image directly, in other line of retrieval based research, 4.2 Captioning of Image Based on Multimodal Learning
retrieved sentences are used for the composition of new v/s Visual Space
description for image query.
Dataset is provided with paired imaged and captions Stanford Captioning based on deep neural network can create captions
coreNLP toolkit is used by Y. Verma et al. [18] to process from visual and multimodal space. Dataset of image
sentences in dataset to derive a sentence list for every single captioning have captions in text form for corresponding
image. First image retrieval is achieved based upon global image. In multimodal space methods a shared multimodal
features of an image to retrieve a set of images for the query in space is learn from images and matching captions, then this
order to produce a description for a query image. A model is multimodal depiction is pass to language decoder. In contrast
trained to predicate phrase relevance is used to select phrases in a visual space based method the matching captions and
from the ones related with retrieved images subsequently. image feature are passed to language decoder independently.
Lastly a description is produced which is based upon selected Several image captioning technique use visual space to
relevant phrases. generate captions for image. This approach is discussed
There are certain limitations also of this method. However briefly in section 4.4.
this method creates general and syntactically correct captions
for image i.e. captions generated by this method are 4.2.1 Multimodal Space
grammatically correct and simple but this method is unable to
create image specific captions and the captions which are Multimodal space method architecture containing vision part,
semantically correct i.e. in some conditions generated language encoder, language decoder and multimodal space
captions may be unrelated to the content of the image. part. To extract image feature vision part use deep
Convolutional Neural Network (CNN) as extractor of feature
4. DEEP NEURAL NETWORKS BASED IMAGE and language decoder extracts word features and learn dense
CAPTIONING feature embedding for every word then it passed to recurrent
layer. Multimodal space maps features of image with word
Due to excessive evolution made in arena of deep learning, features to a common space, then resultant feature map are
work start to relay on deep neural networks to captioning of handed to a language decoder that produce captions by
image. Deep neural network is now taken on broadly to decoding the feature map.
handle image captioning task. Categorisation is done on deep Initially work is done in this field by Kiros et al. [21] the
neural network based approaches to subclasses on the basis of technique uses a CNN to take out the feature of image to
key structures & framework. generate the captions and uses multimodal space that
represent text and image both together for multimodal
4.1 Using Neural Networks Retrieval and Template Based learning and generation of caption, also it propose
Method multimodal neural network model that was conditioned on
image input to create images captions. In their approach log
Increasing the progresses in the arena of deep neural network bilinear language model is adapted to multimodal space. This
we can perform captioning task with ease. The problem of method depends on high level image feature and
ranking and embedding can be resolved with retrieval based representation of word learned from multimodal neural
method, to make image captioning as multimodality for use in language modal and deep neural network.
deep modelling recommended by researchers. “To retrieve a “A Karpathy et al. [22] proposed deep, multimodal model,
description for a query image Socher et al. [19] proposed to embedding of natural language and image data for
use dependency tree recursive neural network to represent bidirectional image and caption embedding task previously
sentences or phrases as compositional vectors. Another deep mention multimodal based approach use common embedding
neural network model is used as visual model to extract the space that maps image and caption directly, however this
features from the images. Obtained multimodal features are method work at finer level and embed fragments of captions
mapped to a common space using max margin objective and fragments of images this method breaks down images
function. Correct sentence and image pair in common space into number of objects and captions into dependency tree
has larger inner product after the training” [20]. At the end relation. This approach achieves some improvement in
sentence retrieval is done which is based upon similarities retrieval task as compared to other methods. This approach
between representation of sentences and images in common has some drawback also like in term of modelling the
space. dependency tree can model relation easily but they are not
always correct” [23].

1120
Urja Bahety et al., International Journal of Emerging Trends in Engineering Research, 9(8), August 2021, 1118 – 1123

A multimodal recurrent neural network (MRNN) language CNN is used get scene type i.e. to notice objects and
model is used by J. Mao et al. [24] to create novel captions for relationship between them and output of this are then used by
image. There are two subnetwork in this method deep language model to transform them into words and combined
Recurrent Neural Network (RNN) used for captions & deep phrases that create caption of image.
CNN used for images both of these networks interacts with Vinyals et al. [25] describes “a method called neural image
one another in multimodal space to form MRNN model. In caption generator, this method uses LSTM for caption
this method both fragments of captions and images are given generation and CNN for image representation. This CNN
as input and it computes probability distribution in order to uses novel method for batch normalisation and output of its
produce subsequent word of captions, in this model five more last layer is used as input to the LSTM decoder. LSTM is able
layers are there- a recurrent layer, two layers of word to keep track of objects that already described using text” [23]
embedding, a multimodal layer and a softmax layer. Kiros and neural image caption generator trained on maximum
[21] proposed a technique that is based on log bilinear model likelihood estimation.
and alexnet for feature extraction. This MRNN approach is To generate a caption for image, image statistics is included
some interrelated to method of kiros [21] which uses fixed to initial state of LSTM and the succeeding word which would
length context where as in “this method temporal context is be generated is based on preceding hidden state and current
stored in recurrent architecture which allows variable context time step. Till it catches the end of the token of sentence this
length. Two word embedding layer use vector in order to procedure is continues. Below figure shows encoder decoder
generate a dense word representation. It encodes both architecture-
semantic and syntactic meaning of words. Semantically
relevancy of word can be get by calculate Euclidian distance
between the two dense word vectors in embedding layer. Most
of the image sentence based multimodal methods use pre
computed word embedding vectors in order to initialise their
model. On the other hand this method randomly initialises
word embedding layer and learn from training dataset which
helps to generate good caption for image” [23]. Below figure
shows multimodal based image captioning-

Figure 4.3.1: Encoder-Decoder Architecture based Image

Captioning

4.3.2 Image Captioning Based on Compositional

Architecture

This method is composition of various separate functional

building blocks. Convolutional neural network is used to get
the semantic meaning from the image first afterword
language model is used to create a set of candidate captions
and these candidate captions are re ranked with the help of
deep multimodal similarity model in order to generate final
caption. Below figure shows compositional architecture based
captioning-

Figure.4.2.1: Multimodal Based Image Captioning

4.3 Captioning of Image Based on Encoder- Decoder

Architecture v/s Compositional Architecture

4.3.1. Captioning Based on Encoder Decoder

Architecture

In this image features are extracts from activation of

Figure 4.3.2: Compositional Network Based Image
convolutional neural network and to create sequence of words
Captioning
it then fed to Long Short Term Memory (LSTM). Basically

1121
Urja Bahety et al., International Journal of Emerging Trends in Engineering Research, 9(8), August 2021, 1118 – 1123

All the process of this methods follows steps like first image are added to different hidden states of language generation
features are obtained using convolutional neural network then model. At last language generation model will generate
attributes are gotten from visual features, multiple captions captions with semantic concepts” [23]. Below figure shows
are created by language model in order to follow above two semantic caption based captioning-
steps, at last captions that are generated is re-ranked using a
model called deep multimodal similarity model to select great
excellence captions of image.

4.4 Some Other Methods

These are some categories of methods which are independent

of other methods like semantic concept based, novel object
based, and attention based captioning are all assemble in this
category.
Figure 4.4.2: Semantic Caption Based Image Captioning
4.4.1 Attention Based Captioning
4.4.3 Novel Object Based Captioning
Basically in captioning CNN is used to get features from
image and RNN is used as decoder to transform this “Deep learning based captioning methods have achieved good
representation word by word. In that method they do not results and they highly depend on paired image and sentence
deliberate spatial aspects of image that is pertinent to the part caption dataset. These types of method only generate
of captions and captions are generated by considering a scene description of object within the context therefore methods
as a whole. This method address this limits because it focus need large set of training dataset consisting of image-text
upon several part of input image dynamically whereas output pairs. Novel object based captioning method generate
sequences are being generated. Firstly use CNN so that description of novel object that are not there in paired image
information of image is acquired that is derived from whole caption datasets” [23]. In this method a language model and a
scene, based on output of above step language generation separate lexical classifier are trained upon unpaired text and
phase generate words, after this main areas of given image are data of image. On paired image caption data a deep captioned
concentrated on every time step of language generation model model is trained, at last both the models are joined together to
which is based upon words that are generated, and till the end train combinely to produce caption for novel object. Below
state of language generation model dynamically the captions figure shows novel object based captioning-
are reorganised.

Figure 4.4.3: Novel Object Based Image Captioning

5. CONCLUSION
The proposed survey paper describes the captioning for image
models and techniques adopted in each methods. The survey
Figure 4.4.1: Attention Based Image Captioning paper has also described some strength and limitations of
discussed approaches. First this paper describes primary
4.4.2 Semantic Caption Based Captioning- captioning for image workings which are template and
retrieval based also it described that retrieval methods are
“This method selectively attend to a set of semantic concept good in generating simple and grammatically correct captions
proposal extracted from the image, these concepts are then
but also has some limitation i.e. unable to generate
combined to a hidden states and output of RNN. In this
semantically correct caption. Afterword the key attention is
method first CNN based encoder is used to encode semantic
dedicated on the approaches which are based on deep neural
concepts image feature. After this image feature are given as
input to language generation model then semantic concepts network which gives prominent results in generating good

1122
Urja Bahety et al., International Journal of Emerging Trends in Engineering Research, 9(8), August 2021, 1118 – 1123

captions, then further divide them in subcategories since 13. V. Ordonez, G. Kulkarni, T. L. Berg., Img2text:
distinct agendas are used in deep neural network based Describing images using 1 million captioned
techniques and some other methods related to image photographs, in: Advances in Neural Information
captioning. Processing Systems, 2011, pp. 1143–1151.
14. M. Hodosh, P. Young, J. Hockenmaier, Framing image
REFERENCES description as a ranking task: data, models and
evaluation metrics, Journal of Artificial Intelligence
1. T. Yang, C. Gan, B. Gong, Learning attributes equals Research 47 2013, pp. 853–899.
multisource domain generalization, in: IEEE Conference 15. F. R. Bach, M. I. Jordan, Kernel independent component
on Computer Vision and Pattern Recognition, Miami, analysis, Journal of Machine Learning Research 3 2002,
FL, USA, 2016, pp. 87–97. pp. 1–48.
2. H. Nickisch, C. H. Lampert, S. Harmeling, Learning to 16. D. R. Hardoon, S. R. Szedmak, J. R. Shawe-Taylor,
detect unseen object classes by between class attribute Canonical correlation analysis: An over view with
transfer, in: IEEE Conference on Computer Vision and application to learning methods, Neural Computation 16
Pattern Recognition, Miami, FL, USA, 2009, pp. 2004, pp. 2639–2664.
951–958. 17. R. Mason, E. Charniak, Nonparametric method for data
3. P. F. Felzenszwalb, R. B. Girshick, D. McAllester, D. driven image captioning, in: Proceedings of the 52nd
Ramanan, Object detection with discriminatively trained Annual Meeting of the Association for Computational
part based models, IEEE Transactions on Pattern Linguistics, 2014.
Analysis and Machine Intelligence 32 (9) 2010, pp.
18. A. Gupta, Y. Verma, C. V. Jawahar., Choosing
1627–1645.
linguistics over vision to describe images, in: AAAI
4. R. Girshick, J. Donahue, T. Darrell, J. Malik, Rich
Conference on Artificial Intelligence,Vol.5, 2012.
feature hierarchies for accurate object detection and
19. R. Socher, A. Karpathy, Q. V. Le, C. D. Manning, A. Y.
semantic segmentation, in: IEEE Conference on
Ng, Grounded compositional semantics for finding and
Computer Vision and Pattern Recognition, Columbus,
describing images with sentences, TACL 2 2014, pp.
OH, USA, 2014, pp. 580-587.
207–218.
5. P. Lakshmi Prasanna, D. Raghava Lavanya, T. Sasidhar,
20. Shuang Bai , Shan An, A Survey On Automatic Image
B. Sekhar Babu, Image Classification Using
Caption Generation, Article in Neurocomputing, May
Convolutional Neural Network, in: International Journal
2018, pp. 291-304.
of Emerging Trends in Engineering Research, October
21. Kiros R, Salakhutdinov R, Zemel R, Multimodal neural
2020, pp. 6816-6820.
language models. In: International conference on
6. S.Vimala, P.Haritha, S.Malathi, A Systematic literature
machine learning, 2014, pp. 595–603.
review on story telling for kids using image
22. A Karpathy, A Joulin, and FeiFeiFLi. Deep fragment
captioning-deep learning, in: 4th International
embeddings for bidirectional image sentence mapping.
Conference on Electronics, Communication and
Advances in neural information processing systems,
Aerospace Technology(ICECA), 2020.
2014, pp. 1889–1897.
7. Shubham Chaturvedi, S. Iyer, Tirthraj Dash, Image
23. Md. Zakir Hossain, Ferdous Sohel, Mohd Fairuz
captioning based image search engine: An alternative to
Shiratuddin, and Hamid Laga. A Comprehensive Survey
retrieval by metadata: SocProS 2017, pp. 181-191.
of Deep Learning for Image Captioning. ACM Comput.
8. B. Makav, V. Kilic, A new image captioning approach Surv. Vol. 51, 2019, pp. 1-36.
for visually impaired people, in: 11th International 24. Junhua Mao et al., Deep captioning with multimodal
Conference on Electrical and Electronics recurrent neural networks (m-rnn). In International
Engineering(ELECO), 2019. Conference on Learning Representations (ICLR), 2015.
9. A.Farhadi et al., Every picture tells a story: Generating 25. O. Vinyals, A. Toshev, S. Bengio, D. Erhan, Show and
sentences from images, in: European Conference on tell: A neural image caption generator, in: Proceedings of
Computer Vision, 2010, pp. 15–29. the IEEE Conference on Computer Vision and Pattern
10. D. Lin, An information-theoretic definition of similarity, Recognition, 2015, pp. 3156–3164.
in: Proceedings of the Fifteenth International Conference
on Machine Learning, pp. 296–304.
11. J. Curran, S. Clark, J. Bos, Linguistically motivated
large-scale nlp with cc and boxer, in: Proceedings of the
45th Annual Meeting of the ACL on Interactive Poster
and Demonstration Sessions, pp. 33–36.
12. Himanshu Sharma, Manmohan Agrahari, Mohd Firoj,
Image Captioning: A Comprehensive Survey, in:
International Conference on Power Electronics and IoT
Applications in Renewable Energy and its Control
(PARC), 2020, pp. 325-328.
1123