International Journal of Computer Sciences and Engineering Open Access

Survey Paper Vol.-7, Special Issue-14, May 2019 E-ISSN: 2347-2693

“Click-n-Purchase: A Shopping guide with Image Retrieval based on

Mobile Visual Search in Fashion Domain”: A Survey

Nikhil S. Tengli1*, Suvarna Nandyal2

1,2
Department of Computer Science, PDA College of Engineering, Kalaburagi, Karnataka
*
Corresponding Author: [email protected], Tel.: +91-9164499131

DOI: https://doi.org/10.26438/ijcse/v7si14.8892 | Available online at: www.ijcseonline.org

Abstract— In the recent years, the use of e-commerce based applications via Internet has grown rapidly, thus increasing the
volume of data in the web. Therefore it necessary to have faster retrieval of required data from the web. This paper provides a
comprehensive review of various image retrieval techniques with their problems. The survey presents various techniques used
so far for the Image Retrieval from the Web based applications, in order to make more efficient way of retrieving the
information by using image retrieval techniques. The survey describes which techniques are used for image retrieval and the
problem faced during the retrieval process. Finally, based on the use of existing techniques and the demand from the real-time
applications a shopping guide will be presented with enhanced features of image retrieval techniques named as Click-n-
Purchase, where the input for this application is taken from the mobiles and the visual search of the related images can be
extracted from web based fashion domain based applications, so that user can be able to search their favourite items in less
amount of time.

Keywords— Image Retrieval Techniques, Mobile Visual Search, Fashion domain, Click-n-Purchase.

I. INTRODUCTION recognized. A variety of techniques are available for the

purpose of target recognition. Some of them are briefly
Recently, there has been an increase in the number of described here. The most important points to be considered
multimedia applications including text, images, video and in object recognition tasks are the extraction of features that
audio data in every field with equal number of processing are independent of rotation and translation [2]. The shape
requirements. The coming up of huge sized image datasets invariant feature based recognition of target objects involves
calls for optimal techniques for storing, searching, indexing following three steps-
and retrieving desired data at a faster speed. Target object  Data Pre-processing
recognition is one of the most popular and widely researched  Feature extraction
topics in computer vision field that has gained a lot of  Classification
momentum. The goal of this topic is to recognize the objects In the first step, the input image is filtered using noise
belonging to the target object class based on its features. removal methods and processed to improve its quality for
Shape of an object is a visually distinguishable feature that recognition purpose. Image processing aims to enhance the
can be easily utilized for recognition task but it has its own image for obtaining good measures of its features [19]. In the
challenges like pose variations, occlusions in the scene, next step, feature extraction from pre-processed image is
lighting conditions, resolution etc [1]. A number of methods performed where the number and type of features extracted
are developed for achieving recognition of target based on depends on the end goal of the research. The extracted
shape cues. The basic means of object recognition involves features of all input images in the dataset are stored in the
matching the extracted features of objects in the image with a database to be matched further for recognition of similar
sample target object features. target. Thus, feature extraction plays an important role in the
entire object recognition task that decides the performance of
This feature matching step is carried on all the objects in the system developed.
every test image to find potential target object present.
Search technique is included in object recognition II. FEATURE EXTRACTION TECHNIQUES
methodology where the feature set extracted will be put to
comparison with the already extracted features of the target Many image retrieval systems rely on features like shape,
object for easy and accurate recognition. To ensure good intensity, and texture etc. [2]. The shape of an object is a
recognition rate, the features extracted must be physical descriptor. Shape representation can be done in the
distinguishable well and relevant for the target object to be form of interior region, border, moment, edges etc. These are

© 2019, IJCSE All Rights Reserved 88

 International Journal of Computer Sciences and Engineering Vol. 7(14), May 2019, E-ISSN: 2347-2693

used to compare the target as well as input image objects by obtaining from Internet users. The system is found to be
noticing the similarity in making measurements. Texture is feasible for real time application.
also a structural descriptor which is a characteristic of object
surfaces such as glass, grains, sand, and textile. The name The application works on client-server architecture. The
texture is derived from the primitive texture elements client’s role is to present the end user with the graphical user
organized in a pattern which is called Texel’s. A Texel may interface asking query image dynamically. The client then
itself have many pixels, with periodic or random sends the final query formed to the server. Initially, an XML
arrangement depending on the object. Naturally occurring file is downloaded that includes the different product names,
objects have random texture while synthetic objects have their underlying category and the focus masks.
predefined ordered textures.
Jan Cychnerski, Adam Brzeski et.al [4] In this work, the
Texture may be of any category such as coarse, fine, regular, authors have ddeveloped a computer vision system for
irregular, or linear. In image processing, texture has two efficient detection and classification of clothing images for e-
broad classes- statistical and structural. shopping images. A combined architecture involving
convolution neural networks that cover Residual networks,
The Statistical category includes the textures usually random Squeeze Net and Single Shot MultiBox Detector (SSD) is
in nature. The structural textures are those which are certain considered. The training experiments for detecting the
and that recur based on few rules be it deterministic or clothing images was done on Deep Fashion dataset
random. Another method is a combination of the above two containing cloth location annotated by rectangular box. The
patterns which we call as mosaic models. system was evaluated against various cloth images available
online on e-commerce sites. Ground truth for the image
The mosaic models depict randomized geometrical dataset were obtained using the online shopping catalog
processes. Visual Seek considers object layout as an image containing metadata for five properties of color, style, sleeve-
feature [2]. The visually distinguishing features like object type, neck design and hemline. Automatic annotation
shape, colour or edges are very good descriptors for image collection showed up mean rate of 83% rate accuracy. In the
retrieval but they are vulnerable in case the query image is a analysis part, the authors have sought the effects of a number
grayscale image that too, a sketch of the object [2]. of improvements attempted like data augmentation under
Since almost every object recognition system concentrates on varying backgrounds, network size increase and ensemble
obtaining rotation and translation invariant features, there are usage on the classification rate. Also, the classification rate
methods that classify these into the following: 1. Image versus processing ease is computed in this work. The most
alignment 2. Invariant features. optimal network organizations for accurately classifying the
input clothing images are also presented.
III. RELATED WORK
The authors have carried out experiments by incrementing
This section discusses the related work in various papers the network model size gradually to test and record the best
with respect to image retrieval in order to analyze the possible classification results but the computational
previous approaches and come up with a better system for complexity is compromised. Apart from a single network,
designing a shopping guide with image retrieval using a ensembles of 5 networks are considered. The popular pre-
mobile interface. trained GoogLeNet classifier is used for fashion textile
classification by eliminating the last fully-connected layer
A. Nodari et.al [3] The authors have proposed a mobile app and pre-training on the Image Net dataset. Image Net dataset
that works in sync with a Content-Based Image Retrieval is a huge sized image collection containing around 1.2
system for online shopping of fashion products. The million images with 1000 categories. Fine tuning the last
application is able to retrieve most matched products of a fully-connected layer with the fashion image dataset
textile image given as an input. The proposed method works collected is finally done. The system classifies a training
by manual selection of the product name by the end user image into one of the classes from a possible 24 fine grained
framed by the camera image which is then sent to the server. classes.
Next, a search for image retrieval is initiated that
automatically recognizes and maps the object similar to the Dibin Zhou, Baokun Hu et.al [5] In this work, the authors
requested one reducing the effort of a prolonged manual have carried out a study on mobile phone based image
search. To measure the performance of the image retrieval retrieval system for the design of shopping guide for online
system designed, the authors have validated against three shopping website. This guide helps in various items to be
datasets- First set involves products of clothing category retrieved from the online repository of products using mobile
from various online shopping websites and the other two interface. The design is aimed at providing easy to use
datasets are the images as well as video frames of clothing interactive methods and innovative layout for image retrieval

© 2019, IJCSE All Rights Reserved 89

 International Journal of Computer Sciences and Engineering Vol. 7(14), May 2019, E-ISSN: 2347-2693

so that the drawbacks of existing image retrieval systems are described. The authors have extended the usual MIL method
resolved. The developed system has a decent practical value by incorporating DD-SVM, where a bag is labelled based on
and simplicity. a few instances satisfying stated rules. Diverse Density (DD)
function is used for learning instance prototype categories in
Liu Shuguang et.al [6] the authors have applied the wavelet a DD-SVM. Since every instance prototype is representative
transform and frequency gain for developing a good clothing of the class of instances to appear in bags with a fixed label
classification system based on the texture features in this than in the unlabeled bags, a nonlinear mapping to map each
work. According to them, the wavelet packet algorithm bag to a point in bag feature space is done. Support vector
works very well in case of classification of clothing types machines are used for training in the bag feature space.
efficiently for the following facts- The middle and low
frequency regain respectively contain the core energy of the P.F. Li, J. Wang et.al [9] In this work, the authors
texture image and usual image. So, with an increase in the concentrate on the task of automatic woven fabric
image resolution, the wavelet transform is driven towards classification using a novel feature extraction technique. The
low frequency range, while wavelet packet is focused on any local binary pattern and the gray level co-occurrence matrix
available frequency range. The authors have exploited the features are initially extracted for the fabric images in the
wavelet packet with Back Propagation neural network to data collection. To reduce high dimensional feature
classify the fabric types based on the texture. The developed information, principal component analysis is performed.
system showed 98% classification accuracy. Support vector machine classifier is employed for
recognition of the fabric image. The method is tested against
Tom Yeh1 et.al [7] Here, the authors have developed a three different types of woven fabric categories namely plain,
mobile interface based image search system producing the twill and satin weave fabrics.
relevant web pages on the Internet containing matching
objects similar to the query image. Image matching produces The below table 1 describes comprehensive analysis of
very accurate results when matching whole image or the research work done so far in image retrieval techniques.
scene is carried out like landmarks etc. or also in cases where Since the accuracy of the methods used in the existing work
the object to be matched has a distinguishable outline. The are more comparative and effective in detecting the object
authors have devised an interactive interface to obtain a from the images, these techniques can be used for our
segmented object boundary with shape matching algorithm research work in improving the accuracy for the detection of
to spot the desired objects of interest on Web pages. the images.
Yixin Chen et.al [8] In this paper, a novel Multiple-Instance
Learning (MIL) based technique for image classification is

Table 1: Comprehensive Analysis of Image retrieval Techniques used in Existing Research Works
Existing work Methodology used in Existing Advantages Limitations Percentage of
details work Accuracy
A mobile visual In this paper, the authors describe a Content based image No image based retrieval 87.434
search application mobile app that works in sync with a retrieval system. technique
for content based Content-Based Image Retrieval
image retrieval in system for online shopping of Dynamic query image No visual words fusion
the fashion domain fashion products. The proposed processing for Image CLEF and
By method works by manual selection image collections using
A. Nodari et.al [3] of the product name by the end user Automated search deep learning
framed by the camera image which technique
is then sent to the server.
Clothes detection In this paper, the authors have Proposed Convolution The retrieval of the query 76
and classification developed a computer vision system neural networks that cover image attributes should
using for efficient detection and Residual network still be improved.
convolutional classification of clothing images for
neural networks e-shopping images. A combined Data Augmentation Network size increased.
By Jan architecture involving convolution
Cychnerski, neural networks that cover Residual Ease of processing speed.
Adam Brzeski networks, Squeeze Net and Single
et.al.[4] Shot MultiBox Detector (SSD) is
considered.
Design of In this paper, the authors have The developed system has The proposed CBIR is Performance is
Shopping Guide carried out a study on mobile phone a decent practical value not extended with a more measured in
System with Image based image retrieval system for the and simplicity. variety of image features. terms of time,

© 2019, IJCSE All Rights Reserved 90

 International Journal of Computer Sciences and Engineering Vol. 7(14), May 2019, E-ISSN: 2347-2693

Retrieval Based on design of shopping guide for online which retrieves

Mobile Platform shopping website. This shopping Providing easy to use The weight assignment images from 0.01
By guide helps in various items to be interactive methods and process is not refined to 0.3 seconds.
Zhou, Dibin et. retrieved from the online repository innovative layout for based on detailed
al.[5] of products using mobile interface. image retrieval. analysis of feature
vectors.
Fabric Texture The authors have presented the Wavelet packet algorithm Low frequency 99.1 for D17D55
Classification wavelet transform and frequency works very well in case of transformation.
Based on Wavelet gain for developing a good clothing classification of clothing.
Packet By classification system based on the
Liu Shuguang texture features in this work. Back Propagation neural.
et.al [6]
A Picture is Worth The authors have developed a Search system producing Problem with similar Accuracy is
a Thousand mobile interface based image search the relevant web pages. object detection. measured based
Keywords: Image- system producing the relevant web on the shape of
Based Object pages on the Internet containing Accurate object matching. object detection
Search on a matching objects similar to the
Mobile Platform query image. Image matching
By produces very accurate results when
Tom Yeh1 et. matching whole image or the scene
al.[7] is carried out like landmarks etc. or
also in cases where the object to be
matched has a distinguishable
outline.
MILES: Multiple- This paper presents a novel Diverse Density (DD) Nonlinear mapping is a 97.17
Instance Learning Multiple-Instance Learning (MIL) function is used for big concern.
By based technique for image learning instance
Yixin Chen et.al classification is described. The prototype categories in a Proposed MIL failed in
[8] authors have extended the usual DD-SVM. classifying overall image
MIL method by incorporating DD- retrieval.
SVM, where a bag is labelled based Multiple-Instance
on a few instances satisfying stated Learning (MIL) based
rules. technique

IV. CONCLUSION REFERENCES

This paper presents a comprehensive review analysis of [1] Mehmood, Zahid and Abbas, Fakhar and Mahmood, Toqeer and
various existing image retrieval techniques with their Javid, Muhammad Arshad and Rehman, Amjad and Nawaz,
problems. This survey provides the analysis of how the Tabassam, Content-Based Image Retrieval Based on Visual Words
various image retrieval techniques can be applied over Fusion Versus Features Fusion of Local and Global Features,
fashion domain and the problem faced during the existing Arabian Journal for Science and Engineering, 2018, pp. 1-20.
[2] Katrien Laenen, Susana Zoghbi, and Marie-Francine Moens, Web
retrieval process. Once the existing problems are analyzed, In Search of Fashion Items with Multimodal Querying, Eleventh
future by using this work novel solution can be provided by ACM International Conference on Web Search and Data
implementing new image retrieval guide called “Click-n- Mining (WSDM '18), 2018, pp. 342-350.
Purchase: A shopping guide” , which provides an improved [3] Angelo Nodari, Matteo Ghiringhelli, Alessandro Zamberletti,
image retrieval based searching of the various interested items Marco Vanetti, Simone Albertini, Ignazio Gallo, “A mobile visual
search application for content based image retrieval in the fashion
from the fashion domain. The accuracy of the new application domain”, 10th International Workshop on Content-Based
will be measured by comparing with the existing techniques Multimedia Indexing, 2012.
used and presented in this work. [4] J. Cychnerski, A. Brzeski, A. Boguszewski, M. Marmolowski and
M. Trojanowicz, "Clothes detection and classification using
convolutional neural networks," 2017 22nd IEEE International
ACKNOWLEDGMENT
Conference on Emerging Technologies and Factory Automation
(ETFA), Limassol, 2017, pp. 1-8. doi:
The author would like to acknowledge the esteemed guidance 10.1109/ETFA.2017.8247638
given by Dr. Suvarna Nandyal, Professor and Head, PDA [5] Zhou, Dibin & Hu, Baokun & Wang, Qihui & Hu, Bin & Jia,
College of Engineering, Kalaburagi, Karnataka to complete Leiping & Wu, Yingfei & Xie, Lijun. “Design of Shopping Guide
this work. System with Image Retrieval Based on Mobile Platform”.
10.2991/3ca-13.2013.37, 2013.
[6] Liu Shuguang, Qu Pingge “Fabric Texture Classification Based on
Wavelet Packet”, The Eighth International Conference on
Electronic Measurement and Instruments,2017.

© 2019, IJCSE All Rights Reserved 91

 International Journal of Computer Sciences and Engineering Vol. 7(14), May 2019, E-ISSN: 2347-2693

[7] Tom Yeh1, Kristen Grauman1, Konrad Tollmar2, Trevor Darrell, in International Conference on Computer Vision, 2011, pp. 209–
“A Picture is Worth a Thousand Keywords: Image-Based Object 216.
Search on a Mobile Platform”, CHI 2005, April 2-7, 2005. [25] L. Zheng, S. Wang, and Q. Tian, “Coupled binary embedding for
Portland, Oregon, USA. large-scale image retrieval,” IEEE Transactions on Image
[8] Yixin Chen, Member, IEEE, Jinbo Bi, Member, IEEE, and James Processing (TIP), vol. 23, no. 8, pp. 3368–3380, 2014.
Z. Wang, Senior Member, IEEE, “MILES: Multiple-Instance [26] Y. Cao, C. Wang, L. Zhang, and L. Zhang, “Edgel index for
Learning”, IEEE TRANSACTIONS ON PATTERN ANALYSIS largescale sketch-based image search,” in IEEE Conference on C
AND MACHINE INTELLIGENCE, VOL. 28, NO. 12, Vision and Pattern Recognition (CVPR), 2011, pp. 761–768.
DECEMBER 2006 [27] J.-P. Heo, Y. Lee, J. He, S.-F. Chang, and S.-E. Yoon, “Spherical
[9] P. F. Li, J. Wang, H. H. Zhang and J. F. Jing, "Automatic woven hashing,” in IEEE Conference on Computer Vision and Pattern
fabric classification based on support vector machine," Recognition (CVPR). IEEE, 2012, pp. 2957–2964.
International Conference on Automatic Control and Artificial [28] J. Tang, Z. Li, M. Wang, and R. Zhao, “Neighborhood
Intelligence (ACAI 2012), Xiamen, 2012, pp. 581-584. discriminant hashing for large-scale image retrieval,” IEEE
[10] Min, Weiqing and Jiang, Shuqiang and Wang, Shuhui and Xu, Transactions on Image Processing (TPI), vol. 24, no. 9, pp. 2827–
Ruihan and Cao, Yushan and Herranz, Luis and He, Zhiqiang,”A 2840, 2015.
survey on context-aware mobile visual recognition, Multimedia [29] L. Wu, K. Zhao, H. Lu, Z. Wei, and B. Lu, “Distance preserving
Systems, 23(6), 2017, pp. 647-665. marginal hashing for image retrieval,” in IEEE International
[11] Weiqing Min, Shuqiang Jiang, Shuhui Wang, Ruihan Xu, Yushan Conference on Multimedia and Expo (ICME), 2015, pp. 1–6.
Cao, Luis Herranz, and Zhiqiang He, “A survey on context-aware [30] K. Jiang, Q. Que, and B. Kulis, “Revisiting kernelized
mobile visual recognition”. Multimedia Systems, 2017, pp. 647- localitysensitive hashing for improved large-scale image
665. retrieval,” in IEEE Conference on Computer Vision and Pattern
[12] Mitul Kumar Ahirwal, Anil Kumar, and Girish Kumar Singh, “An Recognition (CVPR), 2015, pp. 4933–4941.
Approach to Design Self Assisted CBIR System”, International
Conference on Graphics and Signal Processing (ICGSP'17),pp. 21- Authors Profile
25.
[13] Xin Ji, Wei Wang, Meihui Zhang, and Yang Yang, “Cross- Mr. Nikhil. S. Tengli received his B. E. Degree in Computer
Domain Image Retrieval with Attention Modelling”, ACM on Science from VTU, Belagavi, M.Tech. degree in Computer
Multimedia Conference(MM'17),2017,pp.1654-1662. Science and Engineering from VTU, Belagavi. He is
[14] C. Huang, S. Zhang, X. Lin, X. Liu and R. Ji, "Deep-based fisher pursuing Ph.D. under VTU, Belagavi in the area of Image
vector for mobile visual search“, IEEE International Conference Processing and Machine Learning. He has experience of 5
on Image Processing (ICIP), 2017, pp. 3430-3434.
[15] Y. H. Kuo and W. H. Hsu, "Dehashing: Server-Side Context- years in teaching in Computer Science and Engineering and
Aware Feature Reconstruction for Mobile Visual Search“, IEEE 1 year in Industry. His research interests are: Image
Transactions on Circuits and Systems for Video Technology, processing, Cloud computing, Machine learning and Big
27(1), 2017, pp. 139-148. data. He has published 8 papers in UGC indexed journals.
[16] A. Rahman, E. Winarko and M. E. Wibowo, "Mobile content
based image retrieval architectures,“ 4th International Conference
Dr. Suvarna Nandyal has completed BE and M.Tech from
on Electrical Engineering, Computer Science and Informatics
(EECSI), 2017, pp.1-4. PDA College of Engg. Gulbarga and completed Ph.D. from
[17] C. Corbière, H. Ben-Younes, A. Ramé and C. Ollion, "Leveraging JNT University, Hyderabad. She has taken charge of HOD
Weakly Annotated Data for Fashion Image Retrieval and Label computer science and engg. department from 8th may 2015.
Prediction,“ IEEE International Conference on Computer Vision Her area of research is Digital Image Processing. She has
Workshops (ICCVW), 2017, pp. 2268-2274. published more than 50 research papers in International and
[18] J. Sivic and A. Zisserman, “Video Google: A text retrieval
approach to object matching in videos,” in IEEE Conference on
National Journals. She has attended many national and
Computer Vision and Pattern Recognition, 2003, pp. 1470–1477. international conferences. She has organized workshops on
[19] J. Philbin, O. Chum, M. Isard, J. Sivic and A. Zisserman, Object current topics like Digital Image Processing, Cloud
retrieval with large vocabularies and fast spatial matching, 2007 computing and many more. She has presented a paper at
IEEE Conference on Computer Vision and Pattern Recognition, International conference on Digital Image Processing at
Minneapolis, MN, 2007, pp. 1-8. Singapore. She has worked as P.G. Coordinator. She has
[20] W. Zhou, Y. Lu, H. Li, Y. Song, and Q. Tian, “Spatial coding for
large scale partial-duplicate web image search,” in ACM organized workshops on Digital Image Processing and Cloud
International Conference on Multimedia, 2010, pp. 511–520. Computing.
[21] O. Chum, J. Philbin, J. Sivic, M. Isard, and A. Zisserman, “Total
recall: Automatic query expansion with a generative feature model
for object retrieval,” in International Conference on Computer
Vision, 2007, pp. 1–8.
[22] D. Nister and H. Stewenius, “Scalable recognition with a
vocabulary tree,” in IEEE Conference on Computer Vision and
Pattern Recognition, vol. 2, 2006, pp. 2161–2168.
[23] Z. Wu, Q. Ke, M. Isard, and J. Sun, “Bundling features for large
scale partial-duplicate web image search,” in IEEE Conference on
Computer Vision and Pattern Recognition, 2009, pp. 25–32.
[24] X. Wang, M. Yang, T. Cour, S. Zhu, K. Yu, and T. X. Han,
“Contextual weighting for vocabulary tree based image retrieval,”

© 2019, IJCSE All Rights Reserved 92