26th International Computer Conference, Computer Society of Iran, Tehran, Iran

Content-based Clothing Recommender System using Deep Neural Network

Narges Yarahmadi Gharaei Chitra Dadkhah Lorence Daryoush

Faculty of Computer Engineering, Faculty of Computer Engineering, Amirkabir Robotic Research Institute(ARRI)
K. N. Toosi university of Technology K. N. Toosi university of Technology Amirkabir University of Technology
Tehran, Iran Tehran, Iran (TehranPolytechnic)
[email protected] [email protected] Tehran, Iran
[email protected]
2021 26th International Computer Conference, Computer Society of Iran (CSICC) | 978-1-6654-1241-4/20/$31.00 ©2021 IEEE | DOI: 10.1109/CSICC52343.2021.9420544

Abstract—A recommender system primary purpose is to available in the system, offers a list of recommendation
provide a series of item suggestions on a topic to its user. Deep to the user. The cold start problem is also present.
learning is used in many fields and solved difficult and complex 3) Knowledge-based recommender system is appropriate
problems with large volumes of data. Deep learning can also be for special items that are not included in daily basket of
used in referral systems. Today, online shopping systems are
online shopping such as car, apartment. In this kind of
looking for a method that can recommend items according to
the user preference and interest in order to increase their sales. systems user's preferences will be fed to system to make
Clothing sales systems offer a set of recommendation based on a recommendation based on user requirement. For
the needs and interests of the users. Today, due to the current instance in case of purchasing a car, information like car
situation caused by the Coronavirus, the majority of tasks are mileage, price, and color will be defined by users to filter
done online. In this paper, we propose a content-based clothing database according to user desire. This system has no
recommender system using deep neural network. In content- cold start problem.
based systems, product features are required for prediction of 4) Hybrid recommender systems are combination of
unobserved items ratings. In our proposed system by using a different recommender systems; the use of Hybrid
deep neural network, the cloth category is obtained and the need
recommender systems has become much more prevalent
to manually extract the product features is eliminated by
producing the required features with a large and useful volume. due to their high accuracy.
The advantage of this system is that it uses the same network to
specify gender as a feature in making suggestions then shows the Machine learning (ML) is a portion of artificial
results to the user. Different machine learning algorithms are intelligence that makes system learned from data without
tested and analyzed with and without considering demographic explicitly programming. Among the application of machine
information such as gender. The experimental results show that learning, we can mention business [2], advertising [3] and
the loss of our proposed system is lower than the other related medicine [4].
systems and solves the cold start problem for new items. Our Deep learning the most common method to model a
proposed system also recommends novel, relevant and
problem, is a branch of ML. it has the ability to solve many
unexpected items.
challenging problems in machine learning [5]. It can be used
Keywords—Clothing, Recommender System, Deep learning, in variety of fields such as text processing, image processing,
Demographic, Feature Extraction, Cold start, Content, audio and signal processing and is also applicable in
Coronavirus recommender systems. CBRS needs its products features and
I. INTRODUCTION extracts them in different ways, which will be explained in
the next section.
Recommender systems are used in a variety of fields, In this paper, we use deep neural networks to identify the
including music, movies, and books. The main purpose of product category and gender and exploit the same network to
these systems is helping the users reach their needs faster and extract features.
it is also useful in online sales systems. Recommender The structure of the paper is as follows: we referred the
systems are divided into 4 general categories as follows [1]. related works in Section II and explained the problem in
Section III. The dataset used in our system is introduced in
1) Collaborative Filtering Recommender Systems (CFRS) Section IV. Section V is dedicated to explaining the strategy
which find users with close behaviors and tastes. If one and the used deep neural networks. Section VI shows the
of the two users with similar behavior has bought a experimental results. Finally Section VII presents the
product and has given it a positive rating, the system conclusion.
recommends this product to the other user guessing with
high probability that this user will like this product as II. RELATED WORKS
well. One of the problems that these systems face is the Here are some of the recent developments in content-based
cold start problem in which when there is a new recommender systems based on machine learning.
user/item in the system with no background so no Younus et al in [6] proposed Content-Based Image
neighbor can be found for user/item. Retrieval (CBIR) method aim to retrieve images accurately
2) Content-based Recommender Systems (CBRS) unlike from large image databases similar to the query image based
the CFRS are not interested in the similarity between on the similarity between image features. A new hybrid
users. These systems basically deal with the resemblance method has been proposed for image clustering using the
and closeness of the items according to the data given to combination of the Particle Swarm Optimization (PSO) with
them. Based on target user's history, the items they have k-means clustering algorithms. Their method uses the color
bought and the similarity of them with the items and texture images as visual features to represent the images.

978-1-6654-1241-4/21/$31.00 ©2021 IEEE

Authorized licensed use limited to: East Carolina University. Downloaded on June 21,2021 at 01:25:16 UTC from IEEE Xplore. Restrictions apply.
Their proposed method is based on four feature extractions based on the user's requests and tastes. Sales systems also
for measuring the similarity, which are color histogram, color need a system that improves the speed, quantity, and quality
moment, co-occurrence matrices, and wavelet moment. of their sales. Therefore, clothing recommender systems
Ali et al [7] uses SIFT feature extraction algorithm for bring mutual benefit for both customers and the merchant
feature extraction, which gives the key point in an image for systems. So, we provided a recommender system that meets
implementing the features that CBIR needs. They use the all these goals.
optimization technique BFOA (Bacteria Foraging
Optimization Algorithm) to reduce the complexity, cost, Content-Base recommender systems require product
energy, and Time consumption because of not being valuable features. In a sense, due to the fact that the data volume of
the SIFT image feature algorithm. these products is considerably high, entering the features of
There have been a lot of researches and works in content these products manually into system is extremely time-
based recommender systems, but in a lot of them deep consuming. One of the most important benefits of using deep
learning is not used, for example the researches in [8, 9, 10] learning is its ability to execute feature extraction. Another
so they are not included in our review. benefit of it is the elimination of the need for data labeling
Yun-Rou Lin et al in [11] presented (appropriate) clothing and maximum utilization of unstructured data. So these
with proper size recommendation by considering factors such reasons encouraged us to use deep learning.
as gender, body height and clothing features. Online texture
modeling was implemented to produce variety in the texture We have presented a recommender system that eliminates
of clothing so their proposed system can create logical and the need to provide the product features manually. Our
diverse options for consumers. They used CNN system extracts its required features with high volume from
(Convolutional Neural Network) for gender recognition, Dlib just one product image using a Deep Neural Network (DNN).
for face detection [12], and InceptionV3 (GoogleNet) for These features are more explained in section V. So, for each
clothing attributes recognition. Their attributes contain five new product in the system, the feature extraction process is
clothing categories with eight dimensions. performed only by presenting an image of the product. Also
Batuhan AŞIROĞLU et al in [13], developed a our system has solved the cold start problem for new items.
recommender system by taking just a single photo of the user
with their scalable embedded system. Their recommendation For recommendation, it is better to identify the gender of the
is made regardless of shopping history of the user. The product so that the clothes we offer are suitable for them. The
utilized hardware is a Linux based system and a low-cost gender recognition section of our system is also performed by
Raspberry Pi Zero W. They developed two inceptions based a DNN. The systems in [13] performed gender identification
CNN, one for the prediction part and one for feed-forward solely on the user's face image. Gender of the items in their
NN as a recommender system. The prediction part exploits system are already available and they do not perform
from Haar-Cascade of OPENCV python library for user's diagnostic operations. These systems when compared to the
face gender detection. The personal information extraction proposed system in [13] have the following differences.
is done by inception based on deep CNN from the 1. Our system does not need the face of the model to
processed image. The last step is recommending the best- recognize the gender. The procedure is done based on
fitted cloth to the user by another CNN. Their cloth dataset the style and the type of clothing.
contains 132 samples. 2. We have eliminated the need of defining the gender
Yufan Wen et al in [14],, constructed a knowledge graph attribute for the product.
of the user, a knowledge graph of clothing, and a knowledge Our proposed system also uses the same DNN for
graph of context. They used the Apriori algorithm to capture product gender recognition and feature extraction and it
the correlation between clothing attributes and context also recommends novel, relevant and unexpected items.
attributes. They generated recommendation results directly
by taking into account the top-N algorithm and searching and IV. DATASET
matching the established knowledge graph which is based on The utilized data set used in the gender detection, feature
user's requirements. extraction and suggestion is Fashion Product Images (Small)
from Kaggle site. This database contains 44,000 images of
III. MOTIVATION different types of products. 50% of the images in this
Today, due to the current situation caused by the database are related to men's products, 42% belongs to
Coronavirus, the majority of tasks are done online. Online women and the remaining 8% belongs to Unisex, Girls, and
clothes shopping systems are one of them. Although they Boys. 48% of images are related to Apparel, 52% belong to
have excited before but because of the current situation a lot Accessories and the remaining 26% belong to other
more attentions have been turned towards them. Therefore, categories. In this paper, we have only used images related to
the users of these systems have increased compared to the Apparel. The total number of categories in this database is
past so now we are dealing with big data. LUISAVIAROMA, 143, which we have only used the categories mentioned in the
Fashionphile, Runway Catalog, and Amazon are examples of table I. Therefore, we have worked on 14932 images. Fig. 1
these systems. and Fig. 2 display data scatter based on article type and gender
and the numbers written in the pie chart is their percentage.
So, the main challenge of the sale industries is creating a More detailed information about this database is available on
system to deal with big data. Users usually get confused when the Kaggle website at
they face a wide range of options. Therefore, there is a need https://www.kaggle.com/paramaggarwal/fashion-product-
for a system that immediately offers the right target product images-small.

Authorized licensed use limited to: East Carolina University. Downloaded on June 21,2021 at 01:25:16 UTC from IEEE Xplore. Restrictions apply.
 fully connected multilayer perceptron. That means each
TABLE I. ARTICLE TYPES IN DATASET neuron in each layer is connected to all neurons in the next
layer. Each convolution layer convolves the input by a
Article type counts specific filter and passes its result to the next layer [15]. As
Waistcoat 15
Stocking 32 images have high dimensions, every pixel of an image is
Skirts 128 considered as a feature, so when we want to pass it to a
Jackets 258 multilayer perceptron the volume of computations and
Track pants 304 parameters grows rapidly. CNNs are an efficient solution for
Dresses 464 reducing the number of parameters without losing quality on
Trousers 530
Shorts 547
the models. Since our data is an image, we also use CNNs.
Jeans 608 We have implemented our design model with Keras for all
Tops 1761 three purposes as shown in the Fig. 3.
Shirts 3215
T-shirts 7060 The names of the methods used in different layers in the
model is written on the guide on the right side of the figure
according to their color used in the schematic. This network
has 13 layers. The kernel size for all layers is 4*4. Activation
function of all layers except the last one is RELU. The last
layer activation function is SIGMOID. This network is only
trained once for learning the gender of the products.
Input of this network is images and its labels are based on
the gender category of dataset and we call the trained model
GenderModel. Then, to identify the clothing article type, we
train this network by images as input and articles types as
labels, and this time we save the model weights in a separate
Fig. 1. Pie chart of data scatter rate based on article type.
file from the previous model named ArticleTypeModel. The
training is done with 50 epochs. Our loss function is
Categorical_focal_loss with alpha 0.25 and gamma equals to
2. We minimized the error by Nadam [16] optimization
algorithm. The mentioned alpha and gamma value gave us
the best output among the various tests. Our metric for
evaluation is accuracy.

Two models created by this network are used to obtain the

features of the items. We have considered the output obtained
by the weights of the 12th layer as extracting features layer in
Fig. 2. Pie chart of data scatter rate based on gender type. this network. The size of the resulting vector by both
networks is (14932, 68). Therefore, we have merged the two
V. PROPOSED METHOD vectors of the extracted features together and used them as
Our proposed system uses a deep convolution neural image features. So the size of the vectors becomes (14932,
network to detect categories, product gender, and feature 136) by considering feature extraction on the network, it can
extraction from an image. A Convolutional Neural Network be pointed out that, this vector is a black box. Considering the
(CNN, or ConvNet) is a class of deep neural networks, mostly high dimension of the obtained vector, the features we have
used to analyze images and videos. CNNs are a version of achieved are certainly better than the features extracted by

Fig. 3. Fully convolutional neural network schema in our proposed system

Authorized licensed use limited to: East Carolina University. Downloaded on June 21,2021 at 01:25:16 UTC from IEEE Xplore. Restrictions apply.
humans. Each feature that we have some information about it B. Recommendation precision
in dataset can be included in the training phase. We just tested In each row of, some examples of recommended images
2 explicit feature (gender and article type) and the result was according to user's purchase history can be seen. The images
satisfying. Including more explicit features like brand and that have the original label are the images of the purchased
quality are guessed to achieve better results. Although the product by the user and the rest of images are the top-K items
model may extract these features from the image in spite of recommended by our proposed system. As it is shown, the
involving them explicitly. recommended clothes are very similar to the original selected
Weights are already set by the training phases on the clothes by the user. Also, among them, there are several
available data. After adding new data to our data set the unexpected recommendations which have some similar
training process will be continued by the previous weights so features and some very different ones. For testing the
less time is spent for training the new data. accuracy of our recommendations, we have used the ground-
The similarity between the extracted feature vectors of the truth evaluation, so the feedbacks were given by real users
images is calculated using the cosine similarity as Eq. 1. To and they were analyzed as it is shown in Table IV. The
receive a recommendation from the system, it is enough to recommendation precision of our proposed system is about
obtain the feature vector of the user image by both models 73.7% which means the users liked 73.7% of our
and then combining them, using the cosine similarity, find recommendations. According to the recommended items in
and arrange the products with the most similarity and present Fig. 6, you can see the degree of novelty with the cosine
the Top-N of the similar products. similarity metric in table IV. As shown in Fig. 6 by
considering that the similarity of these 10-Top recommended
⃗. ⃗ ∑ ∗
cos = ‖ ⃗‖ ⃗
= (1) item don’t equal to 1, it can be concluded that the system
∑ ∗ ∑ never recommends the items in user's Purchase history. By
using datasets containing more clothes in each category, the
recommendation accuracy will increase. The average time
VI. RESULTS AND EXPERIMENTS required to assign recommendations per user is 0.07 second.
The experimental result has been divided into two parts:
prediction and recommendation accuracy.
TABLE IV. THE PRESICION OF RECOMMENDATION

A. Prediction accuracy User Number of Liked items by Precision

ID recommends items user
We have implemented the explained algorithms and 1 500 358 71.6 %
ResNet-50 [17] in python on the dataset. 80% of data is used 2 500 313 62.6 %
for train and 20% for test. As we have showed in table II loss 3 500 354 70.8 %
of ResNet-50 model for gender detection is 0.136 but for our 4 520 418 80.38 %
5 410 295 71.90 %
GenderModel is 0.0334. In table III loss of ResNet-50 for total 2430 1793 73.7 %
Article type detection is 0.1216 and loss of our model is about
0.0334. Also we have presented the confusion matrix and a VII. CONCLUSION
test of all implementations in Fig. 4 and Fig. 5. According to Nowadays introducing new products to prevent the user
Fig. 1, we have realized that we have a lot of heterogeneity in
from getting bored is a challenge. In this article, we have
terms of numbers in different categories of data. Therefore,
presented a method to recommend clothes. Our proposed
we cannot expect high accuracy from the algorithm. system solves the cold start problem for new items by
The elapsed time for training the ArticleTypeModel is considering their features. Our proposed system uses the
652 seconds and the GenderModel is 735 seconds for training DNN for extracting the features from the image of the items
data by our proposed method. As you can see, the training and our system is evaluated using feedbacks from real users
time is acceptable.
using precision metrics and the results show that the system
is both efficient and precise with 73.7% precision.
TABLE II. COMPARING LOSS AND ACCURACY OF MODELS IN
Recommending serendipitous item is one of the benefits of
GENDER DETECTION our proposed system. Also, it has high speed due to its
simplicity. In the future, we intend to compare our system
Model name Loss Accuracy with other content base clothing recommendation algorithms
1 ResNet-50 0.136 86.24% which doesn't use deep learning methods and measure and
2 Our Keras model 0.0334 83.29% improve the performance of our proposed method. The
source code is available at
https://colab.research.google.com/drive/1_fhBCR1jbg3cnoh
4Afmz7iZaK6xC8Bbo?usp=sharing.
TABLE III. COMPARING LOSS AND ACCURACY OF MODELS IN
ARTICLE DETECTION

Model name Loss Accuracy

1 ResNet-50 0.1216 68.26%
2 Our Keras model 0.0334 66%

Authorized licensed use limited to: East Carolina University. Downloaded on June 21,2021 at 01:25:16 UTC from IEEE Xplore. Restrictions apply.
 TABLE V. COMPARING THE NOVELTY OF RECOMMENDED [7] A. Ali and S. Sharma, "Content based image retrieval using feature
ITEMS extraction with machine learning," 2017 International Conference on
Intelligent Computing and Control Systems (ICICCS), pp. 1048-1053,
Original 2017.
images 13770 52940 32210 11559 59589
Recommended [8] P. K. Gopalan, L. Charlin and D. Blei, "Content-based
items recommendations with Poisson factorization.," Advances in Neural
Item 0 0.999468 0.999129 0.999672 0.999389 0.999493 Information Processing Systems, p. 3176–3184, 2014.
Item 1 0.999462 0.999065 0.999578 0.999373 0.999417 [9] A. Pujahari and V. Padmanabhan, "An Approach to Content Based
Item 2 0.999409 0.999050 0.999459 0.999379 0.999280 Recommender Systems Using Decision List Based Classification with
Item 3 0.999377 0.999029 0.999432 0.999364 0.999233
k-DNF Rule Set," in IEEE, Bhubaneswar, India, 22-24 Dec. 2014.
Item 4 0.999377 0.998996 0.999422 0.999274 0.999217
Item 5 0.999363 0.998978 0.999422 0.999273 0.999217 [10] L. Yu-Chu, Y. Kawakita, E. Suzuki and H. Ichikawa, "Personalized
Item 6 0.999327 0.998696 0.999422 0.999249 0.999180 Clothing-Recommendation System Based on a Modified Bayesian
Item 7 0.999308 0.998693 0.999363 0.999162 0.999128 Network," in IEEE, Izmir, Turkey, 16-20 July 2012.
Item 8 0.999292 0.999620 0.999310 0.999162 0.999128
[11] Y.-R. Lin, W.-H. Su, C.-H. Lin, B.-F. Wu, C.-H. Lin, H.-Y. Yang and
Item 9 0.999273 0.998618 0.999308 0.999131 0.999120
M.-Y. Chen, "Clothing Recommendation System based on Visual
Average 0.999366 0.998987 0.999439 0.999276 0.999241
Information Analytics," in 2019 International Automatic Control
Conference (CACS), Keelung, Taiwan, Taiwan, 05 March 2020.
REFERENCES
[12] D. E. King, "Dlib-ml: A Machine Learning Toolkit," Journal of
Machine Learning Research, vol. 10, pp. 1755-1758, December 2009.
[1] C. C. Aggarwal, Recommender Systems, Springer, 2016. [13] B. AŞIROĞLU, M. İ. ATALAY, A. BALKAYA, E. TÜZÜNKAN,
[2] C. Apte, "The role of machine learning in business optimization," In M. Dağtekin and T. ENSARİ, "Smart Clothing Recommendation
ICML, 2010. System with Deep Learning," in IEEE, Ankara, Turkey, Turkey, 11-
[3] Q. Cui, F.-S. Ba, B. Gao and T.-Y. Liu, "Global Optimization for 13 Oct. 2019.
Advertisement Selection in Sponsored Search," Journal of Computer [14] Y. Wen, X. Liu and B. Xu, "Personalized Clothing Recommendation
Science and Technology 30, vol. no. 2, pp. 295-310, 2015. Based on Knowledge Graph," in IEEE, Shanghai, China, 06
[4] A. Rajkomar, J. Dean and I. Kohane, "Machine Learning in September 2018.
Medicine," New England Journal of Medicine 380, vol. no. 14, pp. [15] K. O'Shea and R. Nash, "An introduction to convolutional neural
1347-1358, 2019. networks.," arXiv preprint arXiv:1511.08458, 2015.
[5] Y. LeCun, Y. Bengio and G. Hinton, Deep learning, Nature, 2015. [16] T. Dozat, "Incorporating Nesterov Momentum into Adam," ICLR
[6] Y. Zeyad Safaa , M. Dzulkifli , S. Tanzila , Mohammed Hazim Workshop, 2016.
Alkawaz, R. Amjad , Mznah Al-Rodhaan and Abdullah Al-Dhelaan, [17] K. He, S. R. X. Zhang and J. Sun, "Deep residual learning for image,"
"Content-based image retrieval using PSO and k-means clustering The IEEE Conference on Computer Vision and Pattern, June 2016.
algorithm.," Arabian Journal of Geosciences 8, vol. no. 8, pp. 6211-
6224, 2015.

a b

c d
Fig. 4. a) Confusion matrix of our keras model for article type b) Our keras ArticleTypeModel test

c) Confusion matrix of ResNet-50 for article type detection d) ResNet-50 artice type detection model test

Authorized licensed use limited to: East Carolina University. Downloaded on June 21,2021 at 01:25:16 UTC from IEEE Xplore. Restrictions apply.
 a b

c d

Fig. 5. a) Confusion matrix of our keras model for gender detection b) Our keras GenderModel test

c) Confusion matrix of ResNet-50 model for gender detection d) ResNet-50 gender detection model test

Fig. 6. Top-K recommendation based on original user clothing

Authorized licensed use limited to: East Carolina University. Downloaded on June 21,2021 at 01:25:16 UTC from IEEE Xplore. Restrictions apply.