Clothing Recommendation System based on Visual

Information Analytics
Yun-Rou Lin Wei-Hsiang Su Chub-Hsien Lin Bing-Fei Wu
Institute of Electrical and Institute of Electrical and Institute of Electrical and Institute of Electrical and
Control Engineering Control Engineering Control Engineering Control Engineering
National Chiao Tung National Chiao Tung National Chiao Tung National Chiao Tung
University University University University
Hsinchu, Taiwan Hsinchu, Taiwan Hsinchu, Taiwan Hsinchu, Taiwan
lilzoe @cssp.cn.nctu.edu.tw [email protected] [email protected] [email protected]

Chang-Hong Lin Hsin-Yeh Yang Ming-Yen Chen

Information and Information and Information and
Communications Research Communications Research Communications Research
Laboratories Laboratories Laboratories
Industrial Technology Industrial Technology Industrial Technology
Research Institute Research Institute Research Institute
Hsinchu, Taiwan Hsinchu, Taiwan Hsinchu, Taiwan
[email protected] [email protected] [email protected]

Abstract—Due to the short fashion style life circle, much Second, it takes too much time to process the massive
more different clothing designs show up. It is hard for information of all consumers. Contented-based approach, [6],
consumers to find the suitable clothes effectively. To solve this [7], can avoid the problems as stated above. In the content-
problem, an automatic and reliable recommendation system is based system, the complex correlation among consumer
in great demand. In this paper, the clothing attributes preferences and past behaviors will not be considered.
recognition, gender recognition, and body height are considered Features of the item are abstracted to represent the item. And
to design the recommendation system. Based on the clothing the system will learn from interests of consumers to
style, gender and body height, the system can recommend the recommend items. There is still a problem that potential
proper clothes with suitable size. On-line texture modeling is
interests of consumers will be ignored due to the
proposed to produce the variation of the clothing texture so that
the recommendation system can give reasonable and diversified
recommendation strategy.
choices for the consumers. Besides, the data of consumers’ In this research, because of the difficulty in collecting
wearing style is also useful to make the better marketing enough profiles of preferences and activities from consumers,
strategy. According to the reasons above, the clothing collaborative filtering is not the best way to establish a
recognition and recommendation system can create a win-win recommendation system. In addition, the features of the
situation between the consumers and the fashion industry. clothing can be easily represented by attributes so that content-
based system is utilized with several parts of visual
Keywords—deep learning, clothing recognition, content-based
filtering, recommendation system
information, which contains personal and clothing
information. The main stages of our research are:
I. INTRODUCTION (1) Personal information prediction
Nowadays, the rapid growing of fashion industry
Considering data of gender and body height of the
contributes to complex and great amount of clothing design.
consumer can help implement a robust recommendation
Too much information of clothing design leads to dilemma of
system. A gender recognition prediction model is applied to
choosing from different clothes when consumers are shopping
predict the gender used for determining the clothing category,
at a apparel store or on a website. Hence, it is necessary to
and the body height used for estimating the proper clothing
apply an automatic and efficient way to filter information so
size is measured based on face detection.
that consumers can buy suitable clothes easier. Applying
recommendation systems will be the solution to deal with (2) Clothing attributes recognition
these problems.
While content-based filtering is adopted, clothing
For the related researches on recommendation systems, [1], attributes can be defined as features of clothing, so a clothing
[2], [3], recommendation systems can be categorized into attribute recognition model is a necessary part in the system.
three main types: collaborative filtering, contented-based Convolutional Neural Network (CNN) is known as the best
approach and hybrid approach. Collaborative filtering, [4], [5] , approach to object classification, and it is easier to realize if
is based on calculating the correlation which refers to the training data is ready; therefore, it is employed to
relationships among preferences, and behaviors of all recognize what kinds of attributes on the clothing of the
consumers. Through computing similarity of subjective consumer. These attributes will be the key factors for
evaluations to other consumers to obtain the correlation, it can searching the similar clothes in the clothing gallery.
predict what consumers will prefer. A major advantage of
(3) Recommendation system
collaborative filtering is that it is able to process complex
items which are not easy to extract appropriate features. When the system gets personal information and clothing
However, there are two problems when collaborative filtering attributes in the two previous stages, it can figure out an
is adopted. First, a new item will be rarely recommended. similarity score which is computed by similarity calculation

XXX-X-XXXX-XXXX-X/XX/$XX.00 © 20XX IEEE

among all these factors. Besides, a random generator is categories with eight dimensions, including Collar, Lapel,
designed to increase the diversity before the recommendation Neck, Neckline, Sleeve, Coat, Pants and Skirt, of fifty-four
result is generated, so the system will not only recommend attributes labels as shown in Fig. 2. The configuration of the
clothes which is the most similar to the wearing of the database is listed in Table I.
consumer but also make sure the potential interests of the
consumer are considered by adding randomness of
recommendation result.
The framework of our research is organized as follows.
Section II explains the details of method in our research.
Section III analyze the results of experiments. Section IV is
the conclusion of this paper.
II. METHODOLOGY
A. Personal information prediction
Recent developments in the field of gender recognition
have led to a renewed interest in CNN. The classic CNN
which is called AlexNet, [8], had been used for gender
recognition in [9] and [10], and the result is a major
breakthroughs. Based on the training method in [10], CelebA,
[11], is chosen to be the training dataset of the gender
recognition model. Instead of using the AlexNet, the
InceptionV3(GoogleNet), [12], [13], [14], is used as a
template in pursuit of a better performance. The accuracy
exceeds 98% on the LFW dataset, [15].

Fig. 1. A schematic diagram for our body height prediction: true view (left),
image view (right)

In order to measure the body height from an image, face

detection from Dlib, [16], is utilized to localize the vertex of
the consumer. As shown in Fig. 1, to find the body height, the
vertical field of view needs to be computed first, which is
defined as:
 (1) Fig. 2. Examples of FashionAI dataset

where v is the vertical field of view, x is the horizontal distance It will take less training time and the model will be more
between the consumer and the camera, θ is the vertical angle robust if the pre-trained model is applied. After testing some
of view of the camera. Moreover, the ratio of vertical field of state-of-the-arts CNN models, the InceptionV3(GoogleNet)
view to the distance between the vertex of the consumer and performs the best and its parameter quantity is relatively low.
the height of the camera, r, can be defined as:
However, the issue is that some of the attributes, like
(2)
Collar Design, Neck Design and Neckline Design, account
where ℎ𝑖 is the height of the image, ℎ𝑣 , which is called for a really small percentage of the full image. It is hard for
vertex height in Fig. 1, is the height of the face forecast the model to learn to extract the representable features of
position in the image. Hence, combine the height of the these attributes on an image. An effect method, [18], is
camera, y, and (1), (2), the prediction of the body height, h, proposed to localize the area where the model focuses on by
can be estimated as: analyzing the confidence maps of the model, so the same idea
is adopted to estimate the region of the clothing attributes. For
(3) the smaller sizes of the clothing attributes, the recognition
B. Clothing attributes recognition models are trained by both the whole image and the estimated
region. In this way, the model can learn more information
To recognize the clothing attritubes, FashionAI dataset, about these attributes so the accuracy can be improved from
[17], is utilized to train a CNN model. The FashinAI dataset 83.80% to 87.59%. .
which is built by Alibaba’s team contains five clothing
 The texture of the clothing is another important factor attribute, color features of tops and color features of bottoms
for describing the clothing style. Related research on color are labeled for the clothes in the clothing gallery, as shown in
expansion method, [19], shows a simple way that define the Table II. Each kind of labels is defined by different methods.
color by computing its value in HSV color space, and it helps Keywords from web crawler are used to label the gender. The
calculate the similarity in recommendation system. labels of Design type and Length type are determined by the
Consequently, texture extracting is proposed to extract the corresponding attribute recognition model. Labels of the
color feature from the clothing image. With the color feature, clothing gallery will be transformed into one-hot encoding
the color information of the clothing can be summarized. vector. Texture extracting method is applied to gain the color
features. The properties of clothing gallery are shown in Fig 3.
There are two steps for texture extracting. The first step is to
Since the clothing gallery contains more than ten thousand
find the location of the clothing by confidence map, and the
images, the variety of the clothing gallery can be achieved.
second step is to crop the area of the clothing into several
patches then calculate the average HSV value of all patches. TABLE II. CLOTHING ATTRIBUTES OF RECOMMENDATION SYSTEM
But the texture of tops tends to be more complex than bottoms
in general, it cannot represent the true composition of texture Main
accurately while calculating the average HSV value. In order Labels of attributes
types
to deal with the cases of tops, K-means is utilized to find the
major component of the HSV value among these patches. All
pixels captured from the patches are categorized into K Collar Shirt, Peter Pan, Puritan, Rib
clusters. And the centers of the cluster will then be the color
features.
Notched, Collarless,
Lapel
Shawl Collar, Plus Size Shawl
TABLE I. CLOTHING ATTRIBUTES FROM FASHIONAI DATASET
Design
Main Neck Turtle
Labels of attributes
types

Strapless Neck, Deep V, Straight Neck, V,

Collar Shirt, Peter Pan, Puritan, Rib Neckline Square, Round, Sweat Heart Neck, One
Shoulder

Notched, Collarless, Sleeve Sleeveless, Short, Elbow, Long, Extra Long

Lapel
Shawl Collar, Plus Size Shawl
Design
Turtle, Ruffle Semi-High Collar, High Waist, Regular, Long, Knee, Ankle &
Neck Coat
Low Turtle, Draped Collar Floor
Length
Strapless Neck, Deep V, Straight Neck, V,
Neckline Square, Off Shoulder, Round, Sweat Heart Pants Short, Mid, Cropped, Full
Neck, One Shoulder

Sleeveless, Cup, Short, Elbow, 3/4, Wrist, Skirt Short, Knee, Ankle, Floor
Sleeve
Long, Extra Long

High Waist, Regular, Long, Micro, Knee,

Coat
Midi, Ankle & Floor
Length

Pants Short, Mid, 3/4, Cropped, Full

Skirt Short, Knee, Midi, Ankle, Floor

C. Recommendation system
There are three steps in our recommendation system,
including building clothing gallery, similarity calculation, and
recommendation list generation. Fig. 3. Clothing gallery properties: clothing type(left), gender(middle),
Design type(right)
First, it is necessary to build a clothing gallery for
recommendation. Most of our clothing gallery is composed of The second step is to combine the clothing feature vectors
the data from FashionAI dataset and Large-scale Fashion into the similarity between the clothing of consumer and the
dataset, which is well-known as DeepFashion dataset, and the clothing gallery. The similarity is defined as the sum of
other parts are obtained by web crawler from well-known products among the clothing features:
shopping websites. Gender, Design type attribute, Length type
 (4) TABLE III. MEAN ABSOLUTE ERROR (MAE) AND ROOT MEAN
SQUARED ERROR (RMSE) OF HEIGHT PREDICTION
where s is similarity, 𝑤𝑖 is adjustable weights for the feature
𝑔
vector products, 𝑓𝑖 is the feature vector of the clothes from
MAE RMSE
clothing gallery, and 𝑓𝑖𝑐 is the feature vector of the consumer
clothes. The 𝑤𝑖 is set to be the importance of each clothing
feature vector, since the consumer concentrates on the gender 1.72 cm 2.22 cm
and the Design type of the clothing, the weight for gender is
set to 10 and the weight for Design type is set to 5, other
C. Clothing attributes recognition
weights are set to 1.
In the last step, a random generator will elect clothes from TABLE IV. ACCURACY OF RECOGNITION SYSTEM
the result of second step, these clothes will be treated as a
recommendation list and give the consumer as a reference. Main Accuracy
Attributes
The Euclidean distance of color features between clothes can types (%)
be considered as the similarity of the clothing texture, so that
it can be a calculable factor of the random generator. The
random generator is based on the probability density function Collar Shirt, Peter Pan, Puritan, Rib 90.09
of the normal distribution, which is defined as:

(5) Notched, Collarless,

Lapel 84.26
Shawl Collar, Plus Size Shawl
where 𝑝𝑖 is the probability of being recommended, 𝑥𝑖 is the
Euclidean distance of color features, 𝛼 is the parameter Design
which makes sure the sum of probability is 1, 𝛽 is an Turtle, Ruffle Semi-High
adjustable parameter, and 𝜎 is the standard deviation of the Neck Collar, 74.74
Low Turtle, Draped Collar
Euclidean distance of color features. To put it briefly, the
shorter Euclidean distance of color features is, the higher Strapless Neck, Deep V,
probability that the clothing being selected by the random Neckline
Straight Neck, V, Square, Off
84.73
generator. Shoulder, Round, Sweat Heart
Neck, One Shoulder
III. EXPERIMENT AND RESULT
Sleeveless, Short, Elbow, Long,
In this Section, the proposed clothing recommendation Sleeve
Extra Long
74.75
system will be evaluated.
A. Evaluation method
High Waist, Regular, Long,
The mean absolute error and root mean squared error Coat 69.18
Knee, Ankle & Floor
between estimated body height and ground truth body height
Length
are used to evaluate our personal height information
prediction. Precision is selected to be the evaluation index for Pants Short, Mid, Cropped, Full 78.04
our recommendation system. The more detailed information
will be explained in each section.
B. Personal height information prediction Skirt Short, Knee, Ankle, Floor 75.12
The result of height prediction, which is shown in Fig 4,
shows that the height prediction system has the capability to
measure body height of the consumers.
FashionAI dataset is used to test the recognition system.
The accuracy of each attribute model is listed in Table IV,
which the accuracy represents the ratio of correct
classification testing data set to the total testing data set.
According to this result, it can be found that recognition
accuracy of Length type is lower than that of in Design type.
Since the body proportions of models in the FashionAI dataset
are different, the Length type attributes will be over or under
estimated significantly. Moreover, the number of the
categories of attributes is too much so that the differences
among these attributes are too small to be distinguished. It is
also the reason why the recognition accuracy in Neck attribute
is lowest in Design type. However, the recognition system still
has great performance in general.
D. Recommendation system
To show the effectiveness of our recommendation system,
this experiment will compare four types of recommendation
Fig. 4. Body height prediction result. system. One is the system that all features are considered, and
the second system consults the features without gender and color features of clothing. The result shows that the system is
color features of the clothing. The third type system is based reliable and robust to the consumers, especially when the
on the idea of using semantic parsing from released source personal information and the color features are considered.
code, [20]. CNN, LSTM, spatial transformer and texture In the future, prediction of more different physiological
encoding layer were used to classify clothing features. And the information of consumers may be added to this system. Also,
pre-trained model which is directly implemented in this paper, the clothing patterns and the collaborative filtering may be
contains 59 attributes. For the last type recommendation employed to build a better and complete clothing
system, color of clothing will be the main factor of clothing recommendation system.
recommendation, [19].
ACKNOWLEDGMENT
As shown in Fig. 5, the system will ask the subject to stand
at specified position and predict clothing attributes of the This work was supported by the Ministry of Science and
subject. This experiment gives consumers four Technology under Grand no. MOST 108-2217-E-009 -001 -
recommendation clothing sets, 6 clothes in each set which are MY3.
generated by four types of recommendation system mentioned REFERENCES
before, and the subjects need to answer their preference.
[1] Francesco Ricci, Lior Rokach, Bracha Shapira, Paul B. Kantor,
Table V shows the feedback of four different “Recommender Systems Handbook,” May 2010.
recommendation clothing sets from 23 people including males [2] F.O. Isinkaye, Y.O. Folajimi, B.A. Ojokoh, “Recommendation systems:
and females. Since there are more than ten thousand clothes in Principles, methods and evaluation,” Egyptian Informatics Journal 16,
2015, pp.261–273.
our clothing gallery, the feedback shows that the system is
capable of providing suitable clothing to subjects. Compare [3] Gediminas Adomavicius, Alexander Tuzhilin, “Toward the Next
Generation of Recommender Systems: A Survey of the State-of-the-
the feedback of four recommendation clothing results, it is Art and Possible Extensions,” IEEE Transactions on Knowledge and
obviously that more features of clothing can help Data Engineering, VOL. 17, NO. 6, JUNE 2005.
recommendation system give more suitable clothing for the [4] Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl,
consumers. “Item-Based Collaborative Filtering Recommendation Algorithms,”
WWW , 2001.
TABLE V. FEEDBACK OF TWO DIFFERENT RECOMMENDATION [5] Paul Resnick, Neophytos Iacovou, Mitesh Suchak, Peter Bergstrom,
CLOTHING RESULTS John Riedl, “GroupLens: An Open Architecture for Collaborative
Filtering of Netnews,” Proceedings of the ACM CSCW’94 Conference
on Computer-Supported Cooperative Work, 1994.
Precision (%)
[6] Robin Burke, “Hybrid Recommender Systems: Survey and
Considered features Experiments,” UserModelling and User-Adapted Interaction, vol. 12,
Tops Bottoms no. 4, 2002.
[7] Fujiang Sun1, Yu Shi, Weiping Wang, “Content-based
Recommendation System Based on Vague Sets,” Fifth International
All features 71.80 64.10 Conference on Intelligent Human-Machine Systems and Cybernetics,
2013.
Features without gender and color 43.58 41.03 [8] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton, “ImageNet
Classification with Deep Convolutional Neural Networks,” NIPS'12
Proceedings of the 25th International Conference on Neural
Semantic parsing method[20] 46.73 53.85 Information Processing Systems, vol. 1, pp. 1097-1105, December
2012.
[9] Gil Levi , Tal Hassncer, “Age and gender classification using
Color features[19] 41.03 33.33 convolutional neural networks,” 2015 IEEE Conference on Computer
Vision and Pattern Recognition Workshops (CVPRW), pp. 34-42, June
2015.
[10] Rajeev Ranjan, Vishal M. Patel, and Rama Chellappa, “HyperFace: A
Deep Multi-Task Learning Framework for Face Detection, Landmark
Localization, Pose Estimation, and Gender Recognition,” IEEE
Transactions on Pattern Analysis and Machine Intelligence, vol. 41, pp.
121-135, December 2017.
[11] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang, “Deep
Learning Face Attributes in the Wild,” 2015 IEEE International
Conference on Computer Vision (ICCV), pp. 3730-3738, December
2015.
[12] Christian Szegedy, Wei Liu, and Yangqing Jia, “Going deeper with
convolutions,” 2015 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), pp. 1-9, June 2015.
[13] Sergey Ioffe, and Christian Szegedy, “Batch normalization:
accelerating deep network training by reducing internal covariate shift,”
ICML'15 Proceedings of the 32nd International Conference on
International Conference on Machine Learning, vol. 37, pp.448-456,
Fig. 5. The snapshot of our recommendation system experiment July 2015.
[14] Christian Szegedy, Vincent Vanhoucke, and Sergey Ioffe, “Rethinking
IV. CONCLUSION the Inception Architecture for Computer Vision,” 2016 IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), pp.
A new clothing recommendation system is now firstly 2818-2826, June 2016.
proposed in this paper. There are two main contributions: the [15] Gary B. Huang, Marwan Mattar, Tamara Berg, Eric Learned-Miller.
clothing recognition and the clothing recommendation system Labeled Faces in the Wild: A Database for Studying Face Recognition
in Unconstrained Environments. University of Massachusetts, Amherst,
which combines gender, body height, clothing attributes and Technical Report 07-49, October, 2007.
[16] Davis E. King, “Dlib-ml: A Machine Learning Toolkit, ”Journal of International Conference on Data Science and Business Analytics, pp.
Machine Learning Research, vol. 10, pp. 1755-1758, December 2009. 482-487, December 2018.
[17] Fashionai dataset. URL http://fashionai.alibaba [20] Sagar Verma, Sukhad Anand, Chetan Arora, Atul Rai, “Diversity in
[18] Sachin Sudhakar Farfade, Mohammad Saberian, Li-Jia Li, “ Multi- Fashion Recommendation Using Semantic Parsing, ” 25th IEEE
view Face Detection Using Deep Convolutional Neural Networks,” In International Conference on Image Processing(ICIP), pp. 500-504,
Proceedings of the 5th ACM on International Conference on October 2018.
Multimedia Retrieval, pages 643–650. ACM, 2015 .
[19] Tao Yang, Xiaoyi Man, Jiao Feng, Jie Chen, Ran Tao, “ Research on
color expansion method in clothing recommendation system,” 2nd