Accepted Manuscript

The Use of Machine Learning Algorithms in Recommender Systems:

A Systematic Review

Ivens Portugal, Paulo Alencar, Donald Cowan

PII: S0957-4174(17)30833-3
DOI: 10.1016/j.eswa.2017.12.020
Reference: ESWA 11721

To appear in: Expert Systems With Applications

Received date: 27 May 2016

Revised date: 7 December 2017
Accepted date: 8 December 2017

Please cite this article as: Ivens Portugal, Paulo Alencar, Donald Cowan, The Use of Machine Learning
Algorithms in Recommender Systems: A Systematic Review, Expert Systems With Applications (2017),
doi: 10.1016/j.eswa.2017.12.020

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service
to our customers we are providing this early version of the manuscript. The manuscript will undergo
copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please
note that during the production process errors may be discovered which could affect the content, and
all legal disclaimers that apply to the journal pertain.
 ACCEPTED MANUSCRIPT

The Use of Machine Learning Algorithms in

Recommender Systems: A Systematic Review
Ivens Portugal Paulo Alencar Donald Cowan
David R. Cheriton School David R. Cheriton School David R. Cheriton School
of Computer Science of Computer Science of Computer Science
University of Waterloo University of Waterloo University of Waterloo
200 University Avenue West 200 University Avenue West 200 University Avenue West
Waterloo, Canada - N2L 3G1 Waterloo, Canada - N2L 3G1 Waterloo, Canada - N2L 3G1

T
[email protected] [email protected] [email protected]

IP
Abstract—Recommender systems use algorithms to provide (Martens, 1959), with the emergence of the field of AI. Today,

CR
users with product or service recommendations. Recently, these there is a plethora of ML algorithms (k-nearest neighbor
systems have been using machine learning algorithms from (Patrick & Fischer III, 1970), clustering (Jain, Murty, & Flynn,
the field of artificial intelligence. However, choosing a suitable
machine learning algorithm for a recommender system is difficult 1999), Bayes network (Friedman, Geiger, & Goldszmidt,
because of the number of algorithms described in the literature. 1997), to name a few types), which are used in applications
Researchers and practitioners developing recommender systems
are left with little information about the current approaches in
algorithm usage. Moreover, the development of recommender
systems using machine learning algorithms often faces problems
and raises questions that must be resolved. This paper presents
US that range from vacuum cleaner robots (Burhans & Kandefer,
2004) and assistance for disabled people (Karimanzira, Otto,
& Wernstedt, 2006) to pattern recognition in images (Torralba,
Fergus, & Weiss, 2008), or self-driving vehicles (Thrun, 2007).
AN
a systematic review of the literature that analyzes the use The potential application of ML algorithms is vast and the field
of machine learning algorithms in recommender systems and looks very promising.
identifies new research opportunities. The goals of this study are ML algorithms are being used in RSs to provide users with
to (i) identify trends in the use or research of machine learning al- better recommendations. However, the ML field does not have
gorithms in recommender systems; (ii) identify open questions in
a clear classification scheme for its algorithms, mainly because
M

the use or research of machine learning algorithms; and (iii) assist

new researchers to position new research activity in this domain of the number of approaches and the variations proposed in the
appropriately. The results of this study identify existing classes literature (Lv & Tang, 2011). As a consequence, it becomes
of recommender systems, characterize adopted machine learning difficult and confusing to choose an ML algorithm that fits
approaches, discuss the use of big data technologies, identify types
ED

one’s need when developing an RS. In addition, researchers

of machine learning algorithms and their application domains,
and analyzes both main and alternative performance metrics. may find it challenging to track the use and the trends of ML
Keywords—systematic review of the literature, recommender algorithms in RSs.
systems, machine learning, machine learning algorithms, appli- One way to assist researchers and practitioners (e.g. soft-
cation domains, performance metrics ware engineers and developers (Pressman, 2015; Isazadeh,
PT

2004)) in choosing which ML algorithm to use in an RS is the

1. I NTRODUCTION study of the RS and ML fields. Research about RSs containing
Recommender systems (RSs) are used to help users find new ML algorithms implemented in the literature can help show
CE

items or services, such as books, music, transportation or even trends and provide a direction for future studies.
people, based on information about the user, or the recom- This paper provides a systematic review to investigate how
mended item (Adomavicius & Tuzhilin, 2005). These systems ML algorithms used in RSs are studied and used; and what
also play an important role in decision-making, helping users are the trends in ML algorithm research and development.
AC

to maximize profits (L.-S. Chen, Hsu, Chen, & Hsu, 2008) or It is expected that, with this systematic review, researchers
minimize risks (Bouneffouf, Bouzeghoub, & Gancarski, 2013). and practitioners can obtain more information about the RS
Today, RSs are used in many information-based companies field, and make better implementation or research decisions.
such as Google (J. Liu, Dolan, & Pedersen, 2010), Twitter The goals of this study are to (i) identify trends in the use
(Ahmed et al., 2013), LinkedIn (Rodriguez, Posse, & Zhang, or research of machine learning algorithms in recommender
2012), and Netflix (Steck, 2013). The field of RS has its origins systems; (ii) identify open questions in the use or research of
in the mid-1990s with the introduction of Tapestry (Goldberg, machine learning algorithms; and (iii) assist new researchers
Nichols, Oki, & Terry, 1992), the first RS. to position new research activity in this domain appropri-
As the RS field evolved, researchers studied the use of ately. The results of this study identify existing classes of
algorithms from machine learning (ML), an area of artificial recommender systems, characterize adopted machine learning
intelligence (AI). ML has been studied since the late 1950s approaches, discuss the use of big data technologies, identify
 ACCEPTED MANUSCRIPT

types of machine learning algorithms and their application do- age, and email. Other forms of explicit user data gathering
mains, and analyze main and alternative performance metrics. (Gemmis et al., 2011; Longo, Barrett, & Dondio, 2009) are
This paper is organized as follows: Section 2 describes when users express their preferences by rating items using a
the theoretical background needed; Section 3 explains the numerical value or a preference such as a Facebook “like.”
systematic review protocol, and Section 4 explains the results Implicit user data gathering accesses information about the
of this study. Section 5 presents conclusions and future work. user indirectly. For example, when visiting an online store,
the server at the online store exchanges messages with the
2. T HEORETICAL BACKGROUND user’s computer, and based on that, the store’s RS may know
This section gives an overview of the two main research the browser the user is using, as well as the user’s country.
fields related to this article, namely recommender systems and More advanced applications monitor user clicks and keystroke
machine learning. logs.
Besides the common recommendation process, in which
2.1 Recommender Systems

T
users are presented with items that might be of interest,
Recommender systems (RSs) use artificial intelligence (AI) recommendations can be provided in other ways. Trust-based

IP
methods to provide users with item recommendations. For recommendations (O’Donovan & Smyth, 2005) take into con-
example, an online bookshop may use a machine learning sideration the trust relationship that users have between them.
(ML) algorithm to classify books by genre and then recom- A trust relationship is a link in a social network to a friend

CR
mend other books to a user buying a specific book. RSs were or a related connection. Recommendations based on trust are
introduced in 1992 when Tapestry, the first RS, appeared. Its worth more than those that do not have trust links. Context-
authors used the term collaborative filtering to refer to the aware recommendations (Adomavicius, Mobasher, Ricci, &
recommendation activity. This term is still used to classify Tuzhilin, 2011) are based on the context of the user. A context
RSs. RSs are divided into three main categories to drive
the recommendations: collaborative, content-based, and hybrid
filtering (Adomavicius & Tuzhilin, 2005).
First, RSs using a collaborative approach consider the user
US is a set of information about the current state of the user,
such as the time at the user location (morning, afternoon,
evening), or their activity (idle, running, sleeping). The amount
of context information to be processed is high, making context-
AN
data when processing information for recommendation. For aware recommendations a challenging research field. Risk-
instance, by accessing user profiles in an online music store, aware recommendations (Bouneffouf et al., 2013) are a subset
the RS has access to all the user data, such as the age, country, of context-aware recommendations and take into consideration
city, and songs purchased. With this information, the system a context in which critical information is available, such as
can identify users that share the same music preference, and user vital signs. It is risk-aware because a wrong decision
M

then suggest songs bought by similar users. may threaten a user’s life or cause damage. Some examples
Second, RSs with a content-based filtering approach base are recommending pills to be taken or stocks the user should
their recommendations on the item data they can access. buy or, sell.
ED

As an example, consider a user who is looking for a new

computer using an online store. When the user browses a 2.2 Machine Learning
particular computer (item), the RS gathers information about Machine Learning (ML) uses computers to simulate hu-
that computer and searchers in a database for computers that man learning and allows computers to identify and acquire
have similar attributes, such as price, CPU speed, and memory knowledge from the real world, and improve performance of
PT

capacity. The result of this search is then returned to the user some tasks based on this new knowledge. More formally, ML
as recommendations. is defined as follows: “A computer program is said to learn
The third category describes RSs that combine the two pre- from experience E with respect to some class of tasks T and
CE

vious categories into a hybrid filtering approach, recommend- performance measure P, if its performance at tasks in T, as
ing items based on the user and the item data. For example, measured by P, improves with experience E” (R. Michalski,
on a social network, an RS may recommend profiles that Carbonell, & Mitchell, 1985). Although the first concepts of
are similar to the user (collaborative filtering), by comparing ML originated in the 1950s, ML was studied as a separate
AC

their interests. In a second step, the system may consider the field in the 1990s (R. S. Michalski, Carbonell, & Mitchell,
recommended profiles as items and thus access their data to 2013). Today, ML algorithms are used in several areas besides
search for new similar profiles (content-based filtering). In the computer science, including business (Apte, 2010), advertising
end, both sets of profiles are returned as recommendations. (Cui, Bai, Gao, & Liu, 2015) and medicine (Kononenko,
When using a collaborative or a hybrid filtering approach, 2001).
RSs must gather information about the user in order to Learning is the process of knowledge acquisition. Humans
develop recommendations. This activity can be done explicitly naturally learn from experience because of their ability to
or implicitly. Explicit user data gathering (Sutton & Barto, reason. In contrast, computers do not learn by reasoning, but
1998) happens when users are aware they are providing their learn with algorithms. Today, there are a large number of ML
information. For instance, when registering for a new online algorithms proposed in the literature. They can be classified
service, users usually fill in a form that asks their name, based on the approach used for the learning process. There
 ACCEPTED MANUSCRIPT

are four main classifications: supervised, unsupervised, semi- which they are designed. Some examples of classification
supervised, and reinforcement learning. can be found in (Shalev-Shwartz & Ben-David, 2013) and
Supervised learning (Kotsiantis, 2007; D. Zhang & Tsai, (Kulkarni, 2012), although the field still does not have any
2006) happens when algorithms are provided with training standard classification.
data and correct answers. The task of the ML algorithm is to
3. S YSTEMATIC R EVIEW
learn based on the training data, and to apply the knowledge
that was gained using real data. As an example consider an When developing RSs, software engineers must decide on
ML learning algorithm being used for book classification in the specific recommender algorithm of all those available. This
a bookstore. A training set (training data + answers) can be choice has significant effect on the rationale of the RS, on
a table relating information about each book to a correct the data that will be needed from users and recommendation
classification. Here, information about each book may be title, items, and on performance issues. The number of algorithm
author, or even every word a book contains. The ML algorithm variations and combinations in the literature makes this choice
learns with the training set. When a new book arrives at a challenging task.

T
the bookstore, the algorithm can classify it based on the This large number of recommender algorithms, which ap-
pears to be constantly growing and changing, makes software

IP
knowledge about book classification it has acquired.
In unsupervised learning (Celebi & Aydin, 2016), ML engineering for RSs a continuing challenge. Trying to develop
algorithms do not have a training set. They are presented with tools to make RS development easier is a moving target, as

CR
some data about the real world and have to learn from that data new studies must be done to observe new open problems and
on their own. Unsupervised learning algorithms are mostly fo- trends, and further enrich the knowledge base.
cused on finding hidden patterns in data. For example, suppose For these reasons the authors conducted a systematic review
that an ML algorithm has access to user profile information in to analyze the development of RSs containing ML algorithms.
This systematic review follows the procedures of (Kitchenham,
a social network. By using an unsupervised learning approach,
the algorithm can separate users into personality categories,
such as outgoing and reserved, allowing the social network
company to target advertising more directly at specific groups
US 2004)

2)
and

in RSs,
has, as goals, to:
1) identify trends in the use or research of ML algorithms

identify open questions in the use or research of ML

AN
of users.
ML algorithms can also be classified as semi-supervised. algorithms,
Semi-supervised learning (Chapelle, Schölkopf, & Zien, 2006; 3) assist new researchers to position new research activity
Xu, Mo, & King, 2012) occurs when algorithms work with a in this domain appropriately.
training set with missing information, and still need to learn This systematic review has one restriction. The authors de-
M

from it. An example is when an ML algorithm is provided cided to limit the set of studies investigated to those describing
with movie ratings. Not every user rated every movie and so, an experiment or a validation study. The main reason for this
there is some missing information. Semi-supervised learning restriction is that several publications in the literature propose
algorithms are able to learn and draw conclusions even with new algorithms that are never tested or validated. Thus, by
ED

incomplete data. including this restriction, this systematic review is able to

Lastly, ML algorithms might have a reinforcement learning analyze the performance metrics of the ML algorithms, such
approach. Reinforcement learning (Sutton & Barto, 1998) as precision, recall, and f-measure.
occurs when algorithms learn based on external feedback given This review examines the following three research questions
PT

either by a thinking entity, or the environment. This approach (RQ):

is analogous to teaching dogs to sit or jump. When the dog RQ1. What are the trends in recommender system use and re-
performs the action correctly, the dog receives a small treat search when implementing a machine learning algorithm?
CE

(positive feedback). It does not receive any treat (negative RQ2. What are the trends in machine learning algorithm use
feedback) if it performs the wrong action. As an example and research when developing a recommender system?
in the computer science field, consider an ML algorithm RQ3. What are the main sources of articles of machine learning
that plays games against an opponent. Moves that lead to algorithms research when embedded in recommender
AC

victories (positive feedback) in the game should be learned systems?

and repeated, whereas moves that lead to losses (negative The protocol for this systematic review has three main steps.
feedback) are avoided. The first step is to gather as many publications as possible
ML has become quite popular recently with the increase using scientific search engines. The authors then analyze the
in processor speed and memory size. As a consequence, studies that were retrieved and apply an initial exclusion
the field now has a large number of algorithms that use criteria. The second step is to read the abstract of the remaining
mathematical or statistical analysis to learn, draw conclusions papers and apply an additional exclusion criteria. The third
or infer data. This number continues to increase as evidenced and last step is to read the entire study and gather data from
by the number of scientific publications that propose variations it, or apply a third set of exclusion criteria. All the data is
or combinations of ML algorithms. For that reason, ML then compiled and is used to answer the research questions
algorithms have been categorized based on the purpose for discussed earlier.
 ACCEPTED MANUSCRIPT

To answer the first research question, the authors investi- Table 3.1: Number of studies in this systematic review
gated the type of filtering strategy used in the recommender Label Number
system being described in a study. The approach to answering Total retrieved 215
Initial Not peer-reviewed study 15
the second question involved more data. The publication exclusion Books, letters, notes, or patents 0
proposed in the publications had its classification (supervised, criteria Graduate Thesis 2
unsupervised, etc), their type (clustering, decision tree, etc) Subtotal retained 199
investigated, as well as its support for distributed technologies Additional Excluded after reading the abstract 17
exclusion Not able to access study 5
(Hadoop, MapReduce). The performance metrics that describe criteria Study in foreign language 3
each ML algorithm inspected in this systematic review were Subtotal retained 174
analyzed. The third question is answered by inspecting the Repeated studies 4
conferences and journals in which the studies were published, Additional Not primary studies 10
exclusion Not about recommender systems 6
and the surveys that were returned by the search query. criteria Not about machine learning 0

T
To strengthen the validity of the review the authors applied based on the Does not explain algorithm 6
entire study Does not include validation study 8
certain exclusion criteria (EC) to the studies that were included Does not include performance metrics 18

IP
in this systematic review. These criteria and the rationale are Total retained studies 121
presented next.
EC1. Studies must be peer-reviewed articles, published in a

CR
conference, journal, press, etc. For example, conference keywords, and attempts to find terms that relate to the field
entries are not considered for review. of RS, ML, and provide some indication that the proposed
EC2. Books, letters, notes, and patents are not included in the approach was validated. Studies must also contain the term
review. “machine learning” in the title, abstract, or keywords. To
EC3. Graduate theses are not considered for review.
EC4. The abstract does not provide enough information.
EC5. The authors must have access to the studies, otherwise
studies are not considered for review.
US retrieve studies that were assessed, the search query also looks
for the terms “experiment” or its synonyms.
The search query was used on three popular academic
search engines Scopus1 , Web of Science2 , and IEEEXplore3 on
AN
EC6. Studies must be primarily in English or French. Studies August 26th , 2016. The search returned 215 publication entries
in languages other than English or French are excluded. that were reviewed for quality. Scopus returned 196 studies,
EC7. Studies must be unique. If a study is repeated, other followed by Web of Science with 33 studies, and IEEEXplore
copies of that study are not included in this review. with 31 studies. The titles of the studies were inspected to find
EC8. Only primary studies are included in this review. For duplicates among search engines. After that they were ready
M

example, surveys of the literature are not considered for to be filtered by the exclusion criteria previously explained.
review. The results are summarized on Table 3.1.
EC9. Studies that do not describe a recommender system The number of studies to be read in the systematic review
ED

approach are not considered for review. decreased from 215 to 121 when filtered by the exclusion crite-
EC10. Studies that do not describe a machine learning approach ria. Fifteen of the study entries were conference or proceeding
are not considered for review. descriptions and are excluded because they are not written
EC11. Studies that do not describe a machine learning approach scientific work. After reading the abstract of the studies, the
PT

sufficiently well are not considered for review. authors were confident that 17 studies were not related to
EC12. Studies that do not describe an experiment or validation the goal of this systematic review and decided to exclude
study are not considered for review. them. The authors did not have access to five studies, even
EC13. Studies that do not describe performance metrics (e.g. after asking help from colleagues and visiting libraries. These
CE

accuracy, precision, recall) are not included in this review. studies were then not inspected in this systematic review. Two
There are some synonyms that denote RSs. Based on studies were in Chinese and another one was in Japanese. Four
(Jannach, Zanker, Felfernig, & Friedrich, 2010) this systematic studies had a copy returned by the search string. These studies
review considers RS terms that replace “recommender” by present the same results and were not counted twice. Only the
AC

“recommendation” and it does not consider any “machine original study was considered in this systematic review. After
learning” synonyms. Synonyms for the term “experiment” reading the studies, those who did not focus their proposal on
are “experimentation,” “evaluation,” “assessment,” and “val- the key research fields of this review were excluded. Moreover,
idation.” All of these terms were featured in the search query studies that did not explain the ML algorithm being used, or
(SQ), which is presented as follows: did not describe a validation study, or its results were also
excluded from this systematic review. In the end, 121 primary
SQ. ((“recommender system” OR “recommendation system”)
studies were retained and analyzed. The list of all studies is
AND (“machine learning”) AND (“experiment” OR “ex-
perimentation” OR “evaluation” OR “assessment” OR 1 http://www.scopus.com
“validation”)) 2 http://webofscience.com

This search query inspects the study title, abstract and 3 http://ieeexplore.ieee.org
 ACCEPTED MANUSCRIPT

presented in the Tables A.1 and A.2 in the Appendix. the studies describe a collaborative approach for filtering, with
One last important point to mention is that the studies a stronger emphasis on a neighborhood-based method.
reviewed may propose more than one ML algorithm. As a The authors decided to observe the timeline of the publica-
consequence, some of the results presented on the next chapter tion of each study. The results are shown in Figure 4.1 and
are focused on the number of studies, while others are focused also confirms that collaborative filtering with a neighborhood-
on the number of algorithms. The 121 studies described a total based method is well researched. In the figure, one clearly
of 205 ML algorithms that are either totally new, or modifica- sees a spike in the year 2012 that indicates a trend in this
tions or optimization of existing ones. Finally, algorithms can research area in recent years. One reason might be the real-
be validated in one or more application domains. This also life applicability of collaborative filtering approaches in social
impacts some results shown in the next section. networks for example, or on the web with spatial-temporal
applications such as the online network platform for room
4. S YSTEMATIC R EVIEW R ESULTS renting AirBnb4 or the transportation network company Uber5 .

T
The reading process focused on finding three types of Another important conclusion drawn from Table 4.1 and
information: one that relates to the RS being described (its Figure 4.1 is the minimal research effort focused on hybrid

IP
classification), another that relates to the ML algorithm (its approaches. Hybrid filtering helps overcome limitations of
type, application domain, and performance metrics), and fi- the other two approaches. However, throughout the years,
nally information about the source of the study (publication research on this type of filtering with ML algorithms has

CR
venue). The abstract and introduction of each paper was been low, despite the fact that some studies show that it gives
read, as well as the description of the proposed approach. more accurate recommendations than other types of filtering
Sometimes, when pieces of data were well described the entire (Adomavicius & Tuzhilin, 2005).
section did not need to be read. The conclusion and future
work sections of each study ware also read looking for open
problems or research directions.
The authors developed a spreadsheet with an identification
of each study with many columns for noting the pieces
US 4.2 Machine Learning Algorithms
ML algorithms can initially be classified as supervised,
semi-supervised, unsupervised, or reinforcement learning. It
is worth calculating the number in each category in this
AN
of information previously described. After reading all the systematic review. However, since studies may propose more
studies, the authors processed the information contained in than one ML algorithm, it is more reasonable to do an analysis
the spreadsheet and organized it in a presentable manner. The on the algorithm level, instead of the study one. Therefore,
results and conclusions are presented in the following sections. Table 4.2 shows the number of ML algorithms found in the
studies of this systematic review that described themselves
M

4.1 Recommender Systems

under one of these ML classifications.
Recommender systems can be classified by content-based, There is a clear research interest in supervised learning ML
collaborative, or hybrid filtering. Usually, content-based ap- algorithms for RSs. One main reason for this result is that most
ED

proaches use the following two strategies to recommend of the algorithms analyzed were modifications or optimization
items to users, according to (Weng, 1998): classifier-based or of well-known ML algorithms. Unsupervised learning had also
neighbor methods. In the first method, users are associated an expressive result. Lastly, there is plenty of room for research
with profiles, and a new item is presented to the classifier. in semi-supervised or reinforcement learning for RSs that new
The classifier then decides whether the item should be recom-
PT

researchers may explore.

mended or not based on the item’s contents. Nearest-neighbor The authors also separated ML algorithms into types based
methods store items that the user has checked or rated and on (Shalev-Shwartz & Ben-David, 2013) to present the number
use an underlying network of items (where similar items have of ML algorithms of each type analyzed in this review. Some
CE

similar properties) to discover the user interest for a new item. algorithms had clear classifications because they were small
Collaborative filtering RSs are subdivided in the follow- variations of well-established algorithms (e.g. incremental
ing categories, according to (Ning, Desrosiers, & Karypis, matrix factorization is a variant of the matrix factorization
2015): neighborhood-based and model-based methods. The algorithm). Other algorithms, popular in the field, were not
AC

first method also stores the relationship user-item (the user grouped with the algorithms of the same type (e.g. k Nearest
interest for an item) in a user profile, but it uses a similarity Neighbors is a clustering algorithm, but has its own entry).
network of users to evaluate whether a new item should However, some algorithms do not seem to fit in any category.
be recommended. In contrast, model-based methods use the For these cases, the algorithm was listed under a new category
stored ratings to produce a predictive model for the user. with its own name (e.g. Personality diagnosis).
Hybrid approaches do not seem to follow any categorization. Other important considerations are that some studies de-
Table 4.1 shows how many studies describe at least one scribed approaches that involve many ML algorithms. When
approach in each of the classifications explained in previous identified, these approaches were listed under the “Ensemble”
paragraphs, as well as the studies themselves. Results point
to a significant number of collaborative filtering approaches 4 http://www.airbnb.com

when developing RSs with ML algorithms. More than half of 5 http://www.uber.com

 ACCEPTED MANUSCRIPT

Table 4.1: Classification of recommender systems

Number
Classification of recommendation system Studies
of studies
(Alemeye & Getahun, 2015; Baldominos, Albacete, Saez, & Isasi, 2015; Banerjee,
Bhowmick, Mukherjee, & Misra, 2012; Brouard & Pomot, 2016; Buettner, 2016; Costa,
Furtado, Pires, Macedo, & Cardoso, 2012, 2013; De Gemmis, Lops, Semeraro, & Basile,
2008; Diaby, Viennet, & Launay, 2014; Elmongui et al., 2015; Haiduc et al., 2013;
Hernández del Olmo, Gaudioso, & Martin, 2009; Hussain, Farooq, Luo, & Slack, 2015;
R. Zhang & Tran, 2010; Jin, Mobasher, & Zhou, 2005; Kong, Zhang, & Ding, 2013;
Content-based filtering / Classifier-based 30
Leopairote, Surarerks, & Prompoon, 2013; Q.-C. Li, Dong, & Li, 2008; T. Liu, Fan, Hu,
& Du, 2011; Lops et al., 2009; Marović, Mihoković, Miks̆a, Pribil, & Tus, 2011; Musto,
Narducci, Lops, De Gemmis, & Semeraro, 2010; Pantraki & Kotropoulos, 2015; Pecli
et al., 2015; Pronoza, Yagunova, & Volskaya, 2016; Taghipour, Kardan, & Ghidary, 2007;
Tsuji, Yoshikane, Sato, & Itsumura, 2014; Vialardi et al., 2011; Wang, Wang, Wang, &
Hsu, 2014; Xin et al., 2014)
(Banerjee et al., 2012; Das et al., 2013; Das Dôres, Alves, Ruiz, & Barros, 2016;

T
Kao & Fahn, 2013; R. Liu, Ding, & Xie, 2014; C. Lu, Stankovic, & Laublet, 2015;
Marques, Guilherme, Nakamura, & Papa, 2011; Nicol, Mary, & Preux, 2014; Pecli et al.,
Content-based filtering / Neighbor-based 15

IP
2015; Pronoza et al., 2016; Szymański & Rzeniewicz, 2016; Tsapatsoulis, Agathokleous,
Djouvas, & Mendez, 2015; Tsuji et al., 2014; Wei, Chen, & Liang, 2011; Xuan, Lu, Zhang,
& Luo, 2014)
(Agarwal, 2011; Bjelica, 2010; Bouneffouf, Bouzeghoub, & Gançarski, 2012; Cai et al.,

CR
2010, 2012; Castro-Herrera, Cleland-Huang, & Mobasher, 2009; Devi & Venkatesh, 2013;
Diaby, Viennet, & Launay, 2013; Z. Fan, Chen, Zha, & Yang, 2016; Forsati, Rahbar, &
Mahdavi, 2009; Ghazarian & Nematbakhsh, 2015; Halder, Seddiqui, & Lee, 2014; Hassan,
Karim, Javed, & Arshad, 2010; Jun, 2005; Karahodza & Donko, 2015; Krzywicki et al.,
2015; T.-J. Lee & Tseng, 2012; X. Li, Wang, & Liang, 2014; Liang, Lu, Ji, & Li, 2014;
Collaborative filtering / Neighborhood-based 37

US
Q. Liu, Xiong, & Huang, 2014; Luong, Huynh, Gauch, & Hoang, 2012; Luong, Huynh,
Gauch, Do, & Hoang, 2012; Marović et al., 2011; McLaughlin & Herlocker, 2004; Nie,
Wang, Huang, & Ding, 2013; Oyama, Hayashi, & Kashima, 2012; Roh, Oh, & Han, 2003;
Y. Song, Zhang, & Giles, 2011; I. Song, Dillon, Goh, & Sung, 2011; Szabó, Póczos, &
Lorincz, 2012; Takács, Pilászy, Németh, & Tikk, 2008; Wan, Jamaliding, & Okamoto,
2009; Wang, Yin, Cheng, & Yu, 2012; Zahra et al., 2015; W. Zhang, Begole, Chu, Liu,
AN
& Yee, 2008; Y. Zhang, Zhuang, Wu, & Zhang, 2009; K. Zhao & Pan, 2015)
(Anaissi & Goyal, 2015; Aouay, Jamoussi, & Gargouri, 2014; Bar, Rokach, Shani, Shapira,
& Schclar, 2013; Bauer & Nanopoulos, 2014; Braida, Mello, Pasinato, & Zimbrão, 2015;
Caraballo, Arruda, Nunes, Lopes, & Casanova, 2014; Dinuzzo, Pillonetto, & De Nicolao,
2011; Gedikli, Bağdat, Ge, & Jannach, 2011; Hofmann, 2003, 2004; Huang & Nikulin,
2014; Krohn-Grimberghe, Busche, Nanopoulos, & Schmidt-Thieme, 2011; X. Li & Chen,
M

2013; Q. Liu et al., 2014; J. Lu, Hoi, Wang, & Zhao, 2013; Marović et al., 2011; Montañés,
Collaborative filtering / Model-based 29
Quevedo, Dı́az, & Ranilla, 2009; Moreno, Shapira, Rokach, & Shani, 2012; Paparrizos,
Cambazoglu, & Gionis, 2011; Pessiot, Truong, Usunier, Amini, & Gallinar, 2007; Sun,
Fan, Bakillah, & Zipf, 2015; Takács et al., 2008; Takáes, Pilászy, Németh, & Tikk, 2009;
Yap, Tan, & Pang, 2005; Yuan, Murukannaiah, Zhang, & Singh, 2014; Zhai & Li, 2015;
ED

S.-Z. Zhang, Liu, & Dong, 2007; X.-Z. Zhang, 2007; Q. Zhao, Zhang, Friedman, & Tan,
2015)
(Bellogı́n, Cantador, Castells, & Ortigosa, 2011; Biancalana, Gasparetti, Micarelli, Miola,
& Sansonetti, 2011; Buabin, 2012; Degemmis, Lops, & Semeraro, 2007; T.-K. Fan &
Chang, 2010; Forsati & Meybodi, 2010; Geng, Zhang, Bian, & Chua, 2016; Islam, Ding,
PT

Hybrid filtering 18 & Chi, 2015; Jung & Lee, 2004; W.-P. Lee & Lu, 2003; J. Li & Zaı̈ane, 2004; Marović
et al., 2011; Middleton, Shadbolt, & De Roure, 2004; Murfi & Obermayer, 2009; Nguyen,
Richards, Chan, & Liszka, 2016; Verma, Hart, Bhatkar, Parker-Wood, & Dey, 2016; Yan,
Xu, Yao, & Lu, 2013; Yeh & Wu, 2010)
CE

entry. The ensemble strategy for machine learning has several approaches constitute a quarter of the results. Ensemble meth-
ways of being implemented (e.g. bagging, boosting, random ods are also at the top of the tables, but this result happened
forest). However, this systematic review does not differentiate because many researchers trying different methods opted to
AC

among them in the analysis. Other studies do not follow tradi- combine their methods in an Ensemble as one additional trial.
tional ensemble techniques, and use different ML algorithms Some ML algorithms ranked low in this systematic review
in different parts of a greater recommendation strategy. These despite their popularity. It is the case of the Neural Network
approaches were listed under the “Various” entry. Table 4.3 or the K Means algorithms. Since this systematic review is
shows detailed results, while table 4.4 provides an alternative focused on the application domain of RS development, these
classification. algorithms are not being researched enough, which opens
opportunities for future studies.
When inspecting the tables, one can observe again the
emergence of collaborative filtering approaches with clustering 4.3 Big Data Technologies
algorithms being the one most researched in RS development. ML algorithms, by definition, improve their performance
Together with Support Vector Machines (SVM), collaborative with access to more data. Similarly, the more data that is
 ACCEPTED MANUSCRIPT

Figure 4.1: Timeline of the classification of the studies

Timeline
10
Content-based filtering / Classifier-based
Content-based filtering / Neighbor-based
Collaborative filtering / Neighborhood-based
8 Collaborative filtering / Model-based
Hybrid filtering
Number of studies

T
IP
4

CR
2

0
2,000 2,004
US 2,008 2,012 2,016
AN
Year

Table 4.2: Machine learning approach

M

Number of ML Number
Approach Studies
algorithms of studies
(Agarwal, 2011; Alemeye & Getahun, 2015; Anaissi & Goyal, 2015; Aouay et al., 2014;
Baldominos et al., 2015; Banerjee et al., 2012; Bar et al., 2013; Bauer & Nanopoulos,
ED

2014; Bellogı́n et al., 2011; Biancalana et al., 2011; Braida et al., 2015; Brouard & Pomot,
2016; Buabin, 2012; Cai et al., 2010, 2012; Caraballo et al., 2014; Castro-Herrera et al.,
2009; Costa et al., 2012, 2013; Das et al., 2013; Das Dôres et al., 2016; De Gemmis et al.,
2008; Diaby et al., 2013, 2014; Dinuzzo et al., 2011; T.-K. Fan & Chang, 2010; Forsati
et al., 2009; Forsati & Meybodi, 2010; Gedikli et al., 2011; Geng et al., 2016; Haiduc
et al., 2013; Hernández del Olmo et al., 2009; Hofmann, 2003, 2004; Huang & Nikulin,
PT

2014; Hussain et al., 2015; R. Zhang & Tran, 2010; Islam et al., 2015; Jin et al., 2005; Jun,
2005; Jung & Lee, 2004; Kao & Fahn, 2013; Karahodza & Donko, 2015; Kong et al., 2013;
Krohn-Grimberghe et al., 2011; Krzywicki et al., 2015; W.-P. Lee & Lu, 2003; Leopairote
Supervised learning 156 97 et al., 2013; Q.-C. Li et al., 2008; X. Li & Chen, 2013; Liang et al., 2014; T. Liu et al.,
2011; Q. Liu et al., 2014; Lops et al., 2009; J. Lu et al., 2013; Luong, Huynh, Gauch,
CE

Do, & Hoang, 2012; Marović et al., 2011; Marques et al., 2011; McLaughlin & Herlocker,
2004; Montañés et al., 2009; Moreno et al., 2012; Murfi & Obermayer, 2009; Musto et al.,
2010; Nicol et al., 2014; Nie et al., 2013; Oyama et al., 2012; Pantraki & Kotropoulos,
2015; Paparrizos et al., 2011; Pecli et al., 2015; Pessiot et al., 2007; Pronoza et al., 2016;
Roh et al., 2003; Y. Song et al., 2011; I. Song et al., 2011; Sun et al., 2015; Szabó et al.,
AC

2012; Szymański & Rzeniewicz, 2016; Takács et al., 2008; Takáes et al., 2009; Tsapatsoulis
et al., 2015; Tsuji et al., 2014; Verma et al., 2016; Vialardi et al., 2011; Wan et al., 2009;
Wang et al., 2012; Wei et al., 2011; Xin et al., 2014; Yan et al., 2013; Yap et al., 2005; Yeh
& Wu, 2010; Yuan et al., 2014; Zhai & Li, 2015; S.-Z. Zhang et al., 2007; X.-Z. Zhang,
2007; W. Zhang et al., 2008; Y. Zhang et al., 2009; Q. Zhao et al., 2015)
Semi-supervised learning 1 1 (Das et al., 2013)
(Bar et al., 2013; Bjelica, 2010; Bouneffouf et al., 2012; Buettner, 2016; Degemmis et al.,
2007; Devi & Venkatesh, 2013; Elmongui et al., 2015; Z. Fan et al., 2016; Ghazarian &
Nematbakhsh, 2015; Halder et al., 2014; Hassan et al., 2010; T.-J. Lee & Tseng, 2012; J. Li
Unsupervised learning 46 24
& Zaı̈ane, 2004; X. Li et al., 2014; R. Liu et al., 2014; C. Lu et al., 2015; Luong, Huynh,
Gauch, & Hoang, 2012; Marović et al., 2011; Middleton et al., 2004; Nguyen et al., 2016;
Y. Song et al., 2011; Xuan et al., 2014; Zahra et al., 2015; K. Zhao & Pan, 2015)
Reinforcement learning 2 2 (Taghipour et al., 2007; Wang et al., 2014)
 ACCEPTED MANUSCRIPT

Table 4.3: Types of machine learning algorithms

Number of ML Number
Type of Machine Learning Algorithm Studies
algorithms of studies
Ensemble 22 14 (Aouay et al., 2014; Bar et al., 2013; Biancalana et al., 2011; Braida et al.,
2015; Buabin, 2012; Elmongui et al., 2015; T.-K. Fan & Chang, 2010;
Islam et al., 2015; Kao & Fahn, 2013; Middleton et al., 2004; Szymański
& Rzeniewicz, 2016; Tsuji et al., 2014; Vialardi et al., 2011; Yan et al.,
2013)
K Means 22 4 (T.-J. Lee & Tseng, 2012; Degemmis et al., 2007; Z. Fan et al., 2016; Zahra
et al., 2015)
Support Vector Machines (SVM) 20 17 (Agarwal, 2011; Banerjee et al., 2012; Brouard & Pomot, 2016; Diaby et al.,
2013, 2014; Ghazarian & Nematbakhsh, 2015; Jun, 2005; Kong et al., 2013;
Pecli et al., 2015; Pronoza et al., 2016; I. Song et al., 2011; Sun et al., 2015;
Szymański & Rzeniewicz, 2016; Tsuji et al., 2014; Verma et al., 2016; Yap
et al., 2005; K. Zhao & Pan, 2015)

T
Bayesian 14 12 (Aouay et al., 2014; Banerjee et al., 2012; Costa et al., 2012, 2013; De
Gemmis et al., 2008; Hernández del Olmo et al., 2009; Lops et al., 2009;
Musto et al., 2010; Paparrizos et al., 2011; Pecli et al., 2015; Pronoza et al.,

IP
2016; S.-Z. Zhang et al., 2007)
Decision Tree 14 13 (Alemeye & Getahun, 2015; Aouay et al., 2014; Banerjee et al., 2012;
Bellogı́n et al., 2011; Caraballo et al., 2014; Costa et al., 2013, 2013; Haiduc
et al., 2013; Hernández del Olmo et al., 2009; Hussain et al., 2015; W.-P. Lee

CR
& Lu, 2003; T. Liu et al., 2011; Musto et al., 2010; Wei et al., 2011)
Matrix Factorization 13 7 (Bauer & Nanopoulos, 2014; Huang & Nikulin, 2014; Krohn-Grimberghe
et al., 2011; J. Lu et al., 2013; Takács et al., 2008; Takáes et al., 2009; Zhai
& Li, 2015)
k Nearest Neighbors 11 10 (Aouay et al., 2014; Castro-Herrera et al., 2009; Das Dôres et al., 2016;

Latent Semantic Analysis

Logistic Regression
7
7
4
6
US Hernández del Olmo et al., 2009; Kong et al., 2013; Liang et al., 2014;
R. Liu et al., 2014; Marović et al., 2011; McLaughlin & Herlocker, 2004;
Pecli et al., 2015)
(Hofmann, 2003, 2004; Marović et al., 2011; Y. Zhang et al., 2009)
(Cai et al., 2012; Das et al., 2013; Krzywicki et al., 2015; Montañés et al.,
AN
2009; Pronoza et al., 2016; Sun et al., 2015)
Various 6 6 (Anaissi & Goyal, 2015; Forsati & Meybodi, 2010; Jung & Lee, 2004;
Marović et al., 2011; Murfi & Obermayer, 2009; Roh et al., 2003)
Clustering 5 4 (Bjelica, 2010; Hassan et al., 2010; J. Li & Zaı̈ane, 2004; Xuan et al., 2014)
Slope One 5 1 (Wang et al., 2012)
Association Rule 4 2 (Leopairote et al., 2013; X.-Z. Zhang, 2007)
M

Kernel Methods 4 4 (Devi & Venkatesh, 2013; Dinuzzo et al., 2011; X. Li & Chen, 2013; X. Li
et al., 2014)
Bandit 3 3 (Bouneffouf et al., 2012; Nicol et al., 2014; Wang et al., 2014)
Frequency Counting 3 1 (Luong, Huynh, Gauch, Do, & Hoang, 2012)
Least Squares 3 2 (Gedikli et al., 2011; Takács et al., 2008)
ED

Neural Network 3 3 (Geng et al., 2016; Marović et al., 2011; Aouay et al., 2014)
Regression Tree 3 2 (Marović et al., 2011; Pecli et al., 2015)
Sim. metric - Cosine Similarity 3 3 (Banerjee et al., 2012; Halder et al., 2014; Q.-C. Li et al., 2008)
Dictionary Learning 2 1 (Szabó et al., 2012)
Gradient Descent 2 2 (Cai et al., 2010; Pessiot et al., 2007)
PT

Latent Dirichlet Allocation 2 2 (Xin et al., 2014; Yeh & Wu, 2010)
Linear Model 2 2 (R. Zhang & Tran, 2010; Moreno et al., 2012)
Linear Regression 2 2 (W. Zhang et al., 2008; Q. Zhao et al., 2015)
Pearson Correlation 2 2 (Halder et al., 2014; Wan et al., 2009)
2 1
CE

Staked Regression (Q. Liu et al., 2014)

Cross-temporal Link Prediction 1 1 (Oyama et al., 2012)
Euclidean Distance 1 1 (Buettner, 2016)
Gaussian Processes 1 1 (Y. Song et al., 2011)
Graphical Model 1 1 (Yuan et al., 2014)
Learning Automata 1 1 (Forsati et al., 2009)
AC

Mahalanobis Classifier 1 1 (Tsapatsoulis et al., 2015)

Markov Model 1 1 (Baldominos et al., 2015)
Lagrange Multiplier 1 1 (Nie et al., 2013)
Mixture Model 1 1 (Y. Song et al., 2011)
Optimal Path Forest 1 1 (Marques et al., 2011)
Personality Diagnosis 1 1 (Marović et al., 2011)
Probabilistic Latent Semantic Analysis 1 1 (Jin et al., 2005)
Q-Learning 1 1 (Taghipour et al., 2007)
Regularization Methods 1 1 (Takács et al., 2008)
Shortest Path 1 1 (Luong, Huynh, Gauch, & Hoang, 2012)
Simil. metric - Geosemantic Proximity 1 1 (C. Lu et al., 2015)
Simil. metric - Aggregate Function 1 1 (Karahodza & Donko, 2015)
Simil. metric - WordNet Class Distance 1 1 (Nguyen et al., 2016)
Single Value Decomposition (SVD) 1 1 (Pantraki & Kotropoulos, 2015)
 ACCEPTED MANUSCRIPT

Table 4.4: Types of machine learning algorithms (alternative classification)

Number of ML Number
Type of Machine Learning Algorithm Studies
algorithms of studies
Clustering 38 18 (Bjelica, 2010; Degemmis et al., 2007; Z. Fan et al., 2016; Hassan et al.,
2010; J. Li & Zaı̈ane, 2004; Xuan et al., 2014; Zahra et al., 2015; Aouay
et al., 2014; Castro-Herrera et al., 2009; Das Dôres et al., 2016; Hernández
del Olmo et al., 2009; Kong et al., 2013; Liang et al., 2014; R. Liu et al.,
2014; Marović et al., 2011; McLaughlin & Herlocker, 2004; Pecli et al.,
2015; T.-J. Lee & Tseng, 2012)
Kernel Methods 25 22 (Devi & Venkatesh, 2013; Dinuzzo et al., 2011; X. Li & Chen, 2013; X. Li
et al., 2014; Agarwal, 2011; Banerjee et al., 2012; Brouard & Pomot, 2016;

T
Diaby et al., 2013, 2014; Ghazarian & Nematbakhsh, 2015; Jun, 2005; Kong
et al., 2013; Pecli et al., 2015; Pronoza et al., 2016; I. Song et al., 2011;
Sun et al., 2015; Szymański & Rzeniewicz, 2016; Tsuji et al., 2014; Verma

IP
et al., 2016; Yap et al., 2005; K. Zhao & Pan, 2015; Y. Song et al., 2011)
Ensemble 22 14 (Aouay et al., 2014; Bar et al., 2013; Biancalana et al., 2011; Braida et al.,
2015; Buabin, 2012; Elmongui et al., 2015; T.-K. Fan & Chang, 2010;

CR
Islam et al., 2015; Kao & Fahn, 2013; Middleton et al., 2004; Szymański
& Rzeniewicz, 2016; Tsuji et al., 2014; Vialardi et al., 2011; Yan et al.,
2013)
Matrix Factorization 21 12 (Bauer & Nanopoulos, 2014; Huang & Nikulin, 2014; Krohn-Grimberghe
et al., 2011; J. Lu et al., 2013; Takács et al., 2008; Takáes et al., 2009; Zhai
& Li, 2015; Hofmann, 2003, 2004; Marović et al., 2011; Y. Zhang et al.,

Decision Tree 17 15
US 2009; Pantraki & Kotropoulos, 2015)
(Alemeye & Getahun, 2015; Aouay et al., 2014; Banerjee et al., 2012;
Bellogı́n et al., 2011; Caraballo et al., 2014; Costa et al., 2013, 2013; Haiduc
et al., 2013; Hernández del Olmo et al., 2009; Hussain et al., 2015; W.-P. Lee
& Lu, 2003; T. Liu et al., 2011; Musto et al., 2010; Wei et al., 2011; Marović
AN
et al., 2011; Pecli et al., 2015)
Graphical Model 17 15 (Yuan et al., 2014; Aouay et al., 2014; Banerjee et al., 2012; Costa et al.,
2012, 2013; De Gemmis et al., 2008; Hernández del Olmo et al., 2009;
Lops et al., 2009; Musto et al., 2010; Paparrizos et al., 2011; Pecli et al.,
2015; Pronoza et al., 2016; S.-Z. Zhang et al., 2007; Xin et al., 2014; Yeh
& Wu, 2010)
M

Regression 16 13 (Cai et al., 2012; Das et al., 2013; Krzywicki et al., 2015; Montañés et al.,
2009; Pronoza et al., 2016; Sun et al., 2015; Gedikli et al., 2011; Takács
et al., 2008; W. Zhang et al., 2008; Q. Zhao et al., 2015; Halder et al.,
2014; Wan et al., 2009; Q. Liu et al., 2014)
Similarity Metric 7 7 (Banerjee et al., 2012; Halder et al., 2014; Buettner, 2016; C. Lu et al.,
ED

2015; Karahodza & Donko, 2015; Nguyen et al., 2016; Q.-C. Li et al.,
2008)
Various 6 6 (Anaissi & Goyal, 2015; Forsati & Meybodi, 2010; Jung & Lee, 2004;
Marović et al., 2011; Murfi & Obermayer, 2009; Roh et al., 2003)
Slope One 5 1 (Wang et al., 2012)
PT

Association Rule 4 2 (Leopairote et al., 2013; X.-Z. Zhang, 2007)

Bandit 3 3 (Bouneffouf et al., 2012; Nicol et al., 2014; Wang et al., 2014)
Neural Network 3 3 (Geng et al., 2016; Marović et al., 2011; Aouay et al., 2014)
Frequency Counting 3 1 (Luong, Huynh, Gauch, Do, & Hoang, 2012)
Dictionary Learning 2 1 (Szabó et al., 2012)
CE

Gradient Descent 2 2 (Cai et al., 2010; Pessiot et al., 2007)

Linear Model 2 2 (R. Zhang & Tran, 2010; Moreno et al., 2012)
Cross-temporal Link Prediction 1 1 (Oyama et al., 2012)
Learning Automata 1 1 (Forsati et al., 2009)
Mahalanobis Classifier 1 1 (Tsapatsoulis et al., 2015)
AC

Markov Model 1 1 (Baldominos et al., 2015)

Lagrage Multiplier 1 1 (Nie et al., 2013)
Mixture Model 1 1 (Y. Song et al., 2011)
Optimal Path Forest 1 1 (Marques et al., 2011)
Personality Diagnosis 1 1 (Marović et al., 2011)
Probabilistic Latent Semantic Analysis 1 1 (Jin et al., 2005)
Q-Learning 1 1 (Taghipour et al., 2007)
Regularization Methods 1 1 (Takács et al., 2008)
Shortest Path 1 1 (Luong, Huynh, Gauch, & Hoang, 2012)
 ACCEPTED MANUSCRIPT

Table 4.5: Big Data technologies movie ratings, named MovieLens7 , which is widely used.
Big Data Technologies Number of studies Studies Another source of user ratings is the Internet Movie Database
(Baldominos (IMDb)8 , which contains millions of titles and ratings that can
et al., 2015;
Yes 3 Dinuzzo et al.,
be used to build a testing dataset.
2011; Geng The social domain ranks in the second place. This domain
et al., 2016) accounts for algorithms aimed to work on social networks, or
No 118 Other studies
applications that connects different users. This use confirms
the trend of collaborative approaches in RS development with
ML algorithms. The tourism and the coding domains ranked
provided to an RS, the better should be its recommendations. low, revealing opportunities for research, since data in these
The evolution of technology has spawned research into new domains are rich and easily accessible.
ways of handling data. One such phenomenon is called Big
Data (M. Chen, Mao, & Liu, 2014), which has produced the 4.5 Performance Metrics

T
Hadoop distributed infrastructure (Shvachko, Kuang, Radia, The main goal of this systematic review is to identify trends
& Chansler, 2010) and the MapReduce programming model of ML algorithm use in RS development that can assist future

IP
(Dean & Ghemawat, 2008). Because Big Data has a direct researchers in their studies. The authors decided to take a
impact in RS development and ML algorithms (Leskovec, deeper look at how the algorithms are being used by inspecting
Rajaraman, & Ullman, 2014), the authors decided to look for

CR
the performance metrics that researchers use to describe ML
studies that have a discussion of Big Data in the description algorithms. These metrics may be accuracy metrics, such
of their proposed algorithms. Table 4.5 shows the number of as Precision or Recall, or alternative metrics, such as User
studies that included Big Data in their discussion or proposals. Preference or Coverage.
Among the studies that described some Big Data adapta- The analysis starts with an understanding of some of the
tions, Baldominos et al. (2015) used Big Data for storage.
The proposed architecture that provides on demand tools for
analysis uses the storage technologies HDFS (Hadoop Dis-
tributed File System) (Shvachko et al., 2010) and HBase6 for
USperformance metrics that have been proposed. Figure 4.2
shows a tree containing several metrics at the leaf nodes,
followed by their classifications as one goes up in the tree.
Although not complete, this tree provides an overview of the
AN
persistence logs and structured information about the execution many metrics that can be used to evaluate ML algorithms.
and predictions. Another study (Dinuzzo et al., 2011), in the In this systematic review, the authors found many of the
health domain, uses data from distributed datasets to make metrics expressed by Figure 4.2, but also found many other
predictions. The description of the Big Data technologies used metrics not described in the figure. Tables 4.7 and 4.8 present
M

in the prediction process was not the focus of the study. Lastly, metrics that were used to describe an algorithm. Note that
Geng et al. (2016) proposes a neural network-based algorithm the numbers do not add up to 121 studies or 205 algorithms.
that is applied to the image domain and, according to the The reason is that an algorithm may use one or more metrics
authors, easily scales to large networks. to describe its performance. Therefore, since there is at least
ED

Although as mentioned earlier, it is clear that few studies one metric per algorithm, one should expect the number of
had their proposals adapted for a Big Data reality, with dis- metrics to be greater than the number of algorithms. Another
tributed technologies or performance-optimized programming consideration is sorting of the methods. The authors decided to
paradigms. This Big Data apporach appears to represent a large sort the results, where well-known performance metrics were
PT

research opportunity for RS development. together and specific metrics were at the bottom of the table.
By inspecting Tables 4.7 and 4.8, one may note that
4.4 Application Domains Precision, Recall and F-measure, are among the most popular
performance metrics used in the studies of this systematic
CE

This systematic review investigates the application domains

review, totalling almost 50% of all occurrences. One reason
used in the studies analyzed. A primary study may propose
that may explain this result is that these metrics are the ones
multiple algorithms, which may be validated in many different
most often explained in textbooks. Most of the times, studies
application domains. This means that the authors may inves-
provided the three metrics together, since they are related, but
AC

tigate the application domains on a per algorithm or a per

as seen in the table, it is not for all the cases. Some studies
study basis. The authors opted for the latter approach so that
provide only the Precision, or only the F-measure. The authors
the number of algorithms proposed in a single study does not
did not calculate the missing values so the results would not
affect the final result of the application domain analysis. The
be affected. Two variants of the F-measure metric were used
results of the analysis are shown on Table 4.6.
by studies, created by changing one of the parameters of the
The application domain of Movies is the one mostly used
metrics. The studies does not explain the reason for the change.
with 31 occurrences among the 121 studies. One reason for
Accuracy ranks high as well with 49 occurrences, mainly
this result is the ease of access to data in the movie domain.
because of its intuitive nature when evaluating ML algorithms.
The University of Minnesota maintains a dataset with several
7 http://movielens.org
6 http://hbase.apache.org 8 http://www.imdb.com
 ACCEPTED MANUSCRIPT

Table 4.6: Application Domains

Number
Domain Studies
of studies
(Banerjee et al., 2012; Bar et al., 2013; Biancalana et al., 2011; Bjelica, 2010; Braida et al., 2015; Das et al., 2013;
Degemmis et al., 2007; Devi & Venkatesh, 2013; Gedikli et al., 2011; Halder et al., 2014; Hofmann, 2003, 2004; Jun,
2005; Karahodza & Donko, 2015; W.-P. Lee & Lu, 2003; X. Li et al., 2014; Liang et al., 2014; Q. Liu et al., 2014;
Movie 31
J. Lu et al., 2013; Marović et al., 2011; McLaughlin & Herlocker, 2004; Nie et al., 2013; Pessiot et al., 2007; Roh et al.,
2003; Takács et al., 2008; Takáes et al., 2009; Wang et al., 2012; Yeh & Wu, 2010; Zahra et al., 2015; S.-Z. Zhang
et al., 2007; Y. Zhang et al., 2009)
(Aouay et al., 2014; Cai et al., 2010, 2012; Diaby et al., 2013; Elmongui et al., 2015; Hassan et al., 2010; Islam et al.,
Social 14 2015; Krzywicki et al., 2015; Y. Song et al., 2011; Verma et al., 2016; Wan et al., 2009; Yan et al., 2013; Yuan et al.,
2014; K. Zhao & Pan, 2015)
(Das et al., 2013; Hernández del Olmo et al., 2009; Huang & Nikulin, 2014; Krohn-Grimberghe et al., 2011; Luong,
Academic 12 Huynh, Gauch, & Hoang, 2012; Luong, Huynh, Gauch, Do, & Hoang, 2012; Middleton et al., 2004; Montañés et al.,
2009; Oyama et al., 2012; Pecli et al., 2015; Vialardi et al., 2011; Xin et al., 2014)
(Bellogı́n et al., 2011; Brouard & Pomot, 2016; Buabin, 2012; Das et al., 2013; Z. Fan et al., 2016; W.-P. Lee & Lu,

T
News 11
2003; Leopairote et al., 2013; Q.-C. Li et al., 2008; Nguyen et al., 2016; Nie et al., 2013; Nicol et al., 2014)
(Anaissi & Goyal, 2015; Bauer & Nanopoulos, 2014; Buettner, 2016; Castro-Herrera et al., 2009; R. Zhang & Tran,
E-commerce 10

IP
2010; Nie et al., 2013; Pecli et al., 2015; Wei et al., 2011; Q. Zhao et al., 2015; Zhai & Li, 2015)
(Forsati et al., 2009; Forsati & Meybodi, 2010; Jin et al., 2005; Jun, 2005; Kao & Fahn, 2013; J. Li & Zaı̈ane, 2004;
Webpages 10
T. Liu et al., 2011; R. Liu et al., 2014; Musto et al., 2010; Taghipour et al., 2007)
(Alemeye & Getahun, 2015; Bouneffouf et al., 2012; Caraballo et al., 2014; Jung & Lee, 2004; J. Lu et al., 2013; Murfi

CR
Documents 9
& Obermayer, 2009; Nie et al., 2013; Szymański & Rzeniewicz, 2016; Xuan et al., 2014)
(Bauer & Nanopoulos, 2014; Ghazarian & Nematbakhsh, 2015; Marques et al., 2011; Moreno et al., 2012; Wang et al.,
Music 6
2014; Zahra et al., 2015)
Books 4 (X. Li & Chen, 2013; Tsuji et al., 2014; Xin et al., 2014; Zahra et al., 2015)
Health 4 (Agarwal, 2011; Dinuzzo et al., 2011; Hussain et al., 2015; I. Song et al., 2011)
Images
Tourism
Games
Pictures
Clothing
4
4
3
3
2
US
(Geng et al., 2016; Kong et al., 2013; Pantraki & Kotropoulos, 2015; Pecli et al., 2015)
(Costa et al., 2012, 2013; C. Lu et al., 2015; Sun et al., 2015)
(Baldominos et al., 2015; Castro-Herrera et al., 2009; Moreno et al., 2012)
(De Gemmis et al., 2008; Lops et al., 2009; Musto et al., 2010)
(X. Li & Chen, 2013; W. Zhang et al., 2008)
AN
E-mail 2 (Agarwal, 2011; Oyama et al., 2012)
Industry 2 (Castro-Herrera et al., 2009; Das et al., 2013)
Jobs 2 (Diaby et al., 2014; Paparrizos et al., 2011)
Restaurant 1 (Yap et al., 2005; Pronoza et al., 2016)
Advertisement 1 (T.-K. Fan & Chang, 2010)
Algorithm 1 (Das Dôres et al., 2016)
M

Antenna 1 (Agarwal, 2011)

Code 1 (Haiduc et al., 2013)
Elections 1 (Tsapatsoulis et al., 2015)
Jokes 1 (Szabó et al., 2012)
ED

Mobile Phones 1 (X.-Z. Zhang, 2007)

Railway 1 (Castro-Herrera et al., 2009)
Video 1 (T.-J. Lee & Tseng, 2012)
PT

Another important result is the large number of studies evalu- Some important considerations are as follows. The analysis
ating their proposals in terms of the error in the prediction by of the performance metrics is per algorithm, which means that
using the RMSE (Round Mean Squared Error) and the MAE studies that proposed more than one algorithm are repeated in
(Mean Absolute Error) metrics. MAE had 123 occurrences, the plots. Moreover, as a study may validate its ML algorithm
CE

and ranks in the first position. The simplicity of the calculation with different versions of the same data, the authors decided
of these metrics may be the reason for this result. to report the results related to the richer data. For example,
MovieLens provides three sizes of their movie ratings dataset:
A study of the occurrence of each metric shows the pop- 100K, 1M, and 10M data values, where K is thousands and M
AC

ularity of some metrics, as well introduces other metrics to is millions. This systematic review analyzes the results related
researchers. However, the authors decided to break down the only to the larger dataset. This is done to simulate the real
most popular metrics and observe how the algorithms actually world as much as possible.
performed, as reported by the studies. The authors decided
to plot all of the values for some of the performance metrics A similar decision is taken regarding parameters of the
displayed at the top of Table 4.7 to discover any trends, or algorithms. Many times, studies report the results of an ML
any studies that stand out. However, plotting tens of studies is algorithm assuming many different parameters. In this case,
not feasible. Many values are overwritten by others values and this systematic review considers the best result for analysis.
the figure becomes unreadable. Therefore, the authors display This is done for the benefit of other researchers who may
the best and worst value of each plot, and in another plot, the be searching for an algorithm that is better than a certain
authors present the top 10 studies for each metric analyzed. threshold.
 ACCEPTED MANUSCRIPT

Table 4.7: Performance Metrics

Number of ML Number
Metrics Studies
algorithms of studies
(Agarwal, 2011; Aouay et al., 2014; Baldominos et al., 2015; Banerjee et al., 2012; Biancalana
et al., 2011; Brouard & Pomot, 2016; Cai et al., 2010, 2012; Caraballo et al., 2014; Costa et al.,
2012, 2013; De Gemmis et al., 2008; Devi & Venkatesh, 2013; Elmongui et al., 2015; T.-K. Fan
& Chang, 2010; Z. Fan et al., 2016; Forsati et al., 2009; Forsati & Meybodi, 2010; Ghazarian &
Nematbakhsh, 2015; R. Zhang & Tran, 2010; Islam et al., 2015; Jun, 2005; Kao & Fahn, 2013;
Precision 98 44
Leopairote et al., 2013; J. Li & Zaı̈ane, 2004; X. Li & Chen, 2013; Liang et al., 2014; T. Liu et al.,
2011; Lops et al., 2009; Luong, Huynh, Gauch, Do, & Hoang, 2012; McLaughlin & Herlocker,
2004; Murfi & Obermayer, 2009; Musto et al., 2010; Nguyen et al., 2016; Nie et al., 2013; Pantraki
& Kotropoulos, 2015; Pecli et al., 2015; Y. Song et al., 2011; Szymański & Rzeniewicz, 2016;
Verma et al., 2016; Wan et al., 2009; Yap et al., 2005; Yuan et al., 2014; K. Zhao & Pan, 2015)
(Aouay et al., 2014; Baldominos et al., 2015; Banerjee et al., 2012; Cai et al., 2010, 2012; Caraballo
et al., 2014; Costa et al., 2012, 2013; De Gemmis et al., 2008; Devi & Venkatesh, 2013; Elmongui

T
et al., 2015; T.-K. Fan & Chang, 2010; Z. Fan et al., 2016; R. Zhang & Tran, 2010; Krzywicki et al.,
Recall 58 32 2015; Leopairote et al., 2013; X. Li & Chen, 2013; Liang et al., 2014; T. Liu et al., 2011; Lops
et al., 2009; McLaughlin & Herlocker, 2004; Murfi & Obermayer, 2009; Musto et al., 2010; Nguyen

IP
et al., 2016; Nie et al., 2013; Pantraki & Kotropoulos, 2015; Y. Song et al., 2011; Szymański &
Rzeniewicz, 2016; Verma et al., 2016; Yap et al., 2005; Yuan et al., 2014; K. Zhao & Pan, 2015)
(Banerjee et al., 2012; Buabin, 2012; Caraballo et al., 2014; Costa et al., 2012, 2013; Das et al.,
2013; Elmongui et al., 2015; T.-K. Fan & Chang, 2010; Z. Fan et al., 2016; Hassan et al., 2010;

CR
R. Zhang & Tran, 2010; Leopairote et al., 2013; X. Li & Chen, 2013; X. Li et al., 2014; Liang
F-measure 80 29 et al., 2014; Lops et al., 2009; C. Lu et al., 2015; Montañés et al., 2009; Murfi & Obermayer,
2009; Pantraki & Kotropoulos, 2015; Pronoza et al., 2016; Y. Song et al., 2011; Szymański &
Rzeniewicz, 2016; Tsapatsoulis et al., 2015; Verma et al., 2016; Xuan et al., 2014; Yap et al.,
2005; Yuan et al., 2014; K. Zhao & Pan, 2015)
R-Precision
F(0.5)
F[3,1298]
1
5
1
1
1
1
US
(Brouard & Pomot, 2016)
(De Gemmis et al., 2008)
(W. Zhang et al., 2008)
(Bar et al., 2013; Braida et al., 2015; Dinuzzo et al., 2011; Gedikli et al., 2011; Hofmann, 2003,
2004; Jun, 2005; Karahodza & Donko, 2015; Krohn-Grimberghe et al., 2011; X. Li et al., 2014;
AN
RMSE 56 22 Q. Liu et al., 2014; J. Lu et al., 2013; Marović et al., 2011; Szabó et al., 2012; Takács et al., 2008;
Takáes et al., 2009; Wang et al., 2012, 2014; Yan et al., 2013; Yeh & Wu, 2010; Zhai & Li, 2015;
Q. Zhao et al., 2015)
(Bauer & Nanopoulos, 2014; Braida et al., 2015; Castro-Herrera et al., 2009; Degemmis et al.,
2007; Devi & Venkatesh, 2013; T.-K. Fan & Chang, 2010; Ghazarian & Nematbakhsh, 2015;
MAE 123 20 Hofmann, 2004; Jung & Lee, 2004; Karahodza & Donko, 2015; Q. Liu et al., 2014; J. Lu et al.,
M

2013; McLaughlin & Herlocker, 2004; Moreno et al., 2012; Takáes et al., 2009; Wang et al., 2012;
Wei et al., 2011; Yan et al., 2013; Zahra et al., 2015; Y. Zhang et al., 2009)
Normalized
2 1 (Nie et al., 2013)
MAE
MAP 7 4 (Biancalana et al., 2011; Elmongui et al., 2015; Kao & Fahn, 2013; Tsapatsoulis et al., 2015)
ED

(Alemeye & Getahun, 2015; Banerjee et al., 2012; Bellogı́n et al., 2011; Bjelica, 2010; Cai et al.,
2012; Castro-Herrera et al., 2009; Halder et al., 2014; Hussain et al., 2015; Jin et al., 2005; Kong
et al., 2013; Krzywicki et al., 2015; W.-P. Lee & Lu, 2003; T.-J. Lee & Tseng, 2012; Q.-C. Li et al.,
Accuracy 49 26
2008; T. Liu et al., 2011; R. Liu et al., 2014; Luong, Huynh, Gauch, & Hoang, 2012; Marques
et al., 2011; Middleton et al., 2004; Paparrizos et al., 2011; I. Song et al., 2011; Sun et al., 2015;
Taghipour et al., 2007; Vialardi et al., 2011; Xin et al., 2014; X.-Z. Zhang, 2007)
PT

ROC 9 4 (Costa et al., 2012, 2013; X. Li & Chen, 2013; Roh et al., 2003)
(Agarwal, 2011; Anaissi & Goyal, 2015; Cai et al., 2012; Diaby et al., 2013, 2014; T.-K. Fan &
AUC 14 8
Chang, 2010; Oyama et al., 2012; I. Song et al., 2011)
Click Through
2 1 (Bouneffouf et al., 2012)
CE

Rate (CTR)
Kullback-Leibler
1 1 (S.-Z. Zhang et al., 2007)
divergence
NARG (Intrusion
3 1 (Hernández del Olmo et al., 2009)
Cost)
NDPM
AC

(Normalized
Distance-based 1 1 (Musto et al., 2010)
Performance
Measure)
Shortcut Gain 2 2 (J. Li & Zaı̈ane, 2004; Taghipour et al., 2007)
MSE (Mean
2 1 (Degemmis et al., 2007)
Squared Error)
Jaccard
10 1 (Xuan et al., 2014)
Coefficient
Folkes & Mal-
10 1 (Xuan et al., 2014)
lows
Rank Score Mea-
1 1 (Jung & Lee, 2004)
sure (RSM)
 ACCEPTED MANUSCRIPT

T
IP
CR
US
AN

Figure 4.2: An overview of performance metrics (from (Japkowicz & Shah, 2011))
M
ED

Table 4.8: Performance Metrics (cont.)

Number of ML Number
Metrics Studies
algorithms of studies
PT

PPE (Percentage
of Positive Eval- 2 1 (Tsuji et al., 2014)
uations)
DCG 3 1 (Agarwal, 2011)
NDCG
CE

(Normalized
Discount 4 3 (Elmongui et al., 2015; Geng et al., 2016; Kao & Fahn, 2013)
Cumulative
Gain)
Rank Score 4 2 (Jung & Lee, 2004; X. Li & Chen, 2013)
AC

Average
Absolute 6 1 (Marović et al., 2011)
Deviation (AAD)
Absolute Error 1 1 (Hofmann, 2003)
0/1 loss 4 2 (Hofmann, 2003, 2004)
R 1 1 (W. Zhang et al., 2008)
R2 1 1 (W. Zhang et al., 2008)
Convergence 1 1 (Huang & Nikulin, 2014)
Error function 1 1 (Buettner, 2016)
Mean Ranking
1 1 (Pessiot et al., 2007)
Error (MRE)
 ACCEPTED MANUSCRIPT

Lastly, performance results may be reported using the 0-100 It should be noticed that the study (Costa et al., 2013) also
range or a 0-1 range. For the former case the authors reduced reported a high value of ROC for their algorithms. In terms of
the result to the range of 0-1 simply by dividing the reported AUC, most of the studies performed well and reported high
result by 100. This is done to make results comparable, and values of AUC. Although not very popular, these two metrics
it does not affect the final result of the analysis. can also be used by other researchers in their analysis to
Figure 4.3 presents the plots for Precision (Figures 4.3a and improve the findings or the amount of detail of their proposals.
4.3b), Recall (Figures 4.3c and 4.3d), and F-measure (Figures
4.3e and 4.3f), followed by a discussion of the results. 4.6 Alternative Performance Metrics
Figure 4.3 shows that one study ranked very well on
Precision, Recall, and F-measure. This study (Costa et al., This section presents alternative metrics that can also be
2013) uses Bayesian algorithms (BayesNet and Naive Bayes) used to describe the performance of ML algorithms in RS
as well as Decision Trees (J48 pruned and unpruned) to development. These metrics are well described in another
study (Gunawardana & Shani, 2015) with examples and sug-

T
recommend points of interest (POIs) to users. According to
the authors, the two main differences from other approaches gested ways of capturing data and calculating results. The
eight metrics are user preference, coverage, confidence, trust,

IP
are the use of a user’s context, because at different contexts,
different items may be relevant or not to the user. A multi- novelty, serendipity, diversity, utility, risk, robustness, privacy,
agent system (MAS) is also developed to handle requests. and scalability. Some of them are discussed in the next few

CR
Overall, many algorithms performed well in Precision, paragraphs.
Recall, and F-measure. The best for each metric is shown in User preference, as its name suggests, relates to the opinion
the top 10 plot. of the user about the recommendations made by the RS.
Figure 4.4 shows the results for the Accuracy metric (Fig- Users are more likely to choose approaches that predict items
that match their preferences. Although the description is easy,
ures 4.4a and 4.4b). There is a quick drop in accuracy among
the algorithms, with few ranking above 80% accuracy. The
accuracy metric is one of the most intuitive ones, and should
not be overlooked, since it gives an initial perception of how
US gathering user data to achieve high user preference is not.
The main method to obtain data about user preference is
the use of questionnaires. The coverage metric relates to the
items that can be recommended to the users that can receive
AN
the ML algorithm is performing.
Figure 4.5 shows the breakdown of performance results for recommendations. There are specific ways to calculate the
RMSE (Round Mean Squared Error) (Figures 4.5a and 4.5b), coverage and one should refer to (Gunawardana & Shani,
MAE (Mean Absolute Error) (Figures 4.5c and 4.5d), and 2015) for more details.
MAP (Mean Average Precision) (Figures 4.5e and 4.5f). Two additional alternative metrics are diversity and scal-
M

Two important points to be mentioned are as follows. One ability. To discuss diversity, one must understand similarity,
may notice that plots of the ML algorithms related to error since these two concepts are antagonistic. If the results are not
metrics, such as RMSE and MAE, show the lower values on similar, then that means they are diverse. Lastly, scalability
does not mean much to the user, but important both to
ED

top, instead of the bottom, so that the best performing studies

are shown on the top. The second point relates to MAP. This researchh and performance. Scalability relates to how well-
metric did not have a relevant number of occurrences and for prepared the algorithms is to handle growth in the amount
that reason, the two plots about this metric are very similar. of data. Most of the time, algorithms need more memory or
The MAP plot about all studies was included for completeness. computation power to manipulate large amounts of data.
PT

On the plots for both the RMSE and the MAE metrics, Table 4.9 shows the number of algorithms that included
few studies had an excellent result. In the RMSE case, most a discussion on alternative metrics in its description. The
of the studies that reported this metric had a value greater difference between the “Textual” and “Numeric” entries in
CE

than 0.8. Care should be taken since the greater the error the table is because that discussion can be in the written form,
the larger difference between what is expected and what is with considerations or suppositions, or it can be based on a
predicted. In the RMSE result, Dinuzzo et al. (2011) reported formula. The last column shows the studies that discussed the
the value of 5.2, and in the MAE result, Bauer and Nanopoulos algorithms. The difference between the number of algorithms
AC

(2014) reported the value of 4.0234. Few studies reported presented in the third column, and the number of studies of
MAP values, but the plots were included in this discussion the fourth column exists because a study may propose more
because of the simplicity of the metric and the possible interest than one algorithm.
of researchers in the results. In addition, Table 4.9 displays four new alternative metrics:
Finally, Figure 4.6 shows the breakdown of results for transparency, quality, perplexity, and sensitivity. The papers
the metrics ROC (Receiver Operating Characteristic curve) that reported values for those metrics do not provide a formal
(Figures 4.6a and 4.6b) and AUC (Area Under the ROC Curve) definition. For that reason, they were presented in this sys-
(Figures 4.6c and 4.6d). It should be noted that both metrics tematic review, but not explained. One final note is that the
did not have a large number of occurrences in the studies perplexity metric is the closest one to another metric defined
analyzed in this systematic review, but they were included in in (Gunawardana & Shani, 2015): serendipity, which describes
this discussion owing to their academic importance. how surprising the successful recommendations are.
 ACCEPTED MANUSCRIPT

Figure 4.3: A breakdown of the performance results for Precision, Recall, and F-measure
(a) Precision - Top 10 (b) Precision

(Costa et al., 2013) 0.99665 0.99665

(Costa et al., 2013) 0.99665
(Costa et al., 2013) 0.99425
(Costa et al., 2013) 0.99425
(Baldominos et al., 2015) 0.9893

T
(Devi & Venkatesh, 2013) 0.98
(Aouay et al., 2014) 0.979

IP
(Aouay et al., 2014) 0.979
(Pecli et al., 2015) 0.975
(Pecli et al., 2015) 0.975 0.077

CR
0.9 0.92 0.94 0.96 0.98 1 0 0.2 0.4 0.6 0.8 1
Precision Precision

(Baldominos et al.,
(Costa et al.,
(Costa et al.,
2015)
2013)
2013)
(c) Recall - Top 10

US 0.9978
0.9975
0.9975
(d) Recall

0.9978
AN
(Costa et al., 2013) 0.9856
(Costa et al., 2013) 0.9856
(Aouay et al., 2014) 0.979
(Aouay et al., 2014) 0.979
(K. Zhao & Pan, 2015)
M

0.9718
(Z. Fan et al., 2016) 0.961
(Caraballo et al., 2014) 0.95 0.025
0.9 0.92 0.94 0.96 0.98 1
ED

0 0.2 0.4 0.6 0.8 1

Recall Recall
(e) F-measure - Top 10 (f) F-measure
PT

(Costa et al., 2013) 0.9971 0.9971

(Costa et al., 2013) 0.9971
(Costa et al., 2013) 0.9849
(Costa et al., 2013) 0.9849
CE

(Elmongui et al., 2015) 0.9817

(Z. Fan et al., 2016) 0.952
(Buabin, 2012) 0.95
(Caraballo et al., 2014) 0.95
AC

(Buabin, 2012) 0.9491

(Buabin, 2012) 0.93009 0.0118
0.9 0.92 0.94 0.96 0.98 1 0 0.2 0.4 0.6 0.8 1
F-measure F-measure
 ACCEPTED MANUSCRIPT

Figure 4.4: A breakdown of the performance results for Accuracy

(a) Accuracy - Top 10 (b) Accuracy

(Marques et al., 2011) 0.9861 0.9861

(T. Liu et al., 2011) 0.985
(Kong et al., 2013) 0.9423
(X.-Z. Zhang, 2007) 0.932
(Kong et al., 2013) 0.9176
(I. Song et al., 2011) 0.9
(Castro-Herrera et al., 2009) 0.89
(Bjelica, 2010) 0.87
(Paparrizos et al., 2011) 0.8609
(Vialardi et al., 2011)

T
0.8536 0.11954
0.8 0.85 0.9 0.95 1 0 0.2 0.4 0.6 0.8 1

IP
Accuracy Accuracy

CR
Many studies have a numeric discussion of coverage with systematic review because of the exclusion criteria previously
formulas to describe their values. By inspecting these studies, explained. The studies are shown in Table 4.11.
the authors noticed that they use specific formulas and no The search string used in this systematic review also re-
standard is defined. The same happened to the sensitivity
metric. This table describes many alternative performance
metrics used to evaluate ML algorithms in RS development
and introduces these metrics to those that did not know them.
US turned conference and journal entries. Since these entries
are not peer-reviewed, they were not inspected based on the
exclusion criteria. However, researchers may find it beneficial
to know the conferences or journals that are reporting on the
AN
Other metrics described in (Gunawardana & Shani, 2015) research fields of recommender systems and machine learning
did not have any occurrence in the studies of this systematic algorithms. Table 4.12 lists the sources (e.g. conferences,
review and therefore were not included in the results table. journals) returned by the search string of this systematic review
with the year in which they were held. The list is sorted by
M

4.7 Analysis of the Sources year.

This section describes a different perspective on the anal- Finally, the authors decided to list the sources of the
ysis of the primary studies, and helps researchers find more primary studies inspected in this review and rank them by the
information about ML algorithms for RS development. The number of studies found in each source. Popular sources may
ED

discussion focuses on other surveys and sources (e.g. confer- contain papers with similar interests to the research fields of
ences, journals) related to this systematic review. this systematic review and indicate possible places to submit
This systematic review adopted an exclusion criteria that publications. Tables 4.13 and 4.14 present the sources of the
limited the papers included in our study to primary studies. primary studies with the number of studies retrieved from
PT

This means that secondary studies such as other literature each source. In the table, two of the sources show up as
reviews were not analyzed. However, these secondary studies important sources of RS and ML algorithms: “Lecture Notes in
hold valuable knowledge that improves the research on the Computer Science” and “Expert Systems with Applications”.
CE

field and may be beneficial to other researchers. For that

reason, Table 4.10 presents the secondary studies that were 5. C ONCLUSIONS AND F UTURE W ORK
excluded from this systematic review. Currently, recommender systems (RS) are widely used in
These secondary studies were not assessed for quality, but e-commerce, social networks, and several other domains.
AC

they were returned by the search string of this systematic Since the introduction of RSs in mid 1990s, research in RSs
review and are expected to cover the main research fields of has been evolving. One progressive step in RS history is
interest, such as recommender systems and machine learning the adoption of machine learning (ML) algorithms, which
algorithms. The full reference to each secondary study is found allow computers to learn based on user information and to
at the end of this study. personalize recommendations further. Machine learning is an
Moreover, domain experts that contributed to this work Artificial Intelligence (AI) research field that encompasses
shared other secondary studies that also relate to at least one of algorithms whose goal is to predict the outcome of data
the research fields of this systematic review. They are different processing. ML has made major breakthroughs in the fields
from those presented on Table 4.10 and may be also beneficial of image recognition, search engines, and security. However,
to researchers in the field. Secondary studies suggested by the ML field has several algorithms described in the literature,
domain experts were not included in the analysis of this with varied characteristics. The literature lacks a classification
 ACCEPTED MANUSCRIPT

Figure 4.5: A breakdown of the performance results for RMSE, MAE, and MAP
(a) RMSE - Top 10 (b) RMSE

(Yan et al., 2013) 0.165 0.165

(Q. Zhao et al., 2015) 0.22
(Yeh & Wu, 2010) 0.252
(Krohn-Grimberghe et al., 2011) 0.32778
(Krohn-Grimberghe et al., 2011) 0.3738

T
(Karahodza & Donko, 2015) 0.7926
(Takáes et al., 2009) 0.8275

IP
(Braida et al., 2015) 0.871
(J. Lu et al., 2013) 0.8732
(Takács et al., 2008) 0.8895 5.2

CR
0 0.2 0.4 0.6 0.8 1 0 2 4 6
RMSE RMSE

(Devi & Venkatesh,

(T.-K. Fan & Chang,
(T.-K. Fan & Chang,
2013)
2010)
2010)
(c) MAE - Top 10

0.01
0.028
0.062
US 0.01
(d) MAE
AN
(T.-K. Fan & Chang, 2010) 0.074
(Yan et al., 2013) 0.111
(Jung & Lee, 2004) 0.16
(Zahra et al., 2015) 0.336
(Zahra et al., 2015)
M

0.344
(Zahra et al., 2015) 0.352
(Zahra et al., 2015) 0.354 4.0234
0 0.1 0.2 0.3 0.4 0.5
ED

0 1 2 3 4 5
MAE MAE
(e) MAP - Top 10 (f) MAP
PT

(Tsapatsoulis et al., 2015) 0.8549 0.8549

(Tsapatsoulis et al., 2015) 0.7993

(Tsapatsoulis et al., 2015) 0.7756
CE

(Kao & Fahn, 2013) 0.64

(Kao & Fahn, 2013) 0.57
AC

(Elmongui et al., 2015) 0.455

(Biancalana et al., 2011) 0.298 0.298
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
MAP MAP
 ACCEPTED MANUSCRIPT

Figure 4.6: A breakdown of the performance results for ROC and AUC
(a) ROC - Top 10 (b) ROC

(Costa et al., 2013) 0.9985 0.9985

(Costa et al., 2013) 0.9985
(Costa et al., 2013) 0.9929
(Costa et al., 2013) 0.9929
(Roh et al., 2003) 0.8461
(Costa et al., 2012) 0.745
(X. Li & Chen, 2013) 0.58
(X. Li & Chen, 2013) 0.45
(X. Li & Chen, 2013)

T
0.45 0.45
0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1

IP
ROC ROC
(c) AUC - Top 10 (d) AUC

CR
(Anaissi & Goyal, 2015) 0.96 0.96
(Oyama et al., 2012) 0.951
(I. Song et al., 2011) 0.95
(Agarwal,
(Agarwal,
(Diaby et al.,
(Diaby et al.,
2011)
2011)
2014)
2013)
0.9237
0.92
0.91
US
0.9388
AN
(Agarwal, 2011) 0.8991
(Diaby et al., 2013) 0.845
(Cai et al., 2012) 0.827 0.60
0.8 0.85 0.9 0.95 1 0 0.2 0.4 0.6 0.8 1
M

AUC AUC
ED

system for algorithms showing the environment in which they massive data storage and analysis. The application domain
are most suitable. Therefore, researchers in RSs do not have a of movies ranks as first among others mainly because of
clear view of the trends in ML algorithm usage to decide on MovieLens, a simple dataset available online. Finally, MAE,
where to focus their research efforts. This study then proposes Precision, Recall, and F-measure are the most used perfor-
PT

a systematic review to observe the ML algorithms that are used mance metrics to evaluate ML algorithms in RS development,
in RSs as well as the trends and open questions in this research and Coverage is the most used alternative metric.
field. This systematic review has also included an analysis of the
The systematic review collected 121 primary studies, after sources of the primary studies that were selected. The analysis
CE

filtering out some based on exclusion criteria. All publications presents surveys of the literature as well as conferences and
were read and the conclusions are as follows. There is a journals that may be of interest to researchers working on
trend for collaborative approaches in RS development, espe- similar topics.
cially with the use of neighborhood-based methods. Hybrid
AC

This study has several contributions to research in expert

approaches are still a research opportunity. A timeline with the and intelligent systems. It presents a comprehensive overview
number of primary studies published in recent years confirms of ML algorithms in RSs that assists application developers
the trends and the opportunities mentioned. by helping them to identify the algorithms, their types, and
Regarding the ML algorithms, both supervised and un- trends in the use of specific algorithms. This study also
supervised learning are being well researched. Clustering provides existing classes of evaluation metrics and ranks the
algorithms, as well Ensemble, and Support Vector Machines ML algorithms based on these metrics. From this result,
(SVM) are among the ones most used. One may note again the researchers and practitioners are able not only to be familiar
presence of neighborhood-based approaches among the ML with the most used evaluation metrics, but also to investigate
algorithms. The focus on Big Data technologies still remains further the approaches that have the high rankings. In addi-
a research opportunity, with few studies even mentioning tion, this study identifies and presents trends in the use of
 ACCEPTED MANUSCRIPT

Table 4.9: Alternative Performance Metrics questions involving research topics related to RSs and ML
Number Number algorithms should be investigated, including the application
Textual/
Metrics of ML of Studies of collaborative approaches in social networks and spatial-
Numeric
algorithms studies
Textual 0 0
temporal domains.
User
(Bellogı́n et al.,
Preference
2011; Ghazarian ACKNOWLEDGMENT
& Nematbakhsh, The authors would like to thank the reviewers for their
2015; Q.-C. Li
Numeric 7 6 et al., 2008; valuable comments, which helped to improve our systematic
Middleton et al., review. The authors also thank the Natural Sciences and
2004; Wang Engineering Research Council of Canada (NSERC), and the
et al., 2014; Yap
et al., 2005) Ontario Research Fund of the Ontario Ministry of Research,
(Karahodza & Innovation, and Science for their financial support for this
Textual 1 1
Donko, 2015) research.

T
Coverage (Braida et al.,
2015; Forsati R EFERENCES

IP
et al., 2009;
Forsati & Adomavicius, G., Mobasher, B., Ricci, F., & Tuzhilin, A.
Meybodi, 2010; (2011). Context-aware recommender systems. AI Mag-
Karahodza &

CR
Numeric 27 8 Donko, 2015; azine, 32(3), 67–80.
J. Li & Zaı̈ane, Adomavicius, G. & Tuzhilin, A. (2005). Toward the next gen-
2004; Middleton eration of recommender systems: a survey of the state-
et al., 2004;
Taghipour et al., of-the-art and possible extensions. IEEE Transactions
2007; Zahra on Knowledge and Data Engineering, 17(6), 734–749.

Diversity
Textual
Numeric
0
1
0
1
et al., 2015)

(Geng et al.,
2016)
(Degemmis
US doi:10.1109/TKDE.2005.99
Agarwal, S. (2011). The infinite push: a new support vector
ranking algorithm that directly optimizes accuracy at the
absolute top of the list. (pp. 839–850).
AN
et al., 2007;
Scalability
Textual 2 2
Baldominos Aggarwal, C. C. (2016a). Content-based recommender sys-
et al., 2015) tems. In Recommender systems: the textbook (pp. 139–
Numeric 0 0 166). Cham: Springer International Publishing. doi:10.
(Degemmis
Transparency
Textual 1 1
et al., 2007)
1007/978-3-319-29659-3 4
M

Numeric 0 0 Aggarwal, C. C. (2016b). Model-based collaborative filtering.

Quality
Textual 0 0 In Recommender systems: the textbook (pp. 71–138).
(Das Dôres et Cham: Springer International Publishing. doi:10.1007/
Numeric 1 1
al., 2016)
978-3-319-29659-3 3
ED

Textual 0 0
Perplexity
Numeric 1 1
(Xin et al., Aggarwal, C. C. (2016c). Neighborhood-based collaborative
2014) filtering. In Recommender systems: the textbook (pp. 29–
Textual 0 0
Sensitivity
(Das et al., 70). Cham: Springer International Publishing. doi:10 .
Numeric 12 1 1007/978-3-319-29659-3 2
2013)
PT

Ahmed, A., Kanagal, B., Pandey, S., Josifovski, V., Pueyo, L.,
& Yuan, J. (2013). Latent factor models with additive
ML algorithms for RSs in different application domains. For and hierarchically-smoothed user preferences. (pp. 385–
394). doi:10.1145/2433396.2433445
CE

example, researchers and practitioners can become aware of

the algorithms that have been applied in a specific domain Alemeye, F. & Getahun, F. (2015). Cloud readiness assessment
(e.g. movie, news, e-commerce). Lastly, sources of primary framework and recommendation system. (Vol. 2015-
and secondary studies are provided. These sources can help November). doi:10.1109/AFRCON.2015.7331995
Amatriain, X. & Basilico, J. (2015). Recommender systems in
AC

researchers and developers to position new research activity

in this domain appropriately. industry: a netflix case study. Springer US. doi:10.1007/
In the future, more studies on the use of Clustering, 978-1-4899-7637-11
Ensemble, and SVM algorithms in RSs can be developed Anaissi, A. & Goyal, M. (2015). Svm-based association rules
to observe the implications of their use, performance, and for knowledge discovery and classification. doi:10.1109/
utility. Moreover, RS development lacks studies analyzing APWCCSE.2015.7476236
early stages, such as requirements and design, and late stages, Aouay, S., Jamoussi, S., & Gargouri, F. (2014). Feature based
such as maintenance. Other research opportunities involve the link prediction. (Vol. 2014, pp. 523–527). doi:10.1109/
investigation of Big Data technologies, which offer a wide AICCSA.2014.7073243
variety of methods to support the storage and analysis of Apte, C. (2010). The role of machine learning in business
massive data. The authors also believe that many other open optimization. (pp. 1–2).
 ACCEPTED MANUSCRIPT

Table 4.10: Secondary studies retrieved in this systematic review

Reference Secondary Study Name
(Bouza & Bernstein, 2014) (Partial) user preference similarity as classification-based model similarity
(Lahlou, Mountassir, Benbrahim, & Kassou, 2013) A Text Classification based method for context extraction from online reviews
(Bertin-Mahieux, Eck, & Mandel, 2010) Automatic tagging of audio: The state-of-the-art
(Carbone & Vlassov, 2015) Auto-Scoring of Personalised News in the Real-Time Web: Challenges, Overview and
Evaluation of the State-of-the-Art Solutions
(Lahlou, Benbrahimand, Mountassir, & Kassou, 2013) Context extraction from reviews for Context Aware Recommendation using Text Classi-
fication techniques
(Cremonesi, Garzotto, Negro, Papadopoulos, & Turrin, 2011) Looking for “good” recommendations: A comparative evaluation of recommender systems
(Bagchi, 2015) Performance and quality assessment of similarity measures in collaborative filtering using
mahout
(Feuerverger, He, & Khatri, 2012) Statistical significance of the netflix challenge
(Shani & Gunawardana, 2013) Tutorial on application-oriented evaluation of recommendation systems
(Jannach, Lerche, Gedikli, & Bonnin, 2013) What recommenders recommend - An analysis of accuracy, popularity, and sales diversity

T
effects

IP
Table 4.11: Secondary studies shared by domain experts
Reference Secondary Study Name
(Ning et al., 2015) A comprehensive survey of neighborhood-based recommendation methods

CR
(Elahi, Ricci, & Rubens, 2016) A survey of active learning in collaborative filtering recommender systems
(Rubens, Elahi, Sugiyama, & Kaplan, 2015) Active learning in recommender systems
(Koren & Bell, 2015) Advances in collaborative filtering
(Sarwar, Karypis, Konstan, & Riedl, 2000) Analysis of Recommendation Algorithms for E-Commerce
(Aggarwal, 2016a) Content-Based Recommender Systems
(Breese, Heckerman, & Kadie, 1998)
(Karypis, 2001)
(Deshpande & Karypis, 2004)
(Aggarwal, 2016b)
(Aggarwal, 2016c)
US
Empirical analysis of predictive algorithms for collaborative filtering
Evaluation of item-based top-N recommendation algorithms
Item-based top-N recommendation algorithms
Model-Based Collaborative Filtering
Neighborhood-Based Collaborative Filtering
AN
(Amatriain & Basilico, 2015) Recommender systems in industry: A netflix case study

Table 4.12: Sources retrieved in this systematic review

Source Name Year
17th International Conference on Passive and Active Measurement, PAM 2016 2016
M

8th Asian Conference on Intelligent Information and Database Systems, ACIIDS 2016 2016
14th Mexican International Conference on Artificial Intelligence, MICAI 2015 2015
2014 International Conference on Sensors Instrument and Information Technology, ICSIIT 2014 2014
KDIR 2014 - Proceedings of the International Conference on Knowledge Discovery and Information Retrieval 2014
SIGIR 2014 - Proceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval 2014
ED

9th IFIP WG 12.5 International Conference on Artificial Intelligence Applications and Innovations, AIAI 2013 2013
2012 IEEE International Workshop on Machine Learning for Signal Processing - Proceedings of MLSP 2012 2012
3rd IFIP TC 12 International Conference on Artificial Intelligence, IFIP AI 2010 As part of 21st IFIP World Computer Congress, WCC 2010 2010
Proceedings - Artificial Intelligence Applications and Innovations - 6th IFIP WG 12.5 International Conference, AIAI 2010 2010
2009 2nd International Workshop on Managing Requirements Knowledge, MARK 2009 2009
PT

Proceedings - 7TH. International Conference on Hybrid Intelligent Systems, HIS 2007 2007
16th Brazilian Symposium on Artificial Intelligence, SBIA 2002 2002
1st Asia-Pacific Conference on Web Intelligence: Research and Development, WI 2001 2001
CE

Bagchi, S. (2015). Performance and quality assessment of sim- Intelligence and Lecture Notes in Bioinformatics), 7872
ilarity measures in collaborative filtering using mahout. LNCS, 1–12. doi:10.1007/978-3-642-38067-9 1
(Vol. 50, pp. 229–234). doi:10.1016/j.procs.2015.04.055 Bauer, J. & Nanopoulos, A. (2014). Recommender systems
Baldominos, A., Albacete, E., Saez, Y., & Isasi, P. (2015). based on quantitative implicit customer feedback. De-
AC

A scalable machine learning online service for big data cision Support Systems, 68, 77–88. doi:10.1016/j.dss.
real-time analysis. doi:10.1109/CIBD.2014.7011537 2014.09.005
Banerjee, S., Bhowmick, P., Mukherjee, D., & Misra, P. Bellogı́n, A., Cantador, I., Castells, P., & Ortigosa, Á. (2011).
(2012). Experiments on synopsis-based tv program rec- Discerning relevant model features in a content-based
ommendation. (pp. 258–263). collaborative recommender system. doi:10.1007/978-3-
Bar, A., Rokach, L., Shani, G., Shapira, B., & Schclar, A. 642-14125-6 20
(2013). Improving simple collaborative filtering models Bertin-Mahieux, T., Eck, D., & Mandel, M. (2010). Automatic
using ensemble methods. Lecture Notes in Computer tagging of audio: the state-of-the-art. doi:10.4018/978-
Science (including subseries Lecture Notes in Artificial 1-61520-919-4.ch014
 ACCEPTED MANUSCRIPT

Table 4.13: Sources of the primary studies of this systematic review

Source Name Number
Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 14
Expert Systems with Applications 7
ACM International Conference Proceeding Series 3
CEUR Workshop Proceedings 3
Decision Support Systems 3
2007 International Conference on Machine Learning and Cybernetics 2
ACM Transactions on Information Systems 2
Communications in Computer and Information Science 2
IEICE Transactions on Information and Systems 2
Journal of Machine Learning Research 2
RecSys’08: Proceedings of the 2008 ACM Conference on Recommender Systems 2
User Modeling and User-Adapted Interaction 2
2008 2nd ACM/IEEE International Conference on Distributed Smart Cameras, ICDSC 2008 1

T
2008 International Conference on Machine Learning and Cybernetics 1
2010 2nd International Conference on Computer Engineering and Applications, ICCEA 2010 1
2010 Ninth International Conference on Machine Learning and Applications (ICMLA) 1

IP
2011 International Conference on Machine Learning and Cybernetics (ICMLC) 1
2013 35th International Conference on Software Engineering (ICSE) 1
2014 17th International Conference on Computer and Information Technology, ICCIT 2014 1

CR
2014 IEEE 13th International Conference on Trust, Security and Privacy in Computing and Communications 1
2014 IEEE Fourth International Conference on Big Data and Cloud Computing (BdCloud) 1
2015 2nd Asia-Pacific World Congress on Computer Science and Engineering, APWC on CSE 2015 1
2015 IEEE 25th International Workshop on Machine Learning for Signal Processing (MLSP) 1
31st International Conference on Machine Learning, ICML 2014 1
4th International Multi-Conference on Computing in the Global Information Technology, ICCGI 2009 1

ACM Transactions on the Web

AI Communications
Artificial Intelligence
US
ACM Transactions on Multimedia Computing Communications and Applications

CCIS 2014 - Proceedings of 2014 IEEE 3rd International Conference on Cloud Computing and Intelligence Systems
1
1
1
1
1
AN
Chinese Journal of Electronics 1
Computational Linguistics and Intelligent Text Processing (CICLING 2015), PT II 1
Computers, Environment and Urban Systems 1
Electronic Markets 1
Future Generation Computer Systems 1
M

IEEE Access 1
IEEE AFRICON Conference 1
IEEE Transactions on Consumer Electronics 1
IEEE Transactions on Neural Networks 1
Information Sciences 1
ED

INFORSID 2016 - Actes du 8e Forum Jeunes Chercheurs du Congres 1

Intelligent Techniques in Recommendation Systems: Contextual Advancements and New Methods 1
International Conference on Computational Intelligence and Software Engineering, 2009. CiSE 2009. 1
International Conference on Information Technology: Coding and Computing (ITCC’05) - Volume II 1
International Journal of Human Computer Studies 1
PT

International Journal of Parallel Programming 1

International Journal on Artificial Intelligence Tools 1
Journal of Electronic Commerce Research 1
Journal of Intelligent Information Systems 1
KDIR 2012 - Proceedings of the International Conference on Knowledge Discovery and Information Retrieval 1
CE

Knowledge and Information Systems 1

Lecture Notes in Computer Science 1
LWA 2011 - Technical Report of the Symposium “Lernen, Wissen, Adaptivitat - Learning, Knowledge, and Adaptivity 2011” of the GI 1
Special Interest Groups KDML, IR and WM
Neurocomputing 1
AC

PACIS 2011 - 15th Pacific Asia Conference on Information Systems: Quality Research in Pacific 1
Personal and Ubiquitous Computing 1
Preference Learning 1
Proceedings - 13th IEEE International Conference on Commerce and Enterprise Computing, CEC 2011 1
Proceedings - 2014 IIAI 3rd International Conference on Advanced Applied Informatics, IIAI-AAI 2014 1
Proceedings - 2015 25th International Conference on Information, Communication and Automation Technologies, ICAT 2015 1
Proceedings - 2015 IEEE Symposium Series on Computational Intelligence, SSCI 2015 1
Proceedings - ICEIS 2015 - 17th International Conference on Enterprise Information Systems 1
Proceedings - IEEE International Conference on Data Mining, ICDM 1
Proceedings - IEEE SSCI 2014 - 2014 IEEE Symposium Series on Computational Intelligence - CIBD 2014: 2014 IEEE Symposium on 1
Computational Intelligence in Big Data
Proceedings - MIPRO 2011 - 34th International Convention on Information and Communication Technology, Electronics and Microelec- 1
tronics
Proceedings - Sixth International Conference on Mobile Data Management, MDM’05 1
 ACCEPTED MANUSCRIPT

Table 4.14: Sources of the primary studies of this systematic review (cont.)
Source Name Number
Proceedings - SocialCom 2010: 2nd IEEE International Conference on Social Computing 1
Proceedings of IEEE/ACS International Conference on Computer Systems and Applications, AICCSA 1
Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in 1
Information Retrieval
Proceedings of the 11th SIAM International Conference on Data Mining, SDM 2011 1
Proceedings of the 12th International Society for Music Information Retrieval Conference, ISMIR 2011 1
Proceedings of the 2013 10th International Joint Conference on Computer Science and Software Engineering, JCSSE 2013 1
Proceedings of the 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2013 1
Proceedings of the ACM Symposium on Applied Computing 1
Proceedings of the IEEE International Conference on Computer Vision 1
Proceedings of the IEEE International Conference on Requirements Engineering 1
Proceedings of the International Joint Conference on Neural Networks 1
Proceedings of the ISCA 27th International Conference on Computers and Their Applications, CATA 2012 1

T
RecSys 2014 - Proceedings of the 8th ACM Conference on Recommender Systems 1
RecSys 2015 - Proceedings of the 9th ACM Conference on Recommender Systems 1
RecSys’07: Proceedings of the 2007 ACM Conference on Recommender Systems 1

IP
RecSys’09 - Proceedings of the 3rd ACM Conference on Recommender Systems 1
RecSys’11 - Proceedings of the 5th ACM Conference on Recommender Systems 1
Scientific World Journal 1

CR
SIGIR Forum (ACM Special Interest Group on Information Retrieval) 1
Social Network Analysis and Mining 1
Studies in Computational Intelligence 1
WWW 2015 Companion - Proceedings of the 24th International Conference on World Wide Web 1

Biancalana, C., Gasparetti, F., Micarelli, A., Miola, A., &

Sansonetti, G. (2011). Context-aware movie recommen-
dation based on signal processing and machine learning.
US Retrieved from http : / / dl . acm . org / citation . cfm ? id =
2074094.2074100
Brouard, C. & Pomot, C. (2016). Specificsearch : un
AN
(pp. 5–10). doi:10.1145/2096112.2096114 outil de recommandation automatique pour la veille
Bjelica, M. (2010). Towards tv recommender system: ex- d’information sur le web. (pp. 101–116).
periments with user modeling. IEEE Transactions on Buabin, E. (2012). Hybrid neural architecture for intelligent
Consumer Electronics, 56(3), 1763–1769. doi:10.1109/ recommender system classification unit design. doi:10.
TCE.2010.5606323 4018/978-1-4666-2542-6.ch010
M

Bouneffouf, D., Bouzeghoub, A., & Gancarski, A. (2013). Buettner, R. (2016). Predicting user behavior in electronic
Risk-aware recommender systems. Lecture Notes in markets based on personality-mining in large online so-
Computer Science (including subseries Lecture Notes in cial networks: a personality-based product recommender
Artificial Intelligence and Lecture Notes in Bioinformat- framework. Electronic Markets, 1–19. doi:10 . 1007 /
ED

ics), 8226 LNCS(PART 1), 57–65. doi:10.1007/978-3- s12525-016-0228-z

642-42054-2 8 Burhans, D. & Kandefer, M. (2004). Dustbot: bringing a
Bouneffouf, D., Bouzeghoub, A., & Gançarski, A. (2012). vacuum-cleaner agent to life. (Vol. 1, pp. 82–84).
Exploration / exploitation trade-off in mobile context- Cai, X., Bain, M., Krzywicki, A., Wobcke, W., Kim, Y., Comp-
PT

aware recommender systems. Lecture Notes in Com- ton, P., & Mahidadia, A. (2010). Learning collaborative
puter Science (including subseries Lecture Notes in Arti- filtering and its application to people to people recom-
ficial Intelligence and Lecture Notes in Bioinformatics), mendation in social networks. (pp. 743–748). doi:10 .
CE

7691 LNAI, 591–601. doi:10.1007/978- 3- 642- 35101- 1109/ICDM.2010.159

3 50 Cai, X., Bain, M., Krzywicki, A., Wobcke, W., Kim, Y., Comp-
Bouza, A. & Bernstein, A. (2014). (partial) user preference ton, P., & Mahidadia, A. (2012). Reciprocal and het-
similarity as classification-based model similarity. Se- erogeneous link prediction in social networks. Lecture
AC

mantic Web, 5(1), 47–64. doi:10.3233/SW-130099 Notes in Computer Science (including subseries Lecture
Braida, F., Mello, C., Pasinato, M., & Zimbrão, G. (2015). Notes in Artificial Intelligence and Lecture Notes in
Transforming collaborative filtering into supervised Bioinformatics), 7301 LNAI(PART 2), 193–204. doi:10.
learning. Expert Systems with Applications, 42(10), 1007/978-3-642-30220-6 17
4733–4742. doi:10.1016/j.eswa.2015.01.023 Caraballo, A., Arruda, J., N.M., Nunes, B., Lopes, G., &
Breese, J. S., Heckerman, D., & Kadie, C. (1998). Empirical Casanova, M. (2014). Trtml - a tripleset recommen-
analysis of predictive algorithms for collaborative filter- dation tool based on supervised learning algorithms.
ing. In Proceedings of the fourteenth conference on un- Lecture Notes in Computer Science (including subseries
certainty in artificial intelligence (pp. 43–52). UAI’98. Lecture Notes in Artificial Intelligence and Lecture
Madison, Wisconsin: Morgan Kaufmann Publishers Inc. Notes in Bioinformatics), 8798, 413–417. doi:10.1007/
978-3-319-11955-7 58
 ACCEPTED MANUSCRIPT

Carbone, P. & Vlassov, V. (2015). Auto-scoring of person- Degemmis, M., Lops, P., & Semeraro, G. (2007). A content-
alised news in the real-time web: challenges, overview collaborative recommender that exploits wordnet-based
and evaluation of the state-of-the-art solutions. (pp. 169– user profiles for neighborhood formation. User Mod-
180). doi:10.1109/ICCAC.2015.9 eling and User-Adapted Interaction, 17(3), 217–255.
Castro-Herrera, C., Cleland-Huang, J., & Mobasher, B. (2009). doi:10.1007/s11257-006-9023-4
Enhancing stakeholder profiles to improve recommen- Deshpande, M. & Karypis, G. (2004). Item-based top-n rec-
dations in online requirements elicitation. (pp. 37–46). ommendation algorithms. ACM Transactions on Infor-
doi:10.1109/RE.2009.20 mation Systems, 22(1), 143–177. doi:10.1145/963770.
Celebi, M. & Aydin, K. (2016). Unsupervised learning algo- 963776
rithms. doi:10.1007/978-3-319-24211-8 Devi, M. & Venkatesh, P. (2013). Smoothing approach to
Chapelle, O., Schölkopf, B., & Zien, A. (2006). Semi- alleviate the meager rating problem in collaborative
supervised learning (1st). The MIT Press. recommender systems. Future Generation Computer

T
Chen, L.-S., Hsu, F.-H., Chen, M.-C., & Hsu, Y.-C. (2008). Systems, 29(1), 262–270. doi:10 . 1016 / j . future . 2011 .
Developing recommender systems with the consider- 05.011

IP
ation of product profitability for sellers. Information Diaby, M., Viennet, E., & Launay, T. (2013). Toward the
Sciences, 178(4), 1032–1048. doi:10.1016/j.ins.2007. next generation of recruitment tools: an online social
09.027 network-based job recommender system. (pp. 821–828).

CR
Chen, M., Mao, S., & Liu, Y. (2014). Big data: a survey. doi:10.1145/2492517.2500266
Mobile Networks and Applications, 19(2), 171–209. Diaby, M., Viennet, E., & Launay, T. (2014). Exploration of
doi:10.1007/s11036-013-0489-0 methodologies to improve job recommender systems on
Costa, H., Furtado, B., Pires, D., Macedo, L., & Cardoso, A. social networks. Social Network Analysis and Mining,
(2012). Context and intention-awareness in pois recom-
mender systems. (Vol. 889).
Costa, H., Furtado, B., Pires, D., Macedo, L., & Cardoso,
A. (2013). Recommending pois based on the user’s
US 4(1), 1–17. doi:10.1007/s13278-014-0227-z
Dinuzzo, F., Pillonetto, G., & De Nicolao, G. (2011). Client-
server multitask learning from distributed datasets. IEEE
Transactions on Neural Networks, 22(2), 290–303.
AN
context and intentions. Communications in Computer doi:10.1109/TNN.2010.2095882
and Information Science, 365, 166–177. doi:10 . 1007 / Elahi, M., Ricci, F., & Rubens, N. (2016). A survey of active
978-3-642-38061-7 17 learning in collaborative filtering recommender systems.
Cremonesi, P., Garzotto, F., Negro, S., Papadopoulos, A., & Computer Science Review, 20, 29–50. doi:10 . 1016 / j .
Turrin, R. (2011). Looking for ”good” recommenda- cosrev.2016.05.002
M

tions: a comparative evaluation of recommender sys- Elmongui, H., Mansour, R., Morsy, H., Khater, S., El-
tems. Lecture Notes in Computer Science (including Sharkasy, A., & Ibrahim, R. (2015). Trupi: twitter rec-
subseries Lecture Notes in Artificial Intelligence and ommendation based on users’ personal interests. Lecture
ED

Lecture Notes in Bioinformatics), 6948 LNCS(PART 3), Notes in Computer Science (including subseries Lecture
152–168. doi:10.1007/978-3-642-23765-2 11 Notes in Artificial Intelligence and Lecture Notes in
Cui, Q., Bai, F.-S., Gao, B., & Liu, T.-Y. (2015). Global Bioinformatics), 9042, 272–284. doi:10 . 1007 / 978 - 3 -
optimization for advertisement selection in sponsored 319-18117-2 20
search. Journal of Computer Science and Technology, Fan, T.-K. & Chang, C.-H. (2010). Learning to predict ad
PT

30(2), 295–310. doi:10.1007/s11390-015-1523-4 clicks based on boosted collaborative filtering. (pp. 209–
Das Dôres, S., Alves, L., Ruiz, D., & Barros, R. (2016). A 216). doi:10.1109/SocialCom.2010.37
meta-learning framework for algorithm recommendation Fan, Z., Chen, S., Zha, L., & Yang, J. (2016). A text clustering
CE

in software fault prediction. (Vol. 04-08-April-2016, approach of chinese news based on neural network
pp. 1486–1491). doi:10.1145/2851613.2851788 language model. International Journal of Parallel Pro-
Das, S., Moore, T., Wong, W.-K., Stumpf, S., Oberst, I., gramming, 44(1), 198–206. doi:10.1007/s10766- 014-
McIntosh, K., & Burnett, M. (2013). End-user feature 0329-2
AC

labeling: supervised and semi-supervised approaches Feuerverger, A., He, Y., & Khatri, S. (2012). Statistical signifi-
based on locally-weighted logistic regression. Artificial cance of the netflix challenge. Statistical Science, 27(2),
Intelligence, 204, 56–74. doi:10.1016/j.artint.2013.08. 202–231. doi:10.1214/11-STS368
003 Forsati, R. & Meybodi, M. (2010). Effective page recommen-
De Gemmis, M., Lops, P., Semeraro, G., & Basile, P. (2008). dation algorithms based on distributed learning automata
Integrating tags in a semantic content-based recom- and weighted association rules. Expert Systems with
mender. (pp. 163–170). doi:10.1145/1454008.1454036 Applications, 37(2), 1316–1330. doi:10 . 1016 / j . eswa .
Dean, J. & Ghemawat, S. (2008). Mapreduce: simplified data 2009.06.010
processing on large clusters. Communications of the Forsati, R., Rahbar, A., & Mahdavi, M. (2009). Effective
ACM, 51(1), 107–113. doi:10.1145/1327452.1327492 page recommendation algorithms based on distributed
 ACCEPTED MANUSCRIPT

learning automata. (pp. 41–46). doi:10 . 1109 / ICCGI . and Lecture Notes in Bioinformatics), 8643, 219–230.
2009.14 doi:10.1007/978-3-319-13186-3 21
Friedman, N., Geiger, D., & Goldszmidt, M. (1997). Bayesian Hussain, A., Farooq, K., Luo, B., & Slack, W. (2015). A
network classifiers. Machine Learning, 29(2-3), 131– novel ontology and machine learning inspired hybrid
163. cardiovascular decision support framework. (pp. 824–
Gedikli, F., Bağdat, F., Ge, M., & Jannach, D. (2011). Rf- 832). doi:10.1109/SSCI.2015.122
rec: fast and accurate computation of recommendations Isazadeh, A. (2004). Software engineering: the trend. Infor-
based on rating frequencies. (pp. 50–57). doi:10.1109/ matica (Ljubljana), 28(2), 129–137.
CEC.2011.24 Islam, M., Ding, C., & Chi, C.-H. (2015). Personalized
Gemmis, M., Iaquinta, L., Lops, P., Musto, C., Narducci, F., recommender system on whom to follow in twitter.
& Semeraro, G. (2011). Learning preference models in (pp. 326–333). doi:10.1109/BDCloud.2014.84
recommender systems. doi:10.1007/978-3-642-14125- Jain, A., Murty, M., & Flynn, P. (1999). Data clustering:

T
6 18 a review. ACM Computing Surveys, 31(3), 264–323.
Geng, X., Zhang, H., Bian, J., & Chua, T.-S. (2016). Learning doi:10.1145/331499.331504

IP
image and user features for recommendation in social Jannach, D., Lerche, L., Gedikli, F., & Bonnin, G. (2013).
networks. (Vol. 11-18-December-2015, pp. 4274–4282). What recommenders recommend - an analysis of ac-
doi:10.1109/ICCV.2015.486 curacy, popularity, and sales diversity effects. Lecture

CR
Ghazarian, S. & Nematbakhsh, M. (2015). Enhancing Notes in Computer Science (including subseries Lecture
memory-based collaborative filtering for group rec- Notes in Artificial Intelligence and Lecture Notes in
ommender systems. Expert Systems with Applications, Bioinformatics), 7899 LNCS, 25–37. doi:10.1007/978-
42(7), 3801–3812. doi:10.1016/j.eswa.2014.11.042 3-642-38844-6 3
Goldberg, D., Nichols, D., Oki, B., & Terry, D. (1992). Using

Communications of the ACM, 35(12), 61–70. doi:10 .

1145/138859.138867
US
collaborative filtering to weave an information tapestry.
Jannach, D., Zanker, M., Felfernig, A., & Friedrich, G. (2010).
Recommender systems: an introduction. doi:10 . 1017 /
CBO9780511763113
Japkowicz, N. & Shah, M. (2011). Evaluating learning algo-
AN
Gunawardana, A. & Shani, G. (2015). Evaluating recom- rithms: a classification perspective. Cambridge Univer-
mender systems. Springer US. doi:10.1007/978-1-4899- sity Press. doi:10.1017/CBO9780511921803
7637-6 8 Jin, X., Mobasher, B., & Zhou, Y. (2005). A web recom-
Haiduc, S., Bavota, G., Marcus, A., Oliveto, R., De Lucia, A., mendation system based on maximum entropy. (Vol. 1,
& Menzies, T. (2013). Automatic query reformulations pp. 213–218).
M

for text retrieval in software engineering. (pp. 842–851). Jun, S.-H. (2005). Web usage mining using support vector
doi:10.1109/ICSE.2013.6606630 machine. (Vol. 3512, pp. 349–356).
Halder, S., Seddiqui, M., & Lee, Y.-K. (2014). An enter- Jung, K.-Y. & Lee, J.-H. (2004). User preference mining
ED

tainment recommendation system using the dynamics through hybrid collaborative filtering and content-based
of user behavior over time. (pp. 41–46). doi:10.1109/ filtering in recommendation system. IEICE Transactions
ICCITechn.2014.7073094 on Information and Systems, E87-D(12), 2781–2790.
Hassan, M., Karim, A., Javed, F., & Arshad, N. (2010). Kao, C.-Y. & Fahn, C.-S. (2013). A multi-stage learning
Self-optimizing a clustering-based tag recommender for framework for intelligent system. Expert Systems with
PT

social bookmarking systems. (pp. 601–606). doi:10 . Applications, 40(9), 3378–3388. doi:10 . 1016 / j . eswa .
1109/ICMLA.2010.93 2012.12.046
Hernández del Olmo, F., Gaudioso, E., & Martin, E. (2009). Karahodza, B. & Donko, D. (2015). Feature enhanced time-
CE

The task of guiding in adaptive recommender systems. aware recommender system. doi:10 . 1109 / ICAT. 2015 .
Expert Systems with Applications, 36(2 PART 1), 1972– 7340527
1977. doi:10.1016/j.eswa.2007.12.070 Karimanzira, D., Otto, P., & Wernstedt, J. (2006). Application
Hofmann, T. (2003). Collaborative filtering via gaussian of machine learning methods to route planning and
AC

probabilistic latent semantic analysis. SIGIR Forum navigation for disabled people. (pp. 366–371).
(ACM Special Interest Group on Information Retrieval), Karypis, G. (2001). Evaluation of item-based top-n recommen-
(SPEC. ISS.), 259–266. dation algorithms. (pp. 247–254).
Hofmann, T. (2004). Latent semantic models for collaborative Kitchenham, B. (2004, July). Procedures for Performing
filtering. ACM Transactions on Information Systems, Systematic Reviews. Keele University, Department of
22(1), 89–115. doi:10.1145/963770.963774 Computer Science, and Empirical Software Engineering
Huang, T.-H. & Nikulin, V. (2014). Two algorithms under National ICT Australia Ltd.
stochastic gradient descent framework for recommender Kong, D., Zhang, M., & Ding, C. (2013). Minimal shrinkage
systems. Lecture Notes in Computer Science (includ- for noisy data recovery using schatten-p norm objec-
ing subseries Lecture Notes in Artificial Intelligence tive. Lecture Notes in Computer Science (including
subseries Lecture Notes in Artificial Intelligence and
 ACCEPTED MANUSCRIPT

Lecture Notes in Bioinformatics), 8189 LNAI(PART 2), learning approach. Decision Support Systems, 54(2),
177–193. doi:10.1007/978-3-642-40991-2 12 880–890. doi:10.1016/j.dss.2012.09.019
Kononenko, I. (2001). Machine learning for medical diagnosis: Li, X., Wang, M., & Liang, T.-P. (2014). A multi-theoretical
history, state of the art and perspective. Artificial Intelli- kernel-based approach to social network-based recom-
gence in Medicine, 23(1), 89–109. doi:10.1016/S0933- mendation. Decision Support Systems, 65(100), 95–104.
3657(01)00077-X doi:10.1016/j.dss.2014.05.006
Koren, Y. & Bell, R. (2015). Advances in collaborative filter- Liang, W., Lu, G., Ji, X., & Li, J. (2014). Difference factor’
ing. Springer US. doi:10.1007/978-1-4899-7637-6 3 knn collaborative filtering recommendation algorithm.
Kotsiantis, S. (2007). Supervised machine learning: a review of Lecture Notes in Computer Science (including subseries
classification techniques. Informatica (Ljubljana), 31(3), Lecture Notes in Artificial Intelligence and Lecture
249–268. Notes in Bioinformatics), 8933, 175–184.
Krohn-Grimberghe, A., Busche, A., Nanopoulos, A., & Liu, J., Dolan, P., & Pedersen, E. (2010). Personalized news

T
Schmidt-Thieme, L. (2011). Active learning for tech- recommendation based on click behavior. (pp. 31–40).
nology enhanced learning. (pp. 28–31). doi:10.1145/1719970.1719976

IP
Krzywicki, A., Wobcke, W., Kim, Y., Cai, X., Bain, M., Liu, Q., Xiong, Y., & Huang, W. (2014). Combining user-
Mahidadia, A., & Compton, P. (2015). Collaborative based and item-based models for collaborative filtering
filtering for people-to-people recommendation in online using stacked regression. Chinese Journal of Electron-

CR
dating: data analysis and user trial. International Journal ics, 23(4), 712–717.
of Human Computer Studies, 76, 50–66. doi:10.1016/j. Liu, R., Ding, L., & Xie, L. (2014). Catlinks - a category
ijhcs.2014.12.003 clustering algorithm based on multi-class regression.
Kulkarni, S. (2012). Machine learning algorithms for problem (pp. 323–326). doi:10.1109/CCIS.2014.7175752
solving in computational applications: intelligent tech-
niques. IGI Global. doi:10.4018/978-1-4666-1833-6
Lahlou, F., Benbrahimand, H., Mountassir, A., & Kassou, I.
(2013). Context extraction from reviews for context
US Liu, T., Fan, B., Hu, H., & Du, X.-Y. (2011). Lawyer infor-
mation integration and recommendation by multi-source
information validation. (Vol. 4, pp. 1909–1913). doi:10.
1109/ICMLC.2011.6016956
AN
aware recommendation using text classification tech- Longo, L., Barrett, S., & Dondio, P. (2009). Toward social
niques. doi:10.1109/AICCSA.2013.6616512 search from explicit to implicit collaboration to predict
Lahlou, F., Mountassir, A., Benbrahim, H., & Kassou, I. users’ interests. (pp. 693–696).
(2013). A text classification based method for context Lops, P., De Gemmis, M., Semeraro, G., Musto, C., Nar-
extraction from online reviews. doi:10.1109/SITA.2013. ducci, F., & Bux, M. (2009). A semantic content-based
M

6560804 recommender system integrating folksonomies for per-

Lee, T.-J. & Tseng, S.-S. (2012). Easy-to-explain feature syn- sonalized access. Studies in Computational Intelligence,
thesis approach for recommending entertainment video. 229, 27–47. doi:10.1007/978-3-642-02794-9 2
ED

Neurocomputing, 92, 61–68. doi:10 . 1016 / j . neucom . Lu, C., Stankovic, M., & Laublet, P. (2015). Leveraging
2011.09.034 semantic web technologies for more relevant e-tourism
Lee, W.-P. & Lu, C.-C. (2003). Customising wap-based in- behavioral retargeting. (pp. 1287–1292). doi:10 . 1145 /
formation services on mobile networks. Personal and 2740908.2742001
Ubiquitous Computing, 7(6), 321–330. doi:10 . 1007 / Lu, J., Hoi, S., Wang, J., & Zhao, P. (2013). Second order on-
PT

s00779-003-0247-6 line collaborative filtering. Journal of Machine Learning

Leopairote, W., Surarerks, A., & Prompoon, N. (2013). Evalu- Research, 29, 325–340.
ating software quality in use using user reviews mining. Luong, H., Huynh, T., Gauch, S., Do, L., & Hoang, K. (2012).
CE

(pp. 257–262). doi:10.1109/JCSSE.2013.6567355 Publication venue recommendation using author net-

Leskovec, J., Rajaraman, A., & Ullman, J. (2014). Mining work’s publication history. Lecture Notes in Computer
of massive datasets: second edition. doi:10 . 1017 / Science (including subseries Lecture Notes in Artificial
CBO9781139924801 Intelligence and Lecture Notes in Bioinformatics), 7198
AC

Li, J. & Zaı̈ane, O. (2004). Combining usage, content, and LNAI(PART 3), 426–435. doi:10 . 1007 / 978 - 3 - 642 -
structure data to improve web site recommendation. 28493-9 45
Lecture Notes in Computer Science (including subseries Luong, H., Huynh, T., Gauch, S., & Hoang, K. (2012).
Lecture Notes in Artificial Intelligence and Lecture Exploiting social networks for publication venue rec-
Notes in Bioinformatics), 3182, 305–315. ommendations. (pp. 239–245).
Li, Q.-C., Dong, Z.-H., & Li, T. (2008). Research of informa- Lv, H. & Tang, H. (2011). Machine learning methods and their
tion recommendation system based on reading behavior. application research. (pp. 108–110). doi:10.1109/IPTC.
(Vol. 3, pp. 1626–1631). doi:10 . 1109 / ICMLC . 2008 . 2011.34
4620666 Marović, M., Mihoković, M., Miks̆a, M., Pribil, S., & Tus,
Li, X. & Chen, H. (2013). Recommendation as link predic- A. (2011). Automatic movie ratings prediction using
tion in bipartite graphs: a graph kernel-based machine machine learning. (pp. 1640–1645).
 ACCEPTED MANUSCRIPT

Marques, C., Guilherme, I., Nakamura, R., & Papa, J. Pantraki, E. & Kotropoulos, C. (2015). Automatic image
(2011). New trends in musical genre classification using tagging and recommendation via parafac2. (Vol. 2015-
optimum-path forest. (pp. 699–704). November). doi:10.1109/MLSP.2015.7324363
Martens, H. (1959). Two notes on machine ”learning”. Infor- Paparrizos, I., Cambazoglu, B., & Gionis, A. (2011). Machine
mation and Control, 2(4), 364–379. learned job recommendation. (pp. 325–328). doi:10 .
McLaughlin, M. & Herlocker, J. (2004). A collaborative 1145/2043932.2043994
filtering algorithm and evaluation metric that accurately Patrick, E. & Fischer III, F. (1970). A generalized k-nearest
model the user experience. (pp. 329–336). neighbor rule. Information and Control, 16(2), 128–152.
Michalski, R., Carbonell, J., & Mitchell, T. (1985). Machine doi:10.1016/S0019-9958(70)90081-1
learning: an artificial intelligence approach. Artificial In- Pecli, A., Giovanini, B., Pacheco, C., Moreira, C., Ferreira, F.,
telligence, 25(2), 236–238. doi:10.1016/0004-3702(85) Tosta, F., . . . Goldschmidt, R. (2015). Dimensionality
90005-0 reduction for supervised learning in link prediction

T
Michalski, R. S., Carbonell, J. G., & Mitchell, T. M. (2013). problems. (Vol. 1, pp. 295–302).
Machine learning: an artificial intelligence approach. Pessiot, J.-F., Truong, T.-V., Usunier, N., Amini, M.-R., &

IP
Springer Science & Business Media. Gallinar, P. (2007). Learning to rank for collaborative
Middleton, S., Shadbolt, N., & De Roure, D. (2004). Onto- filtering. (Vol. AIDSS, pp. 145–151).
logical user profiling in recommender systems. ACM Pressman, R. S. (2015). Software engineering: a practitioner’s

CR
Transactions on Information Systems, 22(1), 54–88. approach (8th). McGrawHill Higher Education.
doi:10.1145/963770.963773 Pronoza, E., Yagunova, E., & Volskaya, S. (2016). Aspect-
Montañés, E., Quevedo, J., Dı́az, I., & Ranilla, J. (2009). Col- based restaurant information extraction for the recom-
laborative tag recommendation system based on logistic mendation system. Lecture Notes in Computer Science
regression. (Vol. 497, pp. 173–188).
Moreno, O., Shapira, B., Rokach, L., & Shani, G. (2012). Tal-
mud: transfer learning for multiple domains. (pp. 425–
434). doi:10.1145/2396761.2396817
US (including subseries Lecture Notes in Artificial Intel-
ligence and Lecture Notes in Bioinformatics), 9561,
371–385. doi:10.1007/978-3-319-43808-5 28
Rodriguez, M., Posse, C., & Zhang, E. (2012). Multiple objec-
AN
Murfi, H. & Obermayer, K. (2009). A two-level learning tive optimization in recommender systems. (pp. 11–18).
hierarchy of concept based keyword extraction for tag doi:10.1145/2365952.2365961
recommendations. (Vol. 497, pp. 201–214). Roh, T., Oh, K., & Han, I. (2003). The collaborative filtering
Musto, C., Narducci, F., Lops, P., De Gemmis, M., & Semer- recommendation based on som cluster-indexing cbr. Ex-
aro, G. (2010). Integrating a content-based recommender pert Systems with Applications, 25(3), 413–423. doi:10.
M

system into digital libraries for cultural heritage. Com- 1016/S0957-4174(03)00067-8

munications in Computer and Information Science, 91 Rubens, N., Elahi, M., Sugiyama, M., & Kaplan, D. (2015).
CCIS, 27–38. doi:10.1007/978-3-642-15850-6 4 Active learning in recommender systems. Springer US.
ED

Nguyen, H., Richards, R., Chan, C.-C., & Liszka, K. (2016). doi:10.1007/978-1-4899-7637-6 24
Redtweet: recommendation engine for reddit. Journal of Sarwar, B., Karypis, G., Konstan, J., & Riedl, J. (2000). Anal-
Intelligent Information Systems, 47(2), 247–265. doi:10. ysis of recommendation algorithms for e-commerce. In
1007/s10844-016-0410-y Proceedings of the 2nd acm conference on electronic
Nicol, O., Mary, J., & Preux, P. (2014). Improving offline eval- commerce (pp. 158–167). EC ’00. Minneapolis, Min-
PT

uation of contextual bandit algorithms via bootstrapping nesota, USA: ACM. doi:10.1145/352871.352887
techniques. (Vol. 2, pp. 1390–1398). Shalev-Shwartz, S. & Ben-David, S. (2013). Understanding
Nie, F., Wang, H., Huang, H., & Ding, C. (2013). Joint machine learning: from theory to algorithms. Cambridge
CE

schatten lp-norm robust matrix completion for missing University Press. doi:10.1017/CBO9781107298019
value recovery. Knowledge and Information Systems, Shani, G. & Gunawardana, A. (2013). Tutorial on application-
42(3), 525–544. doi:10.1007/s10115-013-0713-z oriented evaluation of recommendation systems. AI
Ning, X., Desrosiers, C., & Karypis, G. (2015). A compre- Communications, 26(2), 225–236. doi:10 . 3233 / AIC -
AC

hensive survey of neighborhood-based recommendation 130551

methods. Springer US. doi:10.1007/978-1-4899-7637- Shvachko, K., Kuang, H., Radia, S., & Chansler, R. (2010).
6 2 The hadoop distributed file system. doi:10.1109/MSST.
O’Donovan, J. & Smyth, B. (2005). Trust in recommender 2010.5496972
systems. (pp. 167–174). doi:10.1145/1040830.1040870 Song, I., Dillon, D., Goh, T., & Sung, M. (2011). A health
Oyama, S., Hayashi, K., & Kashima, H. (2012). Link predic- social network recommender system. Lecture Notes in
tion across time via cross-temporal locality preserving Computer Science (including subseries Lecture Notes in
projections. IEICE Transactions on Information and Artificial Intelligence and Lecture Notes in Bioinformat-
Systems, E95-D(11), 2664–2673. doi:10.1587/transinf. ics), 7047 LNAI, 361–372. doi:10 . 1007 / 978 - 3 - 642 -
E95.D.2664 25044-6 29
 ACCEPTED MANUSCRIPT

Song, Y., Zhang, L., & Giles, C. (2011). Automatic tag recom- on academic performance. User Modeling and User-
mendation algorithms for social recommender systems. Adapted Interaction, 21(1-2), 217–248. doi:10 . 1007 /
ACM Transactions on the Web, 5(1). doi:10 . 1145 / s11257-011-9098-4
1921591.1921595 Wan, X., Jamaliding, Q., & Okamoto, T. (2009). Discov-
Steck, H. (2013). Evaluation of recommendations: rating- ering social network to improve recommender system
prediction and ranking. (pp. 213–220). doi:10 . 1145 / for group learning support. doi:10 . 1109 / CISE . 2009 .
2507157.2507160 5363890
Sun, Y., Fan, H., Bakillah, M., & Zipf, A. (2015). Road-based Wang, Y., Wang, X., Wang, Y., & Hsu, D. (2014). Exploration
travel recommendation using geo-tagged images. Com- in interactive personalized music recommendation: a
puters, Environment and Urban Systems, 53, 110–122. reinforcement learning approach. ACM Transactions on
doi:10.1016/j.compenvurbsys.2013.07.006 Multimedia Computing, Communications and Applica-
Sutton, R. S. & Barto, A. G. (1998). Reinforcement learning: tions, 11(1). doi:10.1145/2623372

T
an introduction (1st). Cambridge: MIT Press. Wang, Y., Yin, L., Cheng, B., & Yu, Y. (2012). Learning
Szabó, Z., Póczos, B., & Lorincz, A. (2012). Collaborative fil- to recommend based on slope one strategy. Lecture

IP
tering via group-structured dictionary learning. Lecture Notes in Computer Science (including subseries Lecture
Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in
Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 7235 LNCS, 537–544. doi:10 . 1007 /

CR
Bioinformatics), 7191 LNCS, 247–254. doi:10 . 1007 / 978-3-642-29253-8 47
978-3-642-28551-6 31 Wei, C.-P., Chen, H.-C., & Liang, M.-K. (2011). Estimating
Szymański, J. & Rzeniewicz, J. (2016). Identification of cat- trust strength for supporting effective recommendation
egory associations using a multilabel classifier. Expert services.
Systems with Applications, 61, 327–342. doi:10.1016/j.
eswa.2016.05.039
Taghipour, N., Kardan, A., & Ghidary, S. (2007). Usage-
based web recommendations: a reinforcement learning
US Weng, L.-T. (1998, November). Information enrichment for
quality recommender systems (Doctoral dissertation,
Queensland University of Technology, Brisbane, Aus-
tralia).
AN
approach. (pp. 113–120). doi:10.1145/1297231.1297250 Xin, J., Cui, Z., Zhang, S., He, T., Li, C., & Huang, H.
Takács, G., Pilászy, I., Németh, B., & Tikk, D. (2008). (2014). Constructing topic models of internet of things
Matrix factorization and neighbor based algorithms for for information processing. Scientific World Journal,
the netflix prize problem. (pp. 267–274). doi:10.1145/ 2014. doi:10.1155/2014/675234
1454008.1454049 Xu, Z., Mo, M., & King, I. (2012). Semi-supervised learning.
M

Takáes, G., Pilászy, I., Németh, B., & Tikk, D. (2009). Scalable Xuan, J., Lu, J., Zhang, G., & Luo, X. (2014). Extension of
collaborative filtering approaches for large reeommender similarity measures in vsm: from orthogonal coordinate
systems. Journal of Machine Learning Research, 10, system to affine coordinate system. (pp. 4084–4091).
ED

623–656. doi:10.1109/IJCNN.2014.6889693
Thrun, S. (2007). Self-driving cars -an ai-robotics challenge. Yan, G., Xu, F., Yao, Y., & Lu, J. (2013). Enhancing trustwor-
(p. 12). thiness evaluation in internetware with similarity and
Torralba, A., Fergus, R., & Weiss, Y. (2008). Small codes non-negative constraints. doi:10.1145/2532443.2532459
and large image databases for recognition. doi:10.1109/ Yap, G.-E., Tan, A.-H., & Pang, H.-H. (2005). Dynamically-
PT

CVPR.2008.4587633 optimized context in recommender systems. (pp. 265–

Tsapatsoulis, N., Agathokleous, M., Djouvas, C., & Mendez, 272). doi:10.1145/1071246.1071289
F. (2015). On the design of social voting rec- Yeh, J.-H. & Wu, M.-L. (2010). Recommendation based
CE

ommendation applications. International Journal on on latent topics and social network analysis. (Vol. 1,
Artificial Intelligence Tools, 24(3). doi:10 . 1142 / pp. 209–213). doi:10.1109/ICCEA.2010.48
s0218213015500098 Yuan, G., Murukannaiah, P., Zhang, Z., & Singh, M. (2014).
Tsuji, K., Yoshikane, F., Sato, S., & Itsumura, H. (2014). Book Exploiting sentiment homophily for link prediction.
AC

recommendation using machine learning methods based (pp. 17–24). doi:10.1145/2645710.2645734

on library loan records and bibliographic information. Zahra, S., Ghazanfar, M., Khalid, A., Azam, M., Naeem, U.,
(pp. 76–79). doi:10.1109/IIAI-AAI.2014.26 & Prugel-Bennett, A. (2015). Novel centroid selection
Verma, C., Hart, M., Bhatkar, S., Parker-Wood, A., & Dey, S. approaches for kmeans-clustering based recommender
(2016). Improving scalability of personalized recom- systems. Information Sciences, 320, 156–189. doi:10 .
mendation systems for enterprise knowledge workers. 1016/j.ins.2015.03.062
IEEE Access, 4, 204–215. doi:10.1109/ACCESS.2015. Zhai, H. & Li, J. (2015). Refine social relations and differenti-
2513000 ate the same friends’ influence in recommender system.
Vialardi, C., Chue, J., Peche, J., Alvarado, G., Vinatea, B., Es- Lecture Notes in Computer Science (including subseries
trella, J., & Ortigosa, Á. (2011). A data mining approach Lecture Notes in Artificial Intelligence and Lecture
to guide students through the enrollment process based
 ACCEPTED MANUSCRIPT

Notes in Bioinformatics), 9468, 504–514. doi:10.1007/ Zhang, X.-Z. (2007). Building personalized recommendation
978-3-319-26832-3 47 system in e-commerce using association rule-based min-
Zhang, D. & Tsai, J. (2006). Advances in machine learning ing and classification. (Vol. 7, pp. 4113–4118). doi:10.
applications in software engineering. doi:10.4018/978- 1109/ICMLC.2007.4370866
1-59140-941-1 Zhang, Y., Zhuang, Y., Wu, J., & Zhang, L. (2009). Applying
Zhang, R. & Tran, T. (2010). Helpful or Unhelpful: A Linear probabilistic latent semantic analysis to multi-criteria
Approach for Ranking Product Reviews. JOURNAL OF recommender system. AI Communications, 22(2), 97–
ELECTRONIC COMMERCE RESEARCH, 11(3), 220– 107. doi:10.3233/AIC-2009-0446
230. Zhao, K. & Pan, L. (2015). A machine learning based
Zhang, S.-Z., Liu, L., & Dong, Y.-Z. (2007). An online trust evaluation framework for online social networks.
bayesian networks model for e-commercial personal- (pp. 69–74). doi:10.1109/TrustCom.2014.13
ized recommendation system. (Vol. 6, pp. 3506–3511). Zhao, Q., Zhang, Y., Friedman, D., & Tan, F. (2015). E-

T
doi:10.1109/ICMLC.2007.4370754 commerce recommendation with personalized promo-
Zhang, W., Begole, B., Chu, M., Liu, J., & Yee, N. (2008). tion. (pp. 219–225). doi:10.1145/2792838.2800178

IP
Real-time clothes comparison based on multi-view vi-
sion. doi:10.1109/ICDSC.2008.4635727

CR
US
AN
M
ED
PT
CE
AC
 ACCEPTED MANUSCRIPT

A PPENDIX
Table A.1: The list of the studies inspected in this systematic review
Reference Title
(Agarwal, 2011) The infinite push: A new support vector ranking algorithm that directly optimizes accuracy at the absolute top of the
list
(Alemeye & Getahun, 2015) Cloud readiness assessment framework and recommendation system
(Anaissi & Goyal, 2015) SVM-based association rules for knowledge discovery and classification
(Aouay et al., 2014) Feature based link prediction
(Baldominos et al., 2015) A scalable machine learning online service for big data real-time analysis
(Banerjee et al., 2012) Experiments on synopsis-based TV program recommendation
(Bar et al., 2013) Improving simple collaborative filtering models using ensemble methods
(Bauer & Nanopoulos, 2014) Recommender systems based on quantitative implicit customer feedback
(Bellogı́n et al., 2011) Discerning relevant model features in a content-based collaborative recommender system
(Biancalana et al., 2011) Context-aware movie recommendation based on signal processing and machine learning

T
(Bjelica, 2010) Towards TV recommender system: Experiments with user modeling
(Bouneffouf et al., 2012) Exploration / exploitation trade-off in mobile context-aware recommender systems
(Braida et al., 2015) Transforming collaborative filtering into supervised learning

IP
(Brouard & Pomot, 2016) SpecificSearch : Un outil de recommandation automatique pour la veille d’information sur le web
(Buabin, 2012) Hybrid neural architecture for intelligent recommender system classification unit design
(Buettner, 2016) Predicting user behavior in electronic markets based on personality-mining in large online social networks: A

CR
personality-based product recommender framework
(Cai et al., 2010) Learning collaborative filtering and its application to people to people recommendation in social networks
(Cai et al., 2012) Reciprocal and heterogeneous link prediction in social networks
(Caraballo et al., 2014) TRTML - A tripleset recommendation tool based on supervised learning algorithms
(Castro-Herrera et al., 2009) Enhancing stakeholder profiles to improve recommendations in online requirements elicitation
(Costa et al., 2012) Context and intention-awareness in POIs recommender systems
(Costa et al., 2013)
(Das Dôres et al., 2016)
(Das et al., 2013)
(De Gemmis et al., 2008)
(Degemmis et al., 2007)
US
Recommending POIs based on the user’s context and intentions
A meta-learning framework for algorithm recommendation in software fault prediction
End-user feature labeling: Supervised and semi-supervised approaches based on locally-weighted logistic regression
Integrating tags in a semantic content-based recommender
A content-collaborative recommender that exploits WordNet-based user profiles for neighborhood formation
AN
(Devi & Venkatesh, 2013) Smoothing approach to alleviate the meager rating problem in collaborative recommender systems
(Diaby et al., 2014) Exploration of methodologies to improve job recommender systems on social networks
(Diaby et al., 2013) Toward the next generation of recruitment tools: An online social network-based job recommender system
(Dinuzzo et al., 2011) Client-Server multitask learning from distributed datasets
(Elmongui et al., 2015) TRUPI: Twitter recommendation based on users’ personal interests
(T.-K. Fan & Chang, 2010) Learning to predict ad clicks based on boosted collaborative filtering
M

(Z. Fan et al., 2016) A Text Clustering Approach of Chinese News Based on Neural Network Language Model
(Forsati & Meybodi, 2010) Effective page recommendation algorithms based on distributed learning automata and weighted association rules
(Forsati et al., 2009) Effective page recommendation algorithms based on distributed learning automata
(Gedikli et al., 2011) RF-REC: Fast and accurate computation of recommendations based on rating frequencies
(Geng et al., 2016) Learning image and user features for recommendation in social networks
ED

(Ghazarian & Nematbakhsh, Enhancing memory-based collaborative filtering for group recommender systems
2015)
(Haiduc et al., 2013) Automatic query reformulations for text retrieval in software engineering
(Halder et al., 2014) An entertainment recommendation system using the dynamics of user behavior over time
(Hassan et al., 2010) Self-optimizing a clustering-based tag recommender for social bookmarking systems
(Hofmann, 2003) Collaborative Filtering via Gaussian Probabilistic Latent Semantic Analysis
PT

(Hofmann, 2004) Latent semantic models for collaborative filtering

(Huang & Nikulin, 2014) Two algorithms under stochastic gradient descent framework for recommender systems
(Hussain et al., 2015) A novel ontology and machine learning inspired hybrid cardiovascular decision support framework
(Islam et al., 2015) Personalized recommender system on whom to follow in twitter
(Jin et al., 2005) A Web recommendation system based on maximum entropy
CE

(Jun, 2005) Web usage mining using support vector machine

(Jung & Lee, 2004) User preference mining through hybrid collaborative filtering and content-based filtering in recommendation system
(Kao & Fahn, 2013) A multi-stage learning framework for intelligent system
(Karahodza & Donko, 2015) Feature enhanced time-aware recommender system
(Kong et al., 2013) Minimal shrinkage for noisy data recovery using Schatten-p norm objective
AC

(Krohn-Grimberghe et al., 2011) Active learning for technology enhanced learning

(Krzywicki et al., 2015) Collaborative filtering for people-to-people recommendation in online dating: Data analysis and user trial
(W.-P. Lee & Lu, 2003) Customising WAP-based information services on mobile networks
(T.-J. Lee & Tseng, 2012) Easy-to-explain feature synthesis approach for recommending entertainment video
(Leopairote et al., 2013) Evaluating software quality in use using user reviews mining
(X. Li & Chen, 2013) Recommendation as link prediction in bipartite graphs: A graph kernel-based machine learning approach
(J. Li & Zaı̈ane, 2004) Combining usage, content, and structure data to improve web site recommendation
(Q.-C. Li et al., 2008) Research of information recommendation system based on reading behavior
(X. Li et al., 2014) A multi-theoretical kernel-based approach to social network-based recommendation
(Liang et al., 2014) Difference factor’ KNN collaborative filtering recommendation algorithm
(T. Liu et al., 2011) Lawyer information integration and recommendation by multi-source information validation
(R. Liu et al., 2014) Catlinks - A category clustering algorithm based on multi-class regression
(Q. Liu et al., 2014) Combining user-based and item-based models for collaborative filtering using stacked regression
 ACCEPTED MANUSCRIPT

Table A.2: The list of the studies inspected in this systematic review (cont.)
Reference Title
(Lops et al., 2009) A semantic content-based recommender system integrating folksonomies for personalized access
(J. Lu et al., 2013) Second order online collaborative filtering
(C. Lu et al., 2015) Leveraging semantic web technologies for more relevant e-tourism behavioral retargeting
(Luong, Huynh, Gauch, & Exploiting social networks for publication venue recommendations
Hoang, 2012)
(Luong, Huynh, Gauch, Do, & Publication venue recommendation using author network’s publication history
Hoang, 2012)

T
(Marović et al., 2011) Automatic movie ratings prediction using machine learning
(Marques et al., 2011) New trends in musical genre classification using optimum-path forest
(McLaughlin & Herlocker, 2004) A collaborative filtering algorithm and evaluation metric that accurately model the user experience

IP
(Middleton et al., 2004) Ontological user profiling in recommender systems
(Montañés et al., 2009) Collaborative tag recommendation system based on logistic regression
(Moreno et al., 2012) TALMUD: Transfer learning for multiple domains
(Murfi & Obermayer, 2009) A two-level learning hierarchy of concept based keyword extraction for tag recommendations

CR
(Musto et al., 2010) Integrating a content-based recommender system into digital libraries for cultural heritage
(Musto et al., 2010) Integrating a content-based recommender system into digital libraries for cultural heritage
(Nguyen et al., 2016) RedTweet: recommendation engine for reddit
(Nicol et al., 2014) Improving offline evaluation of contextual bandit algorithms via bootstrapping techniques
(Nie et al., 2013) Joint Schatten lp-norm robust matrix completion for missing value recovery
(Hernández del Olmo et al.,
2009)
(Oyama et al., 2012)
(Pantraki & Kotropoulos, 2015)
(Paparrizos et al., 2011)
(Pecli et al., 2015)
US
The task of guiding in adaptive recommender systems

Link prediction across time via cross-temporal locality preserving projections

Automatic image tagging and recommendation via PARAFAC2
Machine learned job recommendation
Dimensionality reduction for supervised learning in link prediction problems
AN
(Pessiot et al., 2007) Learning to rank for collaborative filtering
(Pronoza et al., 2016) Aspect-based restaurant information extraction for the recommendation system
(Roh et al., 2003) The collaborative filtering recommendation based on SOM cluster-indexing CBR
(I. Song et al., 2011) A health social network recommender system
(Y. Song et al., 2011) Automatic tag recommendation algorithms for social recommender systems
(Sun et al., 2015) Road-based travel recommendation using geo-tagged images
M

(Szabó et al., 2012) Collaborative filtering via group-structured dictionary learning

(Szymański & Rzeniewicz, 2016) Identification of category associations using a multilabel classifier
(Taghipour et al., 2007) Usage-based web recommendations: A reinforcement learning approach
(Takács et al., 2008) Matrix factorization and neighbor based algorithms for the netflix prize problem
ED

(Takáes et al., 2009) Scalable collaborative filtering approaches for large reeommender systems
(Tsapatsoulis et al., 2015) On the design of social voting recommendation applications
(Tsuji et al., 2014) Book recommendation using machine learning methods based on library loan records and bibliographic information
(Verma et al., 2016) Improving Scalability of Personalized Recommendation Systems for Enterprise Knowledge Workers
(Vialardi et al., 2011) A data mining approach to guide students through the enrollment process based on academic performance
(Wan et al., 2009) Discovering social network to improve recommender system for group learning support
PT

(Wang et al., 2014) Exploration in interactive personalized music recommendation: A reinforcement learning approach
(Wang et al., 2012) Learning to recommend based on slope one strategy
(Wei et al., 2011) Estimating trust strength for supporting effective recommendation services
(Xin et al., 2014) Constructing topic models of internet of things for information processing
(Xuan et al., 2014) Extension of similarity measures in VSM: From orthogonal coordinate system to affine coordinate system
CE

(Yan et al., 2013) Enhancing trustworthiness evaluation in internetware with similarity and non-negative constraints
(Yap et al., 2005) Dynamically-optimized context in recommender systems
(Yeh & Wu, 2010) Recommendation based on latent topics and social network analysis
(Yuan et al., 2014) Exploiting sentiment homophily for link prediction
(Zahra et al., 2015) Novel centroid selection approaches for KMeans-clustering based recommender systems
AC

(Zhai & Li, 2015) Refine social relations and differentiate the same friends’ influence in recommender system
(X.-Z. Zhang, 2007) Building personalized recommendation system in E-Commerce using association rule-based mining and classification
(R. Zhang & Tran, 2010) Helpful or Unhelpful: A Linear Approach for Ranking Product Reviews
(Y. Zhang et al., 2009) Applying probabilistic latent semantic analysis to multi-criteria recommender system
(W. Zhang et al., 2008) Real-time clothes comparison based on multi-view vision
(S.-Z. Zhang et al., 2007) An online Bayesian networks model for E-commercial personalized recommendation system
(K. Zhao & Pan, 2015) A machine learning based trust evaluation framework for online social networks
(Q. Zhao et al., 2015) E-commerce recommendation with personalized promotion