This article has been accepted for publication in a future issue of this journal, but has not been

fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2889635, IEEE Access

Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.
Digital Object Identifier 10.1109/ACCESS.2017.Doi Number

Ontology-based Personalised Course

Recommendation Framework
MOHAMMED E. IBRAHIM 1,2(Member, IEEE), YANYAN YANG1,2, DAVID NDZI3,
GUANGGUANG YANG1, MURTADHA ALMALIKI1
1
School of Engineering, University of Portsmouth, Portsmouth PO1 3DJ, U.K
2
School of Computing, University of Portsmouth, Portsmouth PO1 3HE, U.K
3
School of Engineering and Computing, University of the West of Scotland,
Paisley, Scotland, U.K

Corresponding author: Mohammed E. Ibrahim (e-mail: [email protected]).

This work was supported in part by the Higher Committee Education Development in Iraq through the Scholarship Program of the Iraqi
Government, and in part by the University of Portsmouth.

ABSTRACT Choosing a higher education course at university is not an easy task for students. A wide range
of courses is offered by individual universities whose delivery mode and entry requirements all differ. A
personalised recommendation system can be an effective way of suggesting relevant courses to prospective
students. This paper introduces a novel approach that personalises course recommendations that will match
the individual needs of users. The proposed approach developed a framework of an ontology-based hybrid-
filtering system called OPCR. This approach aims to integrate information from multiple sources based on
hierarchical ontology similarity with a view to enhancing efficiency and user satisfaction and to provide
students with appropriate recommendations. OPCR combines collaborative based filtering with content-
based filtering. It also considers familiar related concepts that are evident in the profiles of both the student
and the course, determining the similarity between them. Furthermore, OPCR uses an ontology mapping
technique, recommending jobs that will be available following completion of each course. This method can
enable students to gain a comprehensive knowledge of courses based on their relevance, using dynamic
ontology mapping to link course profiles and student profiles with job profiles. Results show that a filtering
algorithm that uses hierarchically related concepts produces better outcomes compared to a filtering method
that considers only keyword similarity. In addition, the quality of the recommendations improved when the
ontology similarity between the items’ profiles and the users’ profiles were utilised. This approach, using a
dynamic ontology mapping, is flexible and can be adapted to different domains. The proposed framework
can be used to filter items for both postgraduate courses and items from other domains.

INDEX TERMS Information Overload, Recommendation Systems, Course Recommender system,

Ontology, Education Domain

I. INTRODUCTION choosing the one that is most appropriate for the student [2].
Finding information regarding higher education from a Furthermore, even though some course titles are similar, they
large number of websites is a challenging and time- can lead to a different career path [3]. Studies have shown
consuming process. Helping students to make the correct that, naturslly, the students’ choices are influenced by their
choice from a myriad of available courses in order to meet background, personal interests and career interests [4].
their individual needs is a real challenge [1]. We have used Researchers Gordon and Cuseo found that three out of every
the term “course” in this paper to refer to any program of four students were uncertain or tentative about their career
study such as undergraduate, postgraduate and so forth. Such choice at the time of college entry [5].
abundant information means that students need to search, The process of choosing a course can be incredibly
organise and use the resources that can enable them to match tedious and extremely complicated. Nowadays, students can
their individual goals, interests and current level of rapidly find information relating to universities and the
knowledge. This can be a time-consuming process as it courses offered by them using online resources [6].
involves accessing each platform, searching for available However, simply because more course information is now
courses, carefully reading every course syllabus and then provided on university websites, this does not automatically

1
2169-3536 (c) 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2889635, IEEE Access

mean that students possess the cognitive ability to evaluate ontology, the user will be able to gain precise knowledge
them all [1]. Instead, they are confronted with a problem that about the course [22]. We have been able to build a
is termed “information overloading” [7]. relationship between the relevant information available
Artificial intelligence methods developed at the beginning through the internet, including the course modules, job
of research are now being applied to information retrieval opportunities and the users’ interests. Ontology provides a
systems. Recommended systems provide a promising vocabulary of classes and properties that can be used to both
approach to information filtering [8] as they help users to describe a domain and emphasise knowledge sharing [29].
find the most appropriate items [9]. Based on the needs of The use of semantic descriptions of the courses and the
each user recommendation system, a series of specific students’ profiles allows there to be both qualitative and
suggestions will be generated [10]. Recommendation quantitative reasoning regarding the matching, as well as the
systems are widely classified into three main techniques in required information about the courses and the student’s
the literature: collaborative-based filtering (CF) [11], [12]– interests which is necessary in order to refine the process of
[15], content-based filtering (CBF) [16]and hybrid filtering deciding which course to select.
[17]–[21]. A novel hybrid filtering is proposed in this study, based on
There are many online systems currently available that can both the CBF and CF methods and using ontology as a way
be used to find and search for courses [22], which use tools by which to overcome the problem of information
based on the users’ prior knowledge of the courses [19], overloading which has been a key challenge when
keyword-based queries [23], [24] collaborative filtering consideration is given to building an effective
based [25] [26], data mining and association rules based [19], recommendation system. This problem is related to the
[27] and content-based filtering models [28]. Despite the sparsity of information that is available (i.e. for users and
high impact of the course recommendation system and how items) in the recommendation filtering algorithms [30]. The
useful it is, there are certain significant limitations, such as: proposed approach uses ontology for data extraction and
 Models based mainly on the keywords integration from multiple data sources. Data integration that
failed to address the individual user’s needs in the is based on ontology is used in the ontology-based metadata.
recommendation process. It utilises a combination of model-based and memory-based
 Although models use collaborative use of ontology in CF to provide a high-quality
filtering, and data mining such as association rule recommendation.
and decision tree, there is often a lack of historical User profiling that is based on ontology, item ontology,
information that makes it challenging to adopt this the semantic similarity between two ontologies and the
approach. For instance, new students who wish to proposed OKNN algorithm is used in the CF to overcome the
use the systems do not have sufficient information new user problem. On the other hand, item-based ontology
about the model and therefore cannot generate any and semantic similarity are both applied in CBF to overcome
recommendations. the new item cold start problem. In order to ensure the
 The shortcoming of models that use measurement of semantic similarity is more accurate, a
content-based filtering is that current approaches heuristic method is used in the CBF. This measures the “IS-
are based only on a specific subject A” degree between the two nodes of item ontology, which
recommendation rather than an entire university was found to yield a more precise recommendation list for
course. Moreover, the similarity calculation in these the target user.
models is based on the weighted average of features This paper is structured as follows. In Section 2, we
and does not take into account user interaction with discuss related work that is relevant to this study. Section 3
the system, such as the rating value of presents the proposed methodology with all of the process
recommendation items. functions. Section 4 presents the implementation and
 Another shortcoming of the current evolution of the methodology, Section 5 describes the
models is that they do not provide comprehensive discussion results and finally, Section 6 includes the
knowledge about the course that is most relevant to conclusion and recommendations for future work.
the student. For example, students need to know
II. RELATED WORK
what future career the course will lead to and
A recommender system is a tool that provides
require information about this aspect, as well as the
personalised recommendations for those items that are most
quality of the facilities of the educational institution
likely to be relevant and interesting to a user in order to help
itself that will be providing the course.
him/her to find the most useful items [13], [31].
Through categorising the needs of students and their areas Recommended items can be any products, services, books,
of interest, it is possible to recommend an appropriate course. news or information in a given application domain.
It is possible to help students to select a course by developing Recommender systems have been applied in different
methods that will both integrate the data from multiple domains, including the traditional e-commerce domain and,
heterogeneous data sources and allow this to rapidly set remarkably, in emerging domains such as education and
valuable course-related information [6]. By using this engineering [13], [32], [33].

2169-3536 (c) 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2889635, IEEE Access

Recommendation systems have more recently offered In this respect, Artificial Intelligence techniques could
personalised and more relevant recommendations. The develop and improve the decision making and reasoning
personalised approach is achieved by using information that process of humans to minimise the amount of uncertainty
is found in certain situations, such as studying various there is in active learning to ensure a lifelong learning
objects, the context and areas of interest, location and careers mechanism [21]. The challenge for recommender systems,
[34]–[36]. For instance, courses that are recommended to a therefore, is to better understand the student’s interest and
student who wishes to work in IT and who searches for the purpose of the domain [1]. An association mining based
“business information” will differ from those that are recommender has been developed for recommending tasks
recommended to a student who aims to become an academic that are related to learning, and are most suitable for learners
member of staff in the same area, since their requirements based on the performance of the targeted student and other
and level of education will be different. It is treated as students who are similar to them [27]. A course
contextual data that is a significant source of the accuracy of recommendation system has been proposed that would check
the recommendations [37], [38]. how similar university course programmes are to the
Various approaches are contained within the students’ profiles.
recommendation system. The main approaches are content- The proposed framework is a comprehensive one that
based filtering (CBF), collaborative-based filtering (CF) and combines CBF and CF with an ontology technique in order
hybrid-based filtering [16], [19], [39]. CBF attempts to to overcome the overloading information problem. It does
recommend items that are related to those which a given user this by using a similar hierarchal ontology to map the courses
has preferred in the past. In CF, however, the system profiles with the user (student) profile. The new approach
identifies users whose preferences are similar to those of the develops two new methods to extract and integrate data from
given user and suggests items they have favoured. A hybrid multiple sources and then line them. This ontology mapping
recommender system is one that combines two or more of the different data improves the ability to obtain a
recommendation approaches to achieve a better performance comprehensive knowledge of the recommended items. The
with fewer disadvantages than an individual approach. CF is approach tackles the new user problem by calculating the
combined, most frequently, with certain other techniques in ontology similarity there is between the users’ profiles by
an attempt to avoid the cold start problem. In the following measuring the user rates for each item. The proposed
sub sections, we focus on literature using recommendation recommender system is used to work out the hierarchy
systems in the education domain and how the use of ontology ontology similarity there is between the item profiles and
will improve the quality of the recommendations. users’ profiles before the student enrolling in the research
program chooses courses to match his/her requirements.
A. RECOMMENDATION SYSTEM IN EDUCATION
DOMAIN B. ONTOLOGY BASED RECOMMENDER SYSTEM
A significant number of recommender systems have been The original definition of “ontology” in computer science
proposed in the education domain, as well as in teaching and was provided by Gruber [42] as being “explicit specification
academic advising. In the education domain, the target users of a conceptualisation”. An ontology is used to represent an
are students, teachers or academic advisors, and the area of knowledge that formally describes a list of terms.
recommendable items are educational materials, universities Each of these items represents an important concept, such as
or information, such as courses, topics, student performance the classes of objects and the relationships that exist between
and the field of study. Sandvug and Burke presented them [29]. Ontologies provide formal semantics that can be
Academic Advisor Course Recommendation Engine used to both process and integrate a range of information on
(AACORN) that used a case-based reasoning approach, the internet. Modelling information is one of the main goals
which utilised knowledge that had been acquired from of using ontologies [43]. The authors in [44] reported that
previous cases in order to solve new problems [40]. Their ontologies are concept properties, disjointedness statements,
system used both the course histories and experience of past value restrictions and specifications of logical relationships
students as the basis of assisting students in course decision between objects. Ontologies provide a tool for the formal
making. modelling of the structure of a system, which is based on the
At the same time, it was noticed that the future career of relationships that emerge from its observation.
students is an essential factor which can influence their The term taxonomy (topic hierarchy) has been used when
decision to choose a particular course [41]. Farzan and the ontology contains only “IS-A” relationships. The use of
Brusilovsky proved this by using a reported course the word ‘ontology’ is usually restricted to systems that
recommendation system that was based on an adaptive support a rich variety of relationships between concepts,
community [3]. They employed a social navigation approach including logical propositions that formally describe the
to analyse the students’ assessment of their career goal in relationship. Many ontology classifications have been
order to provide recommendations for courses. The primary established [45]. For instance, ontology can refer to specific
idea of this approach was to obtain the students’ explicit domains that may provide conceptual modelling of a
feedback implicitly, as part of their natural interaction with particular domain.
the system.

2169-3536 (c) 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2889635, IEEE Access

Various ontology-based recommendation approaches rates for each item. The proposed recommender
have been developed by using a variety of different methods system is used to work out the hierarchy ontology
[46]. Furthermore, the concept of the semantic web is used similarity there is between the item profiles and the
to improve e-learning. In [47], Yang et al proposed a users’ profiles before the student enrols on the
semantic recommender system approach for e-learning use research program and chooses courses to match
to help learners to define suitable learning objectives. his/her requirements.
Moreover, the system could assist instructors by suggesting
new resources that could be adopted to enhance the syllabus III. PROPOSED SOLUTION
of the course. This system has been built with a query A hybrid recommender method based on ontology has
keywords extension and uses both semantic relations and been proposed in this work. The method firstly aims to
ontology reasoning. The authors in [10]presented a extract and integrate information from multiple sources
personalised ontology-based recommendation system, based on ontology. The information sources are classified
which is similar to the two approaches mentioned above. It into three primary sources; course information sources,
represents items and user profiles in order to provide student information sources and career information sources.
personalised services that use semantic web applications. Integrating information using ontology will obtain an
The evaluation shows that the semantics-based methods of optimal result. Moreover, the second objective is to build
the recommender system improve the accuracy of the dynamic ontology mapping between the user profiles and the
recommendations. A recommendation system based on item profiles that will help to reduce information
ontology can also solve the cold start problem, which occurs overloading. In order to recommend an appropriate
when user information from the past is insufficient [30]. recommendation to the users, we have combined two main
Indeed, this problem occurs due to an initial lack of ratings filtering approaches, CBF and CF, and thus the result is a
for new users and hence it becomes impossible to make combination of memory-based and model-based methods. In
reliable recommendations. An ontology-based model has the CF, several techniques, such as user profiling that is
been proposed for e-learning personalisation which would based on ontology, item ontology and k-NN, are used to
recommend learning objectives by judging the past overcome the information overload problem and improve
preference history of learners. Like traditional systems, this scalability and accuracy.
system suffers from a new user problem and is limited to On the other hand, item-based ontology and semantic
learning objectives only [48]. Ontology structure similarity are applied in the content-based filtering to solve
significantly improves the ontology structure, which can lead the new user issue and to also improve accuracy. The final
to increased accuracy [49]. For instance, all of the “IS-A”s objective is to put forward a list of recommendations and ask
relations in the ontology for measuring semantic similarity the user to assign a rating to each recommendation. The user
were considered to be similar in a hierarchical tree in which then gives their feedback on the recommendation list and
the associations between the concepts were shown by “IS- carries out a re-ranking. User feedback has been used to
A”. Calculating the similarity between the two concepts is evaluate the system and improve its accuracy, as is shown in
made less accurate by this. Consequently, this affects how greater detail in the evaluation section. This work aims to
accurate the recommender system is in finding similar items increase the accuracy and performance of the recommender
or users. To avoid this problem in our system, we will invite system by combining the hybrid method (CBF and CF) with
new users to complete their profile by providing their enhanced ontology.
personal information and preferences and by responding to
certain questions. We will then create the user profile model A. FRAMEWORK OVEREIVEW
based on the user ontology model.
The proposed ontology-based personalised course
The main contributions of this work are the following: recommendation framework (OPCR) is focused on
 Develop a comprehensive framework that recommending courses to students by utilising a hybrid
combines CBF and CF with an ontology technique filtering approach that combines both content-based filtering
in order to overcome the overloading information and collaborative-based filtering with ontology support. As
problem. This is achieved by using a similar shown in Fig.1, OPCR consists of four main layers. The first
hierarchal ontology to map the profiles of the layer is data gathering, which consists of all the information
courses with the user (student) profile. resources and the data collection model. This is used to
 Develop a new approach to extract and extract useful information from multiple sources. The second
integrate data from multiple sources and then map layer is the database, which is used to store all of the items
them. This ontology mapping of the different data and user information. The middle layer is the core functional
improves the ability to obtain a comprehensive part, which includes the ontological data model and the
knowledge of the recommended items. recommender engine model. We will explain each model in
 The approach tackles the new user detail in the following sections. The final layer is a user
problem by calculating the ontology similarity there application layer that consists of the user interface model,
is between the users’ profiles by measuring the user which is responsible for user interaction with the framework,

2169-3536 (c) 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2889635, IEEE Access

for searching items and for giving feedback on the information, or Microsoft Excel documents that have been
recommendation list. Every layer and model in the uploaded to the internet, such as statistical information
framework both links and interacts with the others, based on regarding the reputation of educational institutions, for
the input and output of each one. Our framework comprises example the NSS score for universities. The data from both
the following steps: the student and course ontology is prepared and pre-
(1) Extract all the useful information for the system from processed into the correct format for the recommendation
multiple sources. engine by the pre-processing data component. It was a time-
(2) Build the courses’ profiles by extracting all the useful consuming task to obtain information about each course
information regardong course features and sorting that from all the universities’ websites as each university
information in the system database. Consideration is given to publishes its course information in different formats.
the ontology hierarchy of the course features. Extracting precise information from various websites is
(3) Build the student profile by obtaining student always a challenging task in the domain of information
information via both explicit and implicit approaches. We engineering so we customised a web crawler that browses
have identified different user attributes which can be used to the web page automatically. It scrapes information from a
profile the student into our system as well as the user ratings web page and then sorts this into the system database. The
of the recommended courses. reformulated queries are allocated to web crawlers and APIs
(4) Build dynamic ontology mapping in order to link the that search for specific course information and jobs.
user profile and item profile. The web crawler analyses the web page based on a
(5) Analyse user queries and calculate the similarity definition of the features of each course, and then extracts
between the user profile and the course profile by employing feature values. Each extracted feature value belongs to one
ontology matching and cosine similarity. of the features that we have used in this paper. Five features
(6) Use a collaborative filtering technique in order to of the courses are marked in this study: course title, course
obtain top N users that are similar to the current user by using major subject, course fee, university location and the
an ontology-based k nearest neighbour (OKNN) algorithm. language of the course. On the other hand, the feature that
The final step suggests the recommended list of courses to has been constructed in the user ontology is based on the
the user and obtains user feedback. The purpose of each of feature in item ontology. The implicit information, such as
these components is explained in the following sections. the user, feedback and the rates of the recommendations,
have been collected and added to the user profile for later
use, when it is then utilised to locate a top-rated neighbour
that is similar to the target user.

C. CORE FUNCTIONAL
This section is the most important part of the framework
and it consists of two models. Firstly, the ontology model,
which includes construction dynamic ontologies for the user
and the items that map these ontologies in order to gain a
comprehensive knowledge of the recommendations. After
building the ontologies and mapping them, this will be used
as an input in the recommender engine. The recommender
engine model is the second model in the layer. We have
combined both CBF and CF filters to recommend items to
users and utilised ontology in order to enhance the
performance of the recommender engine (see section D and
E for more details).
Ontologies are used in the proposed approach to model
knowledge regarding the course content (the course profile),
FIGURE 1. OPRC main architecture knowledge about the user (the student profile) and domain
knowledge (the taxonomy of the domain being learned).
B. DATA GATHERING Within the domain of knowledge representation, the term
As it was decided that a content-based recommender ontology refers to both the formal and explicit descriptions
system technique should be the primary approach for the of the domain concepts [1]. These are frequently conceived
provision of recommendations, there are different formats of as a set of entities, relations, functions, instances and axioms
information that need to be gathered to support this system. [7]. By enabling the users or contents to share a common
Fortunately, all of these are available through information understanding of the knowledge structure, ontologies give
sources which are publicly available, either through websites applications the ability to interpret the context of the student
in HTML format, such as the universities' websites for profiles and the course content features based on their
course information and recruitment websites for career semantics. In addition, the hierarchical structure of the

2169-3536 (c) 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2889635, IEEE Access

ontologies allows the developers to reuse the domain been used to build the model. In order to construct this
ontologies (for example, in computer science and ontology, the following steps have been considered:
programming language)[50] in order to describe the learning 1. Determine the domain and scope of the ontology
fields and to build a practical model without the need to start In this proposed work, higher education has been
from scratch. determined as the domain and master’s courses in
The present work has constructed three ontologies. Firstly, Computing and Business Management have been
the course ontology; secondly the student ontology; and determined as the scope of the ontology.
thirdly, the job ontology. The protégé tool has been used to 2. Take into account reusing existing ontology
evaluate the ontologies with hierarchical mapping between In education, many ontologies were found that model this
the ontology classes that are used to compute the similarity aspect of the domain. However, no ontology was found that
between them. Knowledge, represented by the ontologies, could be reused to serve our intended purpose. Despite this,
has been combined into one single ontology. The ontology current ontologies have been used as a guideline to model the
model created significantly helps to reduce information common concepts of the new ontology.
overloading. 3. Enumerate the domain terms
The ontology is defined as a taxonomy that helps to
1) Dynamic ontology construction describe different aspects of the domain, such as the student,
The difference between static and dynamic ontology is course and career. Some concepts are further divided into
that the dynamic depends on certain parameters changing subclasses that would improve the classification of the
that can be considered globally to be situations. Static and instances of these classes.
dynamic ontologies are suitable examples of the static and 4. Determine the classes and the class hierarchy
the dynamic from classical physics [43]. There are generally The classes are defined as a group of individuals or
several ways to make a given static ontology become a instances that represent a class where all of the members
dynamic one; it simply depends on what we want to define share the same concepts. When the classes are ordered
as being changing objects. However, ontologies developed hierarchically, this is termed a taxonomy. Inference engines
by static approaches consist of terms that are limited in their use hierarchies to denote inheritance relationships. Classes
knowledge base due to a lack of updating. A dynamic are defined by following the combination development
ontology-based model is proposed to classify the extracted process, which is a combination of both bottom-to-top and
terms and to build a knowledge base for a specific domain. top-to-bottom approaches. When this approach is followed,
It is a challenge to obtain a well-classified corpus. Even if a the important terms are first defined and then generalisation
corpus is available, it may be classified improperly due to and specialisation takes place.
fewer terms being classified because of the limited and static 5. Define the relationships between classes
nature of the classifiers. To overcome this, we propose using The relationship that exists between class members in an
an ontology-based model in order to classify the terms and ontology is termed the properties. There are two types of
prepare the knowledge base. Ontology is a data model that properties: object and data properties. Object properties
characterises knowledge about a set of classes or concepts represent the binary relations that exist between members of
and the relationships between them [44]. The classes define the classes, such as the relationship between a student and
the types of attributes or properties that are common to the courses. Here, we define a property called HasSelected,
individual objects within the class. which is used to represent this relationship. Data properties
The following modules explain our proposed dynamic link an individual to a data literal, such a student’s ID.
ontology model: Document Analysis, Ontology We found that, by analysing users belonging to a
Construction. Fig.2. particular profile, they have a similar interest in course
ontology. Thus attributes such as offereCourse, HasCareer,
etc. can help to decide initial recommendations to a user
according to his/her profile. In addition, in this work we have
focused mainly on the recommendation of courses based on
CBF, and the attributes in the course vector such as course
title, main subject of course and location. The user nodes in
the user profile ontology are linked to course attributes in the
course ontology using a hasFeildOfStudy, HasLocation
relations. The course ontology is linked with job ontology
FIGURE 2. Dynamic Ontology Construction using a LeadTo relation.
There are many existing methods of constructing
ontologies available. In the present work, we follow the 2) Course Ontology
‘‘Ontology Development 101’’ approach developed by Identifying different attributes is necessary for course
Natalya Noy and Deborah McGuinness [51]. The language profiling[31]. In order to construct a course ontology, we
used to write the ontology is the OWL 2 Web Ontology need to identify factors that most influence a student when
Language [36] and the protégé tool (Version 5.2) [52] has they make a decision in choosing a university course. These

2169-3536 (c) 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2889635, IEEE Access

factors then become the main classes of the ontology. We

carried out a survey of students at the University of
Portsmouth to discover the most important factors that had
influenced their choice of university course. More than 200
students participated in this survey. They were given 20
factors that influenced their decision to choose a university
course and were then asked to rank these on a scale of 1-10.
The 20 factors were classified into six categories and the
scores and standard deviations for each category were
computed. The results have been summarised in Table 1.

TABLE 1. Factors and keys constituent elements for selecting

university courses

Factors and key constituent element Mean

Course information( Field of study, Courses , 7.8
major subjects , course structure)
Course Fee 7.5
NNS score 7.4
Prominence (institutional reputation ) 6.4
Location ( institutional location) 6.9
FIGURE .3 Course ontology Structure
Career 7.9
The course profile attributes will match the user profile
A course programme’s title, fees, location and prominence feature through the ontology mapping. Each class of the
were all factors that appeared to be the most important when course profile will be a map to the equivalent class in the user
the students determined their choice of university for higher profile. Ontology reference is used to identify the equivalent
education (HE) study. The students chose computer sciences classes in both the course profile and the user profile. The
and business management programmes, although some protégé tool was used for the construction and evaluation of
differences in the prioritisation of elements within the broad the ontology model. Fig.4 shows the graphical
factors were observed. The following points can be noted: representation of the course ontology in the protégé
Taking 5.5 as the midpoint on a ten-point Likert scale, environment.
three of the seven factors had a mean score that was lower
than this midpoint. It can be assumed therefore that
promotion, people and prospectus elements do not have a
significant influence on the choices that students make
regarding where to study for their higher education.
Among the elements included in the programme factors,
both the field of study and the details regarding course
information appear to exert the most considerable influence
on the students’ choice of university course programme.
The factor that was uppermost in the students’ decision-
making frameworks was the issue of fees, which had the
greatest impact on university choice and the type of career
that could be achieved following completion of the course.
It was found that issues of institutional prominence
maintain a fairly high profile in students’ decision-making.
The overall reputation of the institution and the National
Student Survey score (NSS) of teaching students are both
significant.
The course attributes are considered when extracting the
course profile, including the essential information, course
information, as well as information regarding fees and FIGURE 4. Graphical Representation of the Course Ontology
university rankings and the university’s NSS score. This
information is used for knowledge discovery at a later stage
of the user profiling process. In Fig.3 the main classes and
subclasses of course ontology are shown with instances.
7

2169-3536 (c) 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2889635, IEEE Access

2) Student Ontology system recommends several courses in the faculties of arts,

Firstly, we need to model the student profile before information technology, science, social science,
recommending the appropriate course. The user profile management, commerce, engineering, education and law.
consists of two main parts. The first part is the personal The student obtains a recommendation for any course
attributes and education attributes of the user and the second depending on their eligibility, i.e. if the student has a
is the user’s rating of the previously recommended course. graduation degree, the system can recommend any
The personal attributes include the user’s individual personal postgraduate course and if the student has a postgraduate
information, as well as education and background degree, either a research course or a PhD can be selected,
information, such as their hometown, gender, the field of depending on the faculty. The proposed approach conducts
study, main subject, major subject, interest area, technical an entrance test as an eligibility criterion for admission into
and non-technical skills, as shown in Fig. 5 and in Fig.6, the the undergraduate and postgraduate engineering courses.
graphical representation of student profile ontology in In the proposed system, there are three ontologies: course
protégé environment is shown. ontology, student profile ontology and job ontology. There
Therefore, in this paper, a student profile can be formally are three aspects of the local ontology construction process.
defined as Formula (1) and Formula (2): These are unstructured text documents from structured
relational data sources and semi-structured data sources files.
U= { a1,a2,…………, an} (1) Unstructured text documents include four processes: data
Where U is the user/student, ai represents the users and ith pre-processing, concept clustering, context extraction and
attributes. local ontology construction. For more information about
If a student has obtained an offer from the system in the past local ontology construction from the unstructured text, see
and rated the courses, we can further define that student as: [1].
Ur = {u,r} = { a1,a2,……, an ,r} (2)
Here, Ur is the user that received a recommendation for the
courses from the system and has rated the courses.
Furthermore, in order to make a satisfactory
recommendation, it is important to ensure that the
characteristics of the recommended activities match the
user’s interests. The course ontology is created for all the
courses that are to be recommended to the user/student. The

FIGURE 5. Student Ontology Structure FIRURE 6. Graphical Representation of the Student Ontology

2169-3536 (c) 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2889635, IEEE Access

3) Job Ontology E. RECOMMENDER ENGINE

A student’s future career is an essential factor that can After constructing the ontology models, in this section we
influence their decision making when they are selecting a now discuss the recommender engine. We used a hybrid
university course [3]. Constructing a job ontology is vital if method which combined the CBF and CF filtering
a student is to understand the attributes of the job. This is approaches with supporting ontology model mapping, and
extracted from a recruitment website, such as Indeed.com. this is the core component of the framework. In the following
Job attributes include such information as job title, job sections, we explain in detail how each element of the hybrid
description, job salary, job location and the required approach works
educational qualifications, as shown in Fig.7.
1) CBF METHOD
As previously mentioned, CBF filtering is based on the
similarities that exist between the items (courses) and the
user’s preferences. In order to calculate the similarity, we
need to generate a vector for the features of both the item and
the user. According to the course ontology model, the main
classes are used as the feature of the item vector. The features
include the course title, the major subject of the course, the
course fee and the institution’s location. A constant weight
has been adjusted for each of these features. These are 15%,
15%, 10%, 10%, respectively. An additional feature that was
used in the CBF filtering to recommend the more relevant
course was the university’s reputation and its NSS score. The
weight assigned to each additional feature was 10% and
10%, respectively in the final scoring function.
Different techniques have been used to calculate the
similarity between the user profile and the course profile,
according to the nature of the attributes in the course profile
and the user profile. Hierarchy ontology similarity has been
used for attributes, such as the course subject root and user
preferred subject.
FIFURE 7. Job Ontology Structure Moreover, the matching similarity has been used to
compute the similarity between the user location and the
location of the university that provides the courses.
There is also a graphical representation of the job ontology Additionally, we have matched the user’s city with the
in protégé environment in Fig.8. regions of the universities in order to obtain more results.
The cities are as classified by the United Kingdom. These are
based on 12 regions and each region is formed of many cities.
For example, the South East includes Portsmouth,
Southampton and Kent amongst others. We have used a
different type of similarity in the CBF approach, such as
cosine similarity, matching similarity and normalisation
similarity, depending on the nature of the feature. This is as
follows:
 Used cosine similarity to calculate the course title and
major course subject, according to the formula (3)
⃗⃗⃗⃗⃗⃗
𝐼𝑓𝑎 .𝐼𝑓⃗⃗⃗⃗⃗⃗⃗𝑏 (3)
𝑆𝑖𝑚(𝐼𝑓𝑎 , 𝐼𝑓𝑏 ) =
||⃗⃗⃗⃗⃗⃗
𝐼𝑓𝑎 || × ||⃗⃗⃗⃗⃗⃗⃗
𝐼𝑓𝑏 ||

Where Ifa , Ifb are item features of item a, b.

 Course fee similarity calculation: The similarity

between the university course fees and the user preferred
fees has been calculated by using the following formula
FIGURE 8. Graphical Representation of the Job Ontology (4):

2169-3536 (c) 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2889635, IEEE Access

𝐹𝑢𝑚𝑎𝑥 − 𝐹𝑐 (4) The previous section presented the way in which the CBF is
𝐹𝑆(𝑈, 𝐶) =
𝐹𝑚𝑎𝑥 − (𝐹𝑚𝑖𝑛 − 1) able to calculate the similarity between the user profile and the
item profile based on the available attributes in each profile
Where: vector. In this section, we explain how the CF works within
FS (U, C) = the course fee similarity between the the framework and how using the ontology-enhanced CF
user preferred fee and the course fee for each performs to find the most similar users to the active user. The
university most important aspect of the CF is how to measure the
Fumax = the maximum university course fee that is similarity between the active user and the other users in the
expected from the user database. In addition, a new algorithm has been produced in
Fmin = the minimum university course fee in the order to enhance the KNN algorithm by using the ontology
database similarity called (OKNN). In the following sub-sections, each
Fc = the university course fee part will be presented in detail.
 Location similarity calculation: The matching
similarity has been used to compute the similarity  USER SIMILARITY CALCULATION
between the user location and the location of the
university providing the courses. In order to achieve The user profile vector consists of two parts; the first part is
more results, we also matched the user’s city with the the user attributes, such as personal and academic
regions where the universities are situated. The United information. The second part is the ratings that the user gives
Kingdom has classified the cities, based on 12 regions, the item in the CBF case. In the proposed work, a new
and each of the regions is formed of many cities. For method has been used to calculate the similarity between the
example, the South East includes Portsmouth, target user and other users in the database. The main idea is
to use an ontology hierarchy similarity in the user profile and
Southampton and Kent amongst others.
the user profile attributes. The proposed system has ontology
 University ranking similarity calculation: we support from the user history similarity that enables it to
calculated the ranking attribute in the user query and calculate the similarity between the target user and the other
course profile, according to the formula (5) users in the system, according to the formula (7). The user
𝑅𝑚𝑎𝑥 − 𝑅𝑐 (5)
𝑅𝑆(𝑈, 𝐶) = similarity value range will be between (0, 1) and the weight
𝑅𝑚𝑎𝑥 − 𝑅𝑚𝑖𝑛 for each part 50%.
US (Ua, Un) = ontology similarity + (7)
Where: recommendation history similarity

𝑅𝑆(𝑈, 𝐶) = the university ranking similarity

between the user preferred ranking and university Where:
ranking (US) is a similarity between the target user Ua and the users
Rmax = the maximum university ranking in the in the system Un. The system considers the levels of the
database ontology concepts in the user profile by classifying the
Rmin = the minimum university ranking in the ontology similarity to four levels. Moreover, the given
database weight for each level is based on its importance, as follows:
Rc = the ranking of the university which provides Level 1 (major subject, main subject, the field of study)
the course Level2 (interest area)
Level3 (user location)
 NSS score similarity calculation: to find the Level 4 (user skills), as shown in Fig.9.
similarity between the NSS score of the course and the
NSS score that the user is satisfied with, the following
formula (6) has been used:
𝑈𝑁 − (𝑁𝑚𝑖𝑛 − 1) (6)
𝑁𝑆(𝑈, 𝐶) =
𝑁𝑚𝑎𝑥 − (𝑁𝑚𝑖𝑛 − 1)

Where:
NS (U, C) = the NSS score similarity between the
user and the course
UN = the user preferred NSS score
Nmin = the minimum NSS score in the database
Nmax = the maximum NSS score in the database
FIGURE 9. Hierarchical matching and matching parameters

To compute the similarity between each level of the

ontology, we need to adjust the weight of each level, based
2) CF METHOD
10

2169-3536 (c) 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2889635, IEEE Access

on the importance of the concepts in the levels. The P = the set of courses that have been rated by user Ua and
importance of the concepts in the ontology level has been Ua
adjusted according to the results of the survey of The algorithm firstly calculates the dot product that is the
postgraduate students at the School of Computing and sum of the products of the two vectors. However, as the dot
School of Business at the University of Portsmouth. The product is sensitive to the magnitude, it might show that two
results of the survey showed that the concepts in Level 1 are vectors with a similar direction are dissimilar to each other,
more important when a user decides to choose a university owing to one having a larger magnitude than the other.
course programme. The weight given to the levels is as Following this, we need to normalise the value by dividing
follows: the product of the lengths of the two vectors together and
Level 1 (30%) calculating the cosine similarity by using the unit vector
Level 2 (10%) rather than the normal vector.
Level 3 (5%)
Level4 (5%).  ONTOLOGY BASED K-NEAREST NEIGHBOUR
For instance, if the Ua profile consists of these attributes: ALGORITHM
artificial intelligence as a major subject, computer sciences
as a main subject, information technology as a field of study, The k-nearest neighbour users of the active user (target
management as an interesting area, Portsmouth as a location, user) must be determined in order to make a
programming as a skill, then user Ub profile has these recommendations list by CF. To achieve this result, we
attributes computer programming as a major subject, proposed a new algorithm, OKNN algorithm, that combines
computer sciences as a main subject, information technology the ontology similarity of the user profile attribute and the
as a field of study, management as an interesting area, item rate when the recommendation history is applied. The
Southampton as a location, programming as a skill. The k-nearest neighbour users to the target user are found by
ontology similarity calculation between Ua, Ub will be based searching only those who exist among the same group, rather
on the Eq.(8): than all the users. For instance, if the target user has a main
𝑛
(8) subject of Computer Sciences and their major is Computer
𝑂𝑆(𝑈𝑎, 𝑈𝑏) = ∑ 𝐿𝑚 Programming, the nearest neighbour will search for all the
𝑙=1 users who have Computer Sciences as a main subject in their
profiles. In addition, not all of the groups are searched in the
Where: User-Clustering attribute of the items selected. The user
OS= Ontology similarity similarity, based on Eq. (11), has been used to locate who is
N = number of levels in the ontology the neighbouring user to the target user. To find the top k-
Lm = level concept matching nearest neighbour to the target user, we needed to rank the
users’ similarity score. A common rate problem we faced for
OS (Ua, Ub) = level1 + level2 + level3+ level4 the top k-nearest neighbour was that the same item had been
OS (Ua, Ub) = (0+ 0.1+0.05) + (0.1) + (0.05) + (0.05) rated by different values, respectively. In order to solve this
OS (Ua, Ub) = 0.35 problem, the following formula has been proposed:
Moreover, after computing the ontology similarity it will
be necessary to obtain the recommendation history similarity ARW C∗(KNNW−Omax∗K)
Average weight score = (( ) (11)
between Ua, Ub. In the proposed work, the recommendation KNNW
history includes all the courses that have been rated by the + Oc *K)/100
user in the CBF case. Many algorithms have been applied to
compute the similarity between the user recommendation Where:
histories. Cosine similarity is one of the algorithms that is KNNW = KNN weight in the final scoring function
most widely used in this area [18]. The similarity between ARW c = average weight of the rate for the current course
the users’ recommendation histories has been computed *100%
according to Eq. (9) and Eq. (10), as follows: Omax = the maximum occurrence of the rate in the
recommendation history of all the top N users
⃗⃗⃗⃗
𝑈𝑎 . ⃗⃗⃗⃗
𝑈𝑏 K = constant (e.g. 2)
(9)
𝑆𝑖𝑚(𝑈𝑎 , 𝑈𝑏 ) = Oc = the number of occurrences of the current course has
⃗⃗⃗⃗𝑎 || × ||𝑈
||𝑈 ⃗⃗⃗⃗𝑏 || been rated

The proposed method improves the scalability and accuracy,

∑𝑝∈𝑃 𝑈𝑎 .𝑈𝑏
𝑆𝑖𝑚(𝑈𝑎 , 𝑈𝑏 ) = (10) leading to an improvement in the performance of the
√∑𝑝∈𝑃(𝑈𝑎)2 √∑𝑝∈𝑃(𝑈𝑏)2 algorithm. We present the steps of this algorithm as follows:

Where:
𝑆𝑖𝑚(𝑈𝑎 , 𝑈𝑏 ) = cosine similarity of two vectors
11

2169-3536 (c) 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2889635, IEEE Access

(OKNN) ALGORITHM university rank, NSS score) is 50%, 30%, 10%, 10%,
For user Uc Get user profile and create vector
respectively.
1:
Final Scoring Function (FSF) = (CBF*(50%)) (12)
2: while there are Users to compare U do
+CF*(30%)) + (university rank *(10%)) + (NSS
3: Create vector for U score 8 (10%))
Calculate the Similarity between U and Uc
4:
by using Formula 7
5: Sort the nearest neighbour list
Get the top 5 nearest neighbour
OPCR ALGORITHM
6:
1: Calculate course score based on the user’s profiles and
7: for each user in the top 5 list do query term
for each course in the user’s recommendation 2: Calculate course NSS score similarity using formula (
8: 6)
history
3: Calculate university rank similarity by using formula (5)
9: If C rate >= 3 then
10: add the C to the KNN list 4: Calculate course fees similarity by using formula (4)
5: Rank and get the top 10 courses which have the highest
11: end if
similarity score
12: end for 6: Recommend the user by the top 10 courses
13: end for
7: If the user chooses any of the recommended
14 for each C in the KNN list do
courses then
15: Calculate the C rate using Equation 11 8: Use OKNN algorithm to determine the five nearest
16: Update the KNN list with the new score neighbours of the current user Uc calculate course
end for scores
17:
9: add top 10 courses to the recommendation list
18: end while
10: end if
11: Return the refined recommendations to the user
In this algorithm, the similarity between ontologies is used
to compare the target user profile to other users to obtain k-
NN users. In this method of similarity, the conceptual
similarities are considered when measuring the similarity IV. EXPERMENTAL STUDY
between two ontologies. The conceptual comparison level An experimental prototype system has been designed
includes the comparison between two taxonomies and the based on the OPRC framework. All modules that have been
comparison of relations between the corresponding concepts developed use open source tools which have been organised
of the two taxonomies. After producing the k-nearest in a traditional client and server structure. The main objective
neighbour users, all courses that have been selected by the of the evaluation is to determine whether the proposed
neighbour users, but have not been selected by the target method, which considers ontology data integration and
user, are recommended to the target user. hierarchically-related concepts, is better than the existing
The final step in the method is that the final filtering method, which does not consider hierarchically-
recommendation list can be presented to the active user related concepts.
according to a hybrid recommendation list from both the To achieve the objectives, we organised an experiment in
CBF and CF filters based on a weighted approach. which participants used an experimental system for
evaluating course items. We made sure that user interaction
with the framework was flexible which allowed the
3) Final Scoring Algorithm
participants to select and rate the items of the university
The proposed approach to filtering combines CBF and CF
course in several sessions; for example, they could use the
with ontology to recommend courses to the user. For the new
CBF and CF algorithm individually to see how the results
user, the system will recommend courses based on his/her
changed compared with the OPCR algorithms. The
profile. The recommendation process will begin based on the
OPCR algorithm by creating a vector of users and courses. participants were asked to provide a rating for each item on
the recommendation list and re-rank the position of the item
The final recommendation list is produced by using the
in the recommendation list. The participants’ ratings were
final scoring function (FSF). FSF combines the similarity
then compared with the system’s rankings.
score of a content-based filtering list and a collaborative
filtering list. Moreover, the other factor will be added to the
final score as well, such as the university ranking and the A. DESCRIPTION OF THE EXPRIMENT
NSS score as shown in the Eq. (12). The value of the final We requested students from different academic
score function similarity should be between the range (0-1). backgrounds from the University of Portsmouth to
The weight percentage for each part in FCF (CBF, CF, participate in our framework experiment. A total of 123
students participated in the month-long experiment. The

12

2169-3536 (c) 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2889635, IEEE Access

students were from two different departments, the School of Each participant could use the system in two ways; one
Computing and the School of Business and Management. option was a general search based on keywords, and the
After evaluating the system, the participants were asked to second was a personalised search achieved by building a user
answer questions regarding different aspects of the system’s profile. This is undertaken by the register in the system and
performance. A total of 95 students responded to the gives the system information about the user’s educational
questionnaires, including 50 students from the School of background and interests. After the user profile has been
Computing and 45 from the School of Business and built, the system search will become more personalised. The
Management. The participants were from different levels of system will recommend the five top courses to participants
education and study, including undergraduate, postgraduate that are more relevant to their user profile. The participant is
and PhD students. Table 2 shows the number of students required to rate each course that is on the recommendation
from each level. list, based on their interest in it, on a scale of 0 (not interested
TABLE 2. Number of participants and level of study at all) to 5 (strongly interesting). Several recommender
systems use a 1-5 scale, particularly course filtering systems
Study and news filtering systems, such as NewsWeeder [53] and
Field of study No. students the commercial Amazon system [11]. The final course
level
recommendation list showed that each participant used CBF
PhD 5 and then CF, as shown in Fig.10. The participants registered
Computer Sciences MSc 15 and each defined an initial profile. The initial profile
BSc 30 consisted of two main parts; the first was personal
PhD 5 information, such as username, gender, postal address, user
Business and Management MSc 16
contact. The second part included academic information for
BSc 24
the user, such as field of study, main subject, major subject,
current study level, interest areas, course language preferred
Each participant that registered onto the system and skills.
recommended courses based on his/her profile. The users Each user’s profile was updated implicitly by giving
were asked to give a rating on the recommended courses and consideration to the course that was rated in the
re-rank the recommended positions. The participants were recommendation list by the user. The weight of each level
also asked to use the search criteria to search on the UCAS was increased if the user rated the item relatively highly. The
website and rank the user satisfaction in both cases. The degree of the relevance of the recommended items was
course dataset used in the experiment was from the UCAS adjusted by using a certain threshold of the rating range.
website. Each participant used the system three times in order to
The experimental system used UCAS as the main source create different profiles with a different search. The
for course information on each day of the experiment. We participants’ user profiles were updated by the data collected
collected all of the course items by using the web crawler from the experimental system. This data was also used in
that had been built and customised to extract course different variations of the algorithm’s runs. The system’s
information. Each user was required to rate all the course performance was evaluated against a ranked list of the items,
items that were in the recommendation list provided on the as rated by the participants.
day of the experiment.

FIGURE 10. Course and job recommendations

13

2169-3536 (c) 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2889635, IEEE Access

Several metrics have been used to analyse the results that 2) Online evaluation:
were collected during this experiment. The users’ ratings on In an online evaluation, users interact with a running
the 0-5 scale were saved so as to enable a ranking order of recommender system and receive a recommendation.
the courses, and thereby express the items’ relevance to the Feedback from the users is then collected by either
user. The questionnaire was used to measure user satisfaction questioning them or observing them. Such a live user
and the quality of the recommendations. A benchmark was experiment may be controlled (e.g. randomly assigning users
used to compare OPCR with the current system. to different conditions) or a field study may be used in which
a recommender system is deployed in real life, and the
B. DATA SOURCE AND CONFIGURATION effects of the system then observed. Online evaluation is the
In this study, the data collection of the content of MSc most desirable as it can provide accurate results of how
courses was gathered from the UCAS (Universities and effective our system is with real users [56]. Conducting such
Colleges Admissions Service) website and Indeed.com evaluations is both time-consuming and complicated, but it
website was then used for job information. In order to is inevitable that we must conduct an online evaluation for
achieve this, a web crawler was built and customised. The this research, since it is the only way to measure real user
collected data was used to construct the item ontology satisfaction. Their multiple metrics have been used to
(courses and jobs), based on our knowledge. There was no evaluate factors, such as recovery, the accuracy of relevance
existing dataset for master’s courses at UK universities. We and rank accuracy, as follows.
have created our dataset, called ontologyset, which includes
courses extracted from UCAS.com. However, there was no 1)Recovery
need for an established benchmark dataset to evaluate The recovery metric has been employed to evaluate how the
OPCR’s performance. The system metadata included close recommender algorithms performed in providing a proper
to 21,000 online courses in ontologyset, covering 70 diverse ranking to the whole item set [57]. The user prefers a kind of
subject areas that had been archived from UCAS.com. These system that provides a higher rank for items which are
were focus chosen and downloaded from different relevant to the target user. Items that are relevant to each user
departments at various universities and colleges in the can be extracted, based on her/his ratings in the test dataset.
United Kingdom for testing purposes. The breakdown was to We considered the course selected by a test user and found
select 20 of these subject areas with a number of courses, that the Like rating (ratings 3, 4, 5) in the test dataset was
however we decided to use the computer sciences and relevant to the active user. Therefore, the recovery RC can
business management courses. Courses in ontologyset cover be obtained according to Eq. (13):
every postgraduate academic level, which yields a 1 𝐾𝑢 𝑝𝑖
∑u∈uTestSet ∑𝑖=1
representative set that includes a wide range of courses
𝑅𝐶 = 𝐾𝑢 𝐶𝑢 (13)
offered at different universities. |uTestSet |
We used the Indeed.com website as a source in order to Where Cu is the number of candidate items for a
extract job information. This information included the job recommendation in an item set, Ku is the number of relevant
title, description, salary, location and user reviews. For test items to user u, pi is the place for an item I in the ranked list
purposes, any jobs that related to CS and BAM courses were for user u, and |uTestSet | is the number of users in the test
extracted. dataset. Based on this definition of recovery, the lower the
RC is, the more accurate the system. In Table 3, an example
C. EVALUATION METRICS of measure recovery metric, five users received a list of
There are many approaches to evaluating the recommended courses and they rated (R) these according to
recommendation systems. The evaluation can use either their individual needs. We used Eq. 13 to find the recovery
offline analysis or online user experimental methods or a metric value as following:
combination of these two approaches [54]. The approaches 1 1 2 5 1 2 3 1 1 3 4 1 1 2 1 2
(5+5+5)+2(5+5)+3(5+5+5)+2(5+5)+1(5)
will be discussed in detail in the following subsections. 𝑅=3 = 0.45
5
1) Offline evaluation:
An offline evaluation is achieved by using a pre-gathered TABLE 3. Example of Recovery Metric
dataset of users who choose or rate items. In many cases, the
offline evaluation will be useful as it will enable knowledge Ranki User 1 User 2 User 3 User4 User 5
about user behaviour to be obtained, such as the movie ng list Course R Course R Course R Course R Course R
domain and music domain [55]. However, it will be difficult 1 C1001 4 C1022 1 C1001 3 C1066 5 C1066 2
to obtain accurate results for the user's interests in the
2 C1004 5 C1034 4 C1222 2 C1032 5 C1032 5
education domain because each user needs to choose a
different education path based on their preferences. For this 3 C1012 2 C1012 5 C1432 3 C1032 2 C1032 2
reason, the online evaluation obtained more accurate results 4 C1023 2 C1023 2 C1004 4 C1033 2 C1033 2
because it was possible to obtain a real user interaction with 5 C1009 3 C1055 2 C1012 1 C1012 1 C1012 1
the recommendation system.

14

2169-3536 (c) 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2889635, IEEE Access

2) Accuracy of list relevance relevant items to the target user. The precision of the systems
on a recommendation list with N items can be defined in Eq.
In an ideal information retrieval system, documents (16) as:
should be ranked in order of how probable their relevance or ∑u∈𝑢TestSet 𝑃𝑢 (𝑁) (16)
usefulness is. Most IR and RS follow this principle and will 𝑃(𝑁) =
|uTestSet |
be presented to the user in a list. There are several methods
that have been presented in the past which measure the According to the example in Table 5, the precision will be as
accuracy of the relevance. One of these methods is average the following:
3 2 3 2 1
precision (AP) [57]. This is the average of the precision value + + + +
that is obtained from the set of top k documents that exist 𝑃(𝑁) = 5 5 5 5 5
= 0.44
5
after each relevant document is retrieved for the single query TABLE 5. Example of the percentage of relevant items to user u
(for one recommendation list). If we have a set of queries
(many recommendation lists), then we need to determine the Ranki User 1 User 2 User 3 User4 User 5
mean average precision MAP as shown in Eq. (14) and Eq. ng list Course R Course R Course R Course R Course R
(15). 1 C1001 4 C1022 1 C1001 3 C1066 5 C1066 2
1
𝐴𝑣𝑒𝑟𝑎𝑔𝑒 𝑃𝑒𝑟𝑐𝑖𝑠𝑖𝑜𝑛 (𝐴𝑃) = ∑𝑛𝑖=1 𝑟𝑒𝑙(𝑘) × (14)
2 C1004 5 C1034 4 C1222 2 C1032 5 C1032 5
𝑀
𝑃𝑟𝑒𝑐 @𝑘 3 C1012 2 C1012 5 C1432 3 C1032 2 C1032 2
1 4 C1023 2 C1023 2 C1004 4 C1033 2 C1033 2
𝑚𝑒𝑎𝑛 𝐴𝑣𝑎𝑟𝑎𝑔𝑒 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 (𝑀𝐴𝑃) = ∑ 𝐴𝑃𝑚
𝑀 (15) 5 C1009 3 C1055 2 C1012 1 C1012 1 C1012 1
𝑚
3) Rank accuracy
Where Rank metrics extend recall and precision to take the
M: the total number of relevant documents positions of correct items in a ranked list into account and
n: The list length measure the ability of an algorithm to produce an ordered list
rel (k): 1 if relevant, otherwise 0 of items that match the opinion of the user. Relevant items
Prec@k : precision at rate 3 and above at each rank are more useful when they appear earlier in the
m: number of queries recommendation list than when the item appears at the
According to the example in Table 4, we have five users who bottom of the list and are particularly important in
received a list of recommended courses and they rated (R) recommender systems as lower ranked items may be
this based on their interest. We used Eq. (14) to obtain the overlooked by users. We used the Spearman’s ranking
average precision for each user as the following: correlation r to calculate the ranking metric for the system
TABLE 4. Example of Accuracy of list relevance Metric [55]. The ranking will be more accurate when the r value is
close to (1). For the calculation method of Spearman’s
User 1 User 2 User 3 User4 User 5 ranking correlation we used Eq. (17)
Course R Course R Course R Course R Course R 𝑛
6
Rec. List Rec. Rec. List Rec. Rec. 𝑟 =1− ∑(𝑥𝑖 − 𝑦𝑖 )2 (17)
List List List 𝑛(𝑛2 − 1)
𝑖=1
C1001 4 C1022 1 C1001 3 C1066 5 C1066 5
Where n is number of recommended items
C1004 5 C1034 4 C1222 2 C1032 5 C1032 2
xi is the rank of item i output by RS
C1012 2 C1012 5 C1432 3 C1032 2 C1032 2
yi is the rank of item i offered by the user
C1023 2 C1023 2 C1004 4 C1033 2 C1033 2
In order to explain how to measure rank metrics we have
C1009 3 C1055 2 C1012 1 C1012 1 C1012 1 two cases scenarios, the example of the first case is shown in
1 2 3
+ +
𝑎𝑣𝑒𝑟𝑎𝑔𝑒 𝑝𝑒𝑟𝑐𝑖𝑠𝑖𝑜𝑛 𝑓𝑜𝑟 𝑢𝑠𝑒𝑟 1 = 1 2 5
= 0.86 Table 6 for user U1, all the user rank is different from the
3 system rank. We used Eq. (17) to find the value of rank
1 2
+
𝑎𝑣𝑒𝑟𝑎𝑔𝑒 𝑝𝑒𝑟𝑐𝑖𝑠𝑖𝑜𝑛 𝑓𝑜𝑟 𝑢𝑠𝑒𝑟 2 = 2 3
= 0.58 metrics as the following:
2 TABLE 6. Example of Represent System Ranking and User Ranking case1
1 2 3
+ +
𝑎𝑣𝑒𝑟𝑎𝑔𝑒 𝑝𝑒𝑟𝑐𝑖𝑠𝑖𝑜𝑛 𝑓𝑜𝑟 𝑢𝑠𝑒𝑟 3 = 1 3 4
= 0.8 Recommendation User System User
3
1 2
+ courses for U1 rate rank rank
𝑎𝑣𝑒𝑟𝑎𝑔𝑒 𝑝𝑒𝑟𝑐𝑖𝑠𝑖𝑜𝑛 𝑓𝑜𝑟 𝑢𝑠𝑒𝑟 4 = 1 2
= 1
2 C1001 4 1 2
1
𝑎𝑣𝑒𝑟𝑎𝑔𝑒 𝑝𝑒𝑟𝑐𝑖𝑠𝑖𝑜𝑛 𝑓𝑜𝑟 𝑢𝑠𝑒𝑟 5 = = 1 1 C1004 5 2 1
1 C1012 2 3 5
Many applications have been designed so that they
C1023 2 4 3
recommend N items to users. Precision for the list
C1009 3 5 4
recommended user u, Pu(N) is defined as the percentage of 6 2 (2 2 (3 2
the relevant items to user u in the list recommended to the 𝑟 =1− 2 ((1 − 2) + − 1) + − 5) + (4 −
5((5) −1)
user. We considered items selected by the target user in the 3)2 + (5 − 4)2 ) r= 0.6
test dataset and received Like rating (such as 3,4,5) as
15

2169-3536 (c) 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2889635, IEEE Access

The second case is for user U2 as shown in Table 7. In this obtained from the participants in order to make a comparison
case we noticed that 3 over 5 recommendation ranks are between the traditional CBF and CF filtering algorithms and
similar in both the system and user ranking and when the OPCR algorithms. The result, shown in Fig.12, was that
implemented with Eq.17 the result will be as following: the proposed approach algorithms worked far more precisely
6 than the traditional one. Moreover, when we compared the
𝑟 =1− ((1 − 2)2 + (2 − 1)2 + (3 − 3)2
5((5)2 − 1) proposed approach with a current course finder system, such
+ (4 − 4)2 + (5 − 5)2 ) as UCAS, it showed that OPCR is more accurate and
r= 0.9 provides more personalised results than UCAS. The
performance was also of a higher quality than that provided
TABLE 7. Example of Represent System Ranking and User Ranking by UCAS, as shown in the Fig.13.
case2

Recommendation User System User

courses for U2 rate rank rank
C1001 3 1 2

C1004 5 2 1
C1012 2 3 3
C1023 1 4 4
C1009 1 5 5

Furthermore, to measure the ranking metric for all the users,

it is necessary to calculate the average for all the r-value for
the testing users. FIGURE 11. OPCR User Interface

D. EXPERIMENTAL RESULTS
The experimental data that we collected, i.e. the user
ratings, was used to both train and test the hybrid filtering
algorithms with the ontology technique.
We implemented OPCR in Java and ran it on an Intel(R)
Core(TM)2 Dup CPU processor, with a CPU of 3.20 GHz
and 16 GB of RAM, under Windows 7. HTML was used for
the system interface, as shown in Fig.11, and the MySql
server was used to allocate a system dataset and user rating.
In addition, a protégé tool was used to evaluate the
ontologies built into the system.
The effectiveness of OPCR was assessed in an empirical
study that used a group of university students who played the
role of appraisers at our university in order to evaluate the
performance of OPCR. To recruit the appraisers, they were FIGURE 12. Comparison between POCR and (CBF, CF) performance
metrics
firstly asked to create their user profile and verify the
usefulness of the recommended courses. We presented our
empirical study to two departments at the University of
Portsmouth, CS (Computer Sciences) and BAM (Business
and Management). Since these participants differed in their
majors and their academic standing, they formed a group of
diverse appraisers. Altogether, 123 appraisers were recruited
which represented a range of groups, from undergraduate to
postgraduate level, across 37 different majors. Additionally,
each appraiser was asked to modify his/her profile twice
during the evaluation process so that different courses would
be requested with each modification. This produced a yield
close to 200 cases that was used to verify the performance of
OPCR.
FIGURE 13. Comparison between POCR and UCAS performance
The three performance measure metrics mentioned in the metrics
online evaluation section were used to evaluate the results

16

2169-3536 (c) 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2889635, IEEE Access

In contrast, we used a questionnaire to evaluate both user Q1 The items recommended to me matched my interests
satisfaction and the quality of the items recommended to the Q2 This recommender system gave me good suggestions
participants. The questions were designed according to the The results are shown in Fig.15 and more than 60% of the
design guidelines and principles, and are described in more users were satisfied with the recommended courses
detail by [58]. The Likert-type scales used statements such regarding how the recommended course matched with users’
as: "Please rate the extent to which you agree/disagree with interests.
the following" and 5-point response scales have been used.
The response scales used anchors such as 1 = Strongly
Disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Strongly
Agree, as shown in Fig.14. The sample of the questions was
as follows:
Q1 Overall, I am satisfied with this recommender system.
Q2 I am convinced of the items recommended to me.
Q3 I am confident I will like the items recommended to me.
Q4 This recommender system made me more confident about
my selection/decision.
Q5 The recommended items made me confused about my
choice.
Q6 This recommender system can be trusted. FIGURE 15. Accuracy Questions

1.2. Familiarity
Familiarity captures how well the users know some of the
recommended items. OPCR used an ontology-based
recommendation technique to recommend the most relevant
items to users. The users were asked, “are some of the
recommended items familiar to you?” The responses showed
that 65% had obtained recommendations which included
some familiar items, and 35% of the users said the results
included new items, as shown in Fig.16.

FIGURE 14. Attitudes Questions

The results showed that 81% of the participants were

satisfied with the recommendations they received. Ontology-
based recommendations helped the users to obtain a more
suitable recommendation. Moreover, 66% of the participants
agreed that the recommendation system had helped them to
make the right decision without making them feel confused
about what was an appropriate choice. We have considered
many of the other factors that are required to obtain an FIGURE 16. Familiarity Question
accurate result regarding the quality of the recommended
item, and the user satisfaction of the OPCR as follows: 1.3. Novelty
1. Quality of Recommended Items Novelty is one of the important indicators of user
1.1 Accuracy satisfaction as it helps users in the decision-making
Questions regarding accuracy evaluated how likely it was process[59]. OPCR provided the users with
that users would see that the course recommended to them recommendations that included novel items which were not
matched their interest (e.g. the location of the university, the expected because ontology mapping is able to link all of the
financial budget). The second question about the accuracy attributes in the course profiles and user profiles.
measurement was whether the system recommended good Recommendations were included for novel items and also
suggestions that would help with the decision-making helped the user to discover new items, according to the
process. The accuracy questions were as follows: results of the user's responses to the novelty questions below
as shown in Fig.17,

17

2169-3536 (c) 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2889635, IEEE Access

Q1 The items recommended to me are novel asked the following questions. The results are shown in
Q2 This recommender system helped me discover a new Fig.19.
course Q1 This recommender system allows me to tell what I
like/dislike.
Q2 This recommender system allows me to modify my
taste profile.
Q3 This recommender system explains why the courses
have recommended to me.

FIGURE 17. Novelty Questions

1.4. Diversity
The course domain in the recommendation system is
different from that of other domains, such as news and FIGURE 19. Interaction Adequacy Questions
movies [60]. OPCR mainly recommended courses based on
content-based filtering, which measures the similarity
between the user profile and the item. The recommendations V. CONCLUSION AND FUTURE WORK
are similar to each other because the ontology mapping Searching and finding an item that is relevant to the user
technique will not allow irrelative items to appear with the is a huge challenge. Choosing a higher education course at
recommendation items. We asked two questions to university is a massive decision for students. The
understand whether the recommendations had diverse items recommendation system in education plays a vital role in
and how similar the recommended items were to each other. overcoming the problem of information overloading, and
The results in Fig.18 show that more than 65% of the helps the students to find relevant and useful courses from a
recommendations items have no diversity. large number of online course resources that are available on
Q1 The items recommended to me are diverse. the internet.
Q2The items recommended to me are similar to each other The current approaches to filtering have many limitations.
To generate a comprehensive knowledge of the
recommended items, information from multiple heterogenic
sources needs to be mapped and linked. This paper proposes
a novel approach to the recommendation system, which
combines CBF and CF, supported by ontology similarity.
OPCR algorithms are used to recommend university courses
to a target student based on the user’s interest and the choices
made by similar students. The experiments showed that
ontology matching is a desirable tool for making a
recommendation to a target student, and it can be seen that
the proposed approach can obtain better results, including
greater user satisfaction and accuracy, than other approaches.
Furthermore, the proposed approach can help to combat
information overloading and the problems faced by new
FIGURE 18. Diversity Questions
users by using ontology similarity between the users’
2. Interaction Adequacy profiles. Furthermore, using the ontology-based integration
OPCR is a flexible system that can dynamically modify approach to integrate data from multiple heterogeneous
any part that is related to the recommender engine or user sources will help the system to provide a comprehensive
profile. The user can give a rating for the recommended recommendation to users.
course, with the scale of the rating adjusted from (1-5). To Throughout the experiments, we noticed that building a
measure how interactive the system is with the user and how new hybrid recommendation system which combines CF and
satisfied the user is with the user interface, the users were CBF utilising ontology improves the information

18

2169-3536 (c) 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2889635, IEEE Access

overloading problem. Moreover, the ontology mapping and recommendation system based on ontology,” Proc. - Int.
recommendation filter algorithms were incorporated to Conf. Mach. Learn. Cybern., vol. 3, pp. 1168–1173, 2013.
improve the accuracy of the recommendation and increase
the user stratification of the recommendation. In addition, [7] Z. K. Zhang, T. Zhou, and Y. C. Zhang, “Personalized
this approach improves the new user problem in the CF by recommendation via integrated diffusion on user-item-tag
tripartite graphs,” Phys. A Stat. Mech. its Appl., vol. 389,
incorporating ontology similarity into the proposed method.
no. 1, pp. 179–186, 2010.
It was found that using dynamic ontology mapping to link
the course profiles and student profile with the job profile [8] I. Garcia, L. Sebastia, and E. Onaindia, “On the design of
helped to provide comprehensive knowledge about the individual and group recommender systems for tourism,”
course that was not only more relevant to the student Expert Syst. Appl., vol. 38, pp. 7683–7692, 2011.
oncology-based course recommender system, but also
successfully brought a new dimension of ontology domain [9] D. Jannach, M. Zanker, A. Felfering, and G. Friedrich,
knowledge about the student and the course resources into “Recommender Systems An Introduction,” Teach. RS, pp.
the recommendation process. 1–335, 2011.
In future, we will enrich our repository by absorbing more
course and user information and heterogeneous data sources. [10] R. Ren, L. Zhang, L. Cui, B. Deng, and Y. Shi,
In addition, we plan to incorporate additional user contexts, “Personalized Financial News Recommendation
e.g., available student behaviour, learning style and learning Algorithm Based on Ontology,” Procedia - Procedia
Comput. Sci., vol. 55, pp. 843–851, 2015.
interests into the recommendation process in order to make
the system more comprehensive and intelligent. We may
[11] G. Linden, B. Smith, and J. York, “Amazon.com
employ more feedback information from students for
recommendations: Item-to-item collaborative filtering,”
effective courses and improve the student model based on IEEE Internet Comput., vol. 7, no. February, pp. 76–80,
students’ feedback and consider more aspects and techniques 2003.
related to recommender systems. We plan to carry out more
experiments with a variety of actual students from different [12] W. Y. W. Yang, Z. W. Z. Wang, and M. Y. M. You, “An
departments and from various academic backgrounds in improved collaborative filtering method for
order to prove the flexibility of our proposal. recommendations’ generation,” 2004 IEEE Int. Conf. Syst.
Man Cybern. (IEEE Cat. No.04CH37583), vol. 5, pp.
4135–4139, 2004.

REFERENCES [13] B. Sarwar, G. Karypis, J. Konstan, and J. Reidl, “Item-

based collaborative filtering recommendation algorithms,”
[1] M. E. Ibrahim, Y. Yang, and D. Ndzi, “Using ontology for 2001.
personalised course recommendation applications,” Lect.
Notes Comput. Sci. (including Subser. Lect. Notes Artif. [14] Y. Lee, “Recommendation System Using Collaborative
Intell. Lect. Notes Bioinformatics), vol. 10404, pp. 426– Filtering,” p. 49, 2015.
438, 2017.
[15] X. Jin and B. Mobasher, “Using semantic similarity to
[2] R. G. Apaza, E. V. Cervantes, L. C. Quispe, and J. O. enhance item-based collaborative filtering,” Proc. 2nd
Luna, “Online courses recommendation based on LDA,” IASTED Int. Conf. Inf. Knowl. Shar., no. May, pp. 1–6,
CEUR Workshop Proc., vol. 1318, pp. 42–48, 2014. 2003.

[3] R. Farzan and P. Brusilovsky, “Social navigation support [16] R. Ljung et al., “Content-Based Recommendation
in a course recommendation system,” in Lecture Notes in Systems,” in The Adaptive Web, vol. 9, no. 4, 2010, pp.
Computer Science (including subseries Lecture Notes in 325–341.
Artificial Intelligence and Lecture Notes in
Bioinformatics), 2006, vol. 4018 LNCS, pp. 91–100. [17] J. K. Tarus, Z. Niu, and G. Mustafa, “Knowledge-based
recommendation: a review of ontology-based
[4] F. Le Roux, E. Ranjeet, V. Ghai, Y. Gao, and J. Lu, “A recommender systems for e-learning,” Artif. Intell. Rev.,
Course Recommender System Using Multiple Criteria pp. 1–28, 2017.
Decision Making Method,” International Conference on
Intelligent Systems and Knowledge Engineering, no. 1. pp. [18] P. C. Chang, C. H. Lin, and M. H. Chen, “A hybrid course
346–350, 2007. recommendation system by integrating collaborative
filtering and artificial immune systems,” Algorithms, vol.
[5] J. Cuseo, “Academic Advisement and Student Retention: 9, no. 3, 2016.
Empirical Connections & Systematic Interventions,” Natl.
Acad. Advis. Assoc., p. 25, 2002. [19] H. Zhang, H. Yang, T. Huang, and G. Zhan, “DBNCF:
Personalized courses recommendation system based on
[6] C. Y. Huang, R. C. Chen, and L. S. Chen, “Course- DBN in MOOC environment,” Proc. - 2017 Int. Symp.
Educ. Technol. ISET 2017, pp. 106–108, 2017.
19

2169-3536 (c) 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2889635, IEEE Access

[20] G. Adomavicius and A. Tuzhilin, “Toward the next [35] M. H. Hsu, “A personalized English learning
generation of recommender systems: A survey of the state- recommender system for ESL students,” Expert Syst.
of-the-art and possible extensions,” IEEE Trans. Knowl. Appl., vol. 34, no. 1, pp. 683–688, 2008.
Data Eng., vol. 17, no. 6, pp. 734–749, 2005.
[36] H. Kadima and M. Malek, “Toward ontology-based
[21] J. Bobadilla, F. Ortega, A. Hernando, and A. Gutiérrez, personalization of a recommender system in social
“Recommender systems survey,” Knowledge-Based Syst., network,” Proc. 2010 Int. Conf. Soft Comput. Pattern
vol. 46, pp. 109–132, 2013. Recognition, SoCPaR 2010, vol. 5, pp. 119–122, 2010.

[22] S. Wang and K. Sapporo, “Topic-Oriented Query [37] K. Meehan, T. Lunney, K. Curran, and A. McCaughey,
Expansion for Web Search Categories and Subject “Context-aware intelligent recommendation system for
Descriptors,” in Proceedings of the 15th international tourism,” 2013 IEEE Int. Conf. Pervasive Comput.
conference on World Wide Web, 2006, pp. 1029–1030. Commun. Work. PerCom Work. 2013, no. March, pp. 328–
331, 2013.
[23] L. R. Khan, “Ontology-based Information Selection,” no.
August, 2000. [38] N. Y. Asabere, “Towards a viewpoint of context-aware
recommender systems (CARS) and services,” Int. J.
[24] The Universities and Colleges Admissions Service in Comput. Sci. Telecommun., vol. 4, no. 1, pp. 10–29, 2013.
United Kingdom, “UCAS.” .
[39] M. Balabanović and Y. Shoham, “Fab: content-based,
[25] F. Carballo, “Masters ’ Courses Recommendation : collaborative recommendation,” Commun. ACM, vol. 40,
Exploring Collaborative Filtering and Singular Value no. 3, pp. 66–72, 1997.
Decomposition with Student Profiling,” no. November,
2014. [40] J. Sandvig and R. Burke, “Aacorn: A CBR recommender
for academic advising,” Tech. Rep. TR05-015, 2005.
[26] H. Zhang, T. Huang, Z. Lv, S. Y. Liu, and Z. Zhou,
“MCRS: A course recommendation system for MOOCs,” [41] Y. Huang, “A Hybrid Recommender: Study and
Multimed. Tools Appl., vol. 68, no. Icat2e, pp. 1–19, 2017. implementation of course selection recommender engine,”
2017.
[27] N. Bendakir and E. Aimeur, “Using association rules for
course recommendation,” 2006. [42] T. R. Gruber, “A translation approach to portable ontology
specifications,” Knowl. Acquis., vol. 5, no. 2, pp. 199–220,
[28] M. M. Lotfy and a a Salama, “Subject Recommendation 1993.
Using Ontology for Computer Science ACM Curricula,”
Int. J. Inf. Sci. Intell. Syst., vol. 3, no. 1, pp. 199–205, 2014. [43] N. Guarino, D. Oberle, and S. Staab, “Handbook on
Ontologies,” 2009.
[29] Hongji Yang, Zhan Cui, and P. O’Brien, “Extracting
ontologies from legacy systems for understanding and re- [44] G. Antoniou, F. Van Harmelen, and S. Edition, “A
engineering,” Proceedings. Twenty-Third Annu. Int. semantic Web primer,” Choice Rev. Online, vol. 46, no.
Comput. Softw. Appl. Conf. (Cat. No.99CB37032), pp. 21– 03, pp. 46-1523-46–1523, 2008.
26, 1999.
[45] M. Grazia, D. Bono, G. Pieri, and O. Salvetti, “WP9: A
[30] K. Zhou, S.-H. Yang, and H. Zha, Functional matrix Review of Data and Metadata Standards and Techniques
factorizations for cold-start recommendation. 2011. for Representation of Multimedia Content,” 2004.

[31] Y. Lee, “An Intelligent Course Recommendation System,” [46] D. Werner, C. Cruz, and C. Nicolle, “Ontology-based
Smart Comput. Rev., vol. 1, no. 1, pp. 69–84, 2011. Recommender System of Economic Articles.,” Webist, pp.
725–728, 2013.
[32] C. Obeid, I. Lahoud, H. El Khoury, and P.-A. Champin,
“Ontology-based Recommender System in Higher [47] Q. Yang, J. Sun, J. Wang, and Z. Jin, “Semantic web-based
Education,” Companion Web Conf. 2018 Web Conf. 2018, personalized recommendation system of courses
vol. 2, pp. 1031–1034, 2018. knowledge research,” Proc. - 2010 Int. Conf. Intell.
Comput. Cogn. Informatics, ICICCI 2010, pp. 214–217,
[33] X. Zhang, “Civil engineering professional courses 2010.
collaborative recommendation system based on network,”
2009 1st Int. Conf. Inf. Sci. Eng. ICISE 2009, pp. 3253– [48] R. Ambikapathy, “Ontology Based E-Learning System for
3256, 2009. a University on Semantic Web,” Int. J. Math. Comput.
Appl. Res., vol. 1, no. 2, pp. 18–27, 2011.
[34] H. Imran, M. Belghis-Zadeh, T.-W. Chang, Kinshuk, and
S. Graf, “PLORS: a personalized learning object [49] K. Bagherifard, M. Rahmani, M. Nilashi, and V. Rafe,
recommender system,” Vietnam J. Comput. Sci., vol. 3, no. “Performance improvement for recommender systems
1, pp. 3–13, 2016. using ontology,” Telematics and Informatics, vol. 34, pp.
20

2169-3536 (c) 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2889635, IEEE Access

1772–1792, 2017.

[50] P. Singto and A. Mingkhwan, “Semantic Searching IT

Careers Concepts Based on Ontology,” J. Adv. Manag.
Sci., vol. 1, no. 1, pp. 102–106, 2013.

[51] N. F. Noy and D. L. McGuinness, “Ontology Development

101: A Guide to Creating Your First Ontology,” Stanford
Knowl. Syst. Lab., p. 25, 2001.

[52] S. V Jambhulkar and S. J. Karale, “Semantic web

application generation using Protégé
tool,” in 2016 Online International Conference on Green
Engineering and Technologies (IC-GET), 2016, pp. 1–5.

[53] K. Lang, “NewsWeeder: Learning to Filter Netnews,”

Proc. 12th Int. Mach. Learn. Conf., pp. 331–339, 1995.

[54] J. L. Herlocker, J. A. Konstan, L. G. Terveen, and J. T.

Riedl, “Evaluating collaborative filtering recommender
systems,” ACM Trans. Inf. Syst., vol. 22, no. 1, pp. 5–53,
2004.

[55] G. Shani and A. Gunawardana, “Evaluating

Recommendation Systems,” in Recommender Systems
Handbook, 2011, pp. 257–297.

[56] J. Beel and S. Langer, “A comparison of offline

evaluations, online evaluations, and user studies in the
context of research-paper recommender systems,” Lect.
Notes Comput. Sci. (including Subser. Lect. Notes Artif.
Intell. Lect. Notes Bioinformatics), vol. 9316, no. x, pp.
153–168, 2015.

[57] F. Hernández del Olmo and E. Gaudioso, “Evaluation of

recommender systems: A new approach,” Expert Syst.
Appl., vol. 35, pp. 790–804, 2008.

[58] I. Brace, Questionnaire design: how to plan, structure and

write survey material for effective market research, vol.
42, no. 06. Kogan Page, 2005.

[59] L. Zhang, “The definition of novelty in recommendation

system,” J. Eng. Sci. Technol. Rev., vol. 6, no. 3, pp. 141–
145, 2013.

[60] A. Parameswaran, P. Venetis, and H. Garcia-Molina,

“Recommendation Systems with Complex Constraints: A
Course Recommendation Perspective,” ACM Trans. Inf.
Syst, vol. 29, p. 33, 2011.

21

2169-3536 (c) 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.