Computers and Education: Artificial Intelligence 3 (2022) 100047

Contents lists available at ScienceDirect

Computers and Education: Artificial Intelligence

journal homepage: www.sciencedirect.com/journal/computers-and-education-artificial-intelligence

A systematic review of ontology use in E-Learning recommender system

Nur W. Rahayu a, b, *, Ridi Ferdiana b, Sri S. Kusumawardani b
a
Department of Informatics, Universitas Islam Indonesia, Indonesia
b
Department of Electrical and Information Engineering, Universitas Gadjah Mada, Indonesia

A R T I C L E I N F O A B S T R A C T

Keywords: Ontology and knowledge-based systems typically provide e-learning recommender systems. However, ontology
e-learning recommendation item use in such systems is not well studied in systematic detail. Therefore, this research examines the development
Learning object and evaluation of ontology-based recommender systems. The study also discusses technical ontology use and the
Ontology-based recommender system
recommendation process. We identified multidisciplinary ontology-based recommender systems in 28 journal
Ontology evaluation
Ontology methodology
articles. These systems combined ontology with artificial intelligence, computing technology, education, edu­
Ontology use cation psychology, and social sciences. Student models and learning objects remain the primary ontology use,
followed by feedback, assessments, and context data. Currently, the most popular recommendation item is the
learning object, but learning path, feedback, and learning device could be the future considerations. This
recommendation process is reciprocal and can be initiated either by the system or students. Standard ontology
languages are commonly used, but standards for student profiles and learning object metadata are rarely
adopted. Moreover, ontology-based recommender systems seldom use the methodology of building ontologies
and hardly use other ontology methodologies. Similarly, none of the primary studies described ontology eval­
uation methodologies, but the systems are evaluated by nonreal students, algorithmic performance tests, sta­
tistics, questionnaires, and qualitative observations. In conclusion, the findings support the implementation of
ontology methodologies and the integration of ontology-based recommendations into existing learning tech­
nologies. The study also promotes the use of recommender systems in social science and humanities courses, non-
higher education, and open learning environments.

hybrids (George & Lal, 2019). Secondary literature (George & Lal, 2019;
1. Introduction Tarus et al., 2018) reports that ontology and knowledge-based systems
dominate recommendation making because such systems can solve
Learning technology has grown rapidly and receives significant cold-start, diversity rating, and overspecialized recommendation prob­
support from technology development, online learning, and stake­ lems (Salehi et al., 2013; Tarus et al., 2018). This aspect is supported by
holders with complex needs (Huang et al., 2019). The online learning using ontology to simplify the relationship among data so that the ac­
market is large, so its market potential is estimated to be worth $325 curacy of the profiling process increases by 7%–15% (Middleton et al.,
billion in 2025 (a three-fold increase from 2015). Therefore, it is not 2009).
surprising that many organizations, researchers, and educators are Owing to the increasing scholarly attention to ontology for recom­
actively developing learning technologies. Numerous innovations such mendation systems, there are several comprehensive reviews of this
as wearable devices (Huang et al., 2019), social network analysis (SNA) field. For example (Bhareti et al., 2020), compares recommendation
(Cela et al., 2015), learning analytics (Araka et al., 2020; Huang et al., techniques, while (Javed et al., 2021; Tarus et al., 2018) classifies
2019), and adaptive/personalized learning (Hwang et al., 2020; Zhang recommender systems based on artificial intelligence techniques.
et al., 2020) have been proposed for technology-based learning. Additionally (George & Lal, 2019; Javed et al., 2021; Tarus et al., 2018),
Several studies related to e-learning recommender systems have classify ontology-recommender systems according to tool, ontology
discussed personalization techniques, such as ontology (Al-Yahya et al., type, and ontology representation language. Furthermore (Premlatha &
2015), collaborative filtering (Bremgartner et al., 2015), software agents Geetha, 2015; Tarus et al., 2018; Truong, 2016), categorize the systems
(Bremgartner et al., 2015; Schouten et al., 2018), fuzzy logic based on the recommendation items. Our research complements the
(Luna-Urquizo, 2019), machine learning (George & Lal, 2019), and previous studies; we discuss the development and evaluation of

* Corresponding author. Department of Informatics, Universitas Islam Indonesia, Indonesia.

E-mail addresses: [email protected] (N.W. Rahayu), [email protected] (R. Ferdiana), [email protected] (S.S. Kusumawardani).

https://doi.org/10.1016/j.caeai.2022.100047
Received 16 June 2021; Received in revised form 31 December 2021; Accepted 11 January 2022
Available online 13 January 2022
2666-920X/© 2022 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-
nc-nd/4.0/).
N.W. Rahayu et al. Computers and Education: Artificial Intelligence 3 (2022) 100047

2. Literature review
List of abbreviations
2.1. Ontology use in e-learning
ADL Advanced Distributed Learning
CAI Computer-assisted instruction An ontology is defined as a logical structure of terms used to describe a
CSCL Computer-supported collaborative learning domain of knowledge, including both the definitions of applicable terms and
IMS Instructional Management Systems their relationships (“ISO/IEC/IEEE International Standard - Systems and
ITS Intelligent Tutoring Systems Software Engineering–Vocabulary,” 2017). Ontology has three main
IVET Intelligent Virtual Environments for Training components, i.e., class, individual and property (Hebeler et al., 2009;
KBS Knowledge Based System Pollock, 2009). Examples of classes are Student and LearningObject, while
LIP Learner Information Package a student or a unit of material is an individual example. Furthermore, the
LMS Learning Management System property component describes the individuals (e.g., the student’s name
OWL Web Ontology Language and academic level (Gómez et al., 2016)) or the relationship between
PAPI Public and Private Information individuals (e.g., the property ‘belongsTo’ of a learning object with
PLE Personal Learning Environment respect to its learning level) (Mariño et al., 2018).
PRISMA Preferred Reporting Items for Systematic Reviews and In literature, ontology have been classified from various viewpoints
Meta-Analyses (Al-Yahya et al., 2015; Khadir et al., 2021; Tapia-Leon et al., 2018). In
RDF Resource Description Framework the current study, ontologies are categorized based on their use in
RDFS Resource Description Framework Schema e-learning environments. Examples of such uses are as follows:
SNA Social Network Analysis
W3C World Wide Web Consortium (1) Modeling the curriculum, e.g., mapping the relationships among
learning objects, learning goals, and the objectives of the study
program (Al-Yahya et al., 2015), and providing a correct curric­
ulum representation that can be understood by educators, stu­
ontology-based recommender systems. Second, we detail the ontology dents, and computers (Tapia-Leon et al., 2018).
components, standards related to ontology models, the ontology col­ (2) Data integration, for example, by integrating and expanding the
laborators, and the recommendation techniques. Third, we map the ontology domain from one discipline to another (Tapia-Leon
variety of recommendation items and the recommendation process that et al., 2018).
are specifically generated by ontology-based recommendation systems. (3) A description of the domain and learning activities. The domain
Hence, this review addresses the following research questions: ontology contains a data model of the course content and learning
activities that can be used in many features of the system. Content
• RQ1 How can we build an ontology-based recommendation system retrieval (i.e., ontology-based queries on the learning object’s
for e-learning? metadata and semantics) (Al-Yahya et al., 2015; Tapia-Leon
• RQ2 What are the state-of-the-art recommendation items for et al., 2018), learning assessment (i.e., mapping student learning
ontology-based recommendation systems? outcomes into the ontology domain), and feedback (i.e., student’s
• RQ3 How do e-learning recommender systems work? answer verification and system guidance on the assessment pro­
• RQ4 What is the role of ontology in the recommendation process? cess) (Al-Yahya et al., 2015) are examples of such usage.
• RQ5 What are the ontology collaborators in the e-learning recom­ (4) Data storage of student profiles (personal data) and learning
mender systems? performance (Al-Yahya et al., 2015). Such performance data
• RQ6 What are the evaluation techniques and the evaluation results of stores students’ grades and learning progress, and student pro­
the e-learning recommender systems? files are stored in either a customized or standardized format. The
specification standards are IEEE Public and Private Information
This study discusses and classifies the primary studies related to (PAPI) for Learner and the IMS1 Learner Information Package
ontology for recommendation systems. We conducted the research by (LIP).
applying the statement of the Preferred Reporting Items for Systematic (5) Recommendation tools for personalized learning; for example,
Reviews and Meta-Analyses (PRISMA) (Page et al., 2021). The material recommendation of learning objects (Al-Yahya et al., 2015),
was gathered using a broad automated search method from all journal learning paths (Premlatha & Geetha, 2015), and learning cus­
articles published in Institute of Electrical and Electronics Engineers tomization for students with disabilities (Tapia-Leon et al., 2018).
(IEEE) Xplore, JSTOR, Proquest, SAGE Journals, Science Direct,
SpringerLink, and Taylor & Francis Online during 2010–2020. 2.2. Learning technology
By mapping the latest ontology-based recommender system, this
study aims to contribute to the extant body of knowledge on ontology for Digital technology as an e-learning application is also called
e-learning. Furthermore, the study findings will provide the research instructional technology (Huang et al., 2019) or learning technology
community with a basis to advance the use of ontology and further (Huang et al., 2019; West, 2018). The last term is used in this current
develop e-learning recommender systems, using a comprehensive study as an umbrella term for a wide range of related technologies
methodology. (seeAppendix A). Learning technology and nondigital learning tools (e.
The remaining sections of this paper continue as follows. Sections 2 g., flash cards) have been used to facilitate learning. Numerous learning
highlight literature review related to ontology use, learning technology technology innovations have gained popularity owing to the numerous
and e-learning recommender system. Section 3 and 4 present the related challenges posed by students’ differences in learning capabilities and
work and the review methodology, respectively. In Section 5, we pro­ technological advances.
vide the extraction results and analysis to answer RQ1-RQ6. Section 6 is Some learning technologies do not contain recommendation items
dedicated to discussing implications, research agendas and limitations. (e.g., traditional mind map tools and blogs), but others offer recom­
Finally, Section 7 concludes the paper. mendation processes. Students’ backgrounds, interests, and

1
IMS stands for Instructional Management Systems.

2
N.W. Rahayu et al. Computers and Education: Artificial Intelligence 3 (2022) 100047

expectations appear at different levels and in diverse contexts, while search was carried out on peer reviewed articles published with the
technology develops very rapidly and increasingly needed (Huang et al., keyword “review,” “ontology,” “recommend*,” “personal*,” and “learning.”
2019; Hwang et al., 2020). (Dron & Anderson, 2021) explicitly stated We selected articles published between 2015 and 2021 because a study
that learning analytics, artificial intelligence, and collaborative systems of articles between 2011 and 2015 concluded that e-learning recom­
will be the most significant innovations. mendation systems are mostly collaborative filtering, content-based or
context-based systems (Klašnja-Milicevic et al., 2015). The search
2.3. Recommender system in e-learning domain returned 64 articles, and then abstract selection was performed to find
related studies. We excluded 30 studies not in the e-learning domain, 25
The recommender system is defined as software tools and techniques articles that were not purely systematic reviews, and 3 articles that did
that provide suggestions for items to be of use to a user (Klašnja-Milicevic not discuss recommender systems nor ontology. Thus, we found six very
et al., 2015). The systems choose the best relevant item by using the relevant articles in our study. We also added three other review articles
dependency principle between user-based activities and item-based ac­ that we knew about before. Hence, we examined nine articles in total.
tivities (Aggarwal, 2016). In the e-learning domain, such systems are Below is a brief description of each article to find the research gap for
used for personalization to achieve student engagement (Huang et al., our current study, and a summary is presented in Table 1.
2019), increase academic achievement (Harley et al., 2018), and help (Al-Yahya et al., 2015) has classified ontology by its usage in edu­
problematic students (Huang et al., 2019). cation. The categories are curriculum modeling, object domain
In the literature, various classifications of recommender systems in ontology, task-ontology, student data, and e-learning service. The study
the e-learning domain were found. Using technique-based classification also found that the ontology was commonly used for learning objects
(Sinha & Dhanalakshmi, 2019), proposed six categories, namely, and link/navigation personalization. Likewise (Bremgartner et al.,
content-based, collaborative filtering, knowledge-based systems (KBS), 2015), reviewed the adaptation techniques in virtual learning environ­
demographic systems, community-based, and hybridized systems (Sinha ments (e.g., distance education and blended courses) by using the
& Dhanalakshmi, 2019). Similarly (Tarus et al., 2018), described 11, constructivist learning theory. The study concluded that ontology is one
namely, content-based, collaborative filtering, KBS, demographic-based, of the popular artificial techniques, besides software agent and fuzzy
utility-based, context-aware, trust-aware, fuzzy-based, social logic. Ontology is a well-known technique that uses student profile to
network-based, group-based, and hybrid. Focusing on the e-learning form student groups in the context of collaborative learning. Further,
domain (Klašnja-Milicevic et al., 2015), described four categories, i.e., these findings are similar to (Mohemad et al., 2017), which found that
content-based, collaborative filtering, matrix and tensor factorization, ontology, multimedia, data mining, and neural networks are popular
and association rule mining. The current study adopts the classification techniques for supporting students with learning disabilities. However
given by (Aggarwal, 2016) as below. (Bremgartner et al., 2015), suggested that the development of an
ontology-based system in the e-learning domain should still consider
(1) The basic model of the recommender system uses interaction or pedagogical aspects (educational theory) and technological innovations.
attribute data. Examples are the collaborative filtering recom­ Technically, the benefits and drawbacks of each recommendation
mender, which uses rating data (e.g., ratings of learning objects technique have been discussed in other systematic reviews. For example
and shopping items) and the content-based recommender, which (Bhareti et al., 2020), compares content-based, collaborative filtering,
uses users’ attributes (profiles) or items’ attributes (keywords). KBS, ontology-based, neural network, deep learning, random forest, and
Additionally, KBS recommenders (i.e., those that use active user hybrid techniques (mostly combinations of content-based and collabo­
specifications of their needs and interests), demographic-based rative filtering). Similarly (Javed et al., 2021), also classifies
recommenders (i.e., applying user demographics), and hybrid context-aware recommender systems based on content-based, collabo­
recommenders (i.e., combining various types of recommenders) rative filtering, KBS/ontology-based, and hybrid techniques (mainly
are also included in this category. combinations of content-based and collaborative filtering). Similar
(2) Domain-specific recommender systems. Examples are context- comparison studies were also conducted by (Tarus et al., 2018) but with
based/context-aware recommenders, which use location, sea­ the addition of context-aware, fuzzy-based, and trust-aware techniques.
son, and social data; time-sensitive recommenders, which These studies differ from ours because the scope of our study does not
consider changes in ratings/interests over time; location-based limit ontology collaborators from the field of artificial intelligence only.
recommenders, which use spatial locality; and social recom­ As a multidisciplinary study, the e-learning recommender system should
menders, which use network structures, social cues, and tags. integrate other disciplines (e.g., education and educational psychology).
Moreover, our study discussed the technical details related to the
The KBS has two main components, i.e., systems and knowledge. In development of the recommender system in various learning technolo­
contrast to information retrieval systems or content-based recommender gies and evaluation methods.
systems that rely upon keywords, KBS stores the knowledge component Related to ontology, previous systematic reviews grouped ontology-
in a knowledge base of relational attributes (Aggarwal, 2016). Then, the based recommender systems by tool (George & Lal, 2019), ontology
system component applies an inference mechanism to the knowledge type, and ontology representation language (George & Lal, 2019; Javed
base to compile recommendations (Ertel, 2017). Fig. 1 shows the et al., 2021; Tarus et al., 2018). Furthermore (George & Lal, 2019), also
simplified hierarchy of the relationships among the learning technolo­ described issues solved by the ontology-based recommender system.
gies (digital learning tools), recommender system, and ontology model. Although these studies were similar to ours, we have added details on
the ontology components, standards related to the ontology model, and
3. Related work inference engines and recommendation techniques that perform
reasoning tasks on the ontology.
The study of recommender systems and adaptive/personalized Recommendation items were reviewed in previous systematic re­
learning has grown rapidly. To understand the development of the views. For example, recommendations for learning objects in the form of
recommender system in the e-learning domain, we searched for related content (Premlatha & Geetha, 2015; Tarus et al., 2018; Truong, 2016),
systematic reviews and mapping studies in the Scopus database. The link (Premlatha & Geetha, 2015), format (Premlatha & Geetha, 2015;
Truong, 2016), learning path (Premlatha & Geetha, 2015; Tarus et al.,
2018), student requirement (Javed et al., 2021), and teaching strategy
2
education technology category based on (Huang et al., 2019). (Truong, 2016). Because the source data are not yet specific using
3
categorization of recommender systems based on (Aggarwal, 2016). ontology, our study tries to map the variation of recommendation items

3
N.W. Rahayu et al. Computers and Education: Artificial Intelligence 3 (2022) 100047

Fig. 1. Relationship among the learning technology,2 recommender system,3 and ontology.

generated by ontology-based recommendation systems. Moreover, our artificial intelligence. Therefore, we concentrated our search on journal
research also discusses the recommendation process for each recom­ publications in these two fields. After screening through these re­
mendation item. quirements, we found 1,962 articles.
Abstracts were subsequently scanned to select articles of primary
4. Materials and methods studies delivering an e-learning domain with adaptive/personalized
content, resulting 72 articles left for full-text exclusion review. Some
This research used the PRISMA flow diagram. PRISMA is used to articles were then excluded for several reasons, such as not containing
identify, select, appraise, and synthesize studies of the state of knowl­ an ontology, using ontologies as a modelling tool without recommen­
edge in a field (Page et al., 2021). Since its publication in 2009, PRISMA dation, providing prediction only, and discussing personalization
has been widely adopted in various disciplines, which includes reviews outside of the technical perspective. Finally, as the details are shown in
of learning technology in the education domain (Bedenlier et al., 2020; Figs. 2 and 28 articles containing ontology-based personalization tech­
Palalas & Wark, 2020) and reviews of recommender systems/artificial niques were ready to be extracted (see details in Appendix B).
intelligence in education (Chen et al., 2020; Deschênes, 2020). This
study uses the latest version of PRISMA (“the PRISMA 2020 statement”) 4.3. Data extraction
as a reference for the criteria. PRISMA includes several checklists, such
as eligibility criteria (inclusion and exclusion criteria), search strategy, Paper extraction uses the following criteria to obtain comprehensive
data collection process, study selection, synthesis method, and synthesis information:
result (Page et al., 2021).
(1) Article profile: author(s); author(s) affiliation; journal name;
4.1. Literature search publication title; publication year; publisher; and ScimagoJR
2020 quartile.
This study searches the primary studies using a broad automated (2) Technology: disciplines; model/system’s name; research output
search method. The keyword string4 applied in the metadata search (model, prototype); type of learning technology; and tools used
consisted of three components and included wildcard characters (if for system development.
applicable). Specifically, we searched for5 “(learner OR student) AND (3) Ontology model: learner style; learning object’s metadata;
(adapt* OR personali* OR recommend* OR intelligent*) AND ontology” for ontology methodology; ontology language; ontology models;
journal articles published from 2010 to 2020. This initial search recommendation item; recommendation technique; semantic
retrieved 90,619 articles across 7 databases (i.e., IEEE Xplore, JSTOR, web tool; student model; student model’s data; and student
Proquest, SAGE Journals, Science Direct, SpringerLink, and Taylor & profile.
Francis Online). (4) Inference mechanism: reasoner method; reasoner tool; reasoner
type (OWL,6 RDFS,7 rule); rule language (if any); sample knowl­
edge; and sample rule.
4.2. Article selection
(5) Performance evaluation: baseline characteristics; case study;
control group; country; course/topic/domain; course/experi­
For inclusion in the study, the reviewed papers must fulfill the re­
ment duration; evaluation method; evaluation result; number of
quirements of being both English journal articles and belong to com­
participants; participant type; and study variables.
puter science/artificial intelligence subjects. We believe that ontology
utilization is rarely discussed outside the field of computer science and
5. Result and analysis

4 All 28 extracted articles were found in the Q1 and Q2 quartiles of

Except for the IEEE Xplore database, which incorporates keywords into the
abstract field, and the ScienceDirect database, which incorporates keywords
fourteen journals: Computers & Education, Computers in Human Behavior,
into the title, abstract, and keyword sections. Data & Knowledge Engineering, Educational Technology & Society, E-
5
ScienceDirect database can recognize spelling variants but does not support
wildcards (*); thus, we use the search string: (learner OR student) AND
6
(adaptive OR adaptivity OR personalization OR recommendation OR recom­ OWL is an abbreviation for Web Ontology Language.
7
mend OR intelligent) AND ontology. RDFS stands for “Resource Description Framework Schema.”

4
N.W. Rahayu et al. Computers and Education: Artificial Intelligence 3 (2022) 100047

Table 1
Summary of systematic review of ontology use in the e-learning recommender system.
Authors Database source Keyword search Number Time span Focus
of articles

Al-Yahya et al. ACM, IEEE, Science Direct, and ontology, learning 33 2000–2012 Classification of ontology use (curriculum modelling,
(2015) Web of Science object-domain ontology, task-ontology, student data
and e-learning service)
Bhareti et al. – – – Comparison of the advantages and disadvantages of
(2020) each AI-based recommendation technique
Bremgartner IEEE Xplore, ACM Digital Library, profile, profiling, profilling, 105 out of no time Adaptation techniques in VLE based on the
et al. (2015) Scopus, Elsevier, EI Compendex personalization, personalization, 951 constraint constructivist approach; the role of artificial
adaptation, adaptive, adaptivity, context- intelligence in adaptation
awareness, VLE, virtual learning
environments, constructivism,
constructivist, CSCLa, computer-suported-
collaborative- learning, collaborative
learning
George and Lal Springer, Elsevier, IEEE, ACM recommender system, e-learning, hybrid 108 2010–2018 Ontology-based recommendation system classification
(2019) Digital Library, and Google recommender system, ontology, ontology- by used tool, language, recommendation item, and
Scholar based recommender system, learning ontology type; description of issues and solutions
management system, knowledge, machine addressed by the ontology-based recommender system
learning
Javed et al. – – 39 2000–2014 Classification of context-aware recommender systems
(2021) by artificial intelligence techniques; categorization of
the ontology-based recommendation system by
recommendation technique, ontology type, ontology
representation language and recommendation item
(student requirement, learning object, learning path)
Mohemad et al. ScienceDirect, Scopus, Springer, education, special education, learning 15 2009–2017 Computer techniques to help students with learning
(2017) ACM Digital Library, and IEEE disability, dyslexia, dyscalculia, and disabilities (dyslexia, dyscalculia, and dysgraphia)
Xplore dysgraphia
Premlatha and – – – Various adaptations of learning objects; use of related
Geetha ontologies
(2015)
Tarus et al. Web of Science (SCI), Engineering ontology, recommender, recommendation, 36 out of 2005–2014 Recommendation techniques used in ontology-based
(2018) Index (EI), Science Direct, EBSCO learning, education, knowledge e-learning, 229 recommendation system in e-learning; classification of
Academic Search Premier, and online learning recommended systems based on ontology type and
Springer, IEEE Xplore, and ACM representation language; description of
Digital Library recommendation items
Truong (2016) Google Scholar, Scopus, and learning styles (or) style, measurement | 51 2004–2012 Process of integrating learning styles into adaptive e-
Science Direct. classification | prediction | evaluation | learning; description of potential data sources for
modelling | detection | recognition, learning style prediction; algorithms for classification of
adaptive | personalized | individualized | learning styles; recommendation items by using
personalization, integration | application | learning styles
using, automatic, learning system |
learning management system, intelligent
tutoring system, student | user modelling,
online | e-learning, computer-assisted
learning, adaptive instructions, adaptive
hypermedia, artificial intelligent,
education data mining
a
CSCL is an abbreviation for Computer-supported collaborative learning.

Learning and Digital Media, Education and Information Technologies, En­ 5.1. RQ1 how can we build an ontology-based recommendation system
gineering Applications of Artificial Intelligence, Expert Systems with Appli­ for e-learning?
cations, Future Generation Computer Systems, Human-centric Computing
and Information Sciences, IEEE Access, IEEE Transactions on Learning The recommender systems in the primary studies used ontology as a
Technologies, Interactive Learning Environments, and Knowledge-Based knowledge base. Models or prototypes of recommender systems have
Systems. The extraction results showed that the articles were written by been designed or implemented in various learning technologies (West,
78 unique authors. An article was written by three authors on average, 2018). The primary studies identified 14 learning technologies, as listed
but articles written by five authors were also found (Muñoz et al., 2015; in Table 2. Moreover, the table shows that the number of recommen­
Porcel et al., 2018; Sánchez-Vera et al., 2012). The authors belonged to dation prototypes was twice that of the recommendation model. Among
36 research institutions, most prolifically from the University of Gran­ these technologies, the most popular prototypes are classified as
ada and the University of Alcala (both in Spain). learning management systems (LMS), intelligent tutoring systems (ITS),
The distribution of the primary studies is dynamic but shows an computer-assisted instruction (CAI), and personal learning environ­
increasing trend (see Fig. 3). Furthermore, the number of journal pub­ ments (PLE).
lications during 2018–2019 has almost doubled. The exception occurred Most studies use recommender systems primarily in higher educa­
in 2020 and was possible because the selection process was carried out tion, with the exception of (Mustafa et al., 2019) in preschool. Most of
between October 2020 and February 2021. the studies are computer science courses (eight studies): C++

5
N.W. Rahayu et al. Computers and Education: Artificial Intelligence 3 (2022) 100047

Fig. 2. PRISMA flow diagram for article selection.

Table 2
Type of learning technology (detail description could be found in Appendix A).
Learning technology Model Prototype

Academic advisor – EDUC8 (Iatrellis et al., 2019)

Cloud e-learning Rani et al. (2015) –
CAI – electronic Justice Relationship
Management (Capuano & Toti,
2019); (Benlamri & Zhang, 2014)
CSCL Khaled et al. (2019) Porcel et al. (2018)
Courseware generator Labib et al. (2017) Piagos (Ullrich & Melis, 2010)
E-assessment Belcadhi (2016) Ontology eLEarning (
Sánchez-Vera et al., 2012)
Instructional design Cultural variables –
advisor tool ontology (Savard
et al., 2020)
ITS AC-Ware (Grubišić Cognitive Ontology of
et al., 2013); ( Educational Systems (Caravantes
Fig. 3. Year of publication of 28 selected articles. Clemente et al., & Galán, 2011); CogSkills2 (
2011) Mustafa et al., 2019); IP
Addressing: Problem-Based
programming language (Benlamri & Zhang, 2014), computer as a sys­ Learning (Jevremovic et al.,
tem (Grubišić et al., 2013), computer network (Demaidi et al., 2018; 2017);
Jevremovic et al., 2017), computer security (AlAgha, 2012), database Intelligent Virtual ON-SMMILE (Yago Student Modeling Agent (
Environments for et al., 2018) Clemente et al., 2014)
administration (Muñoz et al., 2015), web technology (Nitchot et al.,
Training (IVET)
2018) and several courses for third year students (Tarus et al., 2017). LMS INES (Fonte et al., OntoPeFeGe (Demaidi et al.,
Other studies use recommender systems in science courses (Ullrich & 2012); (Demaidi 2018); OntoSakai (Muñoz et al.,
Melis, 2010), engineering courses (Clemente et al., 2014; Gómez et al., et al., 2018) 2015)
2016; Sánchez-Vera et al., 2012), and medicine and nursing courses PLE – KnowledgePuzzle (AlAgha,
2012); MyTeleMap (Nitchot et al.,
(Gómez et al., 2016). In comparison, we found only one study that 2018)
utilized recommender systems in the social sciences (Caravantes & Social learning – ILEARN (Khaled et al., 2019),
Galán, 2011), and none in the humanities. network SharingNotes (Porcel et al., 2018)
The whole process of ontology creation utilizes several tools/envi­ Ubiquitous learning Gómez et al. (2016) –
VLE Kurilovas et al. (2014)
ronments. Protégé is still a popular development environment, while
–

some ontologies have been created using the OWL API (Rani et al., 2015)
and designed in Motp (Savard et al., 2020). Moreover (Belcadhi, 2016),
displayed graphs and class hierarchies by using a combined OWL Viz

6
N.W. Rahayu et al. Computers and Education: Artificial Intelligence 3 (2022) 100047

and OntoGraph environment. learning objectives/goals (Huang et al., 2019). Examples of learning ob­
Ontology models in the e-learning domain were developed by jects include the subject matter or content to be learned, narrative text,
ontological engineers (Khadir et al., 2021), pedagogues, and domain and problem sets.
experts (such as educators and researchers) (Boyce & Pahl, 2007). The majority of recommended items help students directly, except
Furthermore, there are various ontology methodologies available, that for the pedagogical scenario, which benefits educators directly and
is, methodologies for building ontologies, ontology reengineering, indirectly benefits students. It is reasonable because a class is commonly
ontology learning, ontology evaluation, and ontology merging (Corcho attended by many students and taught by one or few educators.
et al., 2003). The review study found that the methodologies of NeOn Therefore, recommendations are more focused on the students.
(Mariño et al., 2018; Yago et al., 2018), On-To-Knowledge (Labib et al., Furthermore, this study also finds that learning objects are still the most
2017), METHONTOLOGY (Muñoz et al., 2015), and KACTUS (Rani popular recommendation item.
et al., 2015) were used to build ontologies. This finding is intriguing
because no other ontology methodologies were explicitly mentioned in 5.3. RQ3 how do e-learning recommender systems work?
the primary studies.
A few ontology models are declared to conform to interoperability Each recommender system has its own recommendation process.
standards. For example, two student profiles (as a part of the student According to the category of recommendation items (Table 3), the
ontology) follow IEEE PAPI or IMS LIP standards. Additionally, five procedures are summarized below.
learning objects (i.e., domain ontologies) apply the IMS LOM,8 LRE AP,9
or ADL SCORM10 metadata standards. This finding is also interesting (1) Personalized learning objects: The system chooses the learning
because interoperability standards are in line with the spirit of ontology methods (e.g., blogging) and then provides an alternative
and semantic web technologies that strengthen the internet’s role as a learning object (such as textbooks or glossaries) (Gómez et al.,
web of linked data (Pollock, 2009). 2016; Kurilovas et al., 2014). Alternatively, the system provides
The current study shows that ontology language standards are being exercises according to the user’s competency level (Fonte et al.,
widely used. Although the World Wide Web Consortium (W3C) pub­ 2012; Jevremovic et al., 2017) or provides learning objects that
lishes several language standards, almost half of our primary studies match the extracted concept (Capuano & Toti, 2019).
used the OWL (Belcadhi, 2016; Benlamri & Zhang, 2014; Clemente (2) Personalized learning paths: The system verifies the student’s
et al., 2011; Demaidi et al., 2018; Gómez et al., 2016; Khaled et al., profile and instructional design before selecting the appropriate
2019; Labib et al., 2017; Mariño et al., 2018; Muñoz et al., 2015; Rani activity or procedure. Later, the system selects the sequence of
et al., 2015; Sánchez-Vera et al., 2012; Yaghmaie & Bahreininejad, activities and records each activity in the student model (Clem­
2011; Yago et al., 2018). ente et al., 2011).
(3) Personalized feedback: Students select their own level of exper­
tise and take a test. Later, the system provides feedback that
5.2. RQ2 what are the state-of-the-art recommendation items of ontology- contains a score, encouragement, and references to a specific
based recommendation systems? learning object (Belcadhi, 2016). Alternatively, the system selects
a feedback type that suits the student’s knowledge, the difficulty
Learning objects, learning paths, feedback, learning devices, and level of the test, and the number of correct answers (Demaidi
pedagogical scenarios are common recommendation items found in et al., 2018).
primary studies (see Table 3). In the current study, the term “learning (4) Personalized learning devices: After checking the strengths and
object” is synonymous with “learning resource,” “learning materials,” weaknesses of the user, the system provides related support and
and “learning content.” The learning object itself is defined as verified then recommends suitable activities or devices (Mariño et al.,
information resources (data, facts, pictures, and videos) related to the 2018).
(5) Personalized pedagogical scenarios: The system analyses the ed­
Table 3 ucator’s culture and then provides suggestions for learning sce­
Recommendation items. narios that fit the culture of prospective students. The educator
Recommendation Related Studies can then accept, reject, or change the suggestion, and their
Items response is recorded by the system (Savard et al., 2020).
Learning objects Learning content (Clemente et al., 2014; Fonte et al., 2012;
Gómez et al., 2016; Grubišić et al., 2013; Khaled et al., Personalization initiatives were found either by the system or by the
2019; Kurilovas et al., 2014; Muñoz et al., 2015; Porcel students. Overall, we conclude that the basic recommendation process is
et al., 2018; Rani et al., 2015; Tarus et al., 2017; Yaghmaie
that (1) the system checks the student model and then (2) the system
& Bahreininejad, 2011; Yago et al., 2018)
Narrative text (Capuano & Toti, 2019) provides recommendations according to the student model.
Problem set (Jevremovic et al., 2017)
Learning paths (AlAgha, 2012; Clemente et al., 2011; Iatrellis et al., 2019; 5.3.1. Recommendation technique
Nitchot et al., 2018) The current review found that various AI techniques supported se­
Feedback (Belcadhi, 2016; Demaidi et al., 2018; Sánchez-Vera et al.,
2012)
mantic reasoners in making recommendations. Three studies (Belcadhi,
Learning devices Mariño et al. (2018) 2016; Labib et al., 2017; Sánchez-Vera et al., 2012) employed solely
Pedagogical Savard et al. (2020) semantic reasoners to make recommendations; the others used different
scenarios AI techniques as follows:

(1) Software agents (Clemente et al., 2011, 2014; Fonte et al., 2012;
Kurilovas et al., 2014; Rani et al., 2015; Savard et al., 2020;
Yaghmaie & Bahreininejad, 2011)
8
LOM is an abbreviation for Learning Object Metadata. (2) Collaborative filtering (Tarus et al., 2017)
9
LRE AP is an abbreviation for European Learning Resource Exchange (3) Cognitive diagnosis method (Clemente et al., 2014)
Application Profile. (4) Fuzzy linguistics (Porcel et al., 2018)
10
ADL SCORM stands for “Advanced Distributed Learning Sharable Content (5) Generalized sequential pattern algorithm (Tarus et al., 2017)
Object Reference Model.” (6) Hierarchical Task Network Planning (Ullrich & Melis, 2010)

7
N.W. Rahayu et al. Computers and Education: Artificial Intelligence 3 (2022) 100047

(7) Natural language processing (Capuano & Toti, 2019) terms, and related terms) and educational relationships (e.g., “requires,”
“has part,” and “teaching order”).
Ontology as a knowledge base for e-learning recommender systems Furthermore, ontology models in reviewed articles could be more
has collaborated with other AI techniques, mostly with software agents. specific, such as:
All related primary studies employed multiple agents (as many as 19
agents in one study (Savard et al., 2020)). Typical software agents are (1) Ontology of the assessment rubric which stores assessment
tutoring agents, which receive information on learning objectives from criteria, categories, and assessments (Yago et al., 2018).
the student model (Clemente et al., 2014); conversational agents, which (2) Cognitive ontology which collects information about the learning
communicate with students in their daily language (Fonte et al., 2012); target (Zhong et al., 2015) or the activation states of learning
and advisor agents, which choose the correct instructional design under objects for a student (e.g., repeated, read, viewed, known, eval­
a set of rules (Savard et al., 2020). uated, and learned) (Caravantes & Galán, 2011).
For example (Clemente et al., 2014), stored the student model data in (3) Ontology of cultural variables which covers personal values,
an ontology and used a software agent to receive information about standard practices, and types of human interaction (Savard et al.,
learning objectives from the student model. Due to this collaboration, 2020).
the recommender system becomes a hybrid system. This finding is (4) Emotional ontology that stores positive, negative, ambiguous,
consistent with the findings of a previous review of the literature, which and neutral emotional data (Khaled et al., 2019).
concluded that recent recommender systems employ hybrid techniques, (5) Feedback ontology which contains course content, educators,
including collaborative filtering and other techniques. (George & Lal, exams, and problem questions and answers (Belcadhi, 2016).
2019). Unfortunately, performance comparison among hybrid tech­
niques is rarely studied. Only one primary study evaluated performance 5.4.2. Inference engine
of three techniques: (1) collaborative filtering alone; (2) collaborative Along with an ontology component, recommender systems maintain
filtering and ontology; and (3) a combination of collaborative filtering, a semantic reasoner component. A semantic reasoner, a rules engine, or
ontology, and sequential pattern mining (Tarus et al., 2017). simply a reasoner, is an inference engine that performs reasoning tasks on
the ontology to generate knowledge (Matentzoglu et al., 2015; Parsia et al.,
5.4. RQ4 what is the role of ontology in the recommendation process? 2017). Examples inference mechanisms are first-order predicate calcu­
lus, the tableau reasoning system, and chain-based rules (forward and
5.4.1. Ontology component backward) (Pollock, 2009). Two primary studies extracted in this review
The ontology is used to store data from learning objects, activities, explicitly applied forward and backward chaining methods (Fonte et al.,
cognitive, emotional states, and student data models in primary studies. 2012; Khaled et al., 2019). Furthermore, using the W3C classification,
Ontology models could consist of several classes or subontologies the reviewed studies have RDFS reasoners (Khaled et al., 2019), OWL
(Clemente et al., 2011; Yago et al., 2018) from various data sources. The reasoners (Belcadhi, 2016; Gómez et al., 2016), and rule-based rea­
data sources were either ontological resources (e.g., existing ontologies) soners (Clemente et al., 2014; Fonte et al., 2012; Muñoz et al., 2015).
or non-ontological resources (e.g., a relational database, thesauri). To Among the generated knowledge are finding prerequisite materials
build an ontology, the resources were then extracted (Santoso et al., (Fonte et al., 2012), determining the profile group based on activities
2011) or reengineered using some techniques (e.g., ontology similarity (Muñoz et al., 2015), inferring probabilities for the predicted current
(Zhao et al., 2012)). bandwidth of a student’s device (Benlamri & Zhang, 2014), and esti­
A student model itself usually contains students’ profiles, learning mating the value of the relationship between message exchanges
styles, student knowledge, and socio-technical contexts (Kurilovas et al., (Khaled et al., 2019). The knowledge was acquired using reasoner tools,
2014). Although student models are tailored to each study, such models such as Pellet (Gómez et al., 2016; Muñoz et al., 2015; Sánchez-Vera
typically contain fixed data. However (Grubišić et al., 2013), used a et al., 2012; Yago et al., 2018), HermiT (Mariño et al., 2018; Rani et al.,
Bayesian model to deal with uncertainty in students’ knowledge. For 2015), Java Expert System Shell (Fonte et al., 2012), Fact++ (Mustafa
example (Tarus et al., 2017), built an ontology that stores student et al., 2019), and the RDF11 application programming interface for
models. The model stores the student’s profile (i.e., name, sex, and age), PHP12 (Khaled et al., 2019).
learning style (active or reflective, sensing or intuitive, visual or verbal,
sequential or global), and level of student’s knowledge (beginner,
intermedia, advanced). 5.5. RQ5 what are the ontology collaborators in the e-learning
Two primary studies used student profile standards as part of the recommender systems?
student model, while the rest did not. IEEE PAPI and IMS LIP were used
in studies (Belcadhi, 2016) and (Fonte et al., 2012), respectively. Extraction results show diverse field supports recommender systems.
Furthermore, four articles declared specific learning styles, i.e., the The recommender system can be a standalone application or use envi­
Felder–Silverman Index of Learning Styles (Rani et al., 2015; Tarus ronments such as context-aware systems and cloud computing. Also,
et al., 2017), Honey and Mumford Learning Styles (Kurilovas et al., theories and methods from educational/educational psychology support
2014), and the Kolb Learning Style Inventory (Yaghmaie & Bahreini­ recommender system can be employed. For example, Bloom’s taxonomy
nejad, 2011). is used to recommend learning objects (Yago et al., 2018), feedback
To build domain ontologies of learning objects, two studies reused (Demaidi et al., 2018), and learning paths (Clemente et al., 2011).
ontologies (Mariño et al., 2018; Muñoz et al., 2015) and others reused or Moreover, there exist theories/methods (see Table 4).
re-engineered non-ontological resources from textbooks (Demaidi et al., Ontology in e-learning recommender systems is employed in
2018; Jevremovic et al., 2017) or thesauri (Capuano & Toti, 2019). Five collaboration with other disciplines. Moreover, eight studies incorpo­
studies used metadata standards explicitly, while the remainder did not. rated three distinct fields, e.g., the eJRM prototype combines ontology,
The most used standards are IMS LOM (Benlamri & Zhang, 2014; Fonte natural language processing, and storytelling (Capuano & Toti, 2019),
et al., 2012; Yago et al., 2018), followed by the LRE AP (Kurilovas et al., and ON-SMMILE integrates ontology, Bloom’s taxonomy, and educa­
2014) and the ADL SCORM (Yaghmaie & Bahreininejad, 2011). For tional modeling language (Yago et al., 2018).
example (Capuano & Toti, 2019), created a domain ontology in the legal
domain. The ontology contains thousands of concepts (e.g., legal
mediation, mediation process, and mediator obligations) that are con­ 11
RDF stands for Resource Description Framework.
nected through informative relationships (e.g., narrower terms, broader 12
PHP is an abbreviation for Hypertext PreProcessor.

8
N.W. Rahayu et al. Computers and Education: Artificial Intelligence 3 (2022) 100047

Table 4 Table 5
Ontology collaborators in ontology-based recommender systems for e-learning. Evaluation methods of personalization performance.
Discipline (role) Theory/technology/approach Evaluation Methods Evaluation variables

Computing technology (system Context-aware systems (Gómez et al., 2016; Questionnaires Perceived effectiveness (Capuano & Toti, 2019; Ullrich &
environment) Yaghmaie & Bahreininejad, 2011) Melis, 2010), usefulness (AlAgha, 2012; Ullrich & Melis,
Cloud computing (Rani et al., 2015) 2010), clarity (Demaidi et al., 2018), interestingness,
Service-oriented architecture (Yaghmaie & approachability, adaptability, suitability for diver learners
Bahreininejad, 2011) (Khaled et al., 2019)
Education/educational Learning activity (Bloom taxonomy (Clemente Qualitative Usability, effectiveness (Belcadhi, 2016)
psychology (supporting theories/ et al., 2011; Demaidi et al., 2018; Yago et al., observation
methods) 2018); Krathwohl taxonomy (Clemente et al., Descriptive statistics Pre-test and post-test score comparison (Demaidi et al.,
2011); and Harrow taxonomy (Clemente et al., 2018), activity log (AlAgha, 2012)
2011)) Inferential statistics Student behavior in the pre-test and post-test (Gómez et al.,
Cognitive psychology (AlAgha, 2012; 2016; Jevremovic et al., 2017; Mustafa et al., 2019;
Caravantes & Galán, 2011; Mustafa et al., Sánchez-Vera et al., 2012), activity log (Caravantes &
2019) Galán, 2011)
Instructional design (pedagogical scenario ( Algorithm Similarity (Khaled et al., 2019); Mean Absolute Error,
Ullrich & Melis, 2010); problem-based learning performance test Precision, and Recall (Tarus et al., 2017)
(Jevremovic et al., 2017); teaching/learning
method iCOPER D3.1 (Kurilovas et al., 2014))
Learning theory (constructivism (Kurilovas control group in the same subject. All primary studies produced positive
et al., 2014))
test results, indicating that the recommendation technique improved
Storytelling (Capuano & Toti, 2019)
Other collaborators Anthropology (Savard et al., 2020)
students’ performance, particularly those who initially did not under­
Management (Iatrellis et al., 2019) stand the material (Demaidi et al., 2018). However, it is interesting to
Social network analysis (Khaled et al., 2019) know that students with higher education tend to be more critical of the
Activity-centered design (Mariño et al., 2018) features of the systems (Capuano & Toti, 2019).
Bayesian probability theory (Grubišić et al.,
2013)
Educational Modeling Language (Yago et al., 6. Discussion
2018)
Graph theory (Nitchot et al., 2018) 6.1. Implications

6.1.1. Popular learning technology and ontology methodology

5.6. RQ6 what are the evaluation techniques and the evaluation results of
Most prototypes of the recommendation systems are standalone
the e-learning recommender systems?
software, except for OntoSakai, which is integrated into the LMS (Muñoz
et al., 2015). The recommender environment may be limited by the
As software, the ontology-based recommender system also needs
number of supported courses. We found that one study used six courses,
evaluation. This review study found that no evaluation methodology
but the majority of studies only used one. Furthermore, the resource
was used to systematically assess the quality of the developed ontology.
difficulty of integrating a recommender system with existing technolo­
However, several recommender systems use some evaluation tech­
gies may prevent integration.
niques. The recommender models are evaluated in experiments with
Learning technology can be a recommender system itself (e.g., ITS,
unreal students, while the prototypes can be evaluated by five methods:
courseware generator) or can contain a recommender system (e.g., LMS,
algorithmic performance tests, descriptive statistics, inferential statis­
VLE). Furthermore, the study revealed that LMS, ITS, CAI, and PLE are
tics, questionnaires, and qualitative observations.
the most popular recommendation systems platforms. This finding is
The recommender model might be evaluated in experiments with
consistent with a previous study that reported LMS, e-assessment, VLE,
unreal students. For example (Clemente et al., 2011; Yago et al., 2018),
ITS, and PLE as the most well-known learning technologies
conducted four experiments of their models in different scenarios. On
(Tibaná-herrera et al., 2018).
the contrary, the prototypes of the recommenders were evaluated in
Only five studies documented ontology building methodologies
terms of system usability (Capuano & Toti, 2019; Demaidi et al., 2018;
(Labib et al., 2017; Mariño et al., 2018; Muñoz et al., 2015; Rani et al.,
Gómez et al., 2016; Porcel et al., 2018; Sánchez-Vera et al., 2012),
2015; Yago et al., 2018). Three others created domain ontologies using
student satisfaction (Sánchez-Vera et al., 2012; Tarus et al., 2017),
reused or reengineered non-ontological resources (Capuano & Toti,
processing efficiency (Capuano & Toti, 2019), and system clarity
2019; Demaidi et al., 2018; Jevremovic et al., 2017). Hence, we predict
(Demaidi et al., 2018). Furthermore, the prototypes were tested in
that the remainder of the major research would have created ontologies
various academic environments, from the course level to the exam level
from scratch tailored to the specific courses. This is particularly true for
(Sánchez-Vera et al., 2012). Most of the participants are students, and
domains ontologies. The methodologies were not adequately docu­
the most popular subjects are related to computer science in higher
mented, possibly because the authors emphasized presenting recom­
education (Demaidi et al., 2018; Jevremovic et al., 2017; Muñoz et al.,
mendation techniques rather than ontology building.
2015; Tarus et al., 2017), followed by courses in medicine and nursing
Interoperability standards, on the other hand, are rarely employed.
(Gómez et al., 2016).
Interoperability standards were frequently overlooked because course
None of the primary studies explicitly described ontology evaluation
numbers were limited and most systems were built from the ground up.
methodologies. However, some articles use system comparison
However, this study shows that ontology language standards have been
(Capuano & Toti, 2019), questionnaires (either customized or the Sys­
widely applied. Therefore, the reusability of recommender systems may
tem Usability Scale) (Demaidi et al., 2018; Gómez et al., 2016;
be improved by using other standards, such as student profile standards
Sánchez-Vera et al., 2012), and other methods (see Table 5). Moreover,
and metadata standards for learning objects.
the two studies use several evaluation methods to test a single system
(Demaidi et al., 2018; Khaled et al., 2019).
6.1.2. Recommendation items in learning environments
To evaluate the results, various recommendation techniques were
Most recommendation items (64% of primary studies) were learning
also compared. For example (Jevremovic et al., 2017), compared their
objects, followed by personalized learning paths and feedback. The va­
results with those of previous research over three consecutive semesters.
riety of recommendation items has grown as technology has become
Similarly (Demaidi et al., 2018; Gómez et al., 2016), used a 40-student

9
N.W. Rahayu et al. Computers and Education: Artificial Intelligence 3 (2022) 100047

more user centric. Furthermore, a recent study found that the best courses. These tasks could be expanded by integrating instructional
learning path could lower students’ cognitive surplus (Premlatha & designs, as proposed by (Vidal-Castro et al., 2012). For example, the
Geetha, 2015) and that individualized feedback could boost affective reasoner could execute a new rule such as “Group concepts and sup­
engagement (Chen et al., 2021). porting content into ‘learning episodes’ that are not so large as to make re­
(AlAgha, 2012) benefited from an open learning environment in view and synthesis difficult but not so small as to break the flow of the
which new learning objects might be added to the browsing space. As learning process” (Vidal-Castro et al., 2012).
open learning gains popularity, we may see more learning objects, Our findings expand the use of ontologies in curriculum modeling,
learning paths, feedback, and device alternatives in the future. There­ content search and retrieval (Al-Yahya et al., 2015), and peer recom­
fore, recommended things will evolve and become more reciprocal. mendation in learning communities (Zheng et al., 2015). Furthermore,
The current study found that the recommender system generated ontology-based recommender systems do not rely solely on ontology.
recommendations independently of educators. Additionally, the system Because educational technology has advanced so quickly, the supporting
or students may initiate the recommendation process. Educators and fields are diverse. As technology improves, so do the system environ­
students can assess students’ achievement of learning gains at the end of ments. Table 4 shows how knowledge processing techniques have pro­
a course or experiment (Demaidi et al., 2018; Gómez et al., 2016; gressed from semantic reasoning to hybrid AI (see the resume in Fig. 4).
Sánchez-Vera et al., 2012). However, because learning is such a two-way Therefore, ontology-based recommender systems are more accessible to
exchange between educators and students, educators are expected to be collaborators from various disciplines.
part in the recommendation process. Additionally, despite receiving
recommendations, students may have difficulties while learning. 6.1.4. Performance evaluation
Therefore, educators should support students to resolve motivational or (Demaidi et al., 2018) found that students with higher educational
competence issues. backgrounds tend to be more critical of the recommender system. The
condition is already known as the expertise reversal effect (Sweller et al.,
6.1.3. Inference engine and recommendation technique 2011). To minimize it (Sweller et al., 2011), proposed customization of
The semantic reasoner, used as an inference engine, is responsible for the instructional methods during a learning session. For example, the
the inference tasks such as deciding student groups, selecting the recommender system could apply direct instruction to a novice learner,
learning objects’ sequence, and determining the type of learning a mix of direct instruction and problem-solving practices to an

Fig. 4. The elements and environments of e-learning recommender system.

10
N.W. Rahayu et al. Computers and Education: Artificial Intelligence 3 (2022) 100047

intermediate learner, and minimally guided problem-solving tasks to complexity. For instance, the recommender system can recommend
advanced learners. learning paths, supported services, and individualized feedback based
on students’ self-determined learning approach.
6.2. Research agenda
6.2.3. Hybrid and multidisciplinary recommender system
6.2.1. Future development and adoption of methodologies Learning technology is rooted in psychology, computing technology,
The findings of this review study indicate that ontology-based and educational disciplines. As indicated in the current study, ontology
recommender systems are rarely integrated into existing learning tech­ has been combined not only with artificial intelligence but also with
nologies, particularly closed-source platforms. On the other hand, other computing technologies to deploy hybrid techniques. Hence,
schools and colleges have incorporated a variety of educational tech­ optimal collaboration is required to develop a robust recommender
nologies to assist students. Furthermore, the most researched and used system capable of producing meaningful recommendations. Addition­
learning technologies are LMS, e-assessment, and VLE. For example, ally, the performance of hybrid techniques should be easily identifiable
Moodle LMS is known as the most popular LMS, used by more than 248 and compared to obtain the optimal combination. However, previous
million users in 251 countries (Demaidi et al., 2018). Therefore, such studies show difficulties in performance comparison due to the lack of a
recommender systems can be integrated into existing educational public data set (Tarus et al., 2017) and the difficulty in isolating the
technology to provide students with a more personalized learning recommendation technique from the learning approach (Jevremovic
experience. Among the benefits are that educators can use the same et al., 2017). To establish a benchmark for the quality of hybrid tech­
recommendation techniques for related courses, and students will have niques, future studies may analyze the outcomes of educational linked
quick access to tailored recommendations from a variety of courses. data (Nahhas et al., 2018) or the same learning approach used by one
Most of the primary studies have not explicitly mentioned the generation of students.
methodology for building ontology except for reuse and reengineering: The current recommender system is also multidisciplinary, as the
(1) ontological resources and (2) nonontological resources. To facilitate system collaborates recommendation techniques with education and
future research collaborations, various scenarios could be used, and educational psychology (e.g., learning activities, learning theory,
include the following as examples: reuse and merge ontological re­ instructional design), and others (e.g., anthropology and SNA). As part
sources; reuse, merge, and reengineer ontological resources; reuse of educational research, future e-learning recommender systems will
ontology design patterns; restructuring ontology resources; or localizing extend beyond computing, and ontology could someday interact with
ontological resources (Suárez-Figueroa et al., 2012). science, art, or other humanities (Akker et al., 2006). Therefore, various
Furthermore, the adoption of standards and methodologies is a learning approaches present a new challenge to effectively utilizing
further concern. The interoperability standard will promote open edu­ recommender systems and learning methods. Furthermore, recom­
cation and future research collaborations by simplifying system testing mender systems can also be applied to primary and secondary educa­
and usage. Additionally, ontology methodologies have been established tion, social sciences, and humanities courses.
to aid diverse users in carrying out primary ontology engineering op­
erations. The techniques are increasingly needed as more people go
online and more ontologies exist. To drive adoption, it is necessary to 6.3. Limitation
investigate the researchers’ or developers’ difficulties in using standards
for student profiles and learning objects. Further, identifying the prob­ The current study reviews the primary studies in relation to the
lems with using ontology methodologies is important. technical aspects of ontology development. Therefore, we search into
Recommender system needs to be evaluated using the proper soft­ computer science/artificial intelligence subjects. however, there might
ware testing method. Moreover, the ontology component should be be some relevant papers found in other subjects (e.g., engineering,
assessed using ontology evaluation methodology. This assessment is multidisciplinary).
necessary to assure the technical quality of the ontology utilized in the The development and implementation of the recommender system in
recommender system. Thus, the evaluation methodology is designed to real learning indeed involves other aspects, such as the technology
verify the correctness of the desired ontology model. Two types of readiness and educational theories (Hwang et al., 2020). The techno­
evaluation are available: (1) ontology assessment based on individual logical aspects include access to technology in the classroom (Callum
characteristics (i.e., the “glass box” method) and (2) a performance test et al., 2015), the educator’s adoption (Cochrane et al., 2014), and the
of the ontology used in the application system (i.e., the “black box” students’ digital competence (Agonács et al., 2020). Furthermore, this
method) (McDaniel & Storey, 2019). study did not discuss the barriers to implement recommender systems.
Therefore, future research is required to describe a more complete
6.2.2. Prospective recommended items and learning environment picture.
Recommender systems in the education domain provide direct and This review study is limited to articles in English, so additional pa­
indirect benefits to students. The personalized learning object or pers in other languages would enrich the research results. However,
learning path are examples of direct benefits. Meanwhile, indirect these limitations do not weaken the value of this research; in fact, our
benefits address recommendation items for other stakeholders, which in findings form the basis for further ontology collaborations and more
turn provide personalized teaching services to students; examples are sophisticated e-learning recommender systems.
the authoring tool for educators (Labib et al., 2017) and the pedagogical
scenarios advisor tool for instructional designers (Savard et al., 2020). 7. Conclusion
We conclude from these findings that research opportunities for
recommendation items are rising, owing to the fact that learning tech­ Ontologies are KBS that play an essential role in adaptive learning
nology involves a large number of stakeholders (i.e., student, educator, technology. The current review investigated the role of ontology in
instructional designer, administrator, support personnel). Moreover, personalized learning, explored the development of an ontology-based
such recommendation innovations could accommodate the stake­ recommender system, and surveyed the recommendation techniques
holders’ needs and operate on various learning technology platforms. for such systems. We conducted research using the PRISMA flow dia­
Recommendations may also be used in open educational environ­ gram, and the primary studies were found using a broad automated
ments. These settings, which have grown in popularity over the last search method. The 28 articles extracted were high-quality articles in
decade, present their own set of pedagogical issues and learning mate­ quartiles Q1 and Q2 of their respective journals from 2010 to 2020. The
rials. As a result, the recommendation problem continues to grow in results are summarized below.

11
N.W. Rahayu et al. Computers and Education: Artificial Intelligence 3 (2022) 100047

(1) ITS, LMS, CAI, and PLE are the most popular learning technology popular collaborators in the last ten years, but the use of the
platform for ontology-based recommender systems. The diverse ontology collaborator is increasing. Performance among hybrid
types of technology and multidisciplinary recommendations have techniques could also be compared by considering learning ap­
extended personalized learning beyond technical computing into proaches and educational linked data.
the overall architecture of the system. (7) Ontology-based recommender models and prototypes are evalu­
(2) Some ontology-based recommender systems use the methodology ated in experiments with a specific population using nonreal
of building ontologies, but other ontology methodologies are students, algorithmic performance tests, descriptive statistics,
rarely used. inferential statistics, questionnaires, and qualitative observa­
(3) The most popular recommendation item is the learning object, tions. Therefore, future research needs to evaluate such systems
which can be personalized by referring to internal data (e.g., in a broader context or with more participants, applying appro­
learning style, scores, and learning activities) or external data (e. priate testing methods.
g., location sensors). The learning path and feedback could be the
next popular recommendation items for students in the upcoming Funding
years. Similarly, recommender systems also can provide recom­
mendations for other e-learning stakeholders, such as educators This research did not receive any specific grants from funding
and instructional designers. agencies in the public, commercial, or not-for-profit sectors.
(4) The recommendation process is reciprocal and can be initiated
either by the system or by the students. The recommendation CRediT authorship contribution statement
process is based on the following functions: first, the system
checks the student model and then provides recommendations Nur W. Rahayu: Writing – review & editing. Ridi Ferdiana:
according to the student model. Furthermore, recommendation Methodology, Supervision. Sri S. Kusumawardani: Conceptualization,
items could be expanded according to the students’ needs and Methodology, Supervision.
could also be applied in open learning environments.
(5) Most ontologies store student models and learning objects’ data.
The next most popular types of data are feedback, assessment, Declaration of competing interest
and context data (e.g., emotional, cognitive, and cultural data).
Ontology language standards are widely used in primary studies, The authors declare that they have no known competing financial
but student profile standards and metadata standards for learning interests or personal relationships that could have appeared to influence
objects are rarely adopted in ontology models. the work reported in this paper.
(6) A multidisciplinary approach has become a new trend in the
development of ontology-based recommender systems. Ontology Acknowledgements
is collaborating with concepts and methods from other tech­
niques, utilizing artificial intelligence, computing technology, The authors gratefully acknowledge Directorate of Academic
education, education psychology, and social sciences. Software Development of Universitas Islam Indonesia for proofreading funding
agents and learning activities have emerged as the two most and Prof. Mochamad Teguh and Hendrik for writing assistance.

Appendix A

Learning technology Description of uses

Academic advisor system Help students plan and make decisions about the study (e.g., learning pathways, learning outcomes) (Iatrellis et al., 2019).
Cloud e-learning Store e-learning data/resources in the cloud (Rani et al., 2015).
CAI Support learning interactions in which computers (teaching machines) provide instructional materials to students (Huang et al., 2019)
CSCL Facilitate interactions between students and enables the dissemination of knowledge/expertise within the community (Magnisalis et al., 2011).
Courseware generator Contains learning material and is used by educators to find and create learning objects (Labib et al., 2017; Ullrich & Melis, 2010).
E-assessment Online test tool to assess students’ knowledge/skills (Sánchez-Vera et al., 2012) with the purpose of obtaining information/feedback (Belcadhi,
2016).
Instructional design advisor Used by instructional designers to plan teaching/learning (e.g., learning activities, materials, and assessments) (Savard et al., 2020).
tool
ITS (also called computer tutor (Aleven et al., 2016)), which is able for one-to-one tutoring by adapting the content and appearance of the learning
according to the student’s abilities (Grubišić et al., 2013).
LMS Software to support learning (either online, face-to-face, or blended), both in an academic environment and in the professional environment
(Barreto et al., 2020) to manage, distribute and monitor users, resources, and interactive and collaborative learning activities (Fonte et al., 2012;
Kpolovie & Lale, 2017).
PLE An integrated approach/system that allows students to control and manage their own learning (Al-Abri et al., 2019; Guettat & Farhat, 2016).
Additionally, 28 definitions of PLE were found from both the educational and technological perspectives (Guettat & Farhat, 2016).
Ubiquitous Learning A learning system/approach to recommend physical learning components (e.g., tools, media) according to student behavior (especially the location
of the student and device’s location) (Dron & Anderson, 2021; Gómez et al., 2016).
Virtual Learning Environments An integrated online learning environment containing learning Objects and tools (Kurilovas et al., 2014). A VE platform could be a virtual world, a
(VLE) game machine, or a 3D visualization/interaction tool (Clemente et al., 2014).
IVET Combines VLE and ITS (Clemente et al., 2014).

12
N.W. Rahayu et al. Computers and Education: Artificial Intelligence 3 (2022) 100047

Appendix B

Publication Paper Title Type Model/system name Learning Recommendation Course/topic/domain

year technology item

2020 Savard et al. Considering cultural variables Model Executable Cultural Instructional Pedagogical N/A
(2020) in the instructional design Adaptation Method design advisor scenarios
process: A knowledge-based tool
advisor system
2019 Mustafa et al. Effectiveness of ontology-based Prototype CogSkills2 ITS Learning objects Cognitive skills of
(2019) learning content generation for preschool curriculum
preschool cognitive skills
learning
2019 Khaled et al. Recommendations-based on Prototype ILEARN SLN Learning objects 4 courses of computer
(2019) semantic analysis of social science
networks in learning
environments
2019 Capuano and Experimentation of a smart Prototype eJRM (electronic CAI Learning objects Legal mediation
Toti (2019) learning system for law based Justice Relationship
on knowledge discovery and Management)
cognitive computing
2019 Iatrellis et al. A novel integrated approach to Prototype EDUC8 Academic Learning paths N/A
(2019) the execution of personalized advisor
and self-evolving learning
pathways
2019 Nitchot et al. Personalized learning system Prototype MyTeLeMap PLE Learning paths Web technology
(2018) for visualizing knowledge
structures and recommending
study materials links
2018 Demaidi et al. OntoPeFeGe: Ontology-Based Prototype OntoPeFeGe LMS Feedbacks Data networking,
(2018) Personalized Feedback Computer networks
Generator
2018 Porcel et al. Sharing notes: An academic Prototype SharingNotes SLN Learning objects 9 courses of computer
(2018) social network based on a science
personalized fuzzy linguistic
recommender system
2018 Mariño et al. Accessibility and Activity- Model ACCESIBILITIC N/A Learning devices Accessibility and e-
(2018) Centered Design for ICT Users: inclusion
ACCESIBILITIC Ontology
2018 Yago et al. ON-SMMILE: Ontology Model ON-SMMILE IVET Learning objects Chemistry experiment
(2018) Network-based Student Model (virtual laboratory)
for MultIple Learning
Environments
2017 Tarus et al. A hybrid knowledge-based Prototype N/A LMS Learning objects 12 courses of computer
(2017) recommender system for e- science
learning based on ontology and
sequential pattern mining
2017 Jevremovic et al. IP Addressing: Problem-Based Prototype IPA-PBL (IP ITS Learning objects Computer network
(2017) Learning Approach on Addressing: Problem-
Computer Networks Based Learning)
2017 Labib et al. On the way to learning style Model LOAT (Learning Courseware Learning objects Introduction to Java
(2017) models integration: a Learner’s Object Authoring generator Programming
Characteristics Ontology Tool)
2016 Belcadhi (2016) Personalized feedback for self- Prototype N/A E-assesment Feedbacks 4 computer science
assessment in lifelong learning courses
environments based on
semantic web
2016 Gómez et al. A Contextualized System for Prototype N/A Ubiquitous Learning objects Physiology (Medicine),
(2016) Supporting Active Learning learning Clinical practice
(Nursing) and Physics
(Systems engineering)
2015 Muñoz et al. OntoSakai: On the optimization Prototype OntoSakai LMS Learning objects Database
(2015) of a Learning Management Administration
System using semantics and
user profiling
2015 Rani et al. (2015) An ontology-based adaptive Prototype N/A Cloud e- Learning objects Algorithm
personalized e-learning system, learning
assisted by software agents on
cloud storage
2014 Clemente et al. Applying a student modeling Prototype SMA(Student IVET Learning objects Biotechnology
(2014) with non-monotonic diagnosis Modeling Agent)
to Intelligent Virtual
Environment for Training/
Instruction
2014 Benlamri and Context-aware recommender Prototype N/A CAI Learning objects C++ programming
Zhang (2014) for mobile learners language
(continued on next page)

13
N.W. Rahayu et al. Computers and Education: Artificial Intelligence 3 (2022) 100047

(continued )
Publication Paper Title Type Model/system name Learning Recommendation Course/topic/domain
year technology item

2014 Kurilovas et al. Web 3.0-based personalization Model N/A VLE Learning objects E-Learning Systems
(2014) of learning objects in virtual
learning environments
2013 Grubišić et al. Ontology based approach to Model AC-ware Tutor ITS Learning objects Computer system
(2013) Bayesian student model design (Adaptive-
Courseware Tutor)
2012 AlAgha (2012) KnowledgePuzzle: A Browsing Prototype KnowledgePuzzle PLE Learning paths Computer viruses
Tool to Adapt the Web
Navigation Process to the
Learner’s Mental Model
2012 Sánchez-Vera Semantic Web technologies for Prototype OeLE (Ontology E-assesment Feedbacks Design and Production
et al. (2012) generating feedback in online eLEarning) of Educational
assessment environments Materials
2012 Fonte et al. An intelligent tutoring module Model INES (INtelligent LMS Learning objects N/A
(2012) controlled by BDI agents for an Educational System)
e-learning platform
2011 Yaghmaie and A context-aware adaptive Model N/A LMS Learning objects N/A (simulation)
Bahreininejad learning system using agents
(2011)
2011 Caravantes and Generic Educational Knowledge Prototype COES (Cognitive ITS Learning objects Web design
Galán (2011) Representation for Adaptive Ontology of
and Cognitive Systems Educational Systems)
2011 Clemente et al. A proposal for student modeling Model N/A ITS Learning paths Chemistry experiment
(2011) based on ontologies and (virtual laboratory)
diagnosis rules
2010 Ullrich and Melis Complex Course Generation Prototype Piagos Courseware Learning objects Mathematics
(2010) Adapted to Pedagogical generator
Scenarios and its Evaluation

References Boyce, S., & Pahl, C. (2007). Developing domain ontologies for course content. Journal of
Educational Technology & Society, 10(3), 275–288. https://www.jstor.org/stable/1
0.2307/jeductechsoci.10.3.275.
Aggarwal, C. C. (2016). Recommender systems. Springer International Publishing
Bremgartner, V., de Magalhães Netto, J. F., & de Menezes, C. S. (2015). Adaptation
Switzerland. https://doi.org/10.1007/978-3-319-29659-3
resources in virtual learning environments under constructivist approach: A
Agonács, N., Matos, J. F., Bartalesi-Graf, D., & O’Steen, D. N. (2020). Are you ready? Self-
systematic review. In 2015 IEEE frontiers in education conference (FIE) (pp. 1–8).
determined learning readiness of language MOOC learners. Education and
https://doi.org/10.1109/FIE.2015.7344316
Information Technologies, 25, 1161–1179. https://doi.org/10.1007/s10639-019-
Callum, K. mac, Day, S., Skelton, D., Lengyl, I., & Verhaart, M. (2015). A multiple case
10017-1
study approach exploring innovation, pedagogical transformation and inclusion for
van den Akker, J., Gravemeijer, K., McKenney, S., & Nieveen, N. (2006). Educational
mobile learning. In T. H. Brown, & H. J. van der Merwe (Eds.), The mobile learning
design research. Routledge. https://doi.org/10.4324/9780203088364
voyage - from small ripples to massive open waters. mLearn 2015 (Vol. 560).
Al-Abri, A., Jamoussi, Y., AlKhanjari, Z., & Kraiem, N. (2019). Towards a classification
Communications in Computer and Information Science. https://doi.org/10.1007/
view of personalized e-learning with social collaboration support. International
978-3-319-25684-9_23.
Journal of Advanced Computer Science and Applications, 10(8), 175–183. https://doi.
Capuano, N., & Toti, D. (2019). Experimentation of a smart learning system for law based
org/10.14569/ijacsa.2019.0100823
on knowledge discovery and cognitive computing. Computers in Human Behavior, 92,
Al-Yahya, M., George, R., & Alfaries, A. (2015). Ontologies in E-learning : Review of the
459–467. https://doi.org/10.1016/j.chb.2018.03.034
literature. International Journal of Software Engineering and Its Applications, 9(2),
Caravantes, A., & Galán, R. (2011). Generic educational knowledge representation for
67–84. https://doi.org/10.14257/ijseia.2015.9.2.07
adaptive and cognitive systems. Journal of Educational Technology & Society, 14(3),
AlAgha, I. (2012). KnowledgePuzzle: A browsing tool to adapt the web navigation
252–266. https://www.jstor.org/stable/10.2307/jeductechsoci.14.3.252.
process to the learner’s mental model. Journal of Educational Technology & Society, 15
Cela, K. L., Sicilia, M. A., & Sanchez, S. (2015). Social network analysis in E-learning
(3), 275–287. https://www.jstor.org/stable/10.2307/jeductechsoci.15.3.275.
environments: A preliminary systematic review. Educational Psychology Review, 27
Aleven, V., McLaren, B. M., Sewall, J., van Velsen, M., Popescu, O., Demi, S.,
(1), 219–246. https://doi.org/10.1007/s10648-014-9276-0
Ringenberg, M., & Koedinger, K. R. (2016). Example-tracing tutors: Intelligent tutor
Chen, X., Xie, H., Zou, D., & Hwang, G.-J. (2020). Application and theory gaps during the
development for non-programmers. International Journal of Artificial Intelligence in
rise of artificial intelligence in education. Computers & Education: Artificial
Education, 26(1), 224–269. https://doi.org/10.1007/s40593-015-0088-2
Intelligence, 1(August), 1–20. https://doi.org/10.1016/j.caeai.2020.100002
Araka, E., Maina, E., Gitonga, R., & Oboko, R. (2020). Research trends in measurement
Chen, X., Zou, D., Xie, H., & Cheng, G. (2021). Twenty years of personalized language
and intervention tools for self-regulated learning for e-learning environments -
learning. Educational Technology & Society, 24(1), 205–222. https://www.jstor.
systematic review (2008 – 2018). Research and Practice in Technology Enhanced
org/stable/10.2307/26977868.
Learning, 15(6). https://doi.org/10.1186/s41039-020-00129-5
Clemente, J., Ramírez, J., & de Antonio, A. (2011). A proposal for student modeling
Barreto, D., Rottmann, A., & Rabidoux, S. (2020). Learning management systems: Choosing
based on ontologies and diagnosis rules. Expert Systems with Applications, 38(7),
the right path for your organization. EdTech Books. https://edtechbooks.org/learning
8066–8078. https://doi.org/10.1016/j.eswa.2010.12.146
_management_systems.
Clemente, J., Ramírez, J., & de Antonio, A. (2014). Applying a student modeling with
Bedenlier, S., Bond, M., Buntins, K., Zawacki-Richter, O., & Kerres, M. (2020).
non-monotonic diagnosis to intelligent virtual environment for training/instruction.
Facilitating student engagement through educational technology in higher
Expert Systems with Applications, 41(2), 508–520. https://doi.org/10.1016/j.
education: A systematic review in the field of arts and humanities. Australasian
eswa.2013.07.077
Journal of Educational Technology, 36(4), 126–150. https://doi.org/10.14742/
Cochrane, T., Antonczak, L., Guinibert, M., & Mulrennan, D. (2014). Developing a mobile
AJET.5477
social media framework for creative pedagogies. In Proceedings of the 10th
Belcadhi, L. C. (2016). Personalized feedback for self assessment in lifelong learning
international conference on mobile learning 2014 (pp. 207–214). http://hdl.handle.net
environments based on semantic web. Computers in Human Behavior, 55, 562–570.
/10292/7371.
https://doi.org/10.1016/j.chb.2015.07.042
Corcho, O., Fernandez-Lopez, M., & Gomez-Perez, A. (2003). Methodologies, tools and
Benlamri, R., & Zhang, X. (2014). Context-aware recommender for mobile learners.
languages for building ontologies. Where is their meeting point? Data & Knowledge
Human-Centric Computing and Information Sciences, 4(1), 1–34. https://doi.org/
Engineering, 46, 41–64. https://doi.org/10.1016/S0169-023X(02)00195-7
10.1186/s13673-014-0012-z
Demaidi, M. N., Gaber, M. M., & Filer, N. (2018). OntoPeFeGe : Ontology-based
Bhareti, K., Perera, S., Jamal, S., Pallege, M. H., Akash, V., & Wijeweera, S. (2020).
personalized feedback generator. IEEE Access, 6, 31644–31664. https://doi.org/
A literature review of recommendation systems. IEEE International Conference for
10.1109/ACCESS.2018.2846398
Innovation in Technology (INOCON), 1–7. https://doi.org/10.1109/
INOCON50539.2020.9298450, 2020.

14
N.W. Rahayu et al. Computers and Education: Artificial Intelligence 3 (2022) 100047

Deschênes, M. (2020). Recommender systems to support learners’ agency in a learning logics (DL 2015), athens, Greece, june 6, 2015 (Vol. 1387, pp. 68–79). CEUR-WS.org
context: A systematic review. International Journal of Educational Technology in Higher http://ceur-ws.org/Vol-1387/paper_4.pdf.
Education, 17(1). https://doi.org/10.1186/s41239-020-00219-w McDaniel, M., & Storey, V. C. (2019). Evaluating domain ontologies: Clarification,
Dron, J., & Anderson, T. (2021). The future of E-learning. In The SAGE handbook of E- classification, and challenges. ACM Computing Surveys, 52(4), 1–44. https://doi.org/
learning research (pp. 537–556). SAGE Publications Ltd. https://doi.org/10.4135/ 10.1145/3329124
9781529716696. Middleton, S. E., Roure, D. de, & Shadbolt, N. R. (2009). Ontology-based recommender
Ertel, W. (2017). In Introduction to artificial intelligence (2nd ed.). Springer International systems. In S. Staab, & R. Studer (Eds.), Handbook on ontologies, international
Publishing AG. https://doi.org/10.1007/978-3-319-58487-4. handbooks on information systems (pp. 1648–1686). Springer-Verlag Berlin
Fonte, F. A. M., Burguillo, J. C., & Nistal, M. L. (2012). An intelligent tutoring module Heidelberg. https://doi.org/10.1007/978-3-540-92673-3_35.
controlled by BDI agents for an e-learning platform. Expert Systems with Applications, Mohemad, R., Mamat, N. F. A., Noor, N. M. M., & Alhadi, A. C. (2017). Computational
39(8), 7546–7554. https://doi.org/10.1016/j.eswa.2012.01.161 approaches in supporting special education domain: A review. Journal of
George, G., & Lal, A. M. (2019). Review of ontology-based recommender systems in e- Telecommunication, Electronic and Computer Engineering, 9(3), 61–67. https://jtec.ute
learning. Computers & Education, 142, 1–18. https://doi.org/10.1016/j. m.edu.my/jtec/article/view/2964.
compedu.2019.103642 Muñoz, A., Lasheras, J., Capel, A., Cantabella, M., & Caballero, A. (2015). OntoSakai: On
Gómez, J. E., Huete, J. F., & Hernandez, V. L. (2016). A contextualized system for the optimization of a Learning Management System using semantics and user
supporting active learning. IEEE Transactions on Learning Technologies, 9(2), profiling. Expert Systems with Applications, 42(15), 5995–6007. https://doi.org/
196–202. https://doi.org/10.1109/TLT.2016.2531685 10.1016/j.eswa.2015.04.019
Grubišić, A., Stankov, S., & Peraić, I. (2013). Ontology based approach to Bayesian Mustafa, G., Abbas, M. A., Hafeez, Y., Khan, S., & Hwang, G.-J. (2019). Effectiveness of
student model design. Expert Systems with Applications, 40(13), 5363–5371. https:// ontology-based learning content generation for preschool cognitive skills learning.
doi.org/10.1016/j.eswa.2013.03.041 Interactive Learning Environments, 27(4), 443–457. https://doi.org/10.1080/
Guettat, B., & Farhat, R. (2016). An approach to assist learners to identify their learning 10494820.2018.1484772
objectives in personal learning environment (PLE). In Proceedings of 2015 5th Nahhas, S., Bamasag, O., Khemakhem, M., & Bajnaid, N. (2018). Added values of linked
international conference on information & communication technology and accessibility data in education: A survey and roadmap. Computers, 7(3). https://doi.org/10.3390/
(ICTA), january 2018. https://doi.org/10.1109/ICTA.2015.7426934 computers7030045
Harley, J. M., Taub, M., Azevedo, R., & Bouchet, F. (2018). “Let’s set up some subgoals”: Nitchot, A., Wettayaprasit, W., & Gilbert, L. (2018). Personalized learning system for
Understanding human-pedagogical agent collaborations and their implications for visualizing knowledge structures and recommending study materials links. E-
learning and prompt and feedback compliance. IEEE Transactions on Learning Learning and Digital Media, 16(1), 77–91. https://doi.org/10.1177/
Technologies, 11(1), 54–66. https://doi.org/10.1109/TLT.2017.2756629 2042753018817615
Hebeler, J., Fisher, M., Blace, R., & Perez-Lopez, A. (2009). In M. Dean, & M. Smith Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D.,
(Eds.), Semantic web programming. Wiley Publishing, Inc. Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J.,
Huang, R., Spector, J. M., & Yang, J. (2019). Lecture notes in educational technology. Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E.,
Springer Nature Singapore. https://doi.org/10.1007/978-981-13-6643-7 McDonald, S., & Moher, D. (2021). The PRISMA 2020 statement: An updated
Hwang, G.-J., Xie, H., Wah, B. W., & Gašević, D. (2020). Vision, challenges, roles and guideline for reporting systematic reviews. BMJ, 372. https://doi.org/10.1136/bmj.
research issues of Artificial Intelligence in Education. Computers & Education: n71
Artificial Intelligence, 1, 1–5. https://doi.org/10.1016/j.caeai.2020.100001 Palalas, A., & Wark, N. (2020). The relationship between mobile learning and self-
Iatrellis, O., Kameas, A., & Fitsilis, P. (2019). A novel integrated approach to the regulated learning: A systematic review. Australasian Journal of Educational
execution of personalized and self-evolving learning pathways. Education and Technology, 36(4), 151–172. https://doi.org/10.14742/AJET.5650
Information Technologies, 24(1), 781. https://doi.org/10.1007/s10639-018-9802-7 Parsia, B., Matentzoglu, N., Gonçalves, R. S., Glimm, B., & Steigmiller, A. (2017). The
ISO/IEC/IEEE International Standard. (2017). Systems and software OWL reasoner evaluation (ORE) 2015 competition report. Journal of Automated
engineering–Vocabulary (pp. 1–541). ISO/IEC/IEEE 24765. https://doi.org/10.1109/ Reasoning, 59(4), 455–482. https://doi.org/10.1007/s10817-017-9406-8
IEEESTD.2017.8016712. 2017(E). Pollock, J. T. (2009). Semantic web for dummies. Wiley Publishing, Inc.
Javed, U., Shaukat, K., Hameed, I. A., Iqbal, F., Alam, T. M., & Luo, S. (2021). A review of Porcel, C., Ching-López, A., Lefranc, G., Loia, V., & Herrera-Viedma, E. (2018). Sharing
content-based and context-based recommendation systems. International Journal of notes: An academic social network based on a personalized fuzzy linguistic
Emerging Technologies in Learning, 16(3), 274–306. https://doi.org/10.3991/ijet. recommender system. Engineering Applications of Artificial Intelligence, 75, 1–10.
v16i03.18851 https://doi.org/10.1016/j.engappai.2018.07.007
Jevremovic, A., Shimic, G., Veinovic, M., & Ristic, N. (2017). IP addressing: Problem- Premlatha, K. R., & Geetha, T. V. (2015). Learning content design and learner adaptation
based learning approach on computer networks. IEEE Transactions on Learning for adaptive e-learning environment: A survey. Artificial Intelligence Review, 44(4),
Technologies, 10(3), 367–378. https://doi.org/10.1109/TLT.2016.2583432 443–465. https://doi.org/10.1007/s10462-015-9432-z
Khadir, A. C., Aliane, H., & Guessoum, A. (2021). Ontology learning: Grand tour and Rani, M., Nayak, R., & Vyas, O. P. (2015). An ontology-based adaptive personalized e-
challenges. Computer Science Review, 39, 1–14. https://doi.org/10.1016/j. learning system, assisted by software agents on cloud storage. Knowledge-Based
cosrev.2020.100339 Systems, 90, 33–48. https://doi.org/10.1016/j.knosys.2015.10.002
Khaled, A., Ouchani, S., & Chohra, C. (2019). Recommendations-based on semantic Salehi, M., Kamalabadi, I. N., & Ghoushchi, M. B. G. (2013). An effective
analysis of social networks in learning environments. Computers in Human Behavior, recommendation framework for personal learning environments using a learner
101, 435–449. https://doi.org/10.1016/j.chb.2018.08.051 preference tree and a GA. IEEE Transactions on Learning Technologies, 6(4), 350–363.
Klašnja-Milicevic, A., Ivanovic, M., & Nanopoulos, A. (2015). Recommender systems in e- https://doi.org/10.1109/TLT.2013.28
learning environments: A survey of the state-of-the-art and possible extensions. Sánchez-Vera, M. del M., Fernández-Breis, J. T., Castellanos-Nieves, D., Frutos-
Artificial Intelligence Review, 44, 571–604. https://doi.org/10.1007/s10462-015- Morales, F., & Prendes-Espinosa, M. P. (2012). Semantic Web technologies for
9440-z generating feedback in online assessment environments. Knowledge-Based Systems,
Kpolovie, P. J., & Lale, N. E. S. (2017). Globalisation and adaptation of university 33, 152–165. https://doi.org/10.1016/j.knosys.2012.03.010
curriculum with LMSs in the Changing World. European Journal of Computer Science Santoso, H. A., Haw, S. C., & Abdul-Mehdi, Z. T. (2011). Ontology extraction from
and Information Technology, 5(2), 28–89. https://www.eajournals.org/journals/eu relational database: Concept hierarchy as background knowledge. Knowledge-Based
ropean-journal-of-computer-science-and-information-technology-ejcsit/vol-5-issu Systems, 24(3), 457–464. https://doi.org/10.1016/j.knosys.2010.11.003
e-2-april-2017/globalization-adaptation-university-curriculum-lmss-changing-worl Savard, I., Bourdeau, J., & Paquette, G. (2020). Considering cultural variables in the
d/. instructional design process: A knowledge-based advisor system. Computers &
Kurilovas, E., Kubilinskiene, S., & Dagiene, V. (2014). Web 3.0 – based personalisation of Education, 145, Article 103722. https://doi.org/10.1016/j.compedu.2019.103722
learning objects in virtual learning environments. Computers in Human Behavior, 30, Schouten, D. G. M., Venneker, F., Bosse, T., Neerincx, M. A., & Cremers, A. H. M. (2018).
654–662. https://doi.org/10.1016/j.chb.2013.07.039 A digital coach that provides affective and social learning support to low-literate
Labib, A. E., Canós, J. H., & Penadés, M. C. (2017). On the way to learning style models learners. IEEE Transactions on Learning Technologies, 11(1), 67–80. https://doi.org/
integration: A learner’s characteristics ontology. Computers in Human Behavior, 73, 10.1109/TLT.2017.2698471
433–445. https://doi.org/10.1016/j.chb.2017.03.054 Sinha, B. B., & Dhanalakshmi, R. (2019). Evolution of recommender system over the
Luna-Urquizo, J. (2019). Learning management system personalization based on multi- time. Soft Computing, 23(23), 12169–12188. https://doi.org/10.1007/s00500-019-
attribute decision making techniques and intuitionistic fuzzy numbers. International 04143-8
Journal of Advanced Computer Science and Applications, 10(11), 669–676. https://doi. Suárez-Figueroa, M. C., Gómez-Pérez, A., Motta, E., & Gangemi, A. (2012). In
org/10.14569/IJACSA.2019.0101188 M. C. Suárez-Figueroa, A. Gómez-Pérez, E. Motta, & A. Gangemi (Eds.), Ontology
Magnisalis, I., Demetriadis, S., & Karakostas, A. (2011). Adaptive and intelligent systems engineering in a networked World. Springer-Verlag Berlin Heidelberg. https://doi.org/
for collaborative learning support: A review of the field. IEEE Transactions on 10.1007/978-3-642-24794-1.
Learning Technologies, 4(1), 5–20. https://doi.org/10.1109/TLT.2011.2 Sweller, J., Ayres, P., & Kalyuga, S. (2011). Cognitive load theory. In Cognitive load
Mariño, B. D. R., Rodríguez-fórtiz, M. J., Torres, M. V. H., & Haddad, H. M. (2018). theory. Springer Science+Business Media. https://doi.org/10.1007/978-1-4419-
Accessibility and activity-centered design for ICT users : ACCESIBILITIC ontology. 8126-4.
IEEE Access, 6, 60655–60665. https://doi.org/10.1109/ACCESS.2018.2875869 Tapia-Leon, M., Chicaiza, J., & Lujan-Mora, S. (2018). Application of ontologies in higher
Matentzoglu, N., Leo, J., Hudhra, V., Parsia, B., & Sattler, U. (2015). A survey of current, education: A systematic mapping study. In 2018 IEEE global engineering education
stand-alone OWL reasoners. In M. Dumontier, B. Glimm, R. S. Gonçalves, conference (EDUCON) (pp. 1344–1353). https://doi.org/10.1109/
M. Horridge, E. Jiménez-Ruiz, N. Matentzoglu, B. Parsia, G. B. Stamou, & G. Stoilos EDUCON.2018.8363385
(Eds.), Informal proceedings of the 4th international workshop on OWL reasoner Tarus, J. K., Niu, Z., & Mustafa, G. (2018). Knowledge-based recommendation: A review
evaluation (ORE-2015) co-located with the 28th international workshop on description of ontology-based recommender systems for e-learning. Artificial Intelligence Review,
50(1), 21–48. https://doi.org/10.1007/s10462-017-9539-5

15
N.W. Rahayu et al. Computers and Education: Artificial Intelligence 3 (2022) 100047

Tarus, J. K., Niu, Z., & Yousif, A. (2017). A hybrid knowledge-based recommender system Yaghmaie, M., & Bahreininejad, A. (2011). A context-aware adaptive learning system
for e-learning based on ontology and sequential pattern mining. Future Generation using agents. Expert Systems with Applications, 38(4), 3280–3286. https://doi.org/
Computer Systems, 72, 37–48. https://doi.org/10.1016/j.future.2017.02.049 10.1016/j.eswa.2010.08.113
Tibaná-herrera, G., Fernández-bajón, M. T., & Moya-anegón, F. De (2018). Yago, H., Clemente, J., Rodriguez, D., & Fernandez-de-Cordoba, P. (2018). ON-SMMILE:
Categorization of E-learning as an emerging discipline in the world publication Ontology network-based student model for MultIple learning environments. Data &
system : A bibliometric study in. International Journal of Educational Technology in Knowledge Engineering, 115, 48–67. https://doi.org/10.1016/j.datak.2018.02.002
Higher Education, 15(21). https://doi.org/10.1186/s41239-018-0103-4 Zhang, L., Basham, J. D., & Yang, S. (2020). Understanding the implementation of
Truong, H. M. (2016). Integrating learning styles and adaptive e-learning system: Current personalized learning : A research synthesis. Educational Research Review, 31
developments, problems and opportunities. Computers in Human Behavior, 55, (December 2019), Article 100339. https://doi.org/10.1016/j.edurev.2020.100339
1185–1193. https://doi.org/10.1016/j.chb.2015.02.014 Zhao, Y., Li, Z., Wang, X., & Halang, W. A. (2012). Decision support in e-business based
Ullrich, C., & Melis, E. (2010). Complex course generation adapted to pedagogical on assessing similarities between ontologies. Knowledge-Based Systems, 32, 47–55.
scenarios and its evaluation. Journal of Educational Technology & Society, 13(2), https://doi.org/10.1016/j.knosys.2011.08.020
102–115. https://www.jstor.org/stable/10.2307/jeductechsoci.13.2.102. Zheng, X.-L., Chen, C.-C., Hung, J.-L., He, W., Hong, F.-X., & Lin, Z. (2015). A hybrid
Vidal-Castro, C., Sicilia, M.Á., & Prieto, M. (2012). Representing instructional design trust-based recommender system for online communities of practice. IEEE
methods using ontologies and rules. Knowledge-Based Systems, 33, 180–194. https:// Transactions on Learning Technologies, 8(4), 345–356. https://doi.org/10.1109/
doi.org/10.1016/j.knosys.2012.04.005 TLT.2015.2419262
West, R. E. (2018). Foundations of learning and instructional historical roots and current Zhong, X., Fu, H., Xia, H., Yang, L., & Shang, M. (2015). A hybrid cognitive assessment
trends. EdTechBooks. https://edtechbooks.org/lidtfoundations/. based on ontology knowledge map and skills. Knowledge-Based Systems, 73, 52–60.
https://doi.org/10.1016/j.knosys.2014.09.004

16