Journal Pre-proof

A learning path recommendation model based on a multidimensional

knowledge graph framework for e-learning

Daqian Shi, Ting Wang, Hao Xing, Hao Xu

PII: S0950-7051(20)30085-X
DOI: https://doi.org/10.1016/j.knosys.2020.105618
Reference: KNOSYS 105618

To appear in: Knowledge-Based Systems

Received date : 8 May 2019

Revised date : 26 December 2019
Accepted date : 3 February 2020

Please cite this article as: D. Shi, T. Wang, H. Xing et al., A learning path recommendation model
based on a multidimensional knowledge graph framework for e-learning, Knowledge-Based
Systems (2020), doi: https://doi.org/10.1016/j.knosys.2020.105618.

This is a PDF file of an article that has undergone enhancements after acceptance, such as the
addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive
version of record. This version will undergo additional copyediting, typesetting and review before it
is published in its final form, but we are providing this version to give early visibility of the article.
Please note that, during the production process, errors may be discovered which could affect the
content, and all legal disclaimers that apply to the journal pertain.

© 2020 Published by Elsevier B.V.

 Journal Pre-proof

*Revised Manuscript (Clean Version)

Click here to view linked References

of
A learning path recommendation model based on a

pro
multidimensional knowledge graph framework for e-learning

Daqian Shi1 , Ting Wang1 , Hao Xing1 and Hao Xu ∗1,2,3,4

1 College of Computer Science and Technology, Jilin University
2 School of Management, Jilin University
3 Department of Computer Science and Technology, Zhuhai College of Jilin University
4 Symbol re-
Computation and Knowledge Engineer of Ministry of Education, Jilin University

Abstract

E-learners face a large amount of fragmented learning content during e-learning. How to extract
lP
and organize this learning content is the key to achieving the established learning target, especially for
non-experts. Reasonably arranging the order of the learning objects to generate a well-defined learning path
can help the e-learner complete the learning target efficiently and systematically. Currently, knowledge-
graph-based learning path recommendation algorithms are attracting the attention of researchers in this
field. However, these methods only connect learning objects using single relationships, which cannot
generate diverse learning paths to satisfy different learning needs in practice. To overcome this challenge,
rna

this paper proposes a learning path recommendation model based on a multidimensional knowledge
graph framework. The main contributions of this paper are as follows. Firstly, we have designed a
multidimensional knowledge graph framework that separately stores learning objects organized in several
classes. Then, we have proposed six main semantic relationships between learning objects in the knowledge
graph. Secondly, a learning path recommendation model is designed for satisfying different learning
needs based on the multidimensional knowledge graph framework, which can generate and recommend
customized learning paths according to the e-learner’s target learning object. The experiment results
indicate that the proposed model can generate and recommend qualified personalized learning paths to
improve the learning experiences of e-learners.
Jou

Keywords: learning path recommendation, knowledge graph, e-learning, learning needs.

1 Introduction

With the popularity and improvement of information search technology, increasingly more learners
choose self-learning through the Internet, which is a well-known method for e-learning [1]. Compared with
∗ Corresponding author: [email protected], Tel. +86-133-0431-0608

1
 Journal Pre-proof

of
the traditional teacher-centered learning style, e-learning has advantages: the learning target can be flexibly
searched and viewed by learners anywhere, not only in class. However, since most of the learning content
on the network is fragmented, how to systematically and efficiently learn multiple knowledge objects in a
specific field has always been a problem for e-learning.

pro
Researchers have realized that the order of learning objects has a great impact on learning quality [2, 3].
Methods for organizing fragmented learning content by learning paths to guide learners have gradually been
accepted. This can effectively reduce the time required to collect learning materials and improves learning
efficiency. At the same time, a high-quality learning path is helpful for improving the learners’ understanding
of the learning content. Therefore, how to generate high-quality learning paths is an issue of concern for the
research community.
Recent research [4, 5, 6, 7, 8] has focused on generating high-quality customized learning paths to
re-
satisfy e-learners. These clustering-based learning path generation methods often collect redundant or
irrelevant learning objects because they ignore potential dependencies between learning objects. Other work
[9, 10, 11, 12] used knowledge graphs for applying these dependencies to the learning path recommendation
model and has achieved some success. However, they only built the simplest relationship to link learning
objects but did not explore further by building more complex semantics in the relationship. These works
did not fully exploit the connectivity of the knowledge graph to link learning objects using various semantic
lP
relationships. Therefore, they can only generate single learning paths that cannot satisfy different learning
needs in for e-learning today.
This paper proposes a learning path recommendation model based on the knowledge graph (KG) to satisfy
diverse learning needs. To improve the learning path recommendation model by using semantic relationships,
an innovative knowledge graph framework is designed for storing and presenting learning objects. We
rna

construct a multidimensional knowledge graph by describing the learning object as the hierarchical classes
of “basic knowledge”, “algorithm” and “task”. Based on this type of knowledge graph framework, learning
objects from different subjects and curricula can be appropriately merged and organized. We have also
constructed more complex and logical semantic relationships between these learning objects. Based on this
framework, we have introduced a learning path generation algorithm and a learning path recommendation
algorithm in the model. The learning path generation algorithm generates relationship constraints based on
an individual’s learning need, and then decides how to generate all possible learning paths. To recommend a
logical learning path from the set of all possible learning paths, we proposed a learning path recommendation
Jou

algorithm for scoring learning paths, in which variables and their weighted coefficients consider the different
learning path preferences of the e-learner.
In this paper, we introduce an innovative model for learning path recommendation that is expandable and
reusable. Therefore, our model can be promoted and used in more fields of e-learning to improve learning
efficiency. At the same time, this research will also help researchers develop more effective learning path
recommendation methods for improving e-learning.
This paper is organized as follows. Section 2 introduces related work on learning path recommendation.

2
 Journal Pre-proof

of
Section 3 presents some basic definitions and describes how the multidimensional KG was built (including
the KG structure and data collection). Section 4 presents the details of the algorithms in the learning path
recommendation model. Section 5 provides the structure of the e-learning system based on the proposed
model. Section 6 evaluates the learning path recommendation model. Feedback from participants was used to

pro
verify the quality of the recommended learning paths. Section 7 presents the conclusions and future research.

2 Related Work

Researchers have proposed various methods for improving learning efficiency; constructing systems
[13, 14, 15] for recommending learning materials to e-learners is a leading-edge research field. Durand et al.
[9] have proved that the order of learning objects is important for learners and introduced a graph-theory-based
re-
system that can generate one-way learning paths by linking all necessary learning objects through the
relationship. Chen’s research [16] noted that learning the prior and posterior knowledge is helpful in
understanding the current learning object, which also supports the idea that learning objects should be sorted
into learning paths. Therefore, for systematic and efficient e-learning, individuals should study learning
objects in the suggested learning path order to achieve their learning target.
Based on this context, researchers have tried to explore the dependency of learning content in e-learning
lP
systems. Chen et al. [17] found that prerequisite relations among concepts play an important role in e-learning
and proposed an effective data-driven solution for prerequisite classification. Pan et al. [18] developed a
graph-based propagation algorithm to order the concepts based on the learned representations of course
concepts. At the same time, some researchers [19, 20, 21] tried to obtain concept prerequisite relation from
course dependencies by novel approaches like directed graphs. These works show great potential in exploring
rna

the dependency on learning content in the e-learning environment. But they still lack the consideration of the
concrete classification of relations/dependencies between learning contents, which is important for effectively
organizing learning content for e-learners.
Data mining was widely used by researchers for organizing learning contents into learning paths. Chen
[22] constructed a genetic-based, personalized e-learning system that can generate appropriate learning paths
by mining the individual learner’s pretest and learning performance data. Dwivedi et al. [23] improved
Chen’s approach by constructing learning paths using a variable-length genetic algorithm (VLGA) that
includes the learning path record from predecessors. Chen et al. [24] proposed an improved ant colony
Jou

optimization algorithm (ACO) based on coordinate system to recommend learning path. Bendahmane et al.
[25] presented a competence based approach (CBA) derived from learning data, learner’s expectations. In
this approach, learners were clustered and traced, and finally obtained proper learning paths. Hsieh and Wang
[26] developed an e-learning system using a data mining approach that can create a relationship hierarchy of
learning materials to generate a suitable learning path. With these data-mining-based methods, learners no
longer need to waste time organizing the learning content, but there are still some problems: (1) Updating
new data in this type of learning system will be difficult because these systems are not self-adaptive, so the

3
 Journal Pre-proof

of
learning path has to be generated every time data updates are performed. (2) These systems sometimes
generate redundant results when identical learning content exists.
As a popular research domain, knowledge graphs have been used recently for learning path construction
since knowledge graphs can avoid ambiguities in learning content descriptions. Encouraged by this

pro
characteristic, some researchers [27, 28, 29] tried to build learning systems for learning path recommendation
based on knowledge graphs, and successfully solved problems (1) and (2). Wan et al. [30] provided a
learner-oriented learning recommendation method for generating learning paths based on knowledge, in which
knowledge units are represented by nodes, and the relationships between knowledge units are represented
by the node connections. Ouf et al. [31] proposed the framework for smart e-learning ecosystem using
knowledge graph and SWRL. Shmelev et al. [32] proposed a method that combined the genetic approach and
knowledge graph technology to arrange the learning objects in a sequence. Chu et al. [33] built an e-learning
re-
system based on the concept map that can generate learning paths according to the relationships in the concept
map. Subsequently, Zhu et al. [10] introduced a learning path recommendation method using preset learning
scenarios, considering that learners need different learning paths in different scenarios. They proposed a
learning path generation method that requires specifying starting and ending nodes.
The above research studies focus on how to generate good learning paths using knowledge graphs.
These studies choose to regard the learning objects as nodes, and relationships as paths in a learning path.
lP
However, these knowledge-graph-based methods did not fully exploit the connectivity of the knowledge graph.
The relationships between learning objects should have logical relationship names that represent specific
semantics, and not just display a connection line. At the same time, these studies note that because e-learners
may require different learning paths for the same learning object, a good learning path recommendation model
should generate learning paths that satisfy diverse learning needs in different scenarios. Based on the related
rna

studies described above, we will design a specific knowledge graph for the learning path recommendation
model, and try to recommend logical learning paths that satisfy the diverse learning needs of individuals.

3 Knowledge graph design

We have designed a knowledge graph framework in order to better organize learning objects and to
logically express semantic relationships between learning objects. Based on this framework, we build
a knowledge graph that has been applied to our learning path recommendation model by taking partial
Jou

machine-learning knowledge as a use case.

3.1 Definition of terminology

To better explain the content that follows, we have defined some relevant terminology that will be used
in this paper.

• Learning object (LO): A learning object is the meta-learning material unit, which includes the basic

4
 Journal Pre-proof

of
information (name, description, etc.) and the learning link. Every learning object can be studied
independently, for example, “Sigmoid_Function” is a learning object that belongs to the “basic
knowledge” class.
• Relationship (RE): The relationship refers to the semantic dependencies of the relationships between

pro
different learning objects. For example, Figure 3.1 presents the relationships between three learning
objects.
• Knowledge graph (KG): A knowledge graph is a type of directed graph in which learning objects are
represented by nodes and relationships are represented by the node connections. A knowledge graph is
defined as follows:
KG = (LO, RE) (1)

re-
In which: LO is the set of all the learning objects in the knowledge graph, and RE is the set of all the
relationships in the knowledge graph.
• Learning need: The learning need is how the learner expects to learn the target learning object. For
example, when we have the target learning object “Region_Based_CNN”, then “what is the prior
knowledge of Region_Based_CNN” and “how to achieve Region_Based_CNN by basic knowledge”
are two different learning needs. In this study, we have pre-established a set of six learning needs as
lP
shown in Table 3.1. In the experiment, we use labels (like #1) to represent different learning needs.
• Learning path (LP): A learning path is the sequence of learning objects that is generated for achieving the
specific learning goal. For example, Figure 3.2 shows the learning path for solving entity recognition.
• Target learning object: A target learning object is what the individual intends to learn in the current
learning session. In the learning path recommendation, the target learning object is always the last
node of the learning path.
rna

Figure 3.1: Example of relationships between learning objects

Jou

3.2 The framework of the knowledge graph

In this study, we proposed a multidimensional knowledge graph framework for representing knowledge.
According to a previous study [34], dividing educational resources into multiple classes helps learners to

5
 Journal Pre-proof

of
pro
Figure 3.2: Example of the learning path

Table 3.1: Learning needs and corresponding labels

Label Type of learning need

#1 Requesting the prior knowledge.
#2 Requesting the algorithm prototype.
#3
#4
#5
re-
Requesting the position of the learning object in the hierarchy.
Requesting the single learning object.
Requesting the learning path from algorithm to target.
#6 Requesting the learning path from basic knowledge to target.

efficiently understand learning profiles, and enables them to logically organize and memorize knowledge.
lP
Therefore, we decided to separate different learning objects into three classes:

• Basic Knowledge
This class contains all the necessary basic knowledge learning objects for supporting algorithms, such as
“naïve Bayes” and “Bayesian statistics”, which is the bottom level of this knowledge graph framework.
rna

• Algorithm
The center of this framework is the “algorithm” class that contains all the algorithms related to the
specific knowledge field (e.g., “Bayes classifier”). An algorithm is the method to achieve/solve a
particular task.

• Task
This class contains all the practical tasks such as “natural language processing” and “sentiment analysis”
in the machine learning field. The “task” class is the top level of this knowledge graph framework.
Jou

E-learners can understand what practical tasks they can achieve/solve after they have learned the basic
knowledge/algorithm.

Figure 3.3 shows an example of a multidimensional KG framework. In this framework, each class
consists of a hierarchy and the corresponding instances of the learning object. The hierarchy represents
the knowledge structure of the current class, while the learning objects are the meta-learning materials that
are inserted into the hierarchy and connected by semantic relationships. To better express the semantic
relationships between learning objects, we have designed multiple relationships and divided them into

6
 Journal Pre-proof

of
intraclass relationships and interclass relationships. Intraclass relationships enable intraclass learning objects
to be connected to each other. On the other hand, interclass relationships provide connections between learning
objects from different classes. Details of the relationships are shown in Table 3.2. These relationships
between learning objects connected by semantics are fundamental for generating relationship constraints in

pro
the learning path generation algorithm.

re-
Figure 3.3: (left) One-dimensional KG framework in previous work [10, 35]. (right) Multidimensional KG framework
lP
in our project, where dotted lines represent inter-class relationships, solid lines represent intra-class relationships, and
the nodes with different colours represent learning objects in different classes.

The one-dimensional knowledge graph framework can only present learning objects in a single class
and has no specific relationships between learning objects. Our knowledge graph provides multiple classes
for learning objects and strengthens the connections between interclass learning objects. In this context, the
rna

knowledge graph is able to show how to practically apply the learned knowledge (basic knowledge-algorithm,
algorithm-task), which can expand the e-learners’ understanding of the learned knowledge and enable them to
grasp the practical use of theoretical knowledge.

3.3 Data collection

In this study, we have built a knowledge graph based on machine learning field knowledge as the use
case. The knowledge graph consists of 675 learning objects and 1,033 relationships, where two interclass
Jou

relationships and four intraclass relationships were set to ensure the diversity of connections between the
learning objects.
During the data collection process, we applied data mining methods and expert knowledge validation to
collect learning objects for the knowledge graph. The first step is to crawl for data; several educational websites
were crawled for information about algorithms and tasks, which was then stored as structured data. After
removing duplicates, we collected 392 algorithm objects and 89 task objects and established the corresponding
relationships between these objects. Then, textbooks (on artificial intelligence and mathematics) were crawled
for information about basic knowledge to produce a list of named entities of basic knowledge. A named entity

7
 Journal Pre-proof

of
Table 3.2: Designed relationships in the knowledge graph

Type Description
Subclass Intra-class The relationship indicates that the current LO has a

pro
subclass.
Individual Intra-class The relationship indicates that the current LO has an
individual.
Pre-knowledge Intra-class The relationship indicates that the current LO (basic
knowledge) has prior knowledge that should be learned first.
Ori-algorithm Intra-class The relationship indicates that the current LO (algorithm)
was improved from an original algorithm.
ApplyToAlgorithm Inter-class
re- The relationship indicates that the current LO (basic
knowledge) can be applied in the target LO (algorithm).
ApplyToTask Inter-class The relationship indicates that the current LO (algorithm)
can be applied in the target LO (task).

recognition model was trained using this list to recognize basic knowledge in algorithms and to extract the
lP
relationship between the target basic knowledge and the current algorithm.
To this point, raw data with all the learning objects and relationships has been obtained. However, since
the data from the network is not always reliable, the data need to be processed using expert knowledge. In this
step, we manually proofread the information about learning objects and corrected wrong relationships. The
procedure of manually proofread mainly includes (a) 2 annotators determine whether the learning objects and
relationships are correct in raw data; (b) compare the results from annotators, we regard the agreements with
rna

correct labels as “correct data”, agreements with incorrect labels and all disagreements as “incorrect data”;
(c) the “incorrect data” is checked and annotated by a super-annotator which is the expert. For the procedure
(b), we use Cohen’s kappa coefficient [36] to verify the reliability of the annotation from inter-annotators, the
result is 0.63 which means the annotations are substantial agreement. A knowledge graph was ultimately built
that contained 225 basic knowledge objects, 361 algorithm objects, 89 task objects, and 1,033 relationships.
The accuracy of the learning object information (96.7%) and relationships (94%) proves that our knowledge
graph reliably supports the learning path recommendation. For detailed introducing our knowledge graph,
Jou

Figure 3.3 (right) abstractly shows an example of the constructed knowledge graph, which contains 2 tasks,
3 algorithms, 4 basic knowledge and 5 different types of relationships. At the same time, a more concrete
example of related algorithms for the specific task “Entity Recognition” is shown in Figure 3.4, part of basic
knowledge was also presented for algorithms respectively in the example.

8
 Journal Pre-proof

of
pro
re-
Figure 3.4: Example of constructed knowledge graph

4 Learning path recommendation model

lP
Based on the knowledge graph, we can start designing and developing the learning path recommendation
model. This section first presents how to generate all the possible learning paths according to the specific
learning need, and then presents how to recommend an appropriate learning path from these possible learning
paths. The structure of the recommendation model and detailed algorithms will be introduced at the end.
rna

4.1 Learning path generation algorithm

In order to make our proposed learning path recommendation model more user-friendly, we design an
algorithm based on the multidimensional knowledge graph to generate the possible learning path through the
e-learner’s target learning object and the learning need. This algorithm replaces the learning path generation
algorithm [10] which using the starting and ending (target) learning objects. Because most of the e-learners
only know the target learning object but not the starting node in general. In this algorithm, the semantic
relationships in the multidimensional knowledge graph will be used as constraints for generating the learning
Jou

path.
The first step of this algorithm is to calculate the relationship constraints φ based on the learning need.
The relationship constraints refer to a set of relationships that may appear in the current learning path. A
relationship outside the constraints is not allowed in the learning path. According to the learning need Nu , all
the corresponding relationship constraints φ = (α, β, γ, δ...) will be output by get RelationConstraint(Nu ).
It is worth noting that φ v RE. The second step of the algorithm is to generate the learning path according to
the relationship constraints φ. Learning path generation will start with the target learning object, and search

9
 Journal Pre-proof

of
for the next learning object connected by the relationship under the constraints. Then, the search continues
from the connected learning object. When the current learning object has no connected learning object, this
is the start (the first learning object) of the current learning path. Based on this, the algorithm will start a
greedy search from the target learning object until it generates all the possible learning paths P. The detailed

pro
algorithmic process is presented in algorithm 3.

4.2 Learning path recommendation algorithm

Based on the e-learner’s input, the learning path recommendation model may generate many possible
learning paths. However, the model should not output multiple learning paths all at once as the result
recommended to the e-learner. Therefore, the optimal learning path needs to be selected for the e-learner. In
re-
this context, we have designed a learning path recommendation algorithm as shown in Equation (2). The
target of maxi=0. . . k Score(Pi ) is to output the recommended learning path Pb with the highest score.

n
X
maxi=0...k Score(Pi ) = maxi=0...k (w j ∗ f j (Pi )) (2)
j=1

In Equation (2), Pi is a specific learning path which needs to be scored. k is the total number of these learning
paths. n is the total number of features in feature set. f is a function for calculating the corresponding feature
lP
of Pi . And w j is the weight of the feature f j (Pi ). Considering the model needs to quantize the learning path
in order to select the one with the highest score. According to the previous work [37, 38] on quantizing
algorithms, we propose a feature set (F) which are publication time ( f1 ), citation count ( f2 ), search frequency
( f3 ), the impact of the publisher ( f4 ) and the impact of the author ( f5 ). Table 4.1 shows the detail about these
features, including the formal expression of the feature and the description of the parameters in the expression.
rna

Research [39, 40, 41] present that e-learners concern on the novelty, authority and popularity of the
research. In this paper, we propose these three directions as the learning preference of e-learner. Specific
learning preference represents the characteristic of the learning path the e-learner more concern with. The
weighting method (W) was applied to constrain features in order to satisfy different learning preferences. We
have provided learning preference options where each weight (wi ) corresponds to a particular feature ( fi ).
Details of weight distributions are as follows:
1. Novelty: Learner prefers to solve a problem using novel algorithms. When w1 =1, w2 =0, w3 =0, w4 =0,
Jou

w5 =0, the feature publication time is considered and the learning path with the most current algorithms
will be selected.
2. Popularity: Learner prefers to learn famous algorithms. When weights are set as w1 =0, w2 =0.5, w3 =0.5,
w4 =0, w5 =0, means the selected learning path is famous in the research community and well-known in
the public.
3. Authority: Learner prefers to use algorithms from authoritative resources. When w1 =0, w2 =0, w3 =0,
w4 =0.5, w5 =0.5, the selected learning path considers the impact of publishers and authors which
represent the authority of publishers and authors.

10
 Journal Pre-proof

of
Xn
W(l pi )
calculateW(l p) = i=1
(3)
n
In this context, e-learners can select learning preference to generate learning path according to their needs. At

pro
the same time, we propose a function for calculating the weight distribution when e-learners have multiple
learning preferences, as shown in Equation (3) . The target of calculateW(l p) is to output the weight
distribution. l p is a set of selected learning preferences from e-learner and l pi represents the specific learning
preference. W is the function for calculating corresponding weights for l pi . n is the number of learning
preferences in l p.
According to the learning path recommendation algorithm above, our model can recommend the most
appropriate learning path among multiple learning paths.

Features
re-
Table 4.1: Features for quantizing algorithms

Correlated Factors Parameters

Publication time
Tp −E(Tp )
f1 (Pi ) = Tc −E(Tp ) Tp is the publishing year of the algorithm. E(Tp ) represents the
earliest publishing year for algorithms in the KG. Tc is the current
year.
lP
Citation count f2 (Pi ) = Cc
M ax(Cc ) Cc is the citiation count of the algorithm. Max(Cc ) represents the
maximum Cc in the KG.
Search frequency f3 (Pi ) = Fs
M ax(Fs ) Fs is the search frequency of an algorithm in Google trends [38],
the interval of Fs is [0,1]. Max(Fs ) represents the maximum Fs
in the KG.
Impact of publisher
H p −Min(H p )
f4 (Pi ) = Hp is the H5-index [37] of the publisher. Min(Hp ) and Max(Hp )
rna

M ax(H p )−Min(H p )
respresent the mimimun and maximum H5-index value.
Impact of author f5 (Pi ) = Ha −Min(Ha )
M ax(Ha )−Min(Ha ) Ha is the H5-index of the algorithm’s authors. Min(Ha ) and
Max(Ha ) respresent the mimimun and maximum H5-index value
of the author.

4.3 Learning path recommendation model based on knowledge graph

Jou

The structure of the learning path recommendation model is presented in Figure 4.1. The input query
Q from the learner should be a query sentence in natural language. The output of the model should be the
recommended learning path.
The details of the algorithms in the model are presented below:
1. The main procedures of how to generate the recommended learning path according to the learner’s
query are shown in algorithm 1.
2. The learner’s input query will be sent to the model for extracting the target learning object and learning

11
 Journal Pre-proof

of
pro
Figure 4.1: The structure of the learning path recommendation model
re-
need in algorithm 2. The raw input query Q needs to be converted into the token set by preprocessing.
This step includes removing stop words, tokenization, lemmatization and some other basic natural
language preprocessing methods. Then, the model needs to recognize the target learning object and
its class. We have established a list of frequent requirements that will be used to recognize the query
keywords in step 4. Lastly, the learning need will be obtained from the class of the target learning
lP
object and the query keywords.
3. In algorithm 3, the model will traverse the knowledge graph to find the learning path set that contains
all the eligible learning paths. In step 1, the function φ = get RelationConstraint(Nu ) will constrain
the type of relationship to be used in this round, according to the learning need Nu . Steps 2-14 aim to
construct an iterative algorithm for extracting all the possible learning paths for set P, based on the
rna

target learning object Ot .

4. Algorithm 4 is the learning path scoring algorithm we introduced in the previous section. In this
algorithm, the model will ultimately output the recommended learning path with the highest score.
Jou

12
 Journal Pre-proof

of
Algorithm 1 Learning path recommendation algorithm based on KG. Pb = RecommendationPath(G, Q, F).
Input:
The knowledge graph G = (LO, RE);
The learner’s query Q;

pro
The feature set F;
Output:
The recommended learning path Pb .
1. Ot , Nu = RequireAnal yze(Q); {analyze the learner’s target learning object Ot and learning need Nu .}
2. P = GetPossiblePaths(G, Ot , Nu ); {get the learning path set P which includes all possible learning
paths.}
3. Pb = Get MaxScorePath(P, l p); {select the recommended learning path Pb with the highest score.}
4. return Pb ;
re-
Algorithm 2 Input processing: extract the target learning object and analyze learning need. Ot , Nu =
RequireAnal yze(Q).
Input:
lP
The learner’s query Q;
Output:
The target learning object Ot ;
The learner’s learning need Nu ;
1. T = preprocessing(Q); {convert raw query Q to the token set T.}
2. Ot = getT arget(T); {obtain the target learning object Ot from T.}
rna

3. c = getClass(Ot ); {get the class of the target learning object Ot .}

4. K = getQuer yK(T); {extract the learner’s query keywords set K from T.}
5. Nu = calculateN(c, K); {calculate the learner’s learning need Nu by c and K.}
6. return Ot , Nu ;
Jou

13
 Journal Pre-proof

of
Algorithm 3 Generate possible learning path set. P = GetPossiblePaths(G, Ot , Nu ).
Input:
The knowledge graph G;
The target learning object Ot ;

pro
The learner’s learning need Nu ;
Output:
The possible learning path set P;
1. φ = get RelationConstraint(Nu ); {generate a set of relationship constraints φ = (α, β, γ, δ...).}
2. function f indPath(Ot )
3. R = get Relations(Ot ); {determine the relationships R that are directly related to Ot .}
4. if {r |r ∈ R, r ∈ φ} not exists then
5.
6.
7.
else
for all r ∈ R do
re-
P.add(p); {Put the possible learning path p into learning path set P.}

8. if r ∈ φ then
9. Oi = getO(Ot , r); {get the learning object Oi related to Ot with r.}
10. p.addRelation(r, Oi ); {put r and Oi into the possible learning path set p.}
lP
11. recursively apply f indPath(Oi );
12. end if
13. end for
14. end if
15. end function
rna

16. return P;
Jou

14
 Journal Pre-proof

of
Algorithm 4 Recommended the learning path Pb with the highest score. Pb = Get MaxScorePath(P, l p).
Input:
The possible learning path set P;
The learner’s learning preference l p;

pro
Output:
The recommended learning path Pb ;
1. for all Pi ∈ P do
2. F = calculateF(Pi ); {calculate the corresponding feature set F of the algorithms in the learning path
Pi .}
3. W = calculateW(l p); {generate a weight distribution W according to l p, each feature f j has
a corresponding weight w j .}
re- Xn
4. Score(Pi ) = calculateScore(W, F); {the score s = w j ∗ f j (Pi ), w j ∈ W, f j ∈ F.}
j=1
5. end for
6. Pb = maxi=0...k Score(Pi ); {recommend the learning path Pb with the highest score.}
7. return Pb ;

5 E-learning system design

lP
The quality of learning content is the most important part of the learning path recommendation, the
learning quality can only be guaranteed when the learning content is accurate. However, the way in which
learning content is presented to e-learners is also an important factor in determining the quality of learning.
An e-learning system has been built based on our learning path recommendation model for providing the
rna

e-learning service. This section introduces the design of the e-learning system.
Figure 5.1 shows the structure of the e-learning system. The system consists of three parts: a graph
database, the website backend, and the website interface.

• Graph database: To enable the website to access data, we stored the knowledge graph in a graph
database that can be easily operated from the website backend. The graph database contains all the
learning objects and relationships, which will be called when generating learning paths and searching
learning objects.
Jou

• Website backend: The website backend is responsible for building the function and outputting the target
result to the website interface. After receiving the input query from the text description window, the
semantic processing unit will analyze the learning requirement from the input query. Then, the kernel
algorithm unit will return the possible learning path set P and the recommended learning path Pb .

• Website interface: As a window for interaction with e-learners, the website interface consists of a
text-description window and a graph-display window. The possible learning path set P will be presented

15
 Journal Pre-proof

of
in the graph-display window, and the target recommended learning path and guidance will be presented
in the text-description window.

pro
re-
Figure 5.1: The structure of the e-learning system
lP
As shown in Figure 5.1, the proposed e-learning system combines dynamic graphic display and text
guidance for presenting the learning content. We applied the text description to succinctly demonstrate the
recommended learning content. Furthermore, the target learning object and its related learning object set were
presented through the operable graph-display window to maximize the possibility of discovering potential
learning content. In the dynamic graph-display window, a mouse-click on a learning object will extend to
rna

more learning objects. The mouse-click will also display the introductory information of the corresponding
target learning object. Based on these functions, the system fully exploits the connectivity of the knowledge
graph when presenting the learning content, so that e-learners can explore more potential learning content
while using our learning path recommendation system.

6 Evaluation and analysis

Jou

This section describes how we evaluate the learning path recommendation model, which includes a
description of the experiment process and the experiment data analysis.

6.1 Experiment process description

This learning path recommendation model aims to output the target learning content according to
e-learner’s input query. The quality of output learning content should be evaluated by the e-learner. Therefore,
experiments are designed using the extrinsic evaluation method to evaluate the learning path recommendation

16
 Journal Pre-proof

of
model. In the experiment, we will verify whether (i) the learning path generation algorithm based on the
multidimensional knowledge graph is better (higher search success rate and more user-friendly) than the
algorithm based on the one-dimensional knowledge graph; (ii) e-learners are satisfied with the recommended
learning path.

pro
The experiment process consists of three parts, as shown in Figure 6.1.

re-
lP
Figure 6.1: Experiment process of verifying learning path recommendation model

Control experiment:

We have designed a control experiment to verify the hypothesis (i). Two learning path recommendation models
are used in the control experiment. The test model uses our learning path generation algorithm based on the
rna

multidimensional knowledge graph; this algorithm can generate the learning path by the target learning object.
The control model uses the learning path generation algorithm based on the one-dimensional knowledge
graph, which generates the learning path using the starting and ending learning objects. The control model
was developed to simulate the state-of-the-art learning path generation method [10]. Participants will be
divided into test and control groups without telling which group they are in. Then, they will be asked to use
the corresponding model to conduct 10 learning path searches based on their own learning interests.
Jou

Feedback collecting:

Participants are asked to record and score their search results, in which feedbacks from the test group will be
used to verify the hypothesis (ii). The score represents the participants’ satisfaction with the recommended
learning path. We use the Likert scale to quantify participant satisfaction. Five points represents very satisfied;
four points represents satisfied; three points represents generally satisfied; two points represents unsatisfied;
and one point represents very unsatisfied. Table 6.1 shows some examples of scored results. It noteworthy
that example 3 presents a failure case that did not yield a target learning path based on the input query.

17
 Journal Pre-proof

of
Table 6.1: Examples of participants’ feedback

Learning need Target learning object Recommended learning path Score

CNN, RCNN, Fast RCNN, Faster
Example 1 #2 Faster RCNN 5

pro
RCNN
Maximum Likelihood Esti-
mation, Viterbi Algorithm,
Example 2 #6 Entity Recognition Baum Welch Algorithm, Hid- 4
den Markov Model, Entity
Recognition
Example 3 #5 3D Reconstruction Query failed 0
re-
Next, the participants are asked to complete a questionnaire to collect basic participant information and
feedback on system usability. The feedback should represent participant satisfaction with the learning
experience, also rated using the Likert scale.

Results analysis:
lP
Based on the data collected from the feedback, we will evaluate our learning path recommendation model
through a data analysis. The expected performance of the model will be verified in this step.

6.2 Analysis of experiment results

The experiment included 251 participants and generated 2,440 qualified experiment result records; 1,470
records were from the test group, and 970 records were from the control group. Figure 6.2 and Figure 6.3
rna

show some basic information about the 251 participants, we can find that participants are balanced distributed
in gender and location. It is worth to mention that all participants were randomly divided into two groups
with a similar gender ratio and the ages of all participants are between 19 to 26 years old.
Jou

Figure 6.2: Participants’ gender information

18
 Journal Pre-proof

of
pro
Figure 6.3: Participants’ provincial citizenship information
re-
lP

Figure 6.4: Distribution of records containing different learning needs

Figure 6.4 shows the distribution of records containing different learning needs; the different colors
rna

represent the six learning needs. Learning needs #5 and #6 represent the largest portion of the pie chart,
which means that e-learners are more concerned about applying knowledge learned to practical tasks. At the
same time, e-learners are less likely to directly search for the target learning object (#4) and are also not much
concerned about the position of the learning object in the hierarchy (#3). We are inspired by this analysis that
shows that by using the learning path recommendation model, e-learners favor learning the evolution of the
target knowledge (#1 and #2) and the application of the target knowledge (#5 and #6). Subsequent studies on
learning path recommendation should focus on these two points.
Jou

To verify the hypothesis (i), we need to calculate the success rate based on the learning paths returned
by the two models. During the experiment, all failed queries and incorrectly returned learning paths were
counted as failures; the remaining results were counted as successes. Figure 6.5 shows the success rate of the
learning paths returned by the two models, labels on the horizontal axis represent different learning needs.
Since learning need #4 only requires a single learning object, we did not include it in this statistic. According
to Figure 6.5, the learning path generation success rate of the experiment group is significantly higher than
that of the control group.

19
 Journal Pre-proof

of
We found that although the success rate of the control model is lower, it can still generate correct learning
paths for learning needs #1, #2, #5, and #6. However, it cannot generate learning paths for #3. The reason is
that, unlike learning path generation for #1, the starting node is unknown while the ending node is known.
When the model needs to present the hierarchy for the current learning object in learning need #3, either the

pro
starting node or the ending node is unknown. However, there is no significant decrease in the success rate of
the test model under #3. This indicates that by using the semantic relationship, the learning path generation
algorithm based on the multidimensional knowledge graph is robust and will not be affected by a missing
learning object in the learning path.

re-
lP

Figure 6.5: Success rate of learning path generation under different learning needs
rna

When aggregating the basic data about the participants collected from the questionnaire, we found that
about half of the participants who successfully generated learning paths using the test model do not have
the relevant learning experience. However, most of the successful participants in the control group have
relevant learning experience. This indicates that the test model is user-friendly for inexpert e-learners who
need guidance.
The above evaluation verifies the hypothesis (i): the learning path generation algorithm based on the
multidimensional knowledge graph is better than the algorithm using one-dimensional knowledge graph. This
Jou

indicates that generating learning paths through semantic relationships is more user-friendly and efficient.
Therefore, it proves that the multidimensional knowledge graph framework we designed is more reasonable
for generating learning paths.
To verify the hypothesis (ii), we have counted the participants’ satisfaction with the recommended
learning path results. Table 6.2 shows some statistics of participants’ records in the test group. We find that
the average total satisfaction score is 3.9. The statistical median of the satisfaction score is 4. These results
prove that most participants are satisfied with the recommended learning path in general. When looking

20
 Journal Pre-proof

of
at the satisfaction scores for different learning needs, all the scores are over 3.6 except for learning need
#3; the scores for #1, #2, and #4 are even above 4.0. This indicates that, in most cases, participants are
satisfied with the selected learning paths for different learning needs. Therefore, this proves that our learning
path recommendation model can recommend appropriate learning paths according to the learning need of

pro
e-learners.

Table 6.2: Statistics of participants’ records in the test group

Learning Average Median Number of Average length of

need satisfaction score satisfaction score records Learning path
#1 4.2 4 242 3.4
#2 4.3 4 246 2.6
#3
#4
#5
2.7
4.7
3.7
re- 2
5
4
107
130
390
7.7
1
3.9
#6 3.6 4 355 4.4
Total 3.9 4 1470 3.8
lP
It is worth noting that the relationship constraints of learning needs #1 and #2 are fewer than those for #3,
#5, and #6. Figure 6.5 shows that the test model is good at generating learning paths with fewer relationships,
but the success rate decreases when generating learning paths with more relationships. Table 6.2 presents
a similar trend as Figure 6.5. The satisfaction score is higher for learning needs with simple relationship
constraints such as #1, #2, and #4 (#4 has no relationship constraint). However, the satisfaction score
decreases when the relationship constraints become complex, especially for learning need #3 which has the
rna

lowest average satisfaction score. At the same time, we can also find that the average satisfaction score and
median satisfaction score decrease when the length of the learning path increases according to Table 6.2.
This suggests that our learning path recommendation model still needs to improve in its ability to generate
appropriate learning paths within the complex relationship constraints and longer length.

7 Conclusion and future work

Jou

In this paper, we presented a learning path recommendation model based on a multidimensional

knowledge graph that resulted in e-learners that are more satisfied with the learning paths recommended for
his learning needs. We designed a multidimensional knowledge graph with diverse relationships between
learning objects for presenting learning content. Based on our knowledge graph, we proposed a learning path
generation algorithm which can generate the learning path by the target learning object. This algorithm has
been proven to be more efficient in generating suitable learning paths than previous algorithms. Furthermore,
we proposed a weighted coefficient scoring method for selecting the target learning path considering the

21
 Journal Pre-proof

of
e-learner’s learning needs. In conclusion, our learning path recommendation model can generate satisfactory
learning paths for e-learners and improves their learning efficiency.
This paper introduced a method for efficiently recommending suitable learning paths for e-learners that
can be expanded and used in any other e-learning knowledge field. However, there are still some limitations to

pro
be noted. When applying this model to another knowledge field, the quality of data collection will affect the
results of the learning path recommendation. At the same time, future research can focus on how to improve
the success rate of generating learning paths with complex relationship constraints, which can improve the
quality of the recommended learning path.

Acknowledgement
re-
This work is supported by the Development Project of Jilin Province of China (No. 20160414009GH,
20170203002GX, 20170101006JC, 20160204022GX), the National Natural Science Foundation of China (No.
61472159, 71620107001, 71232011) and Jilin Provincial Key Laboratory of Big Date Intelligent Computing
(20180622002JC). Premier-Discipline Enhancement Scheme supported by Zhuhai Government and Premier
Key-Discipline Enhancement Scheme supported Guangdong Government Funds.
lP
Author Contributions:

Daqian Shi and Ting Wang designed the data schema of the knowledge graph. Daqian Shi and Ting
Wang built the knowledge graph and designed the learning path recommendation algorithm. Daqian Shi and
Hao Xing designed and built the E-learning system of the use case. Hao Xu designed the research and helped
rna

revise the manuscript. All authors have read and approved the final manuscript.

Conflicts of Interest:

The authors declare no conflict of interest.

References
Jou

[1] Marc J Rosenberg and Rob Foshay. E-learning: Strategies for delivering knowledge in the digital age. Performance Improvement,
41(5):50–51, 2002.
[2] Cristobal Romero and Sebastian Ventura. Data mining in e-learning, volume 4. WIT Press, 2006.
[3] Norsham Idris, Norazah Yusof, Puteh Saad, et al. Adaptive course sequencing for personalization of learning path using neural
network. International Journal of Advances in Soft Computing and its Applications, 1(1):49–61, 2009.
[4] Tiffany Y Tang and Gordon G McCalla. Data mining for contextual educational recommendation and evaluation strategies.
Handbook of Educational Data Mining, page 257, 2010.
[5] Daniela Godoy and Analía Amandi. Link recommendation in e-learning systems based on content-based student profiles.
Handbook of Educational Data Mining, pages 273–286, 2010.

22
 Journal Pre-proof

of
[6] Chungho Su. Designing and developing a novel hybrid adaptive learning path recommendation system (alprs) for gamification
mathematics geometry course. Eurasia Journal of Mathematics, Science and Technology Education, 13(6):2275–2298, 2017.
[7] Chengling Zhao, Zhihui Chen, and Zhifang Huang. Recommendation algorithm and application of adaptive learning path [j].
China Educational Technology, 8:85–91, 2015.

pro
[8] Vincent Tam, Edmund Y Lam, and ST Fung. A new framework of concept clustering and learning path optimization to develop
the next-generation e-learning systems. Journal of Computers in Education, 1(4):335–352, 2014.
[9] Guillaume Durand, Nabil Belacel, and François LaPlante. Graph theory based model for learning path recommendation.
Information Sciences, 251:10–21, 2013.
[10] Haiping Zhu, Feng Tian, Ke Wu, Nazaraf Shah, Yan Chen, Yifu Ni, Xinhui Zhang, Kuo-Ming Chao, and Qinghua Zheng. A
multi-constraint learning path recommendation algorithm based on knowledge map. Knowledge-Based Systems, 143:102–114,
2018.
[11] Imane Kamsa, Rachid Elouahbi, Fatima El Khoukhi, Touria Karite, and Hayat Zouiten. Optimizing collaborative learning path
by ant’s optimization technique in e-learning system. In 2016 15th International Conference on Information Technology Based
re-
Higher Education and Training (ITHET), pages 1–5. Institute of Electrical and Electronics Engineers, 2016.
[12] Eugenijus Kurilovas, Inga Zilinskiene, and Valentina Dagiene. Recommending suitable learning scenarios according to learners’
preferences: An improved swarm based approach. Computers in Human Behavior, 30:550–557, 2014.
[13] Chun Fu Lin, Yu-chu Yeh, Yu Hsin Hung, and Ray I Chang. Data mining for providing a personalized learning path in creativity:
An application of decision trees. Computers & Education, 68:199–210, 2013.
[14] Ebru Özpolat and Gözde B Akar. Automatic detection of learning styles for an e-learning system. Computers & Education,
53(2):355–367, 2009.
lP
[15] Mei-Hua Hsu. A personalized english learning recommender system for esl students. Expert Systems with Applications,
34(1):683–688, 2008.
[16] Chih-Ming Chen. Ontology-based concept map for planning a personalised learning path. British Journal of Educational
Technology, 40(6):1028–1058, 2009.
[17] Chen Liang, Jianbo Ye, Shuting Wang, Bart Pursel, and C Lee Giles. Investigating active learning for concept prerequisite
learning. In Thirty-Second AAAI Conference on Artificial Intelligence. Association for the Advancement of Artificial Intelligence,
2018.
rna

[18] Liangming Pan, Xiaochen Wang, Chengjiang Li, Juanzi Li, and Jie Tang. Course concept extraction in moocs via embedding-
based graph propagation. In Proceedings of the Eighth International Joint Conference on Natural Language Processing, pages
875–884, 2017.
[19] Chen Liang, Jianbo Ye, Zhaohui Wu, Bart Pursel, and C Lee Giles. Recovering concept prerequisite relations from university
course dependencies. In Thirty-First AAAI Conference on Artificial Intelligence. Association for the Advancement of Artificial
Intelligence, 2017.
[20] Liangming Pan, Chengjiang Li, Juanzi Li, and Jie Tang. Prerequisite relation learning for concepts in moocs. In Proceedings of
the 55th Annual Meeting of the Association for Computational Linguistics, pages 1447–1456, 2017.
[21] Hanxiao Liu, Wanli Ma, Yiming Yang, and Jaime Carbonell. Learning concept graphs from online educational data. Journal of
Jou

Artificial Intelligence Research, 55:1059–1090, 2016.

[22] Chih-Ming Chen. Intelligent web-based learning system with personalized learning path guidance. Computers & Education,
51(2):787–814, 2008.
[23] Pragya Dwivedi, Vibhor Kant, and Kamal K Bharadwaj. Learning path recommendation based on modified variable length
genetic algorithm. Education and Information Technologies, 23(2):819–836, 2018.
[24] Mengyuan Chen, Mingwen Tong, Chunmiao Liu, Meimei Han, and Ying Xia. Recommendation of learning path using an
improved aco based on novel coordinate system. In 2017 6th International Congress on Advanced Applied Informatics, pages
747–753. IEEE, 2017.
[25] Mohamed Bendahmane, Brahim El Falaki, and Mohammed Benattou. Individualized learning path through a services-oriented

23
 Journal Pre-proof

of
approach. In Europe and MENA Cooperation Advances in Information and Communication Technologies, pages 95–102.
Springer, 2017.
[26] Tung-Cheng Hsieh and Tzone I Wang. A mining-based approach on discovering courses pattern for constructing suitable
learning path. Expert systems with applications, 37(6):4156–4167, 2010.

pro
[27] Tzone I Wang, Kun Hua Tsai, Ming-Che Lee, and Ti Kai Chiu. Personalized learning objects recommendation based on the
semantic-aware discovery and the learner preference pattern. Educational Technology & Society, 10(3):84–105, 2007.
[28] Hsiao-Chien Tseng, Chieh-Feng Chiang, Jun-Ming Su, Jui-Long Hung, and Brett E Shelton. Building an online adaptive
learning and recommendation platform. In International symposium on emerging technologies for education, pages 428–432.
Springer, 2016.
[29] Matteo Gaeta, Francesco Orciuoli, and Pierluigi Ritrovato. Advanced ontology management system for personalised e-learning.
Knowledge-Based Systems, 22(4):292–301, 2009.
[30] Shanshan Wan and Zhendong Niu. A learner oriented learning recommendation approach based on mixed concept mapping
and immune algorithm. Knowledge-Based Systems, 103:28–40, 2016.
re-
[31] Shimaa Ouf, Mahmoud Abd Ellatif, Shaimaa Ezzat Salama, and Yehia Helmy. A proposed paradigm for smart learning
environment based on semantic web. Computers in Human Behavior, 72:796–818, 2017.
[32] Vadim Shmelev, Maria Karpova, and Alexey Dukhanov. An approach of learning path sequencing based on revised bloom’s
taxonomy and domain ontologies with the use of genetic algorithms. Procedia Computer Science, 66:711–719, 2015.
[33] Kuo-Kuang Chu, Chien-I Lee, and Rong-Shi Tsai. Ontology technology to assist learners’ navigation in the concept map
learning system. Expert Systems with Applications, 38(9):11293–11299, 2011.
[34] Fausto Giunchiglia, Biswanath Dutta, and Vincenzo Maltese. Faceted lightweight ontologies. In Conceptual modeling:
lP
foundations and applications, pages 36–51. Springer, 2009.
[35] Yanru Zhu, Peng Wang, Yaqin Fan, and Ying Chen. Research of learning path recommendation algorithm based on knowledge
graph. In Proceedings of the 6th International Conference on Information Engineering, page 11. Association for Computing
Machinery, 2017.
[36] Mary L McHugh. Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276–282, 2012.
[37] Gyula Mester. Rankings scientists, journals and countries using h-index. Interdisciplinary Description of Complex Systems,
14(1):1–9, 2016.
rna

[38] Hyunyoung Choi and Hal Varian. Predicting the present with google trends. Economic Record, 88:2–9, 2012.
[39] You-Na Lee, John P Walsh, and Jian Wang. Creativity in scientific teams: Unpacking novelty and impact. Research Policy,
44(3):684–697, 2015.
[40] JB Arbaugh. Learning to learn online: A study of perceptual changes between multiple online course experiences. The internet
and higher education, 7(3):169–182, 2004.
[41] Lynn Silipigni Connaway, Joyce Kasman Valenza, Christopher Cyr, Tara Tobin Cataldo, Amy G Buhler, Ixchel M Faniel,
Rachael Elrod, Randy A Graff, Samuel R Putnam, Brittany Brannon, et al. Authority, context and containers: Student
perceptions and judgments when using google for school work. World Library and Information Congress, 2017.
Jou

24
 Journal Pre-proof

*Conflict of Interest Form

of
pro
re-
lP
rna
Jou
 Journal Pre-proof

of
pro
re-
lP
rna
Jou
 Journal Pre-proof

*Credit Author Statement

CRediT author statement

of
Daqian Shi: Conceptualization, Methodology, Software, Validation, Formal analysis, Data
Curation, Writing - Original Draft, Writing - Review & Editing.

Ting Wang: Methodology, Data Curation, Writing - Original Draft.

pro
Hao Xing: Software, Data Curation.

Hao Xu: Conceptualization, Writing - Original Draft, Writing - Review & Editing.

re-
lP
rna
Jou