(IJACSA) International Journal of Advanced Computer Science and Applications,

Vol. 11, No. 1, 2020

Hybrid Machine Learning Algorithms for Predicting

Academic Performance
Phauk Sokkhey1 Takeo Okazaki2
Graduate School of Engineering and Science Department of Computer Science and Intelligent Systems
University of the Ryukyus University of the Ryukyus
1 Senbaru, Nishihara, Okinawa, 903-0123, Japan 1 Senbaru, Nishihara, Okinawa, 903-0123, Japan

Abstract—The large volume of data and its complexity in In the age of the information revolution, analysis of the
educational institutions require the sakes from informative database in education environments such as learning analytics,
technologies. In order to facilitate this task, many researchers predictive analytics, educational data mining, and machine
have focused on using machine learning to extract knowledge learning techniques has become a hot area of research [3-5].
from the education database to support students and instructors The supervised learning was used to predict, classify the
in getting better performance. In prediction models, the students’ performance and analyze their learning behaviors to
challenging task is to choose the effective techniques which could follow up on their progress in classes. However, the
produce satisfying predictive accuracy. Hence, in this work, we challenging task is to find the optimal algorithm which could
introduced a hybrid approach of principal component analysis
produce satisfying results. Machine learning algorithms such
(PCA) as conjunction with four machines learning (ML)
algorithms: random forest (RF), C5.0 of decision tree (DT), and
as naïve Bayes, logistic regression, artificial neural networks,
naïve Bayes (NB) of Bayes network and support vector machine decision tree, random forest, support vector machine, k-nearest
(SVM), to improve the performances of classification by solving neighbor, and more, were popularly used to analyze and predict
the misclassification problem. Three datasets were used to academic performance [3-14]. The performance of each model
confirm the robustness of the proposed models. Through the is varied from dataset to dataset, which relies on the
given datasets, we evaluated the classification accuracy and root characteristics and quality of data.
mean square error (RSME) as evaluation metrics of the proposed
In the classification problem, a reason for misclassification
models. In this classification problem, 10-fold cross-validation
that declines the performance of the model is from the quality
was proposed to evaluate the predictive performance. The
proposed hybrid models produced very prediction results which
of data that disturbs the algorithms. Various literature has
shown itself as the optimal prediction and classification focused on using dimensional reduction (feature selection and
algorithms. feature extraction methods) to improve the prediction and
classification performance. In our work, we applied principal
Keywords—Student performance; machine learning component analysis (PCA) as a feature extraction technique to
algorithms; k-fold cross-validation; principal component analysis transform the original dataset into a new dataset of high quality.
We also introduced 10-fold cross-validation is to evaluate the
I. INTRODUCTION predictive performance of the models and to judge how they
The poor performance of students in high school has perform in a new dataset, the testing samples or test data.
become a worried-task for educators as it affects the secondary This paper aims at proposing a novel hybrid approach of
national exam and step to higher education. Mathematics is machine learning for solving the classification problem. The
considered as the basic background for many science subjects, proposed hybrid approach is the combination of four baseline
and give very strongly affect the national exam and for further machine learning algorithms with 10-fold cross-validation and
study in higher education [1]. For example, students who are principal component analysis.
poor in mathematics are much more likely to fail in diploma
national exams in Cambodia [2]. They later found themselves II. RELATED WORKS
harder to choose a major for higher study and hard to survive Supervised learning in machine learning requires an
in the university journey. Early prediction and classification of effective prediction model for solving prediction and
student performance level offers an early warning and gives a classification problems. As mentioned in the Introduction, the
recipe for improving the poor performance of students as well educational data mining (EDM) field has studied different
as for other managerial settings. Hence, we aim to deal with the machine learning techniques to determine these techniques
unknown behavior pattern of students which affects student obtaining a high accuracy to predict the future performance of
performance. There are various factors affect the performance students [3-5].
of students in mathematics; those factors consist of schooling
factors, domestics or home factors, and personal or individual Table I summarized the popular and state-of-the-art
factors. These related factors were used as predictive features classification algorithms, which were used to predict student
in predicting the achievement of students in mathematics. performance in educational datasets. Several works have been
investigated to find the best algorithms to predict future
performance.

32 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 11, No. 1, 2020

TABLE. I. SUMMARY OF COMMON MACHINE LEARNING CLASSIFIERS separate each two different data classes. Once the data is more
WHICH ARE USED IN PREDICTING STUDENT PERFORMANCE
complex, then we create more dimensional space to have a
Ref. Main Results linear separation of data.
(i) C4.5 and Randomtree were proposed. Given a training sample ( xi , yi ), i  1, 2,..., m , where xi  n
[6]
(ii) C4.5 could produce the highest accuracy.
and yi  {1,1} are called the target classes, the classical SVM
(i) The six classifiers are decision tree (DT), random forest (RF),
artificial neural network (ANN), Navie Bayes (NB), logistic classifier is subject to solve the optimization problem:
[7]
regression (LR), and generalized linear model (GLM).
(ii) The RF was found to be the best classifier. 1  m 
min  wT w  C   i  
(i) C4.5, NB, 3-nearest neighbor (3-NN), backpropagation (BP), w , b ,
2  i 1  
[8] sequential minimal optimization (SMO), LR were proposed,
(ii) NB algorithms produced the highest classification result. subject to: yi ( w  ( xi )  b)  1-  i , i  0, i,
T
(1)
(i) Three tree-based classifiers: J48, Random Tree, and REPTree
[9] were used. where  ( x) is treated for nonlinear function case mapping
(ii) J48 was found to be the best prediction model.
x into a higher dimensional space. The parameters w, b and
(i) NB, support vector machine (SVM), C4.5, CART are used to
[10] build the learning model.  i represent the weight, bias, and slack variable, respectively.
(ii) SVM is the best model compared to NB, C4.5, and CART. And the optimal hyperplane is possibly to be solved using
(i) RF, multilayer perceptron (MLP), and ANN were used to Lagrangian and then transform it into a quadratic problem of
[11] classify student performance. the function W ( ) as in (2):
(ii) The RF algorithms generated the highest accuracy.
(i) J48, CART, and RF classifiers were proposed with principal
m
1 m m
[12] component analysis (PCA). max W ( )    i   i j yi y j K ( xi , x j )
(ii) PCA-RF was found to generate the highest accuracy. i 1 2 i 1 j 1
m

[13]
(i) MLP, Radial Bias Function (RBF), SMO, J48, and NB are
proposed to combine with PCA.
subject to:   i yi  0;  i  [0, C], i  1, 2,.., m,
(ii) PCA-NB generated the highest accuracy.
i 1 (2)
(i) Three Boosting algorithms (C5.0, AddaBoost M1., and
where K ( xi , x j )   ( xi )T  ( x j ) is the kernel function and,
[14] AdaBoost SAMME) are proposed.
(ii) The C5.0 outperformed the other two boosting models.   (1 ,  2 ,...,  m ) is a set of Lagrange multipliers.
III. MACHINE LEARNING ALGORITHMS The decision function can be written as:
We proposed hybrid models by a conjunction of machine  m 
learning algorithms with principal component analysis. We f ( x)  sgn 


 y K ( x , x )  b  .
i i i j
first proposed the baseline models. We then improved the i 1 (3)
performance of our proposed baseline models with k-fold
cross-validation. Lastly, we proposed the hybrid machine Different kernel functions are used to help SVM to
learning model by combining it with principal component maximize margin hyperplanes to obtain the optimal solution.
analysis as in Fig. 1. The most popular used kernels are the polynomial function,
sigmoid function, and radial basis function. SVM with radial
A. The Baseline Models bias function (RBF) kernel is one of the most commonly used
There are numerous effective machine learning approaches kernels for the multi-classification problem since it requires
that have been extensively applied to educational environments. fewer parameters comparing to the polynomial kernel.
For various purposes in educational settings, we need to take Consequently, RFB is an appropriate choice to be used kernel.
different machine learning techniques such as association rule Hence, this work applied RBF as a kernel function top to get
mining, regression analysis, classification, and clustering [3]. the optimal solution.
Classification is a common technique in machine learning that
was used in order to classify and predict the categories or
predefined classes of target variables. In this work, we
observed several machine learning classifiers and selected the
four state-of-the-art methods which are popularly used in
predicting academic performances [3-14]. The four proposed
algorithms are support vector machine, naïve Bayes C5.0 of the
decision tree, and random forest.
1) Support vector machine: A Support Vector Machine
(SVM) is a kind of classification algorithm obtained by the
mean of a separating hyperplane [15]. The concept of SVM is
to create a line or a hyperplane to separates the samples into
classes. SVM is used to observe for the optimal hypersurface to Fig. 1. Illustration of Task Procedure.

33 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 11, No. 1, 2020

2) Naïve Bayes (NB): NB is one among the simple but


Sv
effective machine learning algorithms that is preferably used in IG ( S , A) =I ( S )  I ( Sv ).
vValue ( A)
S
many classification problems. NB is a very attractive method (7)
for education research [16]. In the educational domain, an
One complicated matter after splitting is that a split result
assumption of conditional independence is often ignored and in more than one partition that is what we need to compute
disturbed. Considering that variables are inter-connected, the what is called split information in the following equation:
NB classifier can tolerate strong supervising dependence
between independent variables. NB classifier is Bayes  | Si | 
v
| Si |
theorem-based method that used the idea of computing SplitInfo A ( S )    | S | log
i 1
2 
 |S| 
.
posterior probability for decision rule. NB classifier has been (8)
especially popular for educational data mining. Suppose D is Then, using information gain as see formula (7), and
a dataset of n dimensional vector X : ( x1 , x2 , x3 ,..., xn ) splitting information as in (8), we then can compute the
describing attributes of each student and suppose there are k information gain ration using the following equation:
classes: C1 , C2 ,..., Ck . NB classif-ier predicts X belong to a
class Ci if and only if P(Ci | X )  P(C j | X ) for all IG ( S , A)
GainRatio( S , A)  .
1  j  k , i  j . The NB classifier is found on conditional SplitInfoA ( S )
(9)
Bayes probability as in (4):
The C5.0 of a decision tree is one of the most popular
P (Ci )  P ( X | Ci ) machine learning algorithms that has been widely used in
P (Ci | X )  . various applications.
P( X )
(4)
4) Random Forest (RF): As in the name indicates its
The probability P ( X ) is normalizing constant and meaning, the random forest is an algorithm builds the forest
X  ( x1 , x2 ,..., x p ) is the set of features variables with a strong with a number of trees. A random forest algorithm is a tree-
assumption of independent predictors, then (4) can be based tool that grows many classification trees [12]. It is a kind
rewritten as: of ensemble classifier that combines several classification trees
to create a new classifier. The concepts of bootstrap
P (Ci | X )  P (Ci ) P(X | Ci )
aggregation or bagging method is used to grow each tree. To
n
 P (Ci ) P ( x j | Ci ). classify a new example, each decision tree gives a
j 1 classification for the input data which is so-called “voting for a
(5)
class”. The RF algorithm chooses a class with the highest votes.
The naïve Bayes classifier holds many advantages such as The illustration of the process of random forest algorithms is
it is a very simple algorithm, not contain any parameter to shown in Fig. 2.
optimize, efficient for classification, and easy to interpret.
3) C5.0: Decision tree is a "non-parametric white-box
model" which is simple and effective for classification and
regression tasks while C5.0 is one of the most famous
algorithms of decision tree that construct the structure in the
form of tree diagram [14]. This algorithm takes care of various
of the decisions automatically using fairly reasonable defaults.
C5.0 is a successor of C4.5; it builds tree structure from
training set using the idea of Shannon entropy. The algorithm
purifies the subset of samples via the concept of information Fig. 2. Illustration of Random Forest Algorithm.
entropy. Entropy defines the impurity of any subset of an
sample set S at a specific node N is written as: B. The k-fold Cross-Validation
Cross-validation is one of statistical technique that used to
c
test the effectiveness of machine learning algorithms. There are
Entropy ( S )  I ( S )    P(c ) log
i 1
i 2 P (ci ). various methods of cross-validation but the k-fold cross-
(6) validation is chosen since it is popular and easy to understand,
also generally generates a lower bias comparing to the other
The constant c is denoting the number of classes and cross-validation methods. The process of k-fold cross-
P (c i ) is the proportion of values in the class i . After validations is summarized as the following:
obtaining the measure of purity, the algorithm needs to decide
which feature to split next. The algorithm calculates 1) Shuffle the entire samples randomly
homogeneity resulting from a split on each possible feature, 2) Split samples into k sub folds
this procedure of calculation is called information gain (IG) as 3) In the split k sub folds:
shown in (7):  Take 1 fold as a holdout or test set

34 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 11, No. 1, 2020

 Take the remaining k  1 folds as the training set still contains the useful information of data. Removal of such
features can increase the search speed and accuracy rate.
 Retain the evaluation score and discard the model
NB is a classifier that holds many advantages, yet the
4) Repeat the iteration until every single fold was treated greatest weakness of the NB classifier is that it relies on the
as a testing set. Finally, compute the average score of the often-faulty assumption of equally important and independent
recorded scores. features. If there are any features that are irrelevant to some
class Ck then the whole probability goes to zeros for that class
In our study, we chose the 10-fold cross-validation (will be
shortly called 10-CV) to access our proposed algorithms. This because of production in equation (5), which leads to
process is precisely illustrated in Fig. 3. misclassification. In order to solve this problem, feature
extraction will be the best tool to reduce irrelevant features and
C. The Proposed Hybrid Models also improve the classification performance.
The majority task in supervised machine learning is In the tree-based algorithms C5.0 and RF, the major
classification. The classification problem is a hot issue in data problem in the splitting process of the decision tree is
mining and machine learning. We proposed the four most overfitting. Overfitting caused by noisy data and irrelevant
popular classifiers that hold many merits. However, the major features that produce misclassification results. In return,
problem for those classifiers is overfitting and noisy data which overfitting lowering the accuracy of tree-based classifiers. To
leads to misclassification and deduce the accuracy of the reduce high dimensional data which, contains noisy and
classification. To overcome this matter, we try to reduce irrelevant data, a commonly-used technique is to use feature
irrelevant feature and non-correlated features which disturb in extraction in order to obtain a lower-input space that contains
the classification process. In data analysis, it requires more relevant and informative input features.
computational resources and consumes much time when that
data consists of a huge volume. Hence, the feature extraction In order to improve the performance of the proposed
approach to remove noises in data in order to reduce time and machine learning algorithms, we proposed commonly-used
resource usage and regain the high quality of data. The feature extraction approach: principal component analysis
dimensional reduction could improve accuracy and boost up (PCA) in this study. PCA is a statistical method that transforms
the performance by combining it with classification techniques. an original data set to a new dataset of a lower dimension. The
Using more high-quality data and feature reduction is one of original dataset consists of possibly correlated variables are
the effective approaches to improve the performance of converted into a set of linearly uncorrelated variables.
machine learning models. The four proposed models: support
PCA is one of the most popular dimensionality reduction
vector machine using radial basis function kernel (SVMRBF),
algorithm [17]. In the PCA procedure, the data is first
naïve Bayes (NB), decision tree C5.0, and random forest (RF)
transformed into standardized data with zero mean. The idea
are the affective algorithms for the classification problem, yet
behind getting the principle components is the covariance
there is no perfect algorithm in machine learning.
matrix is computed in order to obtain eigenvector and
SVM is a classifier with the use of support vectors called eigenvalues. The eigenvector with the highest eigenvalue is
hyperplanes to separate data into classes. Thus, for a high treated as the principal component of new data which shows
dimensional dataset, the input space is high and can be unclean the most significant relationship of input feature. PCA is less
which is mostly declining the performance of the SVM sensitive to different datasets than other holistic methods, so it
algorithm. Thus, it requires an effective feature extraction is the most widely used technique as one of the effective
method that discards noisy, irrelevant and redundant data, and feature reduction methods.

Fig. 3. Illustration of the K-Fold Cross-Validation Algorithm.

35 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 11, No. 1, 2020

The procedure of transforming original dataset X of l The proposed hybrid models by conjunction machine
dimension consisting of possibly correlated features to a new learning models with PCA are introduced for predicting and
dataset Z of lower dimension m (m  l ) consisting of linear classifying the academic performance. The best benefits of
uncorrelated features is as follows: PCA are summarized as follow:
a) Removing the high noises from samples and
1) Compute mean: From the already processed data, first,
uncorrelated features from the collected dataset in the
find the mean of each attribute using the equation:
preprocessing step.
1 n b) Reducing the high dimensional data to low
  xi
n i 1 dimensional one which remains the important characteristics
(10)
of data that reduce overfitting problems.
2) Compute variance: In order to investigate and deviation c) Enhance the equality of features by getting rid of
of each feature in the dataset, we compute the variance using correlated features that effectively improve the performance of
equation (11): classification.
1
n In this proposed research, we proposed the hybrid models
Var ( X )   x 2 
n 1 (x i -  )2 by a conjunction of four baseline models (SVMRFB, NB, C5.0,
i 1 (11) and RF) with 10-fold cross-validation (10-CV) and principal
component analysis (PCA).
3) Compute covariance: Given two variables, denoted X
and Y , the covariance and correlation are calculated using IV. DATASETS AND PREPROCESSING
equation (12): A. Datasets
1
n
In our study, we tried to collect all unseen features affecting
Cov ( X , Y )   x 2 
n 1  (x i   X )( yi  Y )
student performance in mathematics subjects. Datasets
i 1 (12) contained 43 features describing the information of the
Cov( X , Y ) equals to zero means that the two attributes X learning behaviors of each student and one target variable
describing the performance levels of students based on their
and Y are independent. Using equation (11) and (12), we can score. The predictive features consist of the features observing
obtain covariance matrix S, which the entry sij , i  j , is the from three main affected factors. These main factors contain
covariance between the i th and j th variables, and diagonal sii the forty-three variables and their descriptions are shown in
Table II. Table III described the predefined classes of the target
is the variance of i th variables. variable.
4) Compute Eigenvalues and Eigenvectors: The features in To confirm the robustness and effectiveness of our
the new datasets are characterized by mean of eigenvectors and proposed algorithms, we used three datasets. The first two
eigenvalues. The obtained eigenvectors will tell the direction of datasets are generated datasets namely GDS1 (2000 samples)
new features space while the eigenvalues are its magnitude. and GDS2 (4000 samples) that were constructed based on
The eigenvalues are possible to obtain by solving the equation: proposed structures of predictive features to the output variable
as stated in [18-20]. The third dataset is the actual dataset that
Det ( S -  I )  0, (13) was collected from 22 high schools in Cambodia. The data
collection was made using questionnaires form. Students were
where the covariance matrix S is symmetric,  is the asked to provide their demographical information related to
eigenvalue of the symmetric matrix S , and I is an identity external effects such as domestic factors, individual or student
matrix. The eigenvector v corresponding to each eigenvalue factors, and school factors. The score of mathematics of
 can be computed via the equation: students in the semester I was obtained from the administrative
offices in each school. The dataset was named ADS3 that
( S -  I )v  0 (14) consists of 1204 samples.

We denoted E  {v : ( S   I )v  0} as the Eigen space TABLE. II. THE OUTPUT VARIABLE

containing all eigenvectors. N Performance Levels Score-based discretization
5) Obtain orthonormal eigenvectors: By means of linear 1 Excellent learner 90% and above
algebra concept, we can obtain the nonnegative eigenvalues 2 Good learner 75% to less than 90%
1  2  ...m  0 with corresponding orthonormal 3 Average learner 60% to less than 75%
eigenvectors v1 , v2 ,..., vm . The eigenvectors are called the
4 Slow learner Less than 60%
principal components of the dataset.

36 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 11, No. 1, 2020

B. Preprocessing Tasks
TABLE. III. THE FACTORS AFFECTING STUDENT PERFORMANCE IN
MATHEMATICS Data preprocessing is an integral step in data mining that is
used to transform the raw dataset into a clean and executable
N Variables Description Type format to be ready for implementation. The preprocessing step
Domestic Factors is not only used to ensure the readiness of data suitable and
1 PEDU1 Father's educational level Nominal ready for modeling but also to improve the performance of the
2 PEDU2 Mother's educational level Nominal
models. The preprocessing tasks in this study contain some
operations such as data cleaning or cleansing, data
3 POCC1 Father’s occupational status Nominal transformation, and data discretization. During data collection,
4 POCC2 Mother’s occupational status Nominal the questionnaire completion was done with missing some
5 PSES Family’s socioeconomic Ordinal questions and inputting invalid value (outliers). In our datasets,
6 PI1 Parents’ attention to students’ attitude Ordinal the number of missing values is low, so we used the imputing
method in order to clean our data. We replaced the missing
7 PI2 Parents’ time and money spending Ordinal
value in our categorical variables by its modes or high
8 PI3 Parents’ involvement as education Ordinal frequency-category values. In the output variable, there is a
9 PS1 Parents’ feeling responsive and need Ordinal few missing value and outliers, then we replaced it by the mean
10 PS2 Parents’ respond to children’s attitude Ordinal value. For simplicity, we transformed some numerical features
11 PS3 Parents’ encouragement Ordinal into ordinal types. In our study, we also discretized the output
variables into four performance levels as shown in Table I.
12 PS4 Parents’ compliment Ordinal
13 DE1 Domestic environment for study Ordinal V. EVALUATION METRICS
14 DE2 Distance from home to school Nominal The performance of each proposed model in analyzing and
Student or Individual Factors predicting student performance can be evaluated from the
15 SELD1 Number of hours for self-study Nominal analysis of the graphical confusion matrix. Without loss of
generality, our output variable can be categorized into four
16 SELD2 Number of hours for private math study Ordinal
ordinal categories as mention in Table I. Table IV shows the
17 SELD3 Frequency of doing math homework Ordinal graphical confusion matrix which represents four classes of
18 SELD4 Frequency of absence in math class Ordinal student performance level in mathematics subject. Class 1
19 SELD5 Frequency of preparing for the math exam Ordinal presents the highest class, Class 2 denotes the second upper
20 SIM1 Student’ s interest in math Ordinal class, Class 3 describes the third class lower, and Class 4
denotes the lowest (poor) group of students. The below
21 SIM2 Student’s enjoyment in math class Ordinal
parameters are calculated.
22 SIM3 Student’s attention in math class Ordinal
23 SIM4 Student’s motivation to succeed in math Ordinal
A. Classification Accuracy
24 ANXI1 Student’s anxiety in math class Ordinal Accuracy is used to quantify the percentage of correctly
predicted. Here, we want to evaluate the potential of our
25 ANXI2 Student’s nervous in the math exam Ordinal
prediction model by measuring the percentage of correctly
26 ANXI3 Student’s feeling helpless in math Ordinal predicted the level of student performance as in (15):
27 POSS1 Internet’s use at home Binary
28 POSS2 Possession of computer Binary Accuracy 
a ii
100%
29 POSS3 Student’s study desk at home Binary a ij
(15)
School Factors
30 CENV1 Classroom environment Ordinal
B. Root Mean Square Error (RMSE)
31 CU1 Content’s language in math class Nominal
We aim not only to predict the ability of students'
performance levels but also to estimate how much our
32 CU2 Class session Nominal
prediction is close to their performance level. We encoded
33 TMP1 Teacher mastering in math class Ordinal these ordinal performance levels {slow, average, good,
34 TMP2 Teacher’s absence in math class Ordinal excellent} as {1,2,3,4}, respectively. The RMSE can be
35 TMP3 Teaching methods in math class Ordinal computed as:
36 TMP4 Teacher’s involving in education’s content Ordinal M
( Plia - Plip ) 2
37 TAC1 Math teacher’s ability Ordinal RMSE  
i 1 M
38 TAC2 Teacher’s encouragement to students Ordinal (16)
39 TAC3 Math teacher’s connection with students Ordinal where Pl a  {1, 2,3, 4} is the actual performance level and
40 TAC4 Math teacher’s help Ordinal
Pl  {1, 2,3, 4} is the predicted performance level.
p

41 ARES1 Adequate number of math teacher Nominal

Contrasting with accuracy, the smaller the RMSE, the better
42 ARES2 Adequate use of classroom Nominal the model is. RMSE equal to 0 shows the prediction model is
43 ARES3 Adequate use of math handout Nominal perfect.

37 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 11, No. 1, 2020

TABLE. IV. GRAPHICAL CONFUSION METRIC TABLE. VI. PERFORMANCE OF BASELINE MODELS TO GDS2

Predicted Classes Bassline Models Accuracy RMSE

Slow Average Good Excellent SVMRBF 75.52% 0.489
Slow a11 a12 a13 a14 NB 67.68% 0.664
Actual Classes

C5.0 86.18% 0.372

Average a21 a22 a23 a24
RF 90.37% 0.321
Good a31 a32 a33 a34
Excellent a41 a42 a43 a44 TABLE. VII. PERFORMANCE OF BASELINE MODELS TO ADS3

Bassline Models Accuracy RMSE

VI. EXPERIMENTAL RESULTS
SVMRBF 86.44% 0.823
In our experiments, we proceed in three phases. Phase 1 is
NB 65.02% 1.016
to implement for the result of the baseline models. Phase 2 is to
improve the baseline models by 10-fold cross-validation (10- C5.0 76.55% 0.845
CV). Phase 3 is to execute a hybrid model which is the RF 89.23% 0.516
combination of the baseline models with 10-CV and PCA.
A. Result of Baseline Models TABLE. VIII. PERFORMANCE OF BASELINE MODELS AND BASELINE
MODELS+10-CV TO GDS1
We proposed four most popular machine learning
techniques, random forest (RF), C5.0 of the decision tree, Models Accuracy RMSE
support vector machine using radial basis function kernel SVMRBF 75.01% 0.516
(SVMRBF), and naïve Bayes (NB) of the Bayesian network. SVMRBF + 10-CV 77.08% 0.456
The two performance metrics, classification accuracy, and
RMSE are shown in the tables. NB 35.79% 1.191
NB+ 10-CV 68.03% 0.654
From Table V, VI, and VII, NB was found to be the poorest
model, while the RF technique generates the highest C5.0 78.42% 0.487
performance with respect to both classification accuracy and C5.0+ 10-CV 95.24% 0.185
RMSE, which shown itself as the potential model. RF 80.06% 0.431
B. Results of Baseline Models with k-fold Cross-Validation RF+ 10-CV 96.48% 0.143
The k-fold cross-validation is a technique that is popularly
used in prediction and classification models to split the dataset TABLE. IX. PERFORMANCE OF BASELINE MODELS AND BASELINE
MODELS+10-CV TO GDS2
into k  1 sub folds for training and 1 fold for testing sets, then
rotate the folds. In this experiment, we used 10-fold cross- Models Accuracy RMSE
validation, since it performs best at this split. 90% of the data SVMRBF 75.52% 0.489
was used in the training section, and 10% was used for testing
SVMRBF + 10-CV 94.15% 0.274
purposes as shown in Fig. 3. Lastly, when all interactions were
done, an average of all evaluation metrics is computed. NB 67.68% 0.664
NB+ 10-CV 76.47% 0.498
From Table VIII, the accuracy of SVMRBF was improved
by 2%. The performance of the poor NB classifier was then C5.0 86.18% 0.372
much improved by to 68.03%. The 10-CV technique improved C5.0+ 10-CV 95.69% 0.174
C5.0 and RF with an accuracy increase of 27% and 15%, RF 90.37% 0.321
respectively. RF+ 10-CV 96.58% 0.139
From Table IX, by shuffling the dataset GDS2 with 10-CV,
the accuracy of SVMRBF algorithm was improved from 75.52% TABLE. X. PERFORMANCE OF BASELINE MODELS AND BASELINE
MODELS+10-CV TO ADS3
to 91.15%, which is a very good improvement. NB increased
by an accuracy of 9%. The tree-based classifiers C5.0 and RF Models Accuracy RMSE
were improved by the accuracy of 9% and 6%, respectively. SVMRBF 86.44% 0.823

TABLE. V. PERFORMANCE OF BASELINE MODELS TO GDS1 SVMRBF + 10-CV 90.66% 0.678

NB 65.02% 1.016
Bassline Models Accuracy RMSE
NB+ 10-CV 92.44% 0.145
SVMRBF 75.01% 0.516 C5.0 76.55% 0.845
NB 35.79% 1.191 C5.0+ 10-CV 94.82% 0.114
C5.0 78.42% 0.487 RF 89.23% 0.561
RF 80.06% 0.431 RF+ 10-CV 98.22% 0.113

38 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 11, No. 1, 2020

From Table X, the NB accuracies improved rapidly from In this section, we proposed the hybrid models as the
65.44% to 90.66%. SVMRBF could yields around 4% better combination of 10-CV in the previous section to PCA in order
than the previous baseline SVMRBF. C5.0 and RF are tree- to avoid overfitting and more improvement in predicting
based classifiers that could produce a high risk of over-fitting. performance. Tables XI, XII, and XIII describe the results of
With a 10-CV, we can not only obtain better performance but the proposed models to the three datasets, GDS1, GDS2, and
also avoid overfitting problems too. By mean of 10-CV, ADS3, respectively.
accuracies of C5.0 and RF were improved to 94.82% and 98.22%
which improved 18% and 9%, respectively. We visualized the performance of the proposed models to
the three datasets GDS1, GDS2 and ADS3 in Fig. 4, 5, and 6,
C. Results of Proposed Hybrid Models respectively. In Fig. 4, the accuracy based in dataset GDS1,
Our proposed hybrid models were constructed by combing our proposed hybrid models boost the accuracy of SVMRBF
the baseline models with a feature reduction approach, PCA. from 75.01% to 83.88%, NB from 35.79% to 86.27%, C5.0
Feature extraction is one of the powerful methods in from 78.42% to 98.32%, and RF from 80.06% to 98.92%.
classification models that are used for the purpose of removing In Fig. 5, the hybrid models improved SVMRBF, NB, C5.0,
irrelevant or non-related features. Dimensionality reduction via and RF with accuracies of 20%, 23%, 12%, and 9%,
PCA [13] can definitely serve as regularization in order to respectively. In Fig. 6, the proposed hybrid SVMRBF could
prevent overfitting and improve the model accuracies. Often, improve the classification accuracy from 86.44% to 97.01%.
people end up making a mistake in thinking that PCA selects Classification through NB could yields 30% better than
some features out of the dataset and discards others. The baseline NB. The accuracies of C5.0 and RF were improved to
algorithm actually constructs a new dataset of properties based 99.25% and 99.72% correctly classified.
on a combination of the old ones.

TABLE. XI. PERFORMANCE OF BASELINE MODELS, BASELINE MODELS +10-CV, AND HYBRID MODELS TO GDS1

Average Lowest Highest Average Highest

Models Std. Lowest RMSE Std.
Accuracy Accuracy Accuracy RMSE RMSE
SVMRBF 75.01% 70.27% 77.01% 1.421 0.516 0.460 0.691 0.059
SVMRBF+10-CV 7.08% 75.67% 78.89% 1.124 0.496 0.456 0.524 0.024
Hybrid SVMRBF 83.88% 82.01% 85.05% 1.123 0.414 0.396 0.437 0.016
NB 35.79% 32.41% 37.27% 1.861 1.191 1.045 1.411 0.127
NB+10-CV 68.03% 66.61% 69.82% 1.363 0.645 0.577 0.768 0.070
Hybrid NB 86.27% 83.40% 90.35% 2.695 0.521 0.456 0.608 0.060
C5.0 78.42% 75.41% 82.72% 2.429 0.487 0.449 0.543 0.038
C5.0+10-CV 95.24% 93.18% 96.28% 0.806 0.185 0.158 0.242 0.026
Hybrid C5.0 98.32% 97.18% 99.28% 0.564 0.067 0.043 0.145 0.027
RF 80.06% 77.25% 83.21% 1.860 0.431 0.371 0.495 0.037
RF+10-CV 96.48% 95.21% 97.52% 0.764 0.143 0.122 0.189 0.015
Hybrid RF 98.92% 97.06% 99.78% 0.817 0.056 0.031 0.126 0.026

TABLE. XII. PERFORMANCE OF BASELINE MODELS, BASELINE MODELS+10-CV, AND HYBRID MODELS TO GDS2

Average Lowest Highest Average Highest

Models Std. Lowest RMSE Std.
Accuracy Accuracy Accuracy RMSE RMSE
SVMRBF 75.52% 70.00% 77.80% 1.606 0.489 0.449 0.524 0.023
SVMRBF+10-CV 94.15% 92.89% 95.51% 0.909 0.274 0.234 0.347 0.050
Hybrid SVMRBF 96.32% 95.52% 96.89% 0.591 0.182 0.173 0.202 0.011
NB 67.68% 65.01% 69.82% 1.614 0.664 0.584 0.768 0.059
NB+10-CV 76.47% 73.52% 78.21% 1.624 0.498 0.466 0.550 0.031
Hybrid NB 91.42% 88.71% 95.05% 1.593 0.321 0.288 0.383 0.033
C5.0 86.18% 84.45% 88.41% 1.454 0.372 0.319 0.533 0.059
C5.0+10-CV 95.69% 93.28% 97.28% 1.026 0.174 0.141 0.197 0.017
Hybrid C5.0 98.62% 98.18% 99.54% 0.475 0.067 0.043 0.145 0.028
RF 90.37% 89.01% 91.80% 1.021 0.321 0.286 0.345 0.018
RF+10-CV 96.58% 95.21% 98.50% 0.928 0.139 0.114 0.189 0.016
Hybrid RF 99.08% 97.60% 99.80% 0.732 0.057 0.031 0.126 0.027

39 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 11, No. 1, 2020

TABLE. XIII. PERFORMANCE OF BASELINE MODELS, BASELINE MODELS+10-CV, AND HYBRID MODELS TO ADS3

Average Lowest Highest Average Highest

Models Std. Lowest RMSE Std.
Accuracy Accuracy Accuracy RMSE RMSE
SVMRBF 86.44% 81.06% 90.69% 2.56 0.823 0.691 1.016 0.089
SVMRBF+10-CV 90.66% 86.66% 95.00% 2.86 0.678 0.364 0.813 0.131
Hybrid SVMRBF 97.01% 95.34% 98.67% 1.112 0.178 0.114 0.230 0.042
NB 65.02% 60.24% 69.21% 2.961 1.016 0.834 1.331 0.164
NB+10-CV 94.82% 91.18% 96.27% 1.165 0.154 0.133 0.232 0.035
Hybrid NB 98.94% 98.01% 99.69% 0.607 0.145 0.042 0.230 0.049
C5.0 76.55% 70.43% 80.73% 2.851 0.845 0.703 0.991 0.097
C5.0+10-CV 97.54% 94.79% 99.50% 1.923 0.114 0.070 0.160 0.034
Hybrid C5.0 99.25% 98.21% 100% 0.606 0.073 0.000 0.145 0.045
RF 89.23% 86.71% 92.35% 1.566 0.561 0.411 0.667 0.066
RF+10-CV 98.22% 95.69% 99.52% 1.353 0.113 0.070 0.160 0.034
Hybrid RF 99.72% 99.01% 100% 0.357 0.041 0.000 0.077 0.029

Fig. 7, 8, and 9 demonstrated the performance based on the

accuracy of each model via each phase. We found the
improvement by using 10-CV combined with PCA gives the
best result in predicting student performance. The figures
show the performance of the RMSE of the models in each step.
The proposed hybrid models could generate a very small
RMSE. The hybrid RF algorithm produced the smallest value
of RMSE which shows itself as the best predictive model in
this prediction problem.

Fig. 4. Performance-based on the accuracy of GDS1

Fig. 7. Performance-based on the RMSE of GDS1.

Fig. 5. Performance-based on the accuracy of GDS2

Fig. 8. Performance-based on the RMSE of GDS2.

Fig. 6. Performance-based on the accuracy of ADS3

40 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 11, No. 1, 2020

Performance”, Springer Nature Switzerland AG 2020, AMLTA 2019,

AISC 921, 2019, pp. 229-239.
[8] Kotsiantis S., Piarrekeas C., and Pintelas P., "Predicting Students'
performance in Distance Learning using Machine Learning Techniques",
Applied Artificial Intelligence, Vol. 18, 2007, pp. 411-426.
[9] Hamoud A. K., Hashim A. S., and Awadh W. A., "Predicting Student
Performance in Higher Education Institutes Using Decision Tree
Analysis", International Journal of Interactive Multimedia and Artificial
Intelligent, Vol. 5, No. 2, February 2018, pp. 26-31.
[10] Daud A., Aljonhani N.R., Abbasi R.A., Lytras M.D., Abbas F.,
Alowibdi J.S., “Prediction student performance using advanced learning
analytics”, Proceedings of 26th International Conference on World Wide
Web, Companion, Perth, Australia, April 2016, pp. 416-421.
[11] M.S. Mythili,, A.R.M. Shanavas, “An Analysis of Students'
Performance using Classification Algorithms”, IOSR Journal of
computer Engineering (IOSR-JCE), Vol 16, No. 1, January 2014, pp.
Fig. 9. Performance-based on the RMSE of ADS3. 63-69.
[12] Aung Nway Oo, “Comparative Study of Principle Component Analysis
From the results, by using 10-CV, we can improve the based on Decision Tree Algorithm”, International Journal of Advances
performance of our baseline models. Additionally, we observed in Scientiofic Research and Engineering, Vol. 4, No. 6, June 2018, pp.
that the proposed novel hybrid models could boost up the 122-126.
classification performance to the superior results. This proposed [13] Karthikeyan T., Thangaraju P., “PCA-NB Algorithms to Enhance the
Predictive Accuracy”, International Journal of Engineering and
hybrid models can be regarded as an optimal prediction models Technology, Vol. 6, No. 1, 2014, pp. 381-387.
for solving prediction and classification problems.
[14] Farid J., Ahmad A.S., “Building student’s performance cesissiion tree
classifier using boosting algorithm”, Indonesian Journal of Electrical
VII. CONCLUSION Engineering and Computer Science, Vol. 14, No. 3,2019, pp. 1298-
This paper introduced the four popular classifiers of 1304.
machine learnings to predict student performance. The four [15] Babak M.A., Seyed K.S., Maryam M.M., “Support vector machine-
proposed algorithms are SVMRBF, NB, C5.0, RF. The based arrhythmia classification using reduced features of heart rate
variablitiy singanal”, Ariticial Intelligence in Mechine (Elsevier), vol.
procedure was made with three phases. Firstly, we observed 44, 2008, pp. 51-64.
the performance of those baseline methods. Secondly, we [16] Humera S., Raniah Z., Kavitha G., “Prediction of Student Performance
improved the performance with 10-CV. Lastly, we combined in Semester Exam Using a Naïve Bayes Classifier”, International
the PCA to baseline models, and 10-CV method to improve the Journal of Innovative Research in Science, Engineering and Technology,
classification performance. Based on classification accuracy Vol. 4, No. 10, October 201 5, pp. 9823-9829.
and RMSE as measurement parameters, it shows that the [17] Jollifie I.T, "Principle components analysis and factor analysis",
proposed hybrid models by conjunction of the proposed Springer 1986.
models with PCA and 10-CV produced very satisfying results. [18] Phauk Sokkhey and Takeo Okazaki., "Comparative Study of Prediction
Models on High School Student Performance in Mathematics", Journal
In conclusion, by combining the baseline models with principal of IEIE Transaction on Smart Processing and Computing, Vol. 8, No. 5,
component analysis, and evaluated by k-fold cross-validation, October 2019, pp. 394-404.
the proposed hybrid models produced a high performance [19] Mohamed Z.G. A., Mustafa B. M., Lazim A., and Hamdan A. M., "The
which shows itself as a potential algorithm for solving Factors Influence Students' Achievement in Mathematics: A Case for
prediction and classification problem. Libyan's Students ". Australian Journal of Basic and Applied Science,
Vol. 17, N0. 9, 2012, pp. 1224-1230.
REFERENCES
[20] Uysal S., "Factors affecting the Mathematics achievement of Turkish
[1] Herbert K., "The New Book of Popular Science", World Applied students in PISA 2012", Academic Journals, Vol. 10, June 2015, pp.
Sciences Journal, Daribury, Connecticut: Grolier Inc., 1978. 1670-1678.
[2] Ministry of Education, Youth and Sport, “Education in Cambodia:
Finding from Cambodia's Experience in PISA for Development”, AUTHORS’ PROFILE
Phnom Penh: Author, 2018. Phauk Sokkhey was born in Kompong Thom province,
[3] S. Slater, S. Joksimovic, V. Kovanovic, R.s Baker, and D. Gasevic, Cambodia. He received his bachelor degree in Mathematics
"Tools for Educational Data Mining: A Review", Journal of Educational from Royal University of Phnom Penh (RUPP), Cambodia, in
and Behavioral Statistics, Vol. 42, No. 1, 2016, pp. 88-106. 2010, and later received his Master degree in Applied
[4] Pooja Thakar, Anil Mehta, and Manisha, "Performance Analysis and Mathematics from Suranaree University of Technology (SUT),
Prediction in Educational Data Mining: A Research Travelogue", Thailand, in 2013. He was a lecturer of mathematics at Institute of
Technology of Cambodia (ITC). Sokkhey is currently a PhD candidate at the
International Journal of Computer Application, Vol. 100, No.12, January
University of the Ryukyus, Japan. His currently research are statistical causal
2015, pp. 60-68.
relationship analysis, machine learning, educational data mining, and data
[5] C. Romero and Ventura., “Educational Data Mining: A Review of the science.
State of Art”, IEEE Transactions on Systems, Man, and Cybernetics,
Part C: Applications and Reviews, vol. 40, No. 6, 2010, pp. 601-618. Takeo Okazaki took B.Sc., M.Sc. from Kyushu
University in 1987 and 1989, respectively. He had been a
[6] Akinrotimi A.O, and Aremu D.R, "Student Performance Prediction
research assistant at Kyushu University from 1989 to 1995. He
Using Randomtree and C4.5," Journal of Digital Innovation and
earned his Ph.D. from University of the Ryukyus in 2014. He is
Contemporary Research, Engineering and Technology, Vol. 6, No. 3, currently a professor at the university of the Ryukyus. His
2018, pp. 23-34. research interests are statistical analysis, data analysis, genome informatics,
[7] Amjad A. S., Mostafa Al-Emran, and Khaled S., “Mining Student tourism informatics, geographic information systems, and data science. He is
Information System Records to Predict Students’ Academic a member of JSCS, IEICE, JSS, GISA, and BSJ Japan.

41 | P a g e
www.ijacsa.thesai.org