See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/351946485

An Optimized Stacking Ensemble Model for Phishing Websites Detection

Article in Electronics · May 2021

DOI: 10.3390/electronics10111285

CITATIONS READS

66 532

8 authors, including:

Mohammed Hussein Al-Sarem Zeyad Ghaleb Al-Mekhlafi

Taibah University University of Ha'il
117 PUBLICATIONS 2,015 CITATIONS 68 PUBLICATIONS 1,131 CITATIONS

SEE PROFILE SEE PROFILE

Badiea Abdulkarem Mohammed Al-Shaibani Tawfik Al-Hadhrami

University of Ha'il Nottingham Trent University
70 PUBLICATIONS 1,114 CITATIONS 51 PUBLICATIONS 931 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Mohammed Hussein Al-Sarem on 02 June 2021.

The user has requested enhancement of the downloaded file.

 electronics
Article
An Optimized Stacking Ensemble Model for Phishing
Websites Detection
Mohammed Al-Sarem 1 , Faisal Saeed 1, * , Zeyad Ghaleb Al-Mekhlafi 2, * , Badiea Abdulkarem Mohammed 2 ,
Tawfik Al-Hadhrami 3, * , Mohammad T. Alshammari 2 , Abdulrahman Alreshidi 2
and Talal Sarheed Alshammari 2

1 College of Computer Science and Engineering, Taibah University, Medina 42353, Saudi Arabia;
[email protected]
2 College of Computer Science and Engineering, University of Ha’il, Ha’il 81481, Saudi Arabia;
[email protected] (B.A.M.); [email protected] (M.T.A.); [email protected] (A.A.);
[email protected] (T.S.A.)
3 School of Science and Technology, Nottingham Trent University, Nottingham NG11 8 NS, UK
* Correspondence: [email protected] (F.S.); [email protected] (Z.G.A.-M.);
[email protected] (T.A.-H.)

Abstract: Security attacks on legitimate websites to steal users’ information, known as phishing
attacks, have been increasing. This kind of attack does not just affect individuals’ or organisations’
websites. Although several detection methods for phishing websites have been proposed using
machine learning, deep learning, and other approaches, their detection accuracy still needs to be
enhanced. This paper proposes an optimized stacking ensemble method for phishing website



detection. The optimisation was carried out using a genetic algorithm (GA) to tune the parameters of


several ensemble machine learning methods, including random forests, AdaBoost, XGBoost, Bagging,
Citation: Al-Sarem, M.; Saeed, F.; GradientBoost, and LightGBM. The optimized classifiers were then ranked, and the best three models
Al-Mekhlafi, Z.G.; Mohammed, B.A.; were chosen as base classifiers of a stacking ensemble method. The experiments were conducted on
Al-Hadhrami, T.; Alshammari, M.T.;
three phishing website datasets that consisted of both phishing websites and legitimate websites—the
Alreshidi, A.; Alshammari, T.S. An
Phishing Websites Data Set from UCI (Dataset 1); Phishing Dataset for Machine Learning from
Optimized Stacking Ensemble Model
Mendeley (Dataset 2, and Datasets for Phishing Websites Detection from Mendeley (Dataset 3). The
for Phishing Websites Detection.
experimental results showed an improvement using the optimized stacking ensemble method, where
Electronics 2021, 10, 1285. https://
doi.org/10.3390/electronics10111285
the detection accuracy reached 97.16%, 98.58%, and 97.39% for Dataset 1, Dataset 2, and Dataset 3,
respectively.
Academic Editor: Khaled Elleithy
Keywords: ensemble classifiers; phishing websites; genetic algorithm; optimization methods
Received: 9 April 2021
Accepted: 26 May 2021
Published: 28 May 2021
1. Introduction
Publisher’s Note: MDPI stays neutral One of the most dangerous cybercrimes is phishing, where the user’s information and
with regard to jurisdictional claims in
credentials are stolen using fake emails or websites that are sent to the target and look like
published maps and institutional affil-
legitimate ones. Phishing attacks have been increasing over the years, and affect many
iations.
internet users. In this type of attack, the phisher selects any organisation as a target, and
then develops a phishing website that is similar to the organisation’s legitimate website.
The phisher then sends several spam emails or posts these links using social media or
any communication medium to many internet users, who may click on these links and be
Copyright: © 2021 by the authors. redirected to the phishing website [1].
Licensee MDPI, Basel, Switzerland. Phishing is one type of social engineering attack that targets many organisations’
This article is an open access article
websites on the internet. It can also attack internet of things (IoT) environments, in which
distributed under the terms and
the devices are highly interconnected, and these threats can affect organizations’ privacy
conditions of the Creative Commons
and data. IoT sensors are considered to be an easy medium for attackers. According to [2],
Attribution (CC BY) license (https://
attackers sent several spam emails, and it was found that refrigerators, televisions, and
creativecommons.org/licenses/by/
routers were among the 25% of devices that hosted them. In addition, hackers in the IoT
4.0/).

Electronics 2021, 10, 1285. https://doi.org/10.3390/electronics10111285 https://www.mdpi.com/journal/electronics

Electronics 2021, 10, 1285 2 of 18

environment may not need to send a virus or Trojan, as they can use the software in the
thingbots for spreading spam emails without the user knowing, as this may not affect
the functionality of IoT devices [3]. Many methods have been introduced to make the
IoT environment more secure, but there is currently no effective method for detecting
phishing emails [1,4]. Several studies have been conducted in order to propose approaches
and methods for detecting phishing websites for the IoT environment. For instance, Wei
et al. [5] introduced a lightweight deep learning method in order to provide a phishing
detection sensor that could work in real time with energy-saving features. If using this
proposed system, there is no need to install anti-phishing software on every IoT device.
However, the designed sensor is only needed for one location (such as an office) between
the devices and the router. In addition, this model can be directly installed on the router
because of its high efficiency.
Deep learning methods have been widely investigated for detecting phishing websites.
For instance, Somesha et al. [6] applied several models for phishing detection, which
included convolution neural network (CNN), deep neural network (DNN), and long short-
term memory (LSTM) models. The applied models obtained a good detection rate, with
an accuracy of 99.57% for LSTM. These models used only one third-party service feature,
in order to make the model robust and efficient. In another study, Ali and Ahmed [7]
introduced a hybrid intelligence method for predicting phishing websites, in which a
genetic algorithm (GA) was utilized to identify the optimal weights for website features
and select the most important ones. These features were used to train deep neural networks
to predict the phishing URLs. The results showed that the proposed approach obtained
significant improvements in terms of accuracy, specificity, sensitivity, and other metrics
compared to other state-of-the-art methods.
In a different approach, several machine learning methods were used to detect the
phishing websites. For instance, Chiew et al. [8] introduced a framework based on feature
selection and machine learning methods for detecting phishing, named hybrid ensemble
feature selection. In this method, the primary feature subsets were obtained using the
cumulative distribution function gradient, and these subsets were used to obtain the
secondary feature subsets using a data perturbation ensemble. The proposed model
used only 20.8% of the original features, and obtained an accuracy of 94.6% using the
Random Forests method. Similarly, Rao and Pais [9] introduced an efficient model based
on feature selection and machine learning; in order to improve the limitations of the
currently used phishing detection methods, they obtained the heuristic features from the
websites’ URLs, source codes, and third-party services. Eight machine learning methods
were used to evaluate the proposed model, and Random Forests obtained the best accuracy
(99.31%). In addition, Ali and Malebary [10] proposed a novel phishing detection model by
utilizing the particle swarm optimization method in order to weight the websites’ features,
which helped to identify the importance of their contributions towards differentiating
the phishing websites from legitimate ones. The results showed that this model led to
outstanding enhancements in terms of accuracy and other performance metrics for several
machine learning methods.
This paper proposes a model which is known as an optimized ensemble classification
model for detecting phishing websites. A genetic algorithm (GA) is used to optimize the
performance of several ensemble classifiers. Then, the best optimized classifiers are used as
base classifiers for the stacking ensemble method. The method includes three main phases:
training, ranking, and testing. In the training phase, random forests, AdaBoost, XGBoost,
Bagging, GradientBoost, and LightGBM are trained without applying an optimization
method. These classifiers are then optimized using the genetic algorithm, which selects the
optimal values of parameters for several ensemble models. The optimized classifiers are
then ranked and used as base classifiers for the stacking ensemble method. Finally, new
websites are collected and used as a testing dataset in order to predict the final class label
of these websites.
Electronics 2021, 10, 1285 3 of 18

The rest of this paper is organized as follows: Section 2 gives an overview of the related
work. Section 3 provides details about the materials and methods. Section 4 presents the
experimental results, which are analysed, discussed, and compared with related works. The
paper concludes with a summary of the outcomes of the proposed method and suggestions
for future work.

2. Related Works
2.1. Recognizing Phishing Attacks in the IoT
There are serious issues regarding the security of the IoT web, as there are billions of
devices (network objects and sensors) that are connected to the internet [11]. Thus, there is
a strong need to protect these IoT data from various types of attacks, including phishing.
Gupta et al. [1] illustrated how advanced infrastructures such as the internet of things
(IoT) are considered a target for phishing attacks. Tsiknas et al. [12] reviewed the main
cyber threats to the industrial internet of things (IIoT), and found that they originate from
five types of attacks: phishing, ransomware, system attacks, supply chain, and protocol.
According to Tsiknas et al. [12], for critical infrastructure such as the IoT, phishers apply
compromised attacks—an advanced method that combines social engineering and includes
zero-day malware and other features that are designed on remote websites and then attack
IIoT systems. The malicious attacker uses the front-end level for accessing the IIoT.
Several methods have been proposed to detect phishing websites in the IoT environ-
ment. Parra et al. [13] proposed a cloud- and deep-learning-based framework that includes
two mechanisms: a distributed convolutional neural network, and cloud-based temporal
long short-term memory. The first mechanism was used for detecting phishing as an IoT
microsecurity device, while the second mechanism was used on the back end to detect
notnet attacks and ingest CNN embeddings for detecting distributed phishing attacks on
several IoT devices. The experimental results showed that the first mechanism could obtain
a detection accuracy of 94.3% running the CNN model, and an F-1 score of 93.58% for
phishing attacks.
Mao et al. [14] discussed the main security issues in smart internet of things (IoT)
systems, and found that phishing is one of the most common types of attacks. In order to
detect these phishing websites, they developed an automated page-layout-based method
that includes machine learning methods. The method is based on aggregation analysis for
obtaining the page layout similarity, which helps in detecting phishing websites. Four ML
methods were applied in these experiments, and the results obtained showed enhanced
accuracy.
The security issues in the IoT were discussed in detail by Virat et al. [15], who argued
that the main challenge with IoT security is that its devices are not intelligent, which
makes the task of solving these issues difficult, requiring appropriate detection methods.
In addition, Deogirikar and Vidhate [16] surveyed various vulnerabilities that put the IoT
as a technology in danger. They reviewed various IoT attacks and discussed their efficiency
and damage level in the IoT, and concluded that extensive research is required in order to
come up with effective solutions.
In addition, deep learning methods were also investigated for protecting internet of
things (IoT) devices against several attacks, such as distributed denial-of-service (DDoS),
phishing, and spamming campaigns. In [17], a stacked deep learning method was in-
troduced to detect malicious traffic attacks affecting IoT devices. This proposed method
showed a good ability to detect benign and malicious traffic data, and obtained a higher
detection rate in real time compared with other methods.

2.2. Machine-Learning-Based Detection Methods

Artificial intelligence (AI) and machine learning (ML) have been widely used as
detection methods for several cyber security issues. For phishing website detection, several
AI- and ML-based methods with good detection performance have been proposed. For
instance, Alsariera et al. [18] proposed new schemes based on AI that considered new
Electronics 2021, 10, 1285 4 of 18

methods for the mitigation of phishing. They introduced four meta-learning techniques
based on the extra-tree-based classifier and applied them to phishing website datasets. The
experimental results showed that the proposed models obtained an accuracy of 97%, and
reduced the false positive rate to 0.028.
Jain and Gupta [19] proposed a new method for detecting phishing websites based on
the hyperlinks located in the websites’ HTML code. This method combines several novel
features of hyperlinks, and divides them into 12 types for training ML models. This method
was applied to a phishing website dataset using several ML classifiers. The experimental
results showed that the proposed model obtained 98.4% accuracy using a logistic regression
classifier. This method is a client-side solution, which does not require any third-party
support. Feng [20] introduced a new a model for phishing website detection using a neural
network. The Monte Carlo technique was used in the training phase, and in the testing
phase the accuracy reached 97.71% while the false positive rate reached 1.7%, indicating
that the proposed model is capable of detecting phishing websites effectively compared to
other machine learning methods.
Aburub and Hadi [21] used association rules to detect phishing websites. They used a
dataset containing 10,068 instances of legitimate and phishing websites, and applied the
phishing multi-class association rule method, which was compared to other associative
classification methods. The experimental results showed that the proposed methods ob-
tained an acceptable detection rate. Similarly, other ML-based methods have been applied
utilizing feature selection methods [22,23], ensemble classifiers [24], hybrid methods of
deep learning and machine learning [25], and other methods.
As can be shown from the previous studies on detecting phishing websites, the
effectiveness of the detection still needs to be enhanced. For instance, Azeez et al. [26]
mentioned that the current applied methods to handle phishing websites are not sufficient.
Thus, they introduced the PhishDetect method, which identifies phishing attacks by using
URL consistency features. This proposed method checks the PhishTank database in order
to verify whether the URL exists, then considers it to be a phishing website if not. This
method requires updating the PhishTank database frequently. In addition, Azeez et al. [27]
proposed a system for detecting malicious URLs on Twitter. This study examined the
correlation of URL redirect chains obtained from Twitter, and then a naive Bayes classifier
was used on these data, with an accuracy of 90%. An interesting comparative study was
conducted by Osho et al. [28] to investigate the performance of several machine learning
methods for the detection of phishing websites. They found that the random forests method
outperforms the existing methods, and achieves an accuracy of 97.3%.
However, some proposed methods were applied to small- or medium-sized datasets,
while other proposed methods were applied to only one dataset (websites or emails).
Therefore, there is a need to conduct further analysis on detecting phishing websites using
more datasets with many benign and malicious websites.

3. Materials and Methods

In this section, the proposed genetic-algorithm-based ensemble classifier approach
for improving phishing website detection is presented and explained. Figure 1 presents
the methodology that we followed in this study. The methodology consists of three
main phases: the training, ranking, and testing phases. In the training phase, random
forests, AdaBoost, XGBoost, Bagging, GradientBoost, and LightGBM were trained without
optimization. The reason behind this is twofold: on the one hand, to obtain a general
insight into the performance of ensemble classifiers before optimizing them, and on the
other hand, to explore which of the phishing websites’ characteristics is most useful. The
aforementioned classifiers were then optimized using the genetic algorithm. Here, the
genetic algorithm was used for selecting the optimal values of model parameters in order
to improve the overall accuracy of the proposed model. Later, in the ranking phase, the
optimized classifiers were ranked and used as a base classifier for the ensemble classifier—
the stacking method. In the testing phase, new websites were collected and used as testing
OR PEER REVIEW 5 of 19

Electronics 2021, 10, 1285 5 of 18

data. Figure 1 refers to this phase as the detection phase, as these steps will be applied to
any website in future in order to detect whether it is a benign or malicious website. In
order to extract the features of the websites, we followed the methodology presented in
data. Figure 1 refers to this phase as the detection phase, as these steps will be applied to
[29]. A set of benign and malicious
any website websites
in future was
in order collected
to detect from
whether it is the malware
a benign and phishing
or malicious website. In order
blacklist of the PhishTank database
to extract of verified
the features phishing
of the websites, pagesthe
we followed [30]. In order presented
methodology to extractinthe
[29]. A set
of benign and malicious websites was collected from the malware
same features as those used in the training dataset (HTML- and JavaScript-based features, and phishing blacklist of
the PhishTank database of verified phishing pages [30]. In order to extract the same features
address-bar-based features, domain-based features, and abnormality-based features), a
as those used in the training dataset (HTML- and JavaScript-based features, address-bar-
Python script was written using the
based features, Beautiful Soup,
domain-based ipaddress,
features, urllib, request,
and abnormality-based andaWhois
features), Python script
libraries. Later, all of was
these features
written usingwere fed into
the Beautiful the ipaddress,
Soup, classifiersurllib,
in order to and
request, predict
Whoisthe final Later,
libraries.
class label of the website.
all of these features were fed into the classifiers in order to predict the final class label of
the website.

Figure 1. The proposed optimized stacking ensemble model for phishing website detection.
Figure 1. The proposed optimized stacking ensemble model for phishing website detection.
The Dataset and Experimental Design
The Dataset and Experimental Design
The experimental part of this work was conducted on three publicly available datasets—
The experimental thepart
Phishing Websites
of this work Data Set conducted
was from UCI (Dataset 1) [31],publicly
on three the Phishing Dataset for
available da-Machine
Learning from Mendeley (Dataset 2) [32], and Datasets for Phishing Websites Detection
tasets—the Phishing Websites Data Set from UCI (Dataset 1) [31], the Phishing Dataset for
from Mendeley (Dataset 3) [33]. To conduct the experiment, the script was written in
Machine Learning from Python Mendeley
3.6 using an(Dataset
Anaconda2)environment
[32], and Datasets forWindows
on the 64-bit Phishing Websitessystem.
10 operating
Detection from Mendeley
Dataset (Dataset 3)of[33].
1 consists 44%To conduct
phishing the experiment,
websites (4898) and 56% thelegitimate
script was writ- (6157).
websites
Since the dataset is quite imbalanced, the oversampling
ten in Python 3.6 using an Anaconda environment on the 64-bit Windows 10 operating technique was used to increase
the size of the minority class. The dataset contains 30 features, which can be categorized
system. Dataset 1 consists
into fourofgroups:
44% phishing websites (4898)
(1) 12 address-bar-based and 56%
features, (2) 5 legitimate
HTML- and websites
JavaScript-based
(6157). Since the dataset is quite
features, imbalanced, thefeatures,
(3) 6 abnormality-based oversampling technique was
and (4) 7 domain-based used
features. to in-
Table 1 presents
crease the size of the minority class. The dataset contains 30 features, which can be cate- in the
the names of these features and the Python library used for extracting each one
testing phase.
gorized into four groups: (1) 12 address-bar-based features, (2) 5 HTML- and JavaScript-
based features, (3) 6 abnormality-based features, and (4) 7 domain-based features. Table 1
presents the names of these features and the Python library used for extracting each one
in the testing phase.
Electronics 2021, 10, 1285 6 of 18

Table 1. Feature description for phishing websites.

Feature Category Feature Name Description Python Library Used

having_IP_Address Using the IP Address
Long URL to hide the suspicious
URL_Length
part
Shortening_Service Using shortening service IPaddress
having_At_Symbol URL having @ symbol Urllib
double_slash_redirecting URL uses “//” symbol Re
Address-bar-based
Prefix_Suffix Add prefix or suffix separated by (-) Datetime
Website has subdomain or BeautifulSoup
having_Sub_Domain
multi-subdomain Socket
SSLfinal_State Age of SSL certificate
Domain_registeration_length Domain registration length
Associated graphic image (icon)
Favicon
with webpage
Port Open port
Presence of HTTP/HTTPS in
HTTPS_token
domain name
How many times a website has
Redirect
been redirected
on_mouseover Effect of mouse over on status bar Request
HTML- and JavaScript-based
RightClick Disabling right click BeautifulSoup
Using pop-up window to submit
popUpWindow
personal information
Iframe Using Iframe
% of external objects contained
Request_URL
within a webpage
URL_of_Anchor % of URL Anchor (<a> tag) BeautifulSoup
Abnormality based % of links in <meta>, <script> and Re
Links_in_tags
<link> WHOIS
SFH Server from Handler
Submit user information using mail
Submitting_to_email
or mailto
Abnormal_URL Host name in URL
age_of_domain Age of the website
DNSRecord Website in WHOIS dataset
web_traffic Popularity of the website
WHOIS
Page_Rank Page Rank
Domain-based features Urllib
Google_Index Google Index
BeautifulSoup
Links_pointing_to_page # of links pointing to page
Statistical_report’ found in statistical reports
Website is classified as phishing or
Result
legitimate

In addition, Dataset 2 includes 48 features extracted from 5000 phishing websites

and 5000 legitimate websites, while Dataset 3 includes 111 features extracted from 30,647
phishing websites and 58,000 legitimate websites. More descriptions about these datasets
can be obtained from [32,33].
In order to evaluate the performance of the proposed ensemble model, the following
performance measures were used: classification accuracy, precision, recall (the detection
rate), F1 score, false positive rate (FPR), and false negative rate (FNR). These measures are
commonly used by researchers to evaluate the performance of phishing website detection
systems [10]. In order to precisely assess the proposed method, all of the conducted exper-
iments including optimized and non-optimized classifiers were validated using 10-fold
cross-validation. The results of each fold were also normalized. P = (95.37/(95.37 + 1.2),
R = 95.37/(95.37 + 4.63).
Electronics 2021, 10, 1285 7 of 18

4. Results and Discussion

This section describes the experimental results for each technique, before presenting
and discussing comparisons with the related works.

4.1. Experimental Results of the Ensemble Classifiers without Optimization

As mentioned earlier, a set of ensemble classifiers was trained using 10-fold cross-
validation. We first conducted the experiment without involving the optimization using
the GA. The performance of the classifiers with default configurations is presented in
Tables 2–4 for Dataset 1, Dataset 2, and Dataset 3, respectively. For Dataset 1, the random
forests classifier yielded the best performance compared with the other classifiers in terms
of accuracy, precision, recall, and F-score; it achieved 97.02% accuracy. The Bagging
classifier also achieved good accuracy, with 96.73%, followed by the LightGBM classifier,
with accuracy of 96.53%. The remaining classifiers obtained accuracy between 93% and
94.61%. Meanwhile, in Dataset 2, the LightGBM classifier obtained the best performance
compared to other classifiers in terms of accuracy, precision, recall and F-score. The random
forests classifier obtained the second best performance using all evaluation measures for
this dataset. Similarly to Dataset 1, the performance of Random Forests obtained the best
results for Dataset 3 in terms of accuracy, recall, and F-score, as shown in Table 4.

Table 2. Performance of ensemble classifiers for Dataset 1.

Random GradientBoost
Measure AdaBoost (%) XGBoost (%) Bagging (%) LightGBM (%)
Forests (%) (%)
Accuracy 97.02 93.17 94.45 96.73 94.61 96.53
Precision 96.58 94.70 94.52 94.99 94.87 95.15
Recall 98.08 96.60 96.39 96.73 96.59 96.70
F-Score 97.49 95.71 95.50 95.90 95.76 95.95

Table 3. Performance of ensemble classifiers for Dataset 2.

Random GradientBoost LightGBM

Measure AdaBoost (%) XGBoost (%) Bagging (%)
Forests (%) (%) (%)
Accuracy 98.37 96.88 97.70 97.51 97.67 98.65
Precision 98.54 96.74 97.57 97.55 97.58 98.56
Recall 98.26 97.04 97.85 97.44 97.76 98.74
F-Score 98.39 96.88 97.71 97.46 97.67 98.65

Table 4. Performance of ensemble classifiers for Dataset 3.

Random GradientBoost
Measure AdaBoost (%) XGBoost (%) Bagging (%) LightGBM (%)
Forests (%) (%)
Accuracy 97.15 93.58 95.33 96.78 95.37 96.67
Precision 95.78 90.89 92.75 95.80 93.06 95.08
Recall 96.13 90.52 93.82 94.93 93.58 95.28
F-Score 95.90 90.70 93.28 95.33 93.32 95.18

Figure 2 shows the false positive rate (FPR) and false negative rate (FNP) for Dataset
1. It was notable that RF had the best FPR and FNP, with 0.05 and 0.02, respectively. The
LightGBM classifier was the second best classifier in terms of FPR (0.068), followed by the
GradientBoost classifier (0.07). In terms of FNR, the Random Forests classifier also yielded
the lowest value (0.02), followed by AdaBoost and Bagging. Although the AdaBoost
classifier had a lower FNR (0.032), its FPR values were higher than those of the LightGBM
classifier, which means that there is a probability of raising a false alarm, in which a positive
Electronics 2021, 10, x FOR PEER REVIEW 8 of 19

Random Forests AdaBoost XGBoost Bagging GradienBoost LightGBM

0.08
Electronics 2021, 10, 1285
GradientBoost classifier (0.07). In terms of FNR, the Random Forests classifier also yielded
8 of 18
0.07 the lowest value (0.02), followed by AdaBoost and Bagging. Although the AdaBoost clas-
0.06 sifier had a lower FNR (0.032), its FPR values were higher than those of the LightGBM
0.05 classifier, which means that there is a probability of raising a false alarm, in which a pos-
0.04 resultresult
itive is given whenwhen
is given the true
thevalue is negative.
true value Similarly,
is negative. as shown
Similarly, in Figures
as shown 3 and 34,and
in Figures the
0.03 RFthe
4, model obtained
RF model the best
obtained theFPR
bestand
FPRFNP
andfor Dataset
FNP 2 and 2Dataset
for Dataset 3.
and Dataset 3.
0.02
0.01 Random Forests AdaBoost XGBoost Bagging GradienBoost LightGBM
0
0.08
FPR FNR
0.07
0.06 Figure 2. Comparisons of the FPR and FNR of ensemble methods for Dataset 1.
0.05
0.04 Table 3. Performance of ensemble classifiers for Dataset 2.
0.03 Gradi-
0.02 Random AdaBoost XGBoost Bagging LightGBM
Measure entBoost
0.01
Forests (%) (%) (%) (%) (%)
(%)
0 Accuracy 98.37 96.88 97.70 97.51 97.67 98.65
FPR
Precision 98.54 96.74 97.57 FNR
97.55 97.58 98.56
Recall 98.26 97.04 97.85 97.44 97.76 98.74
Comparisons of the FPR and FNR of ensemble methods for Dataset 1.
Figure 2. F-Score 98.39 96.88 97.71 97.46 97.67 98.65
Table 3. Performance of ensemble classifiers for Dataset 2.
Random Forests AdaBoost XGBoost Bagging GradienBoost LightGBM
Gradi-
Random AdaBoost XGBoost Bagging LightGBM
0.08 Measure entBoost
Forests (%) (%) (%) (%) (%)
0.07 (%)
Accuracy 98.37 96.88 97.70 97.51 97.67 98.65
Electronics 2021, 10,0.06
x FOR PEER REVIEW 9 of 19
Precision 98.54 96.74 97.57 97.55 97.58 98.56
0.05 Recall 98.26 97.04 97.85 97.44 97.76 98.74
0.04 F-Score 98.39 96.88 97.71 97.46 97.67 98.65
Table 4. Performance of ensemble classifiers for Dataset 3.
0.03
Gradi-
0.02 Random Forests AdaBoost
RandomXGBoost BaggingXGBoost
AdaBoost GradienBoost LightGBM
Bagging LightGBM
Measure entBoost
0.01
0.08
Forests (%) (%) (%) (%) (%)
(%)
0
0.07 Accuracy 97.15 93.58 95.33 96.78 95.37 96.67
Precision FPR 95.78 90.89 92.75 FNR 95.80 93.06 95.08
0.06
Recall 96.13 90.52 93.82 94.93 93.58 95.28
0.05 Figure 3. F-Score
Comparisons
Comparisons of95.90
the FPR and FNR
90.70of ensemble methods for
93.28 Dataset 2. 93.32
95.33 95.18
0.04
0.03 Random Forests AdaBoost XGBoost Bagging GradienBoost LightGBM
0.02
0.08
0.01
0.06
0
0.04 FPR FNR

0.02 Figure 3. Comparisons of the FPR and FNR of ensemble methods for Dataset 2.

0
FPR FNR

Figure
Figure 4.
4. Comparisons
Comparisons of
of the
the FPR
FPR and
and FNR
FNR of
of ensemble
ensemble methods for Dataset
methods for Dataset 3.
3.

4.2. Experimental Results of the GA-Based Ensemble Classifiers

Although all of the classifiers showed good performance, there is still a need to adjust
many of their parameters in order to achieve better evaluation scores. Adjusting such pa-
rameters for each classifier is relatively cumbersome. In this study, a genetic algorithm
was used for tuning the classifiers’ parameters. GAs have shown good results in the field
of algorithm parameter searching [34]. We conducted the experiments using different pa-
rameters to configure the GA (which were used in our previous works and other studies),
Electronics 2021, 10, 1285 9 of 18

4.2. Experimental Results of the GA-Based Ensemble Classifiers

Although all of the classifiers showed good performance, there is still a need to adjust
many of their parameters in order to achieve better evaluation scores. Adjusting such
parameters for each classifier is relatively cumbersome. In this study, a genetic algorithm
was used for tuning the classifiers’ parameters. Gas have shown good results in the field
of algorithm parameter searching [34]. We conducted the experiments using different
parameters to configure the GA (which were used in our previous works and other studies),
and the best ones were used in this study, as shown in Table 5.

Table 5. Parameter settings of the GA used in this paper.

Parameter Value
Generations 10
Population size 24
Mutation rate 0.02
Crossover rate 0.5
Early stop 12

Since there are many parameters to adjust, Table 6 shows the list of adjusted parame-
ters of each classifier and the optimized parameters found by the GA. Among all of the
parameters, finding the optimal number of estimators and learning rate are the most critical
parameters, which impact most highly on the performance of the classifier. XGBoost and
GradientBoost gained a considerable improvement compared to the default parameters,
as shown in Table 7. Meanwhile, the performance of both the LightGBM classifier and
Random Forests was decreased.

Table 6. List of optimized parameters of the classifiers.

Classifier Name Adjusted Parameters Best GA-Based Configuration

Criterion: [‘entropy’, ‘gini’]
max_depth: [10–1200] + [None] Criterion: entropy
max_features: [‘auto’, ‘sqrt’,’log2’, None] max_depth: 142
Random Forests
min_samples_leaf: [4–12] min_samples_leaf: 4 min_samples_split: 5
min_samples_split: [5–10] n_estimators: 1200
n_estimators’: [150–1200]
n_estimators: [100–1200] learning_rate: 0.1
AdaBoost
learning_rate: [1 × 10−3 , 1 × 10−2 , 1 × 10−1 , 0.5, 1.0] n_estimators: 711
n_estimators: [100–1200] learning_rate: 0.1
max_depth: [1–11], max_depth: 5
XGBoost learning_rate: [1 × 10−3 , 1 × 10−2 , 1 × 10−1 , 0.5, 1.] min_child_weight: 3.0
subsample: [0.05–1.01] n_estimators: 588
min_child_weight: [1–21] subsample: 0.7
n_estimators: [100–1200] n_estimators: 1077
Bagging max_samples: [0.1, 0.2, 0.3, 0.4, 0.5, 1.0, 1.1] max_samples: 0.5
bootstrap: [True, False] bootstrap: True
n_estimators: [100–1200] n_estimators: 344
learning_rate: [1 × 10−3 , 1 × 10−2 , 1 × 10−1 , 0.5, 1.0] learning_rate: 1.0
subsample: [0.05–1.01] subsample: 1.0
GradientBoost max_depth: [10–1200] + None max_depth: 1067
min_samples_split: [5–10] min_samples_split: 5
min_samples_leaf: [4–12] min_samples_leaf: 12
max_features: [‘auto’, ‘sqrt’,’log2’, None] max_features: ‘auto’
 Electronics 2021, 10, 1285 10 of 18

Table 6. Cont.

Classifier Name Adjusted Parameters Best GA-Based Configuration

boosting_type: ‘gbdt’
boosting_type: [‘gbdt’, ‘dart’, ‘goss’, ‘rf’]num_leaves:
num_leaves: 13
[5–42]
max_depth: 15
max_depth: [10–1200] + None
learning_rate: 0.5
learning_rate: [1 × 10−3 , 1 × 10−2 , 1 × 10−1 , 0.5, 1.]
n_estimators: 500
n_estimators: [100–1200]
min_child_samples: 399
min_child_samples: [100,500]
min_child_weight: 0.1
min_child_weight: [1 × 10−5 , 1 × 10−3 , 1 × 10−2 , 1
LightGBM
× 10−1 , 1, 10, 100, 1000, 10000]
subsample: 0.855
subsample: sp_uniform(loc = 0.2, scale = 0.8)
colsample_bytree: 0.9234
colsample_bytree’: sp_uniform(loc = 0.4, scale = 0.6)
reg_alpha: [0, 10−1 , 1, 2, 5, 7, 10, 50, 100],
reg_alpha: 2
reg_lambda: [0, 10−1 , 1, 5, 10, 20, 50, 100],
reg_lambda: 5
min_split_gain: 0.0,
min_split_gain: 0.0,
subsample_for_bin: 200,000
subsample_for_bin: 200,000

Table 7. The accuracy of the optimized ensemble models for Dataset 1.

GA– GA–
GA–AdaBoost GA–XGBoost GA–Bagging
Fold GA–RF (%) GradientBoost LightGBM
(%) (%) (%)
(%) (%)
cs 2021, 10, x FOR PEER REVIEW 11 of 19
1 97.11 94.85 96.75 96.56 97.11 96.84
2 96.84 93.13 97.02 96.75 96.93 96.47
3 97.20 93.04 96.93 96.56 96.93 95.66
4 5 96.02 96.29 93.76
92.95 97.47
97.02 97.02 97.65 97.02 97.8396.20 96.20
5 6 96.29 96.47 92.95
93.57 97.02
96.92 96.74 97.02 97.01 97.0296.20 96.20
6 96.47 93.57 96.92 96.74 97.01 96.20
7 7 96.74 96.74 92.85
92.85 97.29
97.29 97.01 97.01 97.29 97.2996.83 96.83
8 8 97.83 97.83 95.66
95.66 97.56
97.56 97.47 97.47 97.83 97.8397.47 97.47
9 9 97.01 97.01 92.85
92.85 97.29
97.29 97.01 97.01 97.19 97.1996.56 96.56
10 10 95.93 95.93 93.67
93.67 95.93
95.93 96.20 96.20 96.11 96.1195.75 95.75
Average 96.74
Average 96.74 93.63
93.63 97.01
97.01 96.90 96.90 97.13 97.1396.42 96.42

To explore thisTo further

explore inthis
Dataset 1, in
further theDataset
confusion
1, thematrices
confusionof matrices
RandomofForests,
Random Forests, XG-
XGBoost, Gradient Boost,
Boost, and LightGBM
Gradient Boost, and are shown in
LightGBM areFigures
shown5–8, respectively.
in Figures Table 8
5–8, respectively. Table 8 lists
lists the results the
of the other performance measures.
results of the other performance measures.

(a) (b)

Figure 5. AFigure 5. A normalized

normalized confusion
confusion matrix matrix ofForests
of Random Random for Forests
phishingforwebsite
phishing website
and and website
legitimate legitimate
classification:
website classification: (a) with default parameters; (b) with optimized parameters for Dataset 1.
(a) with default parameters; (b) with optimized parameters for Dataset 1.
 (a) (b)
Electronics 2021, 10, 1285 11 of 18
Figure 5. A normalized confusion matrix of Random Forests for phishing website and legitimate
website classification: (a) with default parameters; (b) with optimized parameters for Dataset 1.

(a) (b)

AFigure 6. A normalized confusion matrix offor

XGBoost forwebsite
phishing website and website
legitimate website
ics 2021, 10, x FOR PEER REVIEW 12 of 19
Figure
cs 2021, 10, x FOR PEER 6. normalized
REVIEW confusion matrix of XGBoost phishing and legitimate classification:
12 of 19 (a) with
classification: (a) with default parameters; (b) with optimized parameters for Dataset 1.
default parameters; (b) with optimized parameters for Dataset 1.

(a) (b)
(a) (b)
Figure
Figure 7. AFigure 7. A
normalized normalized confusion
confusionconfusion matrix
matrix ofmatrix of GradientBoost
GradientBoost for
for phishingphishing website
websitewebsite and legitimate
and legitimate web-
website classification:
7. A normalized of GradientBoost for phishing and legitimate web-
site classification: (a) with default parameters; (b) with optimized parameters for Dataset 1.
(a) with default parameters; (b)
site classification: (a) with
with optimized parameters
default parameters; (b)for Dataset
with 1.
optimized parameters for Dataset 1.

(a) (b)
(a) (b)
Figure 8. A normalized confusion matrix of LightGBM for phishing website and legitimate website
Figure 8. AFigure 8. A normalized
classification:
normalized confusion confusion
(a) with default
matrix matrix of LightGBM
parameters;
of LightGBM (b)
forwith for phishing
optimized
phishing andwebsite
parameters
website andwebsite
legitimate
for Dataset
legitimate website
1. classification: (a) with
classification: (a) with default parameters; (b) with optimized parameters for Dataset 1.
default parameters; (b) with optimized parameters for Dataset 1.
In Figure 6b and Figure 7b, we can note that the GA–XGBoost and GA–GradientBoost
In Figure 6b and Figure 7b, we can note that the GA–XGBoost and GA–GradientBoost
classifiers gained the most benefit from the optimization for Dataset 1. They correctly de-
classifiers gained the most benefit from the optimization for Dataset 1. They correctly de-
tected 95.94% of phishing website instances as “phishing website” class, which represents
tected 95.94% of phishing website instances as “phishing website” class, which represents
the TP measure, and incorrectly detected 4.06% of these instances as “legitimate” class,
the TP measure, and incorrectly detected 4.06% of these instances as “legitimate” class,
which represents the FP measure. In addition, they detected 98.4% of legitimate website
which represents the FP measure. In addition, they detected 98.4% of legitimate website
instances as “legitimate” class, which represents the TN measure, and incorrectly detected
instances as “legitimate” class, which represents the TN measure, and incorrectly detected
Electronics 2021, 10, 1285 12 of 18

In Figures 6b and 7b, we can note that the GA–XGBoost and GA–GradientBoost
classifiers gained the most benefit from the optimization for Dataset 1. They correctly
detected 95.94% of phishing website instances as “phishing website” class, which represents
the TP measure, and incorrectly detected 4.06% of these instances as “legitimate” class,
which represents the FP measure. In addition, they detected 98.4% of legitimate website
instances as “legitimate” class, which represents the TN measure, and incorrectly detected
1.96% of these instances as “phishing website” class, which represents the FN measure. We
can conclude that both classifiers (GA–XGBoost and GA–GradientBoost) achieved a high
TP rate and a low FP rate.

Table 8. Results of performance evaluation measures when detecting phishing and legitimate classes
for Dataset 1.

Classifier Class Name Precision (%) Recall (%) F-Score (%)

Phishing website 96.40 94.10 95.10
GA–Random
Legitimate 95.20 97.30 96.40
Forests
Weighted Average 95.90 95.70 95.90
Phishing website 97.50 95.80 96.50
GA–XGBoost Legitimate 96.70 98.00 97.20
Weighted Average 97.00 97.00 97.00
Phishing website 97.00 95.70 96.40
GA–
Legitimate 96.80 97.50 97.10
GradientBoost
Weighted Average 96.90 96.80 96.80
Phishing website 95.10 94.20 94.70
GA–LightGBM Legitimate 95.50 96.30 95.80
Weighted Average 95.30 95.30 95.30

After conducting the training phase for the ensemble classifiers on Dataset 1, the perfor-
mances of these classifiers were ranked, and the three best models were: GA–GradientBoost,
GA–XGBoost, and GA–Bagging. These models were used in the next step as base classifiers
(base learner) of a stacking ensemble method. For Dataset 1, the classifiers that were used as
meta-learners were Random Forests, GradientBoost, and Support Vector Machine (SVM).
The same experiments were conducted on Dataset 2 and Dataset 3. The performance
of GA-based ensemble classifiers after optimization is shown in Tables 9 and 10. The
results indicate that some classifiers (such as GA–Random Forests, GA–AdaBoost, and
GA–XGBoost) show improvements in terms of accuracy, precision, recall, and F-score for
Dataset 2, while all of the classifiers show improvements for Dataset 3 using all measures.

Table 9. Performance of GA-based ensemble classifiers for Dataset 2.

Measure GA–Random GA–AdaBoost GA–XGBoost GA–Bagging GA– GA–

Forests (%) (%) (%) (%) GradientBoost LightGBM
(%) (%)
Accuracy 98.39 97.21 98.57 97.51 98.50 98.32
Precision 98.46 97.15 98.50 97.24 98.31 98.10
Recall 98.13 97.28 98.64 97.89 98.54 98.56
F-Score 98.43 97.21 98.57 97.52 98.37 98.33

Table 10. Performance of GA-based ensemble classifiers for Dataset 3.

Measure GA–Random GA–AdaBoost GA–XGBoost GA–Bagging GA– GA–

Forests (%) (%) (%) (%) GradientBoost LightGBM
(%) (%)
Accuracy 96.44 94.06 97.35 96.96 97.27 97.21
Precision 94.63 90.91 96.20 95.3 95.68 96.13
Recall 95.08 92.02 96.14 95.96 96.30 95.81
F-Score 94.86 91.46 96.17 95.64 95.78 95.96
Electronics 2021, 10, 1285 13 of 18

Table 11 shows the mean rank calculated for all classifiers for all three datasets. The re-
sults were obtained by 10-fold cross-validation before and after applying GA optimization.

Table 11. Models ranked by accuracy of classifier obtained by 10-fold cross-validation.

Dataset 1 Dataset 2 Dataset 3

ML Classifier Mean Mean Mean
Mean SD Mean SD Mean SD
Rank Rank Rank
RF 3.2 0.970 0.00427 4.2 0.9837 0.00332 3.6 0.971 0.00229
GA–RF 4.4 0.967 0.00554 4.1 0.9839 0.00327 8 0.964 0.00269
AdaB 11.7 0.932 0.00549 11.5 0.9688 0.00477 12 0.936 0.00335
GA–AdaB 11.2 0.936 0.00889 10.2 0.9721 0.00448 11 0.941 0.00299
XGB 9.6 0.945 0.00491 7.4 0.9770 0.00508 9.5 0.9532 0.00339
GA–XGB 3.1 0.970 0.00447 2.5 0.9857 0.00310 1.5 0.974 0.00201
Bagging 5.3 0.967 0.00492 8.9 0.9751 0.00567 5.8 0.968 0.00214
GA–Bagging 4.3 0.969 0.00412 8.1 0.9751 0.00579 4.9 0.969 0.00243
GB 9.2 0.946 0.00578 7.8 0.9767 0.00492 9.5 0.954 0.00330
GA–GB 1.8 0.971 0.00464 2.9 0.9850 0.00293 2.2 0.973 0.00222
LGB 6.1 0.965 0.00561 1.9 0.9865 0.00307 6.7 0.967 0.00227
GA–LGB 6.7 0.964 0.00514 4.5 0.9832 0.00421 2.9 0.972 0.00235

The results show that most of the models with the highest mean accuracy values were
produced when the GA was used. Among all of the selected classifiers, GA–XGB is a good
choice for use as a base classifier for the stacking ensemble method.
Table 12 shows the testing results for the detection accuracy of the proposed model
using 10-fold cross-validation for Dataset 1, Dataset 2, and Dataset 3.

Table 12. The accuracy of the optimized stacking ensemble method.

Dataset RF Level (%) GB (%) SVM (%)

Dataset 1 97.00 96.82 97.16
Dataset 2 98.57 98.47 98.58
Dataset 3 97.22 97.32 97.39

As shown in Table 12 above, the proposed optimized stacking ensemble model ob-
tained good improvements in terms of phishing website detection accuracy for all datasets.
The proposed optimized stacking ensemble obtained the best performance when the op-
timized ensemble classifiers (GA–GradientBoost, GA–XGBoost, and GA–Bagging) were
used as base learners, and SVM was used as meta-learner. The achieved accuracy reached
97.16%, 98.58%, and 97.39% for Dataset 1, Dataset 2, and Dataset 3, respectively, which
surpasses the other ensemble methods in the previous phase.

4.3. Statistical Analysis and Comparison with Previous Studies

Table 13 presents a comparison of the results obtained (using Dataset 1) with the
preliminary settings, where the base classifiers were trained using the default settings of
hyperparameters, and the improvements obtained after applying the GA and adjusting
the hyperparameters of the classifiers. It also summarizes the mean accuracy and variance
values of each classifier. The results also show that the mean of the GradientBoost classifier
using GA optimization exceeded the means of all of the other classifiers, before and after
applying the optimization.
Electronics 2021, 10, 1285 14 of 18

Table 13. The average accuracy and variance values of all of the classifiers, before and after conducting
GA optimization.

Classifier Name Without Optimization With GA Optimization

Random Forests Avg. 97.02% 96.74%
Variance 0.000 0.000
AdaBoost Avg. 93.17% 93.63%
Variance 0.000 0.000
XGBoost Avg. 94.45% 97.01%
Variance 0.000 0.000
Bagging Avg. 96.73% 96.90%
Variance 0.000 0.000
GradientBoost Avg. 94.61 97.13%
Variance 0.000 0.000
LightGBM Avg. 96.53% 96.42%
Variance 0.000 0.000

In addition to the basic statistical measures listed above, we measured the statistical
significance of the results before and after applying optimization. Hence, the paired two
samples were used for the mean t-test. The null hypothesis, h_0, for this comparison is
that the mean accuracy values achieved before and after applying GA optimization to the
classifiers are the same. The p values suggest that the null hypothesis can be rejected in
four cases (out of six), which means that the improvement is significant in most of the cases
(see Table 14).

Table 14. The reported p values for t-tests.

Classifier Name t-Test Result Conclusion

Random Forests t-stat. 1.466706885 No significant
p-value 0.088 improvement
AdaBoost t-stat. −2.100040666 Significant
p-value 0.032556993 improvement
XGBoost t-stat. −13.49130461 Significant
p-value 0.000 improvement
Bagging t-stat. −2.976672182 Significant
p-value 0.008 improvement
GradientBoost t-stat. −11.26647694 Significant
p-value 0.000 improvement
LightGBM t-stat. 0.971025 No significant
p-value 0.178454 improvement

Similarly, the statistical analysis was conducted on the other datasets. It was found
that the improvements obtained by AdaBoost, XGBoost, and GradientBoost with GA
optimization were significant using Dataset 2, while for Dataset 3, the improvements
obtained by all GA-based ensemble classifiers (except Random Forests) were significant.
In addition, the Friedman test results showed a significant difference in accuracy, of
(X 2 = 51.96, d f = 9, p = 2.82 × 10−7 ) for the first data set, and (X 2 = 48.16, d f = 9, p = 1.83
×10−5 ) and (X 2 = 41.26, d f = 9, p = 2.68 ×10−5 ) for the second and third datasets,
respectively. This indicates that it is safe to reject the null hypothesis when a model
performed the same. In addition, we can conclude that at least one model has different
performance values. Therefore, we conducted the Nemenyi post-hoc.
The comparative analysis of all of the models using their mean ranks was carried out.
The calculated values of critical difference for the datasets were CD = 4.9493, CD = 4.4094,
and CD = 3.283 for the first, second, and third datasets, respectively. Figures 9–11 show
the critical difference diagrams where the models with statistically similar values of perfor-
mance are connected to one another.
 FOR PEER REVIEW 16 of 19
Electronics 2021, 10, 1285 15 of 18

Figure 9. Critical difference diagram of Dataset 1 for the Nemenyi test.

Figure 10 shows the results of the statistical comparison of all of the models against
one another by their mean ranks using Dataset 2 (higher ranks, such as 1.9 for LGB, cor-
respond to higher values). Classifiers (only the three classifiers that have the highest val-
ues) that are not connected by a bold line of length equal to the CD have significantly
different
Figure 9. Critical difference mean
Figure 9. Critical
diagram ranks (Confidence
difference
1 for level
diagram of Dataset
of Dataset of 95%).
1 for
the Nemenyi the Nemenyi test.
test.

Figure 10 shows the results of the statistical comparison of all of the models against
one another by their mean ranks using Dataset 2 (higher ranks, such as 1.9 for LGB, cor-
respond to higher values). Classifiers (only the three classifiers that have the highest val-
ues) that are not connected by a bold line of length equal to the CD have significantly
different mean ranks (Confidence level of 95%).

21, 10, x FOR PEER REVIEW 17 of 19

Figure 10. 10.

Figure Critical difference
Critical differencediagram
diagram of Dataset
Dataset22for
forthe
theNemenyi
Nemenyi test.
test.

Figure 11 shows the results of the statistical comparison of all of the models against
one another by their mean ranks using Dataset 3 (higher ranks, such as 1.5 for GA–XGB,
correspond to higher values). Classifiers (only the three classifiers that have the highest
value) that are not connected by a bold line of length equal to the CD have significantly
different mean ranks (Confidence level of 95%).

Figure 10. Critical difference diagram of Dataset 2 for the Nemenyi test.

Figure 11 shows the results of the statistical comparison of all of the models against
one another by their mean ranks using Dataset 3 (higher ranks, such as 1.5 for GA–XGB,
correspond to higher values). Classifiers (only the three classifiers that have the highest
value) that are not connected by a bold line of length equal to the CD have significantly
different mean ranks
Figure(Confidence leveldiagram
11. Critical difference of 95%).
of Dataset 3 for the Nemenyi test.
Figure 11. Critical difference diagram of Dataset 3 for the Nemenyi test.

In addition, a comparison was conducted with the previous studies that used the
same phishing websites (Dataset 1 and Dataset 2), which is presented in Table 15. As Da-
taset 3 was only recently prepared, it was not used in the previous studies. The evaluation
metrics were accuracy, precision, and recall. The results show that the proposed opti-
Electronics 2021, 10, 1285 16 of 18

Figure 9 shows the results of the statistical comparison of all of the models against
one another by their mean ranks using Dataset 1 (higher ranks, such as 1.8 for GA–GB,
correspond to higher values). Classifiers (only the three classifiers that have the highest
values) that are not connected by a bold line of length equal to the CD have significantly
different mean ranks (Confidence level of 95%).
Figure 10 shows the results of the statistical comparison of all of the models against one
another by their mean ranks using Dataset 2 (higher ranks, such as 1.9 for LGB, correspond
to higher values). Classifiers (only the three classifiers that have the highest values) that
are not connected by a bold line of length equal to the CD have significantly different mean
ranks (Confidence level of 95%).
Figure 11 shows the results of the statistical comparison of all of the models against
one another by their mean ranks using Dataset 3 (higher ranks, such as 1.5 for GA–XGB,
correspond to higher values). Classifiers (only the three classifiers that have the highest
value) that are not connected by a bold line of length equal to the CD have significantly
different mean ranks (Confidence level of 95%).
In addition, a comparison was conducted with the previous studies that used the same
phishing websites (Dataset 1 and Dataset 2), which is presented in Table 15. As Dataset 3
was only recently prepared, it was not used in the previous studies. The evaluation metrics
were accuracy, precision, and recall. The results show that the proposed optimized stacking
ensemble method outperformed the other recent and related works [7,10] in using the
accuracy and recall performance measures for Dataset 1, and outperformed [35] in using
the accuracy, precision and recall measures for for Dataset 2.

Table 15. Comparison of the proposed method with the previous studies.

Precision Recall
Paper Classifier Dataset Accuracy%
% %
Ali and Ahmed [7] GA–ANN Dataset 1 88.77 85.81% 93.34%
Ali and Malebary [10] POS–RF Dataset 1 96.83 98.76% 95.37%
The stacking
This study Dataset 1 97.16 96.86% 96.83%
ensemble method
Khan, Khan, and Hussain [35] ANN after PCA Dataset 2 97.13 96.48% 98.03%
The stacking
This study Dataset 2 98.58 98.50% 98.74%
ensemble method

5. Conclusions
This paper has proposed an optimized stacking ensemble model for detecting phishing
websites. In the optimisation method, a genetic algorithm, was used to find the optimized
values for the parameters of several ensemble learning methods. The proposed model
includes three phases: the training, ranking, and testing phases. In the training phase, sev-
eral ensemble learning methods were trained without applying the optimization method
(GA); these included Random Forests, AdaBoost, XGBoost, Bagging, GradientBoost, and
LightGBM. These classifiers were then optimized using a GA that selects the optimal values
of model parameters and improves their overall accuracy. In the ranking phase, the best
three ensemble methods were selected and used as base classifiers for a stacking ensemble
method. The stacking method also used three classifiers as meta-learners: RF, GB, and
SVM. Finally, in the testing phase, new websites were collected and used as a testing
dataset in order to predict the final class label of these websites (phishing or legitimate).
The experimental results showed that the proposed optimized stacking ensemble method
obtained superior performance compared to other machine-learning-based detection meth-
ods; the obtained accuracy reached 97.16%. A statistical analysis was conducted, which
showed that the obtained improvements were statistically significant. In addition, the
proposed methods were compared with recent studies that used the same phishing dataset,
and it was reported that the proposed method surpassed those used in these studies. As
phishing attacks are more dangerous in internet of things (IoT) environments—because IoT
Electronics 2021, 10, 1285 17 of 18

devices are an easy medium for attackers, who can simply use the software in the thingbots
for spreading spam emails without the user knowing—a light detection method will be
proposed in future work to be applied to IoT environments. In addition, deep learning
methods will be investigated in order to improve the detection rate of phishing websites,
and more phishing datasets will be used.

Author Contributions: Conceptualization, Z.G.A.-M. and B.A.M.; methodology, M.A.-S. and F.S.;
software, M.A.-S.; validation, M.A.-S., and T.A.-H., and A.A.; formal analysis, F.S.; investigation, T.A.-
H., M.T.A. and T.S.A.; resources, F.S. and T.A.-H.; data curation, M.A.-S. and F.S.; writing—original
draft preparation, M.A.-S., F.S. and B.A.M.; writing—review and editing, M.A.-S., F.S. and Z.G.A.-M.;
visualization, F.S.; supervision, M.A.-S.; project administration, M.A.-S. and F.S.; funding acquisition,
Z.G.A.-M., B.A.M., M.T.A., A.A and T.S.A.; All authors have read and agreed to the published version
of the manuscript.
Funding: This research has been funded by the Scientific Research Deanship at the University of
Ha’il, Saudi Arabia, through project number RG-20 023.
Data Availability Statement: Data are available in [31–33].
Acknowledgments: We would like to acknowledge the Scientific Research Deanship at the University
of Ha’il, Saudi Arabia, for funding this research.
Conflicts of Interest: The authors declare no conflict of interest.

References
1. Gupta, B.B.; Arachchilage, N.A.G.; Psannis, K.E. Defending against phishing attacks: Taxonomy of methods, current issues and
future directions. Telecommun. Syst. 2018, 67, 247–267. [CrossRef]
2. Gubbi, J.; Buyya, R.; Marusic, S.; Palaniswami, M.S. Internet of Things (IoT): A vision, architectural elements, and future directions.
Futur. Gener. Comput. Syst. 2013, 29, 1645–1660. [CrossRef]
3. Roman, R.; Najera, P.; Lopez, J. Securing the Internet of Things. Computer 2011, 44, 51–58. [CrossRef]
4. Tang, D. Event Detection in Sensor Networks; School of Engineering and Applied Sciences, The George Washington University:
Washington, DC, USA, 2009.
5. Wei, B.; Hamad, R.A.; Yang, L.; He, X.; Wang, H.; Gao, B.; Woo, W.L. A Deep-Learning-Driven Light-Weight Phishing Detection
Sensor. Sensors 2019, 19, 4258. [CrossRef]
6. Somesha, M.; Pais, A.R.; Rao, R.S.; Rathour, V.S. Efficient deep learning techniques for the detection of phishing websites. Sadhana
2020, 45, 1–18. [CrossRef]
7. Ali, W.; Ahmed, A.A. Hybrid intelligent phishing website prediction using deep neural networks with genetic algorithm-based
feature selection and weighting. IET Inf. Secur. 2019, 13, 659–669. [CrossRef]
8. Chiew, K.L.; Tan, C.L.; Wong, K.; Yong, K.S.; Tiong, W.K. A new hybrid ensemble feature selection framework for machine
learning-based phishing detection system. Inf. Sci. 2019, 484, 153–166. [CrossRef]
9. Rao, R.S.; Pais, A.R. Detection of phishing websites using an efficient feature-based machine learning framework. Neural Comput.
Appl. 2018, 31, 3851–3873. [CrossRef]
10. Ali, W.; Malebary, S. Particle Swarm Optimization-Based Feature Weighting for Improving Intelligent Phishing Website Detection.
IEEE Access 2020, 8, 116766–116780. [CrossRef]
11. Khursheeed, F.; Sami-Ud-Din, M.; Sumra, I.A.; Safder, M. A Review of Security Machanism in internet of Things (IoT). In
Proceedings of the 2020 3rd International Conference on Advancements in Computational Sciences (ICACS), Lahore, Pakistan,
17–19 February 2020; pp. 1–9.
12. Tsiknas, K.; Taketzis, D.; Demertzis, K.; Skianis, C. Cyber Threats to Industrial IoT: A Survey on Attacks and Countermeasures.
IoT 2021, 2, 163–186. [CrossRef]
13. Parra, G.D.L.T.; Rad, P.; Choo, K.-K.R.; Beebe, N. Detecting Internet of Things attacks using distributed deep learning. J. Netw.
Comput. Appl. 2020, 163, 102662. [CrossRef]
14. Mao, J.; Bian, J.; Tian, W.; Zhu, S.; Wei, T.; Li, A.; Liang, Z. Phishing page detection via learning classifiers from page layout
feature. EURASIP J. Wirel. Commun. Netw. 2019, 2019, 1–14. [CrossRef]
15. Virat, M.S.; Bindu, S.; Aishwarya, B.; Dhanush, B.; Kounte, M.R. Security and Privacy Challenges in Internet of Things. In
Proceedings of the 2018 2nd International Conference on Trends in Electronics and Informatics (ICOEI), Tirunelveli, India, 11–12
May 2018; pp. 454–460.
16. Deogirikar, J.; Vidhate, A. Security attacks in IoT: A survey. In Proceedings of the 2017 International Conference on I-SMAC (IoT
in Social, Mobile, Analytics and Cloud) (I-SMAC), Palladam, India, 10–11 February 2017; Volume 16, pp. 32–37.
17. Alotaibi, B.; Alotaibi, M. A Stacked Deep Learning Approach for IoT Cyberattack Detection. J. Sensors 2020, 2020, 1–10. [CrossRef]
Electronics 2021, 10, 1285 18 of 18

18. Alsariera, Y.A.; Adeyemo, V.E.; Balogun, A.O.; Alazzawi, A.K. AI Meta-Learners and Extra-Trees Algorithm for the Detection of
Phishing Websites. IEEE Access 2020, 8, 142532–142542. [CrossRef]
19. Jain, A.K.; Gupta, B.B. A machine learning based approach for phishing detection using hyperlinks information. J. Ambient. Intell.
Humaniz. Comput. 2018, 10, 2015–2028. [CrossRef]
20. Feng, F.; Zhou, Q.; Shen, Z.; Yang, X.; Han, L.; Wang, J. The application of a novel neural network in the detection of phishing
websites. J. Ambient. Intell. Humaniz. Comput. 2018, 1–15. [CrossRef]
21. Aburub, F.; Hadi, W. A New Association Classification Based Method for Detecting Phishing Websites. J. Theor. Appl. Inf. Technol.
2021, 99, 147–158.
22. Gandotra, E.; Gupta, D. An Efficient Approach for Phishing Detection using Machine Learning. In Multimedia Security; Giri, K.J.,
Parah, S.A., Bashir, R., Muhammad, K., Eds.; Springer: Singapore, 2021; pp. 239–253. [CrossRef]
23. Shabudin, S.; Sani, N.S.; Ariffin, K.A.Z.; Aliff, M. Feature Selection for Phishing Website Classification. Int. J. Adv. Comput. Sci.
Appl. 2020, 11, 587–595.
24. Subasi, A.; Molah, E.; Almkallawi, F.; Chaudhery, T.J. Intelligent phishing website detection using random forest classifier. In
Proceedings of the 2017 International Conference on Electrical and Computing Technologies and Applications (ICECTA), Ras Al
Khaimah, United Arab Emirates, 21–23 November 2017; pp. 1–5.
25. Yu, X. Phishing Websites Detection Based on Hybrid Model of Deep Belief Network and Support Vector Machine. IOP Conf. Ser.
Earth Environ. Sci. 2020, 602, 012001. [CrossRef]
26. Azeez, N.A.; Salaudeen, B.B.; Misra, S.; Damaševičius, R.; Maskeliūnas, R. Identifying phishing attacks in communication
networks using URL consistency features. Int. J. Electron. Secur. Digit. Forensics 2020, 12, 200–213. [CrossRef]
27. Azeez, N.A.; Atiku, O.; Misra, S.; Adewumi, A.; Ahuja, R.; Damasevicius, R. Detection of Malicious URLs on Twitter. In Advances
in Electrical and Computer Technologies; Springer: Singapore, 2020; pp. 309–318.
28. Osho, O.; Oluyomi, A.; Misra, S.; Ahuja, R.; Damasevicius, R.; Maskeliunas, R. Comparative Evaluation of Techniques for
Detection of Phishing URLs. In Proceedings of the Communications in Computer and Information Science; Springer: Cham, Switzerland,
2019; pp. 385–394.
29. Patil, D.R.; Patil, J.B. Malicious web pages detection using feature selection techniques and machine learning. Int. J. High. Perform.
Comput. Netw. 2019, 14, 473. [CrossRef]
30. PhishTank, Developer Information. Available online: http://phishtank.org/developer_info.php (accessed on 28 February 2021).
31. Dua, D.; Graff, C. UCI Machine Learning Repository; School of Information and Computer Science, University of California:
Irvine, CA, USA. Available online: https://archive.ics.uci.edu/ml/datasets/Phishing+Websites (accessed on 10 January 2021).
32. Tan, C.L. Phishing Dataset for Machine Learning: Feature Evaluation. Mendeley Data 2018, 1. [CrossRef]
33. Vrbančič, G. Phishing Websites Dataset. Mendeley Data 2020, 1. [CrossRef]
34. Jiang, Y.; Tong, G.; Yin, H.; Xiong, N. A Pedestrian Detection Method Based on Genetic Algorithm for Optimize XGBoost Training
Parameters. IEEE Access 2019, 7, 118310–118321. [CrossRef]
35. Khan, S.A.; Khan, W.; Hussain, A. Phishing Attacks and Websites Classification Using Machine Learning and Multiple Datasets
(A Comparative Analysis). In Proceedings of the Transactions on Petri Nets and Other Models of Concurrency XV; Springer: Cham,
Switzerland, 2020; pp. 301–313.

View publication stats