Received January 12, 2022, accepted January 24, 2022, date of publication February 28, 2022, date of current

version March 10, 2022.

Digital Object Identifier 10.1109/ACCESS.2022.3155695

A Malware Detection Approach Using

Autoencoder in Deep Learning
XIAOFEI XING1,2 , XIANG JIN 1, HAROON ELAHI1 , HAI JIANG3 , AND GUOJUN WANG 1
1 School of Computer Science and Cyber Engineering, Guangzhou University, Guangzhou, Guangdong 510006, China
2 School of Information Engineering, Zhengzhou University of Industrial Technology, Zhengzhou, Henan 451100, China
3 Department of Computer Sciences, Arkansas State University, Jonesboro, AR 72467, USA

Corresponding author: Xiang Jin ([email protected])

This work was supported by the Chinese Scholarship Council under Grant No.202008440213, and the National Key Research and
Development Program of China under Grant No.2020YFB1005804.

ABSTRACT Today, in the field of malware detection, the expanding limitations of traditional detection
methods and the increasing accuracy of detection methods designed on the basis of artificial intelligence
algorithms are driving research findings in this area in favour of the latter. Therefore, we propose a novel
malware detection model in this paper. This model combines a grey-scale image representation of malware
with an autoencoder network in a deep learning model, analyses the feasibility of the grey-scale image
approach of malware based on the reconstruction error of the autoencoder, and uses the dimensionality
reduction features of the autoencoder to achieve the classification of malware from benign software. The
proposed detection model achieved an accuracy of 96% and a stable F-score of about 96% by using
the Android-side dataset we collected, which outperformed some traditional machine learning detection
algorithms.

INDEX TERMS Malware detection, autoencoders, malware images, mobile application security.

I. INTRODUCTION artificial intelligence. These detection techniques are more

In recent years, the rapid development of mobile internet accurate, robust and generalisable than traditional malware
technology has rendered the growth of the software industry. detection techniques, and can avoid the risk of false detection
The number of malware is growing with each passing day. for many newly generated malware. Therefore, it is of better
According to the latest China Internet Annual Network Secu- scientific interest to dig into malware detection systems based
rity Report [1], as of 2019, there were as many as 13,510,900 on this algorithms.
cases of mobile Internet malware programs, with nearly There are 2 main phases about malware detection tech-
2,791,300 new cases added this year alone. The Android sys- niques using artificial intelligence algorithms: the data pre-
tem has been the key to many mobile-based malware attacks processing phase, which focuses on the extraction of software
due to the open nature of the Android application market. features, and the model classification phase, which uses the
With the increase in Android malware security threats, it is feature data to train the model to complete the classification
necessary to develop an efficient and novel mobile malware task.
detection method to solve the problem. In the data pre-processing phase, the common extraction
Traditional malware detection techniques are limited by methods include static extraction and dynamic extraction
the number of detection rules that need to be set manu- about feature data. Static extraction of features means extract-
ally. It is impossible to detect many new malware vari- ing features without running the software program [3]–[6],
ants in today’s world of increasing malware [2]. In recent in ways that include extracting bytecode [7], file header infor-
years, malware detection techniques combined with AI algo- mation [8], API call information [9], application interface
rithms have shown better performance with the boom in information [10], application permission information [8], etc.
The main principle of static analysis features is to obtain
The associate editor coordinating the review of this manuscript and the source code or bytecode of the program through soft-
approving it for publication was Marco Martalo . ware decompilation and analyze the semantic features and

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

25696 For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/ VOLUME 10, 2022
X. Xing et al.: Malware Detection Approach Using Autoencoder in Deep Learning

semantic information contained in it. The detection method MDS based on inputs and outputs and adds redundant
using such features is included less overhead and stable, API calls using the adversarial RNN in an adversarial attack.
such as MaMadroid [11] proposed by Mariconti et al. and a Files injected using redundant API calls can easily bypass
malware detection method proposed by Wenjin Li et al. [12] RNN detection [32]. Despite the high accuracy of RNNs, the
which used Android-side application permission informa- reliability of the results generated by RNNs may still be ques-
tion, API call information and other static data for malware tionable in malware detection. Convolutional neural networks
detection. can extract location non-specific local features from fixed-
Compared with static extraction, the dynamic approach size, high-dimensional tensor-type data. As a result, it has
analyzes the behavioral activities of software been used and shown excellent performance in computer
runtime [13]–[18]. Therefore, the extracted features are vision research, and there are also many research applications
more accurate, such as the DL-Droid proposed by in the field of malware detection. Mahoud et al. [33] used
Mohammed et al. [19], who used software log files running 2D-CNN and 3D-CNN as classification models and used
on real devices to extract feature data. They used more than various detection data extracted from dynamic environments
30,000 applications to extract feature data, and the accuracy as feature data, with an accuracy of up to 90%. Xiao et al. [34]
was as high as 99.6%. Tobiyama et al. [20] used recurrent used CNN to understand the characteristics of Android mal-
neural networks to extract feature from the temporal data of ware from Dalvik bytecode. The method is efficient with an
the processes when the malware program was running, and accuracy rate of over 93%. Wang et al. [35] proposed vari-
then used convolutional neural networks to classify them with ous network models for detecting malware, such as CNN-S,
a high accuracy of 96%. However, many malware programs DAE-CNN-S, where malware data representation is extracted
hide their malicious behaviour in a virtual environment [21], software privileged information to generate feature images,
and the dynamic virtual operating environment required to which outperforms most malware detection algorithms based
make them behave maliciously is more demanding and com- on traditional machine learning models. Malware detection
plex, so the classification model using such features is less models using deep learning neural networks show superior
stable in accuracy and more overheads. detection performance, and are more scalable than malware
In the model training and classification phase, the main detection models using machine learning algorithms.
approaches include malware detection methods based on In this work, we extract features from the bytecode of var-
machine learning algorithms and deep learning models. The ious command methods of android software in a static way.
methods based on machine learning algorithms mainly use Then an auto-encoder based on convolutional neural network
common machine learning algorithms as classification mod- framework is used to reconstruct the grey-scale image corre-
els, like Wang et al. [22], used five machine learning models sponding to each malware. Finally, the auto-encoder is exper-
for software classification, namely Support Vector Machine imentally analysed in reconstructing the high-dimensional
(SVM), K-Nearest Neighbour (KNN), Naive Bayes (NB), features of the malware performance. We designed a neural
Classification Regression Tree (CART) and Random For- network based on the auto-encoder structure to perform the
est (RF). Kumar et al. [23] proposed a feature learning classification and detection task for malware. And exper-
model using various machine learning algorithms to achieve iments were conducted using datatsets we collected from
detection of malware with low overhead and high accuracy. VisureShare. The experimental results show that our method
RepassDroid [24] extracted various APIs with sensitive trig- is more accurate than traditional machine learning methods
ger points and basic software permissions as datasets to train and some deep learning malware detection models based on
a machine learning model for detection. They used 24,288 malware images.
samples for training and testing, and the experimental results The main contributions of this paper are as follows:
show that their method had satisfactory results with accuracy • We propose a method for generating feature images
rates of 97.7% and 93.3%, respectively. Yerima et al. [25] corresponding to each malware and benign software.
used a Bayesian classifier as classification model, and The main approach is to convert the bytecodes of the
Li et al. [26] used a decision tree to construct a model to various methods in the software into grey-scale images
achieve classification and detection of malware. for subsequent model training and classification.
Malware detection methods based on deep learning mod- • We used auto-encoder based on convolutional neural
els mainly use neural networks [27], [28], recurrent neural network designed to recognise the high-dimensional fea-
networks and convolutional neural networks to implement tures contained in such grey-scale images, and experi-
malware detection. The malware detection methods applying mentally demonstrated the feasibility of the scheme.
recurrent neural network models are the most common. The • We propose a neural network model based on autoen-
methods based on this network structure usually encode all coder networks for the classification task of malware
API instructions of the malware as one-hot vectors and put detection and experimentally demonstrate the high accu-
them into the model as input data, e.g. [29], [30]. Long- racy of our malware detection model.
term short-term memory (LSTM) networks have shown on The remainder of this paper is summarised as shown fol-
the metric of high accuracy [31]. However, RNNs are vul- lows: Section 2 presents the related work. We propose our the-
nerable to attacks. An attacker mimics the RNN used in the oretical scheme in section 3. Section 4 gives the experimental

VOLUME 10, 2022 25697

 X. Xing et al.: Malware Detection Approach Using Autoencoder in Deep Learning

results of malware detection model. Section 5 sumarises our

liminations and section 6 summarises our work.

II. RELATED WORK

This section describes the work related to the generation of
malware images and the static malware detection method
based on deep learning models. Therefore, this section con-
tains 2 parts, the first part is the malware image generation
scheme and the second part is the static malware detection
scheme based on deep learning models.

A. MALWARE IMAGES
In the feature extraction phase for malware detection, neural
networks can also be used to extract the corresponding fea-
tures of the software in addition to extracting the correspond-
ing static feature information, such as API calls, permission
information, etc, and dynamic feature information, such as
network activity, log files, etc. This feature extraction solution
is more automated and simpler than other manual feature FIGURE 1. The malware grayscale image in [37].
extraction methods.
Automatic extraction of software features using neural nets model consisting of deep learning algorithms and traditional
requires consideration of the data representation. So that it machine learning algorithms to achieve 99.3% accuracy for
can better extract the key features and ensure the accuracy of a 25-class malware classification task using data augmenta-
the test results. A feasibel solution is the use of images [36], tion based on affine image transformation. Singh et al. [42]
where the program is transformed into an image and handed used 15 different combinations of Android malware image to
over to the neural net to extract the features. The similarity of identify and classify Android malware, and machine learning
software structures is reflected by the similarity of textures algorithms were used to analyse grey-scale malware images
between corresponding images, such as the malware picture instead of the Softmax layer of CNN such as K-Nearest
representation scheme proposed by Natarij et al. [37], they Neighbour (KNN), Support Vector Machine (SVM) and Ran-
transformed the binary code of the malware into the form dom Forest (RF). The classification results showed that the
of a 2-dimensional matrix, and it can be represented in the method achieved a correct classification rate of 92.59%.
form of a grey-scale graph since the numerical range of its These works focus on how to convert software of different
transformed matrix is [0, 255], as shown in Fig. 1, where data sizes into images of the same size, and need to consider the
of different structures have different textures. challenge of how to do the best possible job of reducing
Yan et al. [38] generated greyscale images from mal- redundancy in the process of generating the images. The
ware files while decompiling to obtain the software opcode difference between our work and previous work is that we
sequences, trained the greyscale images using convolutional try to extract the binary code of the method field in the
neural networks, learned the opcode sequences using long software and convert some of the information into byte code
and short-term memory networks, and conducted exper- to complete the generation of the grey-scale image. Analysing
iments on more than 40,000 samples with an accuracy the feasibility of such a scheme is a major part of our research.
of 99.88%. They used bilinear interpolation to resize the
images to ensure that the size of the greyscale images B. A STATIC MALWARE DETECTION SOLUTION BASED
input to the training network should be the same size. ON DEEP LEARNING MODELS
K. He et al. [39] proposed a malware detection method based Deep learning models can show better performance on clas-
on image recognition. They converted malware into RGB sification and prediction tasks [43], [44], so they have been
images and classified them using CNN and spatial pyramid widely used in many research areas [45], such as recom-
pooling (SPP) layer. Experimental evaluation showed that the mendation systems [46], privacy protection [47], [48], image
malware detection method designed based on RGB images recognition [49], and natural language processing. In the
is highly accurate and resistant to redundant API injection field of malware detection, deep learning models also have
attacks. ASLAN et al. [40] focused on the design of the a wide range of applications [50].
network architecture of the detection model by converting The proposed malware detection scheme is related to the
PE files of software samples into grey-scale maps of malware, static feature extraction of software samples and the use of
training and detecting them using a hybrid network structure, deep learning networks as classification detection models.
and testing them on the Malimg dataset with an accuracy Therefore, we present some noteworthy work in the area
of 97.98%. Nisa et al. [41] used distinctive pre-trained models of malware detection models based on deep learning mod-
(AlexNet and Inception-V3) for feature extraction, a hybrid els. There are two reasons for choosing the static analysis

25698 VOLUME 10, 2022

X. Xing et al.: Malware Detection Approach Using Autoencoder in Deep Learning

approach to extract software file features. Firstly, static anal- which improved the average accurate detection rate and preci-
ysis is intuitive and comprehensive, as compared to dynamic sion by 11% compared to CNN-based sequence classification
extraction efforts, static analysis does not need to consider and Hidden Markov Model-based methods. The accuracy was
when malware needs any trigger conditions to exhibit mali- as high as 94%. Chai et al. [52] obtained local semantic
cious behaviour, and its underlying source code intuitively features from API call sequence information, learned them
contains the functional features of malware. Secondly, static using cascaded convolutional neural networks and graph con-
analysis is faster and more efficient than dynamic detection, volutional networks, proposed a joint framework for mal-
which takes a long time to run the malware program in order ware detection LGMal, and used the Alibaba Cloud Security
to record all kinds of data and is inefficient when dealing with malware detection dataset to conduct experiments with high
a large number of software samples, whereas static analysis accuracy. ZOU et al. [53] transformed the function call graph
can extract features from a large number of software samples of a program into a complex social network and used the
in a short period of time, which makes practical sense. The centrality analysis of social networks to perform the detec-
deep learning model was chosen because of its ability to tion. The approach is to represent the semantic features of
generalise and detect previously unseen malware samples the graph by calculating the average closeness between sen-
with high accuracy. sitive API calls and the central node. Their detection method
Wang et al. [6] obtained the corresponding manifest demonstrated 99.1% accuracy on 3988 benign and 4265 mali-
files and source code files from Android application files, cious samples, and was also six times faster than MaMaDroid.
extracted the corresponding software permission information In our work, we try a different and novel idea to achieve
and API function call information, then used deep learn- detection. We use autoencoder network to design detection
ing algorithms to identify and classify them. The experi- model, the design process of this network is complex but
ments proved that the proposed method has higher accuracy converges quickly and take less time to train.
and stability compared with the traditional support vector
machine method, and can identify similar features among III. APPROACH
similar malware. Yuan et al. [28] proposed a combination
A. OVERVIEW
of static analysis methods and dynamic analysis methods
We propose a approach to malware detection, which is
for software feature extraction work in response to the cur-
designed based on the automatic encoder network. The Fig. 2
rent severe malware threat environment, statically analyzing
illustrates the overall structure and main tasks of our malware
the manifest configuration file and class execution file of
detection method. First, benign files and malware are trans-
Android software, and dynamically analyzing the log file of
formed into corresponding greyscale images by decompiling
malware programs on the Android side, combining the two to
the APK files, the binary codes are ectracted from methods
extract software feature information. Then, they using deep
in software, then converting them into decimal data by bytes,
learning models for training and classification, experiment-
which are filled with pixel value. Afterwards, the greyscale
ing on more than 20,000 sample programs. The accuracy
images are passed through 2 deep learning networks in order
degree engaged to 96.76%. Kim et al. [27] conducted exper-
to complete 2 tasks. The first deep learning network named
iments on 41260 software samples, decompiled software
automatic encoder network - 1(AE-1), which we use to anal-
executable files, extracted the corresponding software con-
yse the feasibility of using grey-scale images to represent
figuration files, execution files and function library files, used
the corresponding features of softwares, and the second deep
neural networks as classification models. And they proposed
learning network is automatic encoder network - 2(AE-2),
a multimodal deep neural network model by inputting differ-
which we use to perform the task of classifying malicious
ent types of features into different initial neural networks for
softwares from benign softwares. The detailed design pro-
processing for features with different attributes. And finally
cess of AE-1 and AE-2 will be described in the subsequent
the results are aggregated. The accuracy of the experimental
sections.
data reached 98%. Li et al. [12] proposed a malware detection
method based on weight-adjusted deep learning networks,
which combined dangerous API calls and risky permission B. PRE-PROCESSING OF FEATURE DATA
information as feature data, and the experimental results The main task of the Pre-processing of feature data phase is
showed a high accuracy. These works focus on how to com- to provide an input data for the neural network model. We use
bine software feature data with deep learning models, using a grey-scale image of the software bytecode to represent
feature information that is not comprehensive enough and the characteristics of the software, the so-called grey-scale
different from the feature data used in our work. Secondly, the image of the software bytecode is to decompile the software
deep learning models they use are relatively simple, and the to obtain its binary bytecode, then convert it into a decimal
robustness of their classification network detection accuracy type by byte and fill it into a fixed size two-dimensional
for large-scale malware detection is controversial. matrix, since a byte is 8 bits, that corresponds exactly to
Shukla et al. [51] designed a malware detection model the range of data from 0 to 255 and can be composed as
based on recurrent neural networks, using grey-scale images a grey-scale image. The advantages of using this method
and hardware-based performance counters to extract feature, are twofold. Firstly, this method of extracting software

VOLUME 10, 2022 25699

 X. Xing et al.: Malware Detection Approach Using Autoencoder in Deep Learning

FIGURE 2. The overview of our proposed approach.

features is less overhead and intuitive. Secondly, the as shown in Fig. 3. The encoding network achieves the effect
grey-scale image converted from the software file bytecode is of dimensionality reduction and compression, and the decod-
a suitable input to the convolutional network for training and ing network achieves the purpose of reconstructing the input.
classification, as our subsequent network model is composed Its loss function is defined as the error value between the
of a convolutional neural network, which requires a fixed size original input and the model output corresponding to the
multi-dimensional matrix type of data. original input, and minimising its loss function by means
However, the conversion of software binary codes into of training and gradient updating is the operation process of
grey-scale images has some drawbacks. Although the soft- the autoencoder network. Borghesi et al. used autoencoders
ware binary code contains a variety of feature, it also contains to enable anomaly detection in large computer systems [55].
a large amount of works focus on how to convert software Their results show that the autoencoder can monitor anoma-
of different sizes into images of the same size, and need to lies that were never noticed before based on previous
consider the sticking points of how to do the best possible log records with an accuracy of between 88% and 96%.
job of reducing redundancy in the process of generating Angelo et al. propose a malware detection system for
the images. The difference between our work and previous Android based on an autoencoding network [56]. They
work is that we try to extract the binary code of the method put sequences of API calls from the application as input
field in the software and convert some of the information into an autoencoder network to complete feature extraction,
into byte code to complete the generation of the grey-scale then used a neural network to train and classify features.
image. Analysing the feasibility of such a scheme is a major Their system achieves higher accuracy than complex tradi-
part of our research. The redundant information causes high tional machine learning methods such as J48, Naive Bayes
pre-processing overhead and reduces the accuracy and robust- and MLP.
ness of the model classification at the later stage. We designed 2 model structures and named them AE-1 and
For this reason, we decompile the software and instead of AE-2 respectively, the design sequence is AE-1 first and then
converting the software binary data directly into a greyscale AE-2. The main purpose of designing the AE-1 network is to
image. We extract all the methods in the software and convert use it to analyse the feasibility of feature extraction methods
the byte code of methods into a greyscale image, filling in for grey-scale images, and the purpose of designing the
any blank areas with zero. The advantage of this is twofold. AE-2 network is to use it for malware detection. The rea-
Firstly, these methods contain various actions of the software, son for designing the 2 networks is that the AE-1 network
such as sending network data, reading private information exhibited more drawbacks and less stability for the experi-
on the phone, writing data to the phone’s ROM and hard mental aspects of the classification task, so we improved on
drive, and can be used to visually represent malicious actions the AE-1 network and proposed the AE-2 network. It is worth
in a greyscale image without setting up a dynamic runtime noting that the AE-1 network is trained in an unsupervised
environment. The second point is that we have reduced the manner and no software samples are labelled, while the AE-2
redundancy of using images to represent malware compared network is trained in a supervised manner and requires
to previous grey-scale image processing, making the subse- labelling of malicious and benign software samples.
quent classification of the model more accurate and stable. The specific structure of two networks will be described in
the subsequent Part I and Part II.
C. THE STRUCTURE OF OUR AUTOENCODER
The autoencoder network structure is a special kind of unsu- 1) THE FIRST AUTOMATIC ENCODER STRUCTURE (AE-1)
pervised neural network in a deep learning model [54]. The structure of model AE-1 is shown in Fig. 4,
It consists of an encoding network and a decoding network, and consists of convolutional layers, pooling layers and

25700 VOLUME 10, 2022

X. Xing et al.: Malware Detection Approach Using Autoencoder in Deep Learning

this is what the autoencoder network is trained on. On the con-

trary, if a feature image corresponds to a category belonging
to benign software, the similarity between the reconstructed
image generated by model and the original image will be
low and the SimiliarError value will be enlarged, since the
structure of a feature image transformed by benign software
is very different from the structure of a feature image trans-
formed by malware. For example, if we ask an expert in real
life to focus on malware without studying the characteristics
of benign software, he can easily distinguish the difference
between benign and malware if he is knowledgeable in the
high-dimensional characteristics of malware and then looks at
FIGURE 3. The schematic representation of an autoencoder.
benign software. If the error value of SimiliarError generated
by the two types of software after such an autoencoder have
a huge difference, then we can use this to make sure that the
autoencoder network can indeed reconstruct the correspond-
ing feature images of the two types of software better.

2) THE SECOND AUTOMATIC ENCODER STRUCTURE (AE-2)

The structure of model AE-2 is shown in Fig. 5, in which
the autoencoder network structure is similar to model AE-1.
The only difference is that we have an external multi-layer
perceptron network to facilitate classification and experimen-
tal evaluation. We first extract the high-dimensional features
FIGURE 4. The first automatic encoder structure (AE-1). corresponding to malware and benign software from model
AE-1 by pre-training, then extract the output from the hidden
up-sampling layers. The activation function uses the relu layer of model AE-1 and use it for the training of the multi-
function and the MAE loss function with the Equ.(1): layer perceptron network. The multilayer perceptron network
outputs two-dimensional vectors to complete the malware
1 X
lr = (yi − yri )2 (1) and benign software classification task.
2n
i

Model AE-1 analyzes whether it can reconstruct the mal-

ware feature image based on the magnitude of similar-
ity between the original malware feature image and the
reconstructed image that has been reconstructed through the
autoencoder network.
We determine whether AE-1 is able to perform this task
by analyzing the numerical magnitude of its similarity, and
this measure of similarity is expressed by defining the
SimiliarError as the following Equ.(2).
PN i i
i=0 (|yr − yg |)
SimiliarError = (2)
N
where yir is a pixel of the original image, yig is the pixel FIGURE 5. The second automatic encoder structure (AE-2).

corresponding to the original position in the image generated

by the autoencoder, and N is all the pixel points of an image. IV. EXPERIMENT EVALUATION
The theoretical basis for determining whether AE-1 can In this section, we evaluate the proposed approach through
perform this task based on the numerical magnitude of the experiments under different indicators. This section includes
SimiliarError is that only unlabeled malware datasets are four parts, namely experimental setup, data feature extrac-
employed during the training phase of the autoencoder net- tion, effectiveness analysis of reconstructing malware images
work. In the predictive classification phase, if a feature image and performance analysis of the detection model.
corresponds to a category that is malware, then the recon-
structed image it generates via the autoencoder network is the A. EXPERIMENTAL SETUP
same as the original. The similarity between images will be This subsection focuses on information related to the experi-
high and the SimiliarError value will be decreased, because mental setup and, for this purpose, is divided into 3 sections,

VOLUME 10, 2022 25701

 X. Xing et al.: Malware Detection Approach Using Autoencoder in Deep Learning

TABLE 1. Experimental environment setup. TABLE 2. Parameters of AE-1 model.

namely the experimental environment setup, the dataset, and

the training details.

1) EXPERIMENTAL ENVIRONMENT SETUP

The details are shown in table 1 for information on the
experimental environment. Our experiments were conducted
using an Intel CoreTM i5-8300 machine with 16GB RAM, malware software, the malware test set DTest_mal , which con-
and GeForce GTX 1060 MQ. The machine had a 64-bit win- tains a partial dataset of the collected malware, and the benign
dows10 operating system. We used Keras, Tensorflow 2.1, software test set DTest_benign , which contains a dataset of the
and Python 3.7 for programing purposes. collected benign software files.
The AE-1 network use training dataset DTrain for the train-
2) DATASET ing task, then use the malware test set DTest_mal and the benign
To evaluate the performance of the proposed model, we col- software test set DTest_benign for the test task. If the new input
lected benign software from the Google App Store [57] of test set is similar to the input of the dataset used in the
and malware from VirusShare [58], where benign software training phase, then the reconstruction error for this test set
consisted of 10 categories such as office, video, gaming, is very small. Conversely, if the new inputs of test set are
finance, photography and reading, and malware included different from the inputs to the dataset used in the training
datasets for APK categories released in 2016, 2017 and 2018. phase, then this test set will exhibit a very large reconstruction
VirusTotal [59] scans a random sample of software to deter- error. The large difference in the error data produced by these
mine that they are correctly labled. 2 test sets after AE-1 is exactly what we are experimenting
We divided them into 3 types of datasets according to with. Since our hypothesis is based on the theory that malware
their purpose, namely: (1) Dataset-1, this dataset is used is all similar and benign software is not similar to malware,
for training and evaluation of AE-1 models, which includes in practice, the different functional characteristics exhibited
8121 malware and 2000 benign softwares. (2) Dataset-2, between malware families in the malware dataset and the
this dataset is used for training, validation and testing of the large redundancy characteristics contained in the software
AE-2 model and contains 8121 malware and 7015 benign dataset can lead to experimental results exhibiting large insta-
software. (3) Dataset-3, this dataset is used to analyse the bilities. For this reason, we are more interested in the relative
detection performance of the AE-2 model on unseen soft- differences between the 2 test sets than in the absolute errors
ware and includes 5,384 malware and 5,000 benign software. they exhibit.
It is worth noting that when we divided Dataset-2 and The AE-2 network is used for the task of analyzing the
Dataset-3, we deliberately put older software samples into performance of the detection model. For the overall dataset
Dataset-2 for training, e.g. malware from 2016, and newer partitioning, we used 80% of the Dataset-2 as the training set
releases into Dataset-3, e.g. 2017, 2018. The purpose of this is and 20% as the test set. In the training phase, the training
to simulate the scenario when the model detects new softwore set was trained and validated using k-fold cross-validation,
samples released in the future and to faciliate the analysis of with k = 6, meaning that 5/6 of the training set was used for
its performance. training and 1/6 for validation, repeated 6 times, and finally
the average was taken. In the testing phase, the test set is used
3) TRAIN AND TEST DEAILS for testing. Minutes are used as units for training time. The
The AE-1 network is used for the task of analyzing the perfor- Adam optimization algorithm, learning rate of 0.0001 and
mance of the autoencoder to reconstruct feature images, and epoch of 100 are chosed in AE-2’s training.
detailed parameters of model AE-1 are shown in Table 2. The The variety of evaluation metrics such as FPR, TPR, ACC,
Adam optimization algorithm is used in the training phase Precision and F-score are used in the model evaluation test
and we set the learning learning rate to be 1e-4, the epoch phase, which are calculated as shown below.
is 100.
We divide the dataset-1 into 3 parts, the training FPR = FP/(FP + TP) (3)
dataset DTrain , which contains a partial dataset of the collected TPR = Recall = TP/(TP + FN ) (4)

25702 VOLUME 10, 2022

X. Xing et al.: Malware Detection Approach Using Autoencoder in Deep Learning

Acc = (TP + TN )/(TP + TN + FP + FN ) (5) TABLE 3. Quantitative analysis of two datasets.

Precision = TP/(TP + FP) (6)

F1 − score = 2 · Precision · Recall/(Precision + Recall)
(7)

B. DATA FEATURE EXTRACTION not similar to the DTrain , and this quantitative analysis also
We used the Androguard tool to complete the data demonstrated that the network structure would show similar
pre-processing task of the model, extracting the source code results on the invisible data set.
of all the class files in the APK file through the Androguard Based on this experiment, we can then show that the
analysis framework, extracting the bytecodes of all the meth- task of reconstruction can be performed well by the auto-
ods and converting them into the decimal data needed for matic encoder through the pre-processed malware data
the corresponding grey-scale images. In the sample dataset from our data, and that the automatic encoder can identify
of software collected, we chose files with as small a data size high-dimensional features of both benign and malicious soft-
as possible to ensure that we could standardise the size of all ware. Then, we implement the subsequent task of classifying
images. malware and benign software.
Based on this approach, all software is converted into the
feature images we need during the data pre-processing phase.

C. EFFECTIVENESS ANALYSIS OF RECONSTRUCTING

MALWARE IMAGES
In this subsection, we evaluate the effectiveness of recon-
structing malware images by analysing the overall error dis-
tribution in malware and benign reconstruct malware images.
Fig. 6 shows the overall error distribution for the 2 test sets.
The reconstructed error value generated by each software
after the encoder network are normalised and expressed as
value on the Y-axis. We normalise by adding up the error FIGURE 6. Reconstruction error for two datasets.
value for each pixel point corresponding to the feature image
of the malware and dividing by the total. In the line statistics D. PERFORMANCE ANALYSIS OF THE DETECTION MODEL
graph, the blue line represents the error trend for the overall We use the AE-2 model to analyse the classification per-
DTest_mal and the yellow line represents the error trend for formance of autoencoders in this subsection. To the end,
the overall DTest_benign . The error is not exactly zero due we experimented with some similar previous research work
to the inherent variability contained in the dataset and the for comparative analysis, include detection models using tra-
redundancy of the software files. However, as can be seen in ditional machine learning algorithms [22], detection models
Fig. 6, the overall error trend is stable for the malware dataset designed based on recurrent neural network [29], detection
represented by the blue line, whereas the overall error trend models designed based on autoencoder networks and con-
for the benign software test set represented by the yellow line volutional neural networks [35] and detection models using
is unstable and fluctuates widely, and the relative difference Malware Images [39]. Among them, the detection model
between the mean value of the errors presented by the two CNN-SPP designed with malware images and convolutional
datasets is large. Thus, our theory is plausible. neural networks proposed by He and Kim [39] showed high
We conducted a quantitative analysis of the two test sets accuracy on the dataset provided by Seoul National Univer-
and normalised the value and presented them in Table 3, sity [60]. The DAE-CNN model proposed by Wang et al. [35]
where the normalisation was done by calculating the mean uses an autoencoder network as the data preprocessing model
absolute error (MAE) and root mean square error (RMSE) of and the output data of the model is used for training and
the test set after making the error of the training set equal to 1. detection of convolutional neural networks, which is similar
The experimental data, as described in Table 3, showed that to our theoretical model, and they show better results on
the normalised MAE and normalised RMSE produced by the 10,000 benign APPs and 13,000 malicious APPs. We use
malware test set were close to 1, indicating that the DTest_mal these 2 types of models as baseline models for comparison
was similar to the DTrain , while the benign test set produced experiment with AE-2 on different datasets.
a normalised MAE and normalised RMSE greater than 1 and
also greater than the value of the training set, indicating that 1) PERFORMANCE COMPARISON OF DIFFERENT MODELS
the benign software test set was not similar to the training set. The ROC curves in Fig. 7 show the effect of the model on the
The normalised MAE and normalised RMSE for the software training set, from which it can be seen that the model exhibits
test set were greater than 1 and also greater than the value a more stable performance on the training set. The ROC
for the malware test set, indicating that the DTest_benign was curves in Fig. 8 show the model’s performance on the test set

VOLUME 10, 2022 25703

 X. Xing et al.: Malware Detection Approach Using Autoencoder in Deep Learning

FIGURE 7. The ROC curve of AE-2 on training set. FIGURE 9. The ROC curve of different models on the unseen software.

FIGURE 8. The ROC curve of different models on the test set. FIGURE 10. Comparation results in five different models.

connected neural network, we use it as our benchmark model.

which in Dataset-2. We can see that our model outperforms Fig. 10 illustrates the value of accuracy, precision, recall
the other two. and F-score on the five models. We find that the decision
To further examine the detection performance of our tree outperforms the support vector machine model and the
model on unseen malware, the Datasets-3 is used as test set naive Bayes model in terms of accuracy and performance
for AE-2. The ROC curves are shown in Fig. 9, from which for traditional machine learning detection algorithms through
it can be seen that our model shows good accuracy and some cross-sectional comparisons, while the deep learning model
feasibility in detecting unseen malware, but also shows some outperforms the traditional machine learning algorithm in
flaws as the software changes with year iteration. terms of overall performance. Our model achieves better
In the experimental results presented in Table 4, the high results in terms of search accuracy and completeness.
F1-score value for the AE-2 model demonstrate the feasibility Fig. 11 illustrates the performance comparison between the
and stability of our solution. The traditional machine learning two different deep learning models. It can be seen from the
algorithms and autoencoder have higher F-scores due to the figure that AE-2 spends less training time compared to the
use of multiple fine-grained feature extraction methods, and CNN-0 model, and the ACC, recall, Precision, and F-score
the efficient feature extraction method is the key to determine metrics are all very similar. For FPR, AE-2 shows lower
the performance of the model. The lower F1-score shown in value.
recurrent neural network model is due to the fact that its fea- Table 5 shows all the experimental data. The training time
ture extraction method is complex and the model built using for our model is 1407.32 mins, which is about 23.45 hours.
RNN shows unstable performance on the dataset. The lower For the CNN-0 model, the number of parameters in the
F1-score in convolutional neural network model demonstated image data is huge and it takes a significant amount of time,
that this way of constructing feature images based on file 28.14 hours, after performing the convolution and pooling
binary codes contains more redundant information and less operations since the structure contains only one layer of
distinct features. convolution and one layer of pooling.

2) DETAILED COMPARISON OF MULTIPLE INDICATORS V. LIMINATION

After that, we use a variety of common machine learn- There are 2 main limitations of our model. The first point
ing models and deep learning models to conduct exper- is that the data pre-processing method needs to be improved.
iments comparing various evaluation metrics. The com- Although we did our best to reduce the redundant information
mon machine learning algorithms include support vector carried by the software feature representation, there is still the
machines, decision trees and naive bayes, and the deep learn- problem of inefficiency, and the dataset we used is smaller
ing model named CNN-0, which model consists of one layer due to the limitations of our experimental environment.
of convolution, one layer of pooling and one layer of fully The feasibility and effectiveness of this approach on other

25704 VOLUME 10, 2022

X. Xing et al.: Malware Detection Approach Using Autoencoder in Deep Learning

TABLE 4. Performance comparison of different models.

ACKNOWLEDGMENT
An earlier version of this paper was presented at the IEEE
MASS2020, Delhi, India, October 1-4, 2020 [DOI: 10.1109/
MASS50613.2020.00009], and expanded version of a paper
entitled ’A Malware Detection Approach Using Malware
Images and Autoencoders.’

REFERENCES
[1] (2019). China Internet Security Research Report. (Nov. 15, 2020).
[Online]. Available: https://www.cert.org.cn/publish/main/upload/
File/2019Annual%20report.pdf
FIGURE 11. Performance comparison of 2 deep learning models.
[2] Y. Ye, T. Li, D. Adjeroh, and S. S. Iyengar, ‘‘A survey on malware
detection using data mining techniques,’’ ACM Comput. Surv., vol. 50,
no. 3, pp. 1–40, May 2018.
TABLE 5. Detailed comparison of multiple indicators.
[3] S. Rastogi, K. Bhushan, and B. B. Gupta, ‘‘Android applications repack-
aging detection techniques for smartphone devices,’’ Proc. Comput. Sci.,
vol. 78, pp. 26–32, Jan. 2016.
[4] R. Pandita, X. Xiao, W. Yang, W. Enck, and T. Xie, ‘‘WHYPER:
Towards automating risk assessment of mobile applications,’’ in Proc. 22nd
USENIX Secur. Symp. (USENIX Security), 2013, pp. 527–542.
[5] W. Klieber, L. Flynn, A. Bhosale, L. Jia, and L. Bauer, ‘‘Android taint flow
types of malware dataset will require more detailed analy- analysis for app sets,’’ in Proc. 3rd ACM SIGPLAN Int. Workshop State Art
Java Program Anal. (SOAP), 2014, pp. 1–6.
sis and research in the future. The second point is that the [6] Z. Wang, J. Cai, S. Cheng, and W. Li, ‘‘DroidDeepLearner: Identifying
instability of detection performance caused by deep learn- Android malware using deep learning,’’ in Proc. IEEE 37th Sarnoff Symp.,
ing models is difficult to estimate. Although deep learning Sep. 2016, pp. 160–165, doi: 10.1109/SARNOF.2016.7846747.
[7] M. G. Schultz, E. Eskin, F. Zadok, and S. J. Stolfo, ‘‘Data mining methods
algorithms, such as convolutional neural networks, have a for detection of new malicious executables,’’ in Proc. IEEE Symp. Secur.
promising future in areas such as image recognition and text Privacy. (S&P), May 2001, p. 2001, doi: 10.1109/SECPRI.2001.924286.
generation, the use of malware feature data for classification [8] B. P. Sarma, N. Li, C. Gates, R. Potharaju, C. Nita-Rotaru, and I. Molloy,
tasks in deep learning models can lead to unstable detection ‘‘Android permissions: A perspective combining risks and benefits,’’ in
Proc. 17th ACM Symp. Access Control Models Technol. (SACMAT), 2012,
performance, because the models are very dependent on the pp. 13–22.
original training dataset, and the higher the accuracy, the [9] C. Zhao, W. Zheng, L. Gong, M. Zhang, and C. Wang, ‘‘Quick and accurate
greater the dependency, and the detection accuracy for soft- Android malware detection based on sensitive Apis,’’ in Proc. IEEE Int.
Conf. Smart Internet Things (SmartIoT), Aug. 2018, pp. 143–148.
ware samples that are not in the training set will be reduced. [10] H. Fereidooni, M. Conti, D. Yao, and A. Sperduti, ‘‘ANASTASIA:
ANdroid mAlware detection using STatic analySIs of applications,’’ in
VI. CONCLUSION Proc. 8th IFIP Int. Conf. New Technol., Mobility Secur. (NTMS), Nov. 2016,
pp. 1–5.
In this paper, we propose a novel approach to malware detec- [11] E. Mariconti, L. Onwuzurike, P. Andriotis, E. De Cristofaro, G. Ross,
tion, which is based on the principle of using grey-scale and G. Stringhini, ‘‘MaMaDroid: Detecting Android malware by building
images to represent the features of malware and using an Markov chains of behavioral models,’’ 2016, arXiv:1612.04433.
[12] W. Li, Z. Wang, J. Cai, and S. Cheng, ‘‘An Android malware detec-
auto-encoder network to design a classification model to
tion approach using weight-adjusted deep learning,’’ in Proc. Int.
achieve malware detection. Experimental results show the Conf. Comput., Netw. Commun. (ICNC), Mar. 2018, pp. 437–441, doi:
feasibility of our proposed approach of converting the byte- 10.1109/ICCNC.2018.8390391.
code of all methods in software into a greyscale image to [13] B. Amos, H. Turner, and J. White, ‘‘Applying machine learning classifiers
to dynamic Android malware detection at scale,’’ in Proc. 9th Int. Wireless
represent the features in a software sample. Compared to Commun. Mobile Comput. Conf. (IWCMC), Jul. 2013, pp. 1666–1671.
malware detection methods designed based on traditional [14] S. Nari and A. A. Ghorbani, ‘‘Automated malware classification based on
machine learning algorithms, our method is more accurate. network behavior,’’ in Proc. Int. Conf. Comput., Netw. Commun. (ICNC),
Jan. 2013, pp. 642–647.
Our method requires less training time and detection time
[15] G. Cabau, M. Buhu, and C. P. Oprisa, ‘‘Malware classification based on
compared to other malware detection systems designed based dynamic behavior,’’ in Proc. 18th Int. Symp. Symbolic Numeric Algorithms
on deep learning models. In future work, we will continue to Sci. Comput. (SYNASC), Sep. 2016, pp. 315–318.
explore more effective methods for representing malware fea- [16] W. Enck, P. Gilbert, S. Han, V. Tendulkar, B.-G. Chun, L. P. Cox, J. Jung,
P. McDaniel, and A. N. Sheth, ‘‘TaintDroid: An information-flow tracking
ture images and focus our research on the data pre-processing system for realtime privacy monitoring on smartphones,’’ ACM Trans.
stage to explore newer malware detection methods. Comput. Syst., vol. 32, no. 2, pp. 1–29, Jun. 2014.

VOLUME 10, 2022 25705

 X. Xing et al.: Malware Detection Approach Using Autoencoder in Deep Learning

[17] W.-C. Wu and S.-H. Hung, ‘‘DroidDolphin: A dynamic Android malware [37] L. Nataraj, S. Karthikeyan, G. Jacob, and B. S. Manjunath, ‘‘Malware
detection framework using big data and machine learning,’’ in Proc. Conf. images: Visualization and automatic classification,’’ in Proc. 8th Int. Symp.
Res. Adapt. Convergent Syst. (RACS), 2014, pp. 247–252. Visualization Cyber Secur., 2011, pp. 1–7.
[18] M. Egele, T. Scholte, E. Kirda, and C. Kruegel, ‘‘A survey on automated [38] J. Yan, Y. Qi, and Q. Rao, ‘‘Detecting malware with an ensemble
dynamic malware-analysis techniques and tools,’’ ACM Comput. Surv., method based on deep neural network,’’ Secur. Commun. Netw., vol. 2018,
vol. 44, no. 2, pp. 1–42, Feb. 2012. pp. 1–16, 2018.
[19] M. K. Alzaylaee, S. Y. Yerima, and S. Sezer, ‘‘DL-droid: Deep learning [39] K. He and D.-S. Kim, ‘‘Malware detection with malware images using
based Android malware detection using real devices,’’ Comput. Secur., deep learning techniques,’’ in Proc. 18th IEEE Int. Conf. Trust, Secur.
vol. 89, Feb. 2020, Art. no. 101663. Privacy Comput. Commun./13th IEEE Int. Conf. Big Data Sci. Eng. (Trust-
[20] S. Tobiyama, Y. Yamaguchi, H. Shimada, T. Ikuse, and T. Yagi, ‘‘Malware Com/BigDataSE), Aug. 2019, pp. 95–102.
[40] O. Aslan and A. A. Yilmaz, ‘‘A new malware classification frame-
detection with deep neural network using process behavior,’’ in Proc. IEEE
work based on deep learning algorithms,’’ IEEE Access, vol. 9,
40th Annu. Comput. Softw. Appl. Conf. (COMPSAC), vol. 2, Jun. 2016,
pp. 87936–87951, 2021.
pp. 577–582.
[41] M. Nisa, J. H. Shah, S. Kanwal, M. Raza, M. A. Khan, R. Damaše-
[21] D. Shi, X. Tang, and Z. Ye, ‘‘Detecting environment-sensitive vičius, and T. Blažauskas, ‘‘Hybrid malware classification method using
malware based on taint analysis,’’ in Proc. 8th IEEE Int. Conf. segmentation-based fractal texture analysis and deep convolution neural
Softw. Eng. Service Sci. (ICSESS), Nov. 2017, pp. 322–327, doi: network features,’’ Appl. Sci., vol. 10, no. 14, p. 4966, Jul. 2020, doi:
10.1109/ICSESS.2017.8342924. 10.3390/app10144966.
[22] W. Wang, Y. Li, X. Wang, J. Liu, and X. Zhang, ‘‘Detecting [42] J. Singh, D. Thakur, F. Ali, T. Gera, and K. S. Kwak, ‘‘Deep feature extrac-
Android malicious apps and categorizing benign apps with ensem- tion and classification of Android malware images,’’ Sensors, vol. 20,
ble of classifiers,’’ Future Gener. Comput. Syst., vol. 78, pp. 987–994, no. 24, p. 7013, Dec. 2020, doi: 10.3390/s20247013.
Jan. 2018. [43] Y. LeCun, Y. Bengio, and G. Hinton, ‘‘Deep learning,’’ Nature, vol. 521,
[23] A. Kumar, K. S. Kuppusamy, and G. Aghila, ‘‘A learning model to no. 7553, pp. 436–444, 2015.
detect maliciousness of portable executable using integrated feature set,’’ [44] Y. Bengio, ‘‘Learning deep architectures for AI,’’ Mach. Learn., vol. 2,
J. King Saud Univ. Comput. Inf. Sci., vol. 31, no. 2, pp. 252–265, no. 1, pp. 1–127, 2009.
Apr. 2019. [45] X. Yan, Y. Xu, X. Xing, B. Cui, Z. Guo, and T. Guo, ‘‘Trustworthy network
[24] N. Xie, F. Zeng, X. Qin, Y. Zhang, M. Zhou, and C. Lv, ‘‘Repass- anomaly detection based on an adaptive learning rate and momentum
Droid: Automatic detection of Android malware based on essential per- in IIoT,’’ IEEE Trans. Ind. Informat., vol. 16, no. 9, pp. 6182–6192,
missions and semantic features of sensitive Apis,’’ in Proc. Int. Symp. Sep. 2020.
Theor. Aspects Softw. Eng. (TASE), Guangzhou, China, Aug. 2018, [46] X. Li, W. Jiang, W. Chen, J. Wu, G. Wang, and K. Li, ‘‘Directional and
pp. 52–59. explainable serendipity recommendation,’’ in Proc. Web Conf., Apr. 2020,
pp. 122–132.
[25] S. Y. Yerima, S. Sezer, and G. McWilliams, ‘‘Analysis of Bayesian [47] H. Elahi, G. Wang, T. Peng, and J. Chen, ‘‘AI and its risks in Android
classification-based approaches for Android malware detection,’’ IET Inf. smartphones: A case of Google smart assistant,’’ in Dependability in Sen-
Secur., vol. 8, no. 1, pp. 25–36, Jan. 2014. sor, Cloud, and Big Data Systems and Applications, vol. 1123, G. Wang,
[26] Q. Li and X. Li, ‘‘Android malware detection based on static analysis of M. Z. A. Bhuiyan, S. D. C. di Vimercati, and Y. Ren, Eds. Guangzhou,
characteristic tree,’’ in Proc. Int. Conf. Cyber-Enabled Distrib. Comput. China: Springer, Nov. 2019, pp. 341–355.
Knowl. Discovery, Sep. 2015, pp. 84–91. [48] Y. Xu, G. Wang, J. Ren, and Y. Zhang, ‘‘An adaptive and configurable
[27] T. Kim, B. Kang, M. Rho, S. Sezer, and E. G. Im, ‘‘A multimodal deep protection framework against Android privilege escalation threats,’’ Future
learning method for Android malware detection using various features,’’ Gener. Comput. Syst., vol. 92, pp. 210–224, Mar. 2019.
IEEE Trans. Inf. Forensics Security, vol. 14, no. 3, pp. 773–788, Aug. 2019, [49] K. He, X. Zhang, S. Ren, and J. Sun, ‘‘Deep residual learning for image
doi: 10.1109/TIFS.2018.2866319. recognition,’’ in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR),
[28] Z. Yuan, Y. Lu, and Y. Xue, ‘‘Droiddetector: Android malware char- Jun. 2016, pp. 770–778.
acterization and detection using deep learning,’’ Tsinghua Sci. Tech- [50] R. Nix and J. Zhang, ‘‘Classification of Android apps and malware using
nol., vol. 21, no. 1, pp. 114–123, Feb. 2016, doi: 10.1109/TST.2016. deep neural networks,’’ in Proc. Int. Joint Conf. Neural Netw. (IJCNN),
7399288. May 2017, pp. 1871–1878.
[29] R. Pascanu, J. W. Stokes, H. Sanossian, M. Marinescu, and A. Thomas, [51] S. Shukla, G. Kolhe, S. M. Pd, and S. Rafatirad, ‘‘RNN-based classifier
‘‘Malware classification with recurrent networks,’’ in Proc. IEEE Int. to detect stealthy malware using localized features and complex symbolic
Conf. Acoust., Speech Signal Process. (ICASSP), South Brisbane, QLD, sequence,’’ in Proc. 18th IEEE Int. Conf. Mach. Learn. Appl. (ICMLA),
Australia, Apr. 2015, pp. 1916–1920. Dec. 2019, pp. 406–409, doi: 10.1109/ICMLA.2019.00076.
[52] Y. Chai, J. Qiu, S. Su, C. Zhu, L. Yin, and Z. Tian, ‘‘LGMal: A joint
[30] X. Wang and S. M. Yiu, ‘‘A multi-task learning model for malware clas- framework based on local and global features for malware detection,’’
sification with useful file access pattern from API call sequence,’’ 2016, in Proc. Int. Wireless Commun. Mobile Comput. (IWCMC), Jun. 2020,
arXiv:1610.05945. pp. 463–468, doi: 10.1109/IWCMC48107.2020.9148289.
[31] B. Athiwaratkun and J. W. Stokes, ‘‘Malware classification with LSTM [53] D. Zou, Y. Wu, S. Yang, A. Chauhan, W. Yang, J. Zhong, S. Dou, and H. Jin,
and GRU language models and a character-level CNN,’’ in Proc. IEEE Int. ‘‘IntDroid: Android malware detection based on API intimacy analysis,’’
Conf. Acoust., Speech Signal Process. (ICASSP), New Orleans, LA, USA, ACM Trans. Softw. Eng. Methodology, vol. 30, no. 3, pp. 1–32, May 2021,
Mar. 2017, pp. 2482–2486. doi: 10.1145/3442588.
[32] W. Hu and Y. Tan, ‘‘Black-box attacks against RNN based malware detec- [54] (Jan. 15, 2020). The Keras Blog. [Online]. Available:
tion algorithms,’’ in Proc. Workshops 32nd AAAI Conf. Artif. Intell. New https://blog.keras.io/building-autoencoders-in-keras.html
Orleans, LA, USA: AAAI Press, 2018, pp. 245–251. [55] A. Borghesi, A. Bartolini, M. Lombardi, M. Milano, and L. Benini,
[33] M. Abdelsalam, R. Krishnan, Y. Huang, and R. Sandhu, ‘‘Malware detec- ‘‘Anomaly detection using autoencoders in high performance computing
tion in cloud infrastructures using convolutional neural networks,’’ in Proc. systems,’’ in Proc. AAAI Conf. Artif. Intell., vol. 33, 2019, pp. 9428–9433.
IEEE 11th Int. Conf. Cloud Comput. (CLOUD), Jul. 2018, pp. 162–169, [56] G. D’Angelo, M. Ficco, and F. Palmieri, ‘‘Malware detection in mobile
doi: 10.1109/CLOUD.2018.00028. environments based on autoencoders and API-images,’’ J. Parallel Distrib.
[34] X. Xiao, ‘‘An image-inspired and CNN-based Android malware detection Comput., vol. 137, pp. 26–33, Mar. 2020.
[57] (Jan. 15, 2020). Google Play Store. [Online]. Available:
approach,’’ in Proc. 4th IEEE/ACM Int. Conf. Automated Softw. Eng.
https://developer.android.google.cn/distribute/google-play
(ASE), San Diego, CA, USA, Nov. 2019, pp. 1259–1261.
[58] (Jan. 15, 2020). Virusshare. [Online]. Available: http://virusshare.com/
[35] W. Wang, M. Zhao, and J. Wang, ‘‘Effective Android malware detection [59] (Jan. 15, 2020). Virustotal. [Online]. Available: https://www.
with a hybrid model based on deep autoencoder and convolutional neural virustotal.com/ko
network,’’ J. Ambient Intell. Hum. Comput., vol. 10, no. 8, pp. 3035–3043, [60] J.-W. Jang, H. Kang, J. Woo, A. Mohaisen, and H. K. Kim, ‘‘Andro-
Aug. 2019. dumpsys: Anti-malware system based on the similarity of malware creator
[36] T. H.-D. Huang and H.-Y. Kao, ‘‘R2-d2: ColoR-inspired convolutional and malware centric information,’’ Comput. Secur., vol. 58, pp. 125–138,
NeuRal network (CNN)-based AndroiD malware detections,’’ in Proc. May 2016.
IEEE Int. Conf. Big Data (Big Data), Dec. 2018, pp. 2633–2642.

25706 VOLUME 10, 2022