International Journal of Scientific Research in Engineering and Management (IJSREM)

Volume: 08 Issue: 06 | June - 2024 SJIF Rating: 8.448 ISSN: 2582-3930

A New Malware Classification Framework Based on Deep Learning

Algorithms

Dr. Manu Y M Hongirana D

Computer Science and Engineering, Computer Science and Engineering,
BGS Institute of Technology, BGS Institute of Technology,
BG Nagar, Karnataka BG Nagar, Karnataka
[email protected] [email protected]

Abstract — Recent advancements in computer malicious programs designed to infiltrate and compromise
technology have precipitated a shift towards virtual computer systems, often with malicious intent such as data
environments, accelerated by the COVID-19 pandemic. theft, system disruption, or financial gain. Over time,
Cybercriminals have capitalized on this trend, transitioning malware variants have evolved, employing sophisticated
their activities to exploit vulnerabilities in cyberspace. techniques such as obfuscation and packing to evade
Malicious software (malware) has emerged as a preferred tool traditional detection methods. As a result, the task of
for launching cyber-attacks, continually evolving with malware detection and classification has become
sophisticated obfuscation and packing techniques to evade
increasingly challenging, requiring innovative approaches
detection. Traditional machine learning (ML) algorithms, once
to effectively combat emerging threats. Traditional
effective in identifying malware, are now struggling to keep
pace with these advancements. In response, deep learning (DL) artificial intelligence (AI) techniques, particularly machine
algorithms offer a promising solution, leveraging their ability learning (ML) algorithms, have been instrumental in
to discern intricate patterns and correlations within data. This malware detection efforts. However, with the rapid
study proposes a novel hybrid deep-learning-based evolution of malware variants, these conventional
architecture, integrating two pre-trained network models to approaches are no longer as effective in accurately
enhance classification accuracy. Through extensive evaluation identifying and categorizing malicious software. In
on datasets including Malimg, Microsoft BIG 2015, and response to these challenges, deep learning (DL)
Malevis, the proposed method demonstrates significant
algorithms have emerged as a promising solution due to
improvements in accuracy, outperforming existing ML-based
malware detection methods in the literature. Specifically, the
their ability to autonomously learn intricate patterns and
proposed method achieves an impressive accuracy of 97.78% relationships within data. This project aims to address the
on the Malimg dataset, underscoring its effectiveness in shortcomings of traditional malware detection methods by
combating sophisticated malware variants. proposing a novel deep-learning-based framework for
malware classification. By leveraging the power of deep
Keywords — Malware, malware classification, malware neural networks and integrating multiple pre-trained
detection, malware variants, deep neural networks, transfer models, the proposed framework seeks to enhance the
learning, deep learning. accuracy and efficiency of malware classification. Through
rigorous evaluation on diverse datasets, including Malimg,
Microsoft BIG 2015, and Malevis, the effectiveness of the
I. INTRODUCTION
proposed approach will be demonstrated, offering a robust
The evolution of technology has fundamentally solution to the ever-evolving threat landscape of
transformed human interactions and activities, cybersecurity.
progressively shifting them into virtual domains. The onset
of the COVID-19 pandemic has further expedited this The proposed deep-learning-based framework
transition, as remote work, online communication, and represents a paradigm shift in malware detection and
digital transactions have become integral facets of daily classification, offering a comprehensive solution to combat
life. However, alongside these advancements, there has the increasingly sophisticated tactics employed by
been a parallel rise in cyber threats, with cybercriminals cybercriminals. By harnessing the capabilities of deep
exploiting the vulnerabilities inherent in virtual neural networks, the framework aims to not only accurately
environments. Central to their arsenal of tools is malicious identify known malware variants but also effectively detect
software (malware), which poses a significant threat to new and emerging threats.
cybersecurity. Malware encompasses a wide range of The project unfolds in four main stages, each

© 2024, IJSREM | www.ijsrem.com DOI: 10.55041/IJSREM35564 | Page 1

 International Journal of Scientific Research in Engineering and Management (IJSREM)
Volume: 08 Issue: 06 | June - 2024 SJIF Rating: 8.448 ISSN: 2582-3930

crucial to the success of the proposed framework. Firstly,

data acquisition involves gathering comprehensive datasets 2. Static Analysis Techniques
containing diverse samples of malware to train and
evaluate the deep neural network. Next, the design of the Static analysis techniques involve examining the
code or binary of a malware sample without executing it.
deep neural network architecture involves structuring the
Researchers have explored various static analysis approaches,
network to effectively process and analyze malware including signature-based detection, heuristic analysis, and
samples, leveraging insights from pre-trained models. structural analysis. Notably, Christodorescu et al. (2005)
Subsequently, the training phase entails fine-tuning proposed a static analysis framework called BITSHRED,
which analyzed binary code to identify common features and
the deep neural network using the acquired datasets,
similarities among malware samples. While static analysis
allowing it to learn and adapt to the intricacies of various techniques can provide valuable insights into malware
malware variants. Finally, the evaluation stage assesses the characteristics, they are often limited by their inability to
performance of the trained deep neural network on detect polymorphic and metamorphic malware variants.
independent test datasets, validating its efficacy in
accurately classifying malware. 3. Dynamic Analysis Techniques
Through experimentation on benchmark datasets Dynamic analysis techniques involve executing
such as Malimg, Microsoft BIG 2015, and Malevis, the malware samples in a controlled environment to observe their
proposed framework's performance will be benchmarked behavior. By monitoring system calls, network activity, and
against existing state-of-the-art methods. Key metrics file interactions, dynamic analysis can uncover malicious
including accuracy, precision, recall, and F1-score will be behavior indicative of malware. Many studies have explored
analyzed to quantify the framework's effectiveness in dynamic analysis approaches, including sandboxing,
emulation, and behavior-based detection. For instance, Baert
malware classification.
et al. (2015) developed a dynamic analysis framework called
Overall, this project aims to contribute to the Cuckoo Sandbox, which automated the execution and analysis
advancement of cybersecurity by introducing a novel deep- of malware samples in isolated environments. While dynamic
learning-based approach to malware detection and analysis techniques offer greater resilience against obfuscation
and evasion techniques, they can be resource-intensive and
classification. By leveraging the power of deep neural
prone to evasion by sophisticated malware.
networks, the proposed framework offers a robust and
scalable solution to mitigate the growing threats posed by 4. Hybrid Approaches
malware in today's digital landscape.
To overcome the limitations of static and dynamic
II. RELATED WORKS analysis techniques, researchers have proposed hybrid
approaches that combine the strengths of both methodologies.
Malware detection and classification have been Hybrid approaches often integrate static and dynamic analysis
longstanding challenges in the field of cybersecurity. Over the techniques to achieve comprehensive malware detection and
years, researchers have proposed various approaches to classification. For example, Kolter and Maloof (2006)
address these challenges, leveraging advancements in artificial introduced a hybrid approach called MAVMM, which
intelligence, machine learning, and deep learning. This section combined static analysis with machine learning-based
provides an overview of the related work in the domain of dynamic analysis to classify malware. By leveraging features
malware detection and classification, highlighting key extracted from both static and dynamic analysis, MAVMM
methodologies, techniques, and findings. achieved improved detection rates compared to standalone
approaches.
1. Traditional Machine Learning Approaches
5. Deep Learning-Based Approaches
Traditional machine learning (ML) techniques have
been widely employed for malware detection and In recent years, deep learning (DL) has emerged as a
classification. Early approaches focused on extracting features promising paradigm for malware detection and classification.
from malware samples and training classifiers to distinguish DL algorithms, such as convolutional neural networks (CNNs)
between malicious and benign software. One notable study by and recurrent neural networks (RNNs), have demonstrated
Yen et al. (2010) utilized features such as API calls, byte superior performance in various domains, including computer
sequences, and opcode frequencies to train support vector vision, natural language processing, and speech recognition.
machine (SVM) classifiers for malware detection. Despite Researchers have increasingly applied DL techniques to
achieving moderate success, these approaches struggled to malware detection, leveraging the ability of neural networks
keep pace with the rapid evolution of malware variants and the to automatically learn intricate patterns and features from raw
increasing sophistication of obfuscation techniques. data. For instance, Saxe et al. (2015) introduced a deep
learning approach called DEEPLEARNING-Malware, which
© 2024, IJSREM | www.ijsrem.com DOI: 10.55041/IJSREM35564 | Page 2
 International Journal of Scientific Research in Engineering and Management (IJSREM)
Volume: 08 Issue: 06 | June - 2024 SJIF Rating: 8.448 ISSN: 2582-3930

utilized convolutional neural networks to extract features from malware threats necessitates continuous monitoring and
binary code and classify malware samples. By training on adaptation of detection systems to mitigate emerging risks.
large-scale datasets, DEEPLEARNING-Malware achieved Researchers and practitioners must collaborate to address
state-of-the-art performance in malware detection, surpassing these challenges and develop robust, scalable solutions for
traditional ML-based approaches. real-world cybersecurity applications.

6. Transfer Learning and Pre-trained Models 10. Future Directions and Open Research Questions

Transfer learning, a technique that leverages Looking ahead, several avenues for future research in
knowledge gained from one domain to another, has gained malware detection and classification are worth exploring.
prominence in malware detection research. By fine-tuning pre- These include developing explainable AI techniques to
trained deep learning models on malware datasets, researchers enhance the interpretability and trustworthiness of malware
have achieved significant improvements in classification classifiers, exploring ensemble learning approaches to
accuracy. For example, Raff et al. (2017) utilized transfer combine the strengths of multiple classifiers, and investigating
learning with pre-trained convolutional neural networks to federated learning techniques for collaborative and privacy-
classify malware images extracted from executables. By preserving malware detection. Moreover, addressing the
adapting pre-trained models to the task of malware challenges posed by emerging threats such as fileless malware,
classification, Raff et al. achieved high accuracy rates while ransomware, and supply chain attacks will require innovative
reducing the need for extensive feature engineering and solutions and interdisciplinary collaborations across academia,
dataset labeling. industry, and government.

7. Adversarial Attacks and Robustness III. METHODOLOGY

Despite the success of deep learning-based The methodology for the proposed malware
approaches in malware detection, they remain vulnerable to classification framework based on deep learning algorithms
adversarial attacks, wherein attackers manipulate input data to comprises several integral stages aimed at developing an
deceive the classifier. Adversarial attacks can undermine the effective and robust system. Initially, the process involves
robustness and reliability of malware classifiers, leading to acquiring comprehensive datasets containing samples of
misclassifications and false positives. Researchers have malware, crucial for training and evaluating the deep learning
explored techniques to enhance the robustness of deep model. These datasets, such as Malimg, Microsoft BIG 2015,
learning models against adversarial attacks, including and Malevis, offer diverse representations of malware across
adversarial training, defensive distillation, and input various families and variants. With the datasets in hand, the
preprocessing. For instance, Grosse et al. (2017) introduced a next step focuses on designing the architecture of the deep
method called adversarial training, wherein the model is learning model. Here, a hybrid model architecture is proposed,
trained on adversarially perturbed samples to improve its integrating two prominent pre-trained network models:
resilience against attacks. ResNet-50 and AlexNet. Following architecture design, the
model undergoes extensive training using the acquired
8. Evaluation Metrics and Benchmark Datasets malware datasets. Leveraging transfer learning techniques, the
pre-trained network models are fine-tuned on the malware
Evaluating the performance of malware detection and datasets to learn discriminative features specific to malware
classification systems requires robust evaluation metrics and classification. Once trained, the performance of the deep
benchmark datasets. Common evaluation metrics include neural network is evaluated using independent test datasets.
accuracy, precision, recall, F1-score, and area under the Evaluation metrics such as accuracy, precision, recall, F1-
receiver operating characteristic curve (AUC-ROC). score, and AUC-ROC are computed to assess the model's
Researchers often benchmark their models on publicly efficacy in classifying malware accurately. Throughout the
available datasets, such as Malimg, Microsoft BIG 2015, and experimentation and results analysis phase, various
Malevis, to facilitate comparison and reproducibility. For experiments are conducted to analyze the performance of the
example, Zhang et al. (2018) evaluated their deep learning- framework under different configurations, aiming to identify
based malware classifier on the Malimg dataset, achieving the optimal settings that maximize classification accuracy
high accuracy and F1-score. while minimizing false positives and false negatives.

9. Real-World Deployments and Challenges IⅤ. IMPLEMENTATION

While research in malware detection and The implementation phase of the proposed malware
classification has made significant strides, deploying these
classification framework involves the practical execution of
systems in real-world environments presents numerous
challenges. Real-world deployments must contend with the outlined methodology, employing various tools and
factors such as scalability, interoperability, privacy concerns, techniques to develop a robust deep learning model. Python
and regulatory compliance. Moreover, the dynamic nature of was selected as the primary programming language due to
© 2024, IJSREM | www.ijsrem.com DOI: 10.55041/IJSREM35564 | Page 3
 International Journal of Scientific Research in Engineering and Management (IJSREM)
Volume: 08 Issue: 06 | June - 2024 SJIF Rating: 8.448 ISSN: 2582-3930

its versatility and extensive support for machine learning Fig. 2. Flow chart of the proposed work
libraries. Within the Python ecosystem, TensorFlow and
PyTorch emerged as the leading deep learning frameworks,
with PyTorch ultimately chosen for its flexibility and ease of
use. Leveraging PyTorch's capabilities, a hybrid model
architecture integrating ResNet-50 and AlexNet pre-trained
models was designed to extract features and classify
malware samples effectively.

A. Data Set

Fig. 1. Proposed malware classification methodology

The dataset utilized in this project encompasses behavioral
data derived from over 20,000 malware instances executed
within a controlled Cuckoo sandbox environment. This
environment ensured isolation and security while allowing for
the recording of Windows API calls made by the malware during
execution. The resulting behavioral data, capturing the sequence
of API calls, was stored in a MongoDB NoSQL database. Each
entry in the dataset represents a unique sequence of API calls,
amounting to a total of 342 distinct API calls recorded.
Furthermore, an additional public malware dataset comprising
7,101 records with eight different class labels was incorporated.
This dataset specifically focuses on Windows malware API call
sequences, executed within a Windows 7 environment. Each
record in this dataset includes a list of API call strings alongside
a corresponding class label, providing valuable insights for
training and evaluating the proposed malware classification C. Algorithm
framework. 1) Among the seven different malware, select the one to be
classified
B. Workflow 2) The dataset is examined for the particular kind of
The project workflow entails the classification of seven malware. The model labels other categories with label
distinct types of malware—Trojan, AdWare, Virus, BackDoor, 0 and the virus type information with label 1.
Downloader, Worms, and SpyWare—based on their API call 3) A Binary classification model using Decision Tree is
sequences. Initially, classical machine learning algorithms
defined and created.
including K-Nearest Neighbors, Decision Trees, and Support
Vector Machines (SVM) are employed for classification. 4) With 80% training data, the model is trained. The testing
Subsequently, the classification process is refined using deep step made use of the remaining 20% of the data. New
learning algorithms, specifically Long Short-Term Memory software’s API calls are put into the classifier during this
(LSTM) networks and Convolutional Neural Networks (CNN) process, and new instances are given a class label based
with LSTM layers. The accuracy of these deep learning on the trained model.
approaches is then compared against that achieved by classical
5) Results of testing and training are documented. The
machine learning algorithms. Figure 1 provides a visual
representation of the project's workflow, outlining the sequential performance metrics such as accuracy, precision, and
stages involved in data preprocessing, model training, and confusion matrix for each algorithm is computed.
evaluation to attain precise malware classification.

© 2024, IJSREM | www.ijsrem.com DOI: 10.55041/IJSREM35564 | Page 4

 International Journal of Scientific Research in Engineering and Management (IJSREM)
Volume: 08 Issue: 06 | June - 2024 SJIF Rating: 8.448 ISSN: 2582-3930

V. RESULTS AND DISCUSSIONS Additionally, the comparison with state-of-the-art

results provides valuable insights into the progress and
advancements in malware classification techniques. The
The evaluation metrics serve as vital indicators for consistent superiority of the proposed architecture reaffirms its
assessing the effectiveness of classification models, shedding status as a cutting-edge solution in the field, offering improved
light on their performance and efficiency. In this study, a range of accuracy and reliability compared to existing methodologies. This
evaluation metrics including accuracy, sensitivity, specificity, and not only highlights the efficacy of the proposed approach but also
F-score were utilized to gauge the performance of the proposed underscores the need for continual innovation and development in
methods. Figures 3, and 4 illustrate the metric values across the the field of cybersecurity.
Malimg, Microsoft BIG 2015, and Malevis datasets for the
AlexNet, ResNet-50 deep neural network models, and the
proposed models, respectively. Notably, the suggested method
consistently outperforms other deep neural network architectures,
demonstrating superior robustness and performance.

Fig. 4. Quantitative results on Malevis dataset.

VI. CONCLUSION AND FUTURE WORK

Fig. 3. Quantitative results on Microsoft BIG 2015 dataset. In conclusion, this project has presented a
comprehensive framework for classifying various types of
malware based on their API call sequences. Through the
Beyond the evaluation metrics and confusion matrices,
utilization of both classical machine learning and deep learning
it's essential to delve into the implications of the results obtained.
algorithms, the proposed approach has demonstrated promising
The superior performance of the proposed method across multiple
results in accurately categorizing malware samples into distinct
datasets suggests its potential as a robust solution for malware
classes. The classical machine learning models, including K-
classification tasks. By outperforming other deep neural network
Nearest Neighbors, Decision Trees, and Support Vector
architectures, the proposed model demonstrates its ability to
Machines, provided a solid foundation for initial classification,
effectively capture intricate patterns and features within malware
achieving respectable accuracy rates. However, the integration of
data, thereby enhancing classification accuracy and reliability.
deep learning algorithms, specifically Long Short-Term Memory
Furthermore, the observed differences in performance (LSTM) networks and Convolutional Neural Networks (CNN)
across malware variants underscore the importance of with LSTM layers, significantly improved classification
understanding the nuances of different malware types. While the accuracy, surpassing the performance of traditional machine
proposed method excelled in classifying most variants accurately, learning methods.
the variations in detection rates for specific types highlight
Furthermore, the comparative analysis between classical
potential areas for further optimization and refinement. Future
machine learning and deep learning algorithms highlighted the
research could focus on identifying the underlying factors
superior performance of deep learning approaches in handling
contributing to these performance differences and devising
complex patterns and features inherent in malware API call
strategies to mitigate them, thereby enhancing the model's overall
sequences. This underscores the potential of deep learning models
effectiveness.
to enhance malware detection and classification capabilities in
cybersecurity applications.
Overall, the field of malware classification using deep
learning holds immense potential for advancements in
cybersecurity, and further research in this area could contribute

© 2024, IJSREM | www.ijsrem.com DOI: 10.55041/IJSREM35564 | Page 5

 International Journal of Scientific Research in Engineering and Management (IJSREM)
Volume: 08 Issue: 06 | June - 2024 SJIF Rating: 8.448 ISSN: 2582-3930

significantly to enhancing malware detection and mitigation [15] Xin Ma, Shize Guo, Wei Bai, Jun Chen, Shiming Xia,
strategies in the future. Zhisong Pan, “An API Semantics-Aware Malware Detection
Method Based on Deep Learning”, Security and Communication
Networks, vol. 2019, Article ID 1315047, 9 pages, 2019.
https://doi.org/10.1155/2019/1315047
[16] Naveen, B., et al. "An Efficient Electronic Nasal Pod for Air
Pollutants Detection." 2023 International Conference on Recent
REFERENCES Advances in Science and Engineering Technology
(ICRASET). IEEE, 2023.
[17] Ferhat Ozgur Catak, Ahmet Faruk Yazı, Ogerta Elezaj, and
[1] Ucci, D., Aniello, L. and Baldoni, R., 2019. Survey of machine
Javed Ahmed. Deep learning based sequential model for malware
learning techniques for malware analysis. Computers Security,
analysis using windows exe api calls. PeerJ Computer Science,
81, pp.123-147.
6:e285, July 2020.
[2] LeCun, Y., Bengio, Y. and Hinton, G., 2015. Deep learning.
[18] Dhruthi, S., et al. "Litter Classification based on Convnet
nature, 521(7553), pp.436-444.
Artificial Neural Networks." (2023).
[3] "Anomaly Detection in Videos Using Deep Learning
[19] M. Schofield , G. Alicioglu , R. Bianco , P. Turner , C.
Techniques ", International Journal of Emerging Technologies
Thatcher , A. Lam, Bo Sun, ”Convolution Neural network for
and Innovative Research (www.jetir.org | UGC and issn
malware Classification based on API Call Sequence“, 8th
Approved), ISSN:2349-5162, Vol.8, Issue 6, page no. ppc582-
c588, June-2021, Available at: International Conference on Artificial Intelligence and
http://www.jetir.org/papers/JETIR2106349.pdf Applications (AIAP 2021), January 23 24, 2021, Zurich,
Switzerland.
[4] Zhang, Y. and Paxson, V., 2000. Detecting backdoors. In 9th
[20] SNEHA RAJ, N., et al. "A Machine Learning Approach to
USENIX Security Symposium (USENIX Security 00).
Predict Autism Spectrum Disorder." (2021).
[5] K. Shaukat, S. Luo, V. Varadharajan, I. A. Hameed and M.
[21] Gupta, S., Sharma, H. and Kaur, S., 2016, December.
Xu, “A Survey on Machine Learning Techniques for Cyber
Security in the Last Decade,” in IEEE Access, vol. 8, pp. 222310- Malware characterization using windows API call sequences. In
222354, 2020, doi: 10.1109/ACCESS.2020.3041951. International Conference on Security, Privacy, and Applied
Cryptography Engineering (pp. 271-280). Springer, Cham.
[6] Dankan Gowda, V., Swetha, K. R., Namitha, A. R., Manu, Y.
[22] Catak, F.O. and Yazı, A.F., 2019. A benchmark API call
M., Rashmi, G. R., & Veera Sivakumar, C. (2018). IOT Based
dataset for windows PE malware classification. arXiv preprint
Smart Health Care System to Monitor Covid-19 Patients.
arXiv:1905.01999.
[7] Mitchell, T.M. and Mitchell, T.M., 1997. Machine learning
[23] Kumar, K.S. and Mohanavalli, S., 2017, January. A
(Vol. 1, No. 9). New York: McGraw-hill.
performance comparison of document oriented NoSQL
[8] Manu, Y. M., G. K. Ravikumar, and S. V. Shashikala. databases. In 2017 International Conference on Computer,
"Anomaly Alert System using CCTV surveillance." 2022 IEEE Communication and Signal Processing (ICCCSP) (pp. 1-6).
2nd Mysore Sub Section International Conference IEEE.
(MysuruCon). IEEE, 2022.
[24] Peng, P., Yang, L., Song, L. and Wang, G., 2019, October.
[9] Berman, D.S.; Buczak, A.L.; Chavis, J.S.; Corbett, C.L. Opening the blackbox of virustotal: Analyzing online phishing
ASurvey of Deep Learning Methods for Cyber Security. scan engines. In Proceedings of the Internet Measurement
Information 2019, 10, 122. https://doi.org/10.3390/info10040122 Conference (pp. 478-485).
[10] Raksha, P. R., et al. "3* 3 Energy Production and [25] Arunkumar, A. S., Y. M. Manu, and G. K. Ravikumar.
Conversion." (2018). "Cyberbullying Detection Primarily based on Semantic Greater
[11] G. Mahajan, B. Saini and S. Anand, “Malware Classification Marginalized Denoising Automobile-Encoder." (2019).
Using Machine Learning Algorithms and Tools,” 2019 Second
International Conference on Advanced Computational and
Communication Paradigms (ICACCP), Gangtok, India, 2019, pp.
1-8, doi: 10.1109/ICACCP.2019.8882965.
[12] Manu, Y. M., and G. K. Ravikumar. "Survey on Machine
Learning Based Video Analytics Techniques." Journal of
Computational and Theoretical Nanoscience 17.11 (2020): 4989-
4995.
[13] D. Gavrilut, M. Cimpoes¸u, D. Anton and L. Ciortuz,
“Malware detection using machine learning”, Computer Science
and Information Technology 2009. IMCSIT’09. International
Multiconference on, pp. 735-741, October 2009.
[14] Raksha, P. R., et al. "Detection of Underground Water using
Sound Waves (A SURVEY)." (2019).
© 2024, IJSREM | www.ijsrem.com DOI: 10.55041/IJSREM35564 | Page 6