Dynamic Malware Detection Approach Based on API Calls: Machine Learning and Ensemble Learning Models

Ulu A., KARAKAYA A.

INTERNATIONAL JOURNAL OF INFORMATION SECURITY SCIENCE, cilt.13, sa.4, ss.1-20, 2024 (TRDizin)

Özet

The rapid evolution of malware presents significant challenges in cybersecurity. The malware can be detected by using static, dynamic, or hybrid features. Static features are effective for the detection of malware but they are unable to prevent code obfuscation approaches. On the other hand, dynamic features contain code or characters of malware that are obtained while the software is operating on a device. In comparison to static analysis, dynamic analysis is a better method especially to detect the malware containing code obfuscation. The dynamic datasets, contain API calls and permissions, enabling real-time monitoring of malware behavior. Since these datasets are obtained through behavior analysis during execution, methods using dynamic datasets offer a more realistic approach compared to those using static datasets. This study investigates the efficacy of various machine learning and ensemble learning models for malware detection using dynamic analysis. For this purpose, it is used the VirusSample and VirusShare datasets, which consist of API calls and permissions. For both datasets, the Random Forest (RF) model used Principal Component Analysis (PCA) for feature extraction achieved the best results among the machine learning models, with accuracies of 94.83% and 86.27%, respectively. For the VirusSample dataset, the stacking ensemble learning model, which uses RF and Decision Trees (DT) as base classifiers and K-Nearest Neighbors (k-NN) as the meta classifier, achieved the highest accuracy of 94.56% thanks to the use of PCA. In contrast, for the VirusShare dataset, the stacking ensemble learning model, which uses RF, k-NN, and Gradient Boosting (GB) as base classifiers and support vector machine (SVM) as the meta classifier, achieved the highest accuracy of 86.21% with PCA usage for feature extraction. These results underscore the superiority of dynamic analysis and the effectiveness of ensemble methods in enhancing malware detection accuracy. This study contributes to the optimization of machine learning models and the advancement of cybersecurity solutions.