Detecting Obfuscated Malware Infections on Windows Using Ensemble Learning Techniques

Abstract

In the internet and smart devices era, malware detection has become crucial for system security. Obfuscated malware poses significant risks to various platforms, including computers, mobile devices, and IoT devices, by evading advanced security solutions. Traditional heuristic-based and signature-based methods often fail against these threats. Therefore, a cost-effective detection system was proposed using memory dump analysis and ensemble learning techniques. Utilizing the CIC-MalMem-2022 dataset, the effectiveness of decision trees, gradient-boosted trees, logistic Regression, random forest, and LightGBM in identifying obfuscated malware was evaluated. The study demonstrated the superiority of ensemble learning techniques in enhancing detection accuracy and robustness. Additionally, SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-agnostic Explanations) were employed to elucidate model predictions, improving transparency and trustworthiness. The analysis revealed vital features significantly impacting malware detection, such as process services, active services, file handles, registry keys, and callback functions. These insights are crucial for refining detection strategies and enhancing model performance. The findings contribute to cybersecurity efforts by comprehensively assessing machine learning algorithms for obfuscated malware detection through memory analysis. This paper offers valuable insights for future research and advancements in malware detection, paving the way for more robust and effective cybersecurity solutions in the face of evolving and sophisticated malware threats. © 2025 St. Petersburg Federal Research Center of the Russian Academy of Sciences. All rights reserved.