Adversarial Auto encoder Framework for Zero-Day Malware Detection Using Dynamic Behavior Graph Representations

Abstract

The rapid evolution of zero-day malware poses a significant threat to modern cybersecurity infrastructures, as these threats often evade signature-based detection systems by exploiting unknown vulnerabilities or obfuscating code. Traditional approaches lack the adaptability required to detect such previously unseen malware in real-time. This study proposes a robust detection framework that leverages adversarial auto encoders (AAEs) and dynamic behaviour graph representations to identify zero-day malware with high precision and low false positives. The framework constructs dynamic behaviour graphs from runtime system activities such as file access, registry changes, and process creation, captured in sandboxed environments. These graphs are embedded into fixed-size vectors and processed by an adversarial auto encoder trained to differentiate benign and anomalous behaviour based on reconstruction error and latent space regularity. Experiments were conducted using the CIC-MalMem-2022 dataset, comprising over 3,000 samples from multiple malware families and benign software. The proposed AAE framework achieved a detection accuracy of 96.47%, precision of 94.80%, recall of 93.52%, and an AUC-ROC of 97.10%, outperforming baseline models including CNN, Random Forest, and One-Class SVM. It also demonstrated a significantly lower false positive rate (1.42%) compared to other techniques. This research presents a scalable and generalizable malware detection approach suitable for real-world deployment in enterprise and critical infrastructure environments. By combining structural behavioural modeling with adversarial training, the framework provides a proactive solution to the increasing challenge of zero-day malware.