Attention-Based Autoencoder for Anomaly Detection in Privacy-Preserving DNS Traffic

Abstract

The rapid adoption of privacy-preserving DNS protocols, such as DNS over HTTPS (DoH) and DNS over TLS (DoT), has improved the confidentiality of Internet communications by encrypting DNS queries and responses. Although these protocols strengthen user privacy, they also create major challenges for conventional intrusion detection systems, which depend heavily on payload inspection and manually labeled traffic data for identifying malicious activities. As encrypted DNS traffic continues to grow, there is an increasing need for intelligent detection mechanisms capable of identifying anomalies without relying on decrypted content. This study proposes an attention-based autoencoder framework for anomaly detection in encrypted DNS traffic using a self-supervised learning strategy. The proposed model is trained exclusively on benign DNS flows, allowing it to learn normal traffic behavior without requiring labeled attack samples. A Transformer-based autoencoder architecture is employed to capture temporal relationships within DNS flow sequences and reconstruct input patterns from flow-level metadata features. Anomalous behavior is identified through reconstruction errors generated during the decoding process, where higher deviations indicate suspicious traffic patterns. Experimental evaluation is conducted using the ISCX2021 encrypted DNS dataset containing both benign and malicious DNS flows. The proposed framework achieves an accuracy of 93.1%, precision of 91.5%, recall of 89.8%, and an F1-score of 90.6%, outperforming baseline models including PCA, Isolation Forest, and LSTM Autoencoders. In addition, the model attains an AUC score of 0.92 with an average inference latency of 43 ms per DNS flow, demonstrating its suitability for near real-time deployment. The proposed framework provides a scalable and privacy-preserving solution for detecting anomalies in encrypted DNS environments without requiring payload access or extensive manual labeling. Its lightweight and adaptive design makes it suitable for practical deployment in enterprise DNS infrastructures, ISP-level monitoring systems, and edge-based cybersecurity applications