Detection of DGA-based Malware Communications from DoH Traffic Using Machine Learning Analysis

Abstract

Encrypted domain name resolution can reduce the risk of privacy leakage for Internet users, but it may also prevent network administrators from detecting suspicious communications. Since operating systems supporting DNS over HTTPS (DoH) have increased in recent years, malware that uses Domain Generation Algorithm (DGA) can exploit it to hide the generated domain names. In this paper, we propose a system that detects DGA-based malware communications from DoH traffic. Based on the concept of hierarchical machine learning analysis, the proposed system classifies network traffic with Gradient Boosting Decision Tree (GBDT) and tree-ensemble models. The evaluation confirmed that the system was able to detect DoH traffic generated by PadCrypt, Sisron, Tinba, and Zloader with 99.12% accuracy. The results indicate that the system has the ability to detect different DGA-based malware communications from DoH traffic with sufficient accuracy to support network administrators.