Efficient Clustering of Emails Into Spam and Ham: The Foundational Study of a Comprehensive Unsupervised Framework

Abstract

The spread and adoption of spam emails in malicious activities like information and identity theft, malware propagation, monetary and reputational damage etc. are on the rise with increased effectiveness and diversification. Without doubt these criminal acts endanger the privacy of many users and businesses'. Several research initiatives have taken place to address the issue with no complete solution until now; and we believe an intelligent and automated methodology should be the way forward to tackle the challenges. However, till date limited studies have been conducted on the applications of purely unsupervised frameworks and algorithms in tackling the problem. To explore and investigate the possibilities, we intend to propose an anti-spam framework that fully relies on unsupervised methodologies through a multi-algorithm clustering approach. This paper presents an in-depth analysis on the methodologies of the first component of the framework, examining only the domain and header related information found in email headers. A novel method of feature reduction using an ensemble of 'unsupervised' feature selection algorithms has also been investigated in this study. In addition, a comprehensive novel dataset of 100,000 records of ham and spam emails has been developed and used as the data source. Key findings are summarized as follows: I) out of six different clustering algorithms used -Spectral and K-means demonstrated acceptable performance while OPTICS projected the optimum clustering with an average of 3.5% better efficiency than Spectral and Kmeans, validated through a range of validations processes II) The other three algorithms-BIRCH, HDBSCAN and K-modes, did not fare well enough. III) The average balanced accuracy for the optimum three algorithms has been found to be ≈94.91%, and IV) The proposed feature reduction framework achieved its goal with high confidence.

An Unsupervised Approach for Content-Based Clustering of Emails Into Spam and Ham Through Multiangular Feature Formulation

“…The graph facilitates a connection to zi with another node or element zj, which belongs to k nearest neighbors of zi. The Weight Matrix W of T, defined using (3), illustrates the local structure of the data space [7]. , 7 ∈ <== >7 8 ?…”

“…. 3 c is a suitably chosen constant, and a graph Laplacian, L, (additional discussion in the 'Spectral Clustering' section), is constructed from W. For every feature r, LS -a Laplacian score, is then derived using (4), where D is the Diagonal Matrix of W. Features are then ranked accordingly [7].…”

“…Distinctive clusters that are present within the set of data points can be recognized through density based clustering (an unsupervised learning method). The algorithm assumes that clusters in a data space usually form a contiguous zone of high point density, which can be disengaged from other similar dense regions [38] [7]. DBSCAN initiates the process by calculating an approximation of 'density' and then subsequently pushes those data points that are placed in the low dense areas further away from each other, as well as from zones of high density.…”

“…Business Email Compromise (BEC) attacks resulted in a financial damage of around USD 3.5 billion in 2019, as reported by FBI [4]. Australian consumers and businesses, by the end of 2019, had lost over AUD 28 million due to email fraud [7]. An average person spends around 28% of a regular workweek interacting with emails [5].…”

Online Identity Theft, Security Issues, and Reputational Damage

“…[2] Some online identity theft schemes include email phishing attempts to lure victims into revealing personal information (e.g., passwords, addresses) and stealing driver's license, credit card, and checking account numbers by hacking into private websites. [3] It also includes unique biometric data (fingerprints, retina scans, facial geometry, iris images, and other unique physical representations), patient identification numbers, and medical records.…”