Behavior Profiling of Email

Abstract

This paper describes the forensic and intelligence analysis capabilities of the Email Mining Toolkit (EMT) under development at the Columbia Intrusion Detection (IDS) Lab. EMT provides the means of loading, parsing and analyzing email logs, including content, in a wide range of formats. Many tools and techniques have been available from the fields of Information Retrieval (IR) and Natural Language Processing (NLP) for analyzing documents of various sorts, including emails. EMT, however, extends these kinds of analyses with an entirely new set of analyses that model "user behavior". EMT thus models the behavior of individual user email accounts, or groups of accounts, including the "social cliques" revealed by a user's email behavior. The application of this technology to diverse Internet objects and events (e.g., email and web transactions) allows for a broad range of behavior-based analyses including the detection of proxy email accounts and groups of user accounts that communicate with one another including covert group activities. Data mining applies machine learning and statistical techniques to automatically discover and detect misuse patterns, as well as anomalous activities in general. When applied to network-based activities and user account observations for the detection of errant or misuse behavior, these methods are referred to as behavior-based misuse detection. Behavior-based misuse detection can provide important new assistance for counter-terrorism intelligence. In addition to standard Internet misuse detection, these techniques will automatically detect certain patterns across user accounts that are indicative of covert, malicious or counter-intelligence activities. Moreover, behavior-based detection provides workbench functionalities to interactively assist an intelligence agent with targeted investigations and off-line forensics analyses. Intelligence officers have a myriad of tasks and problems confronting them each day. The sheer volume of source materials requires a means of honing in on those sources of maximal value to their mission. A variety of techniques can be applied drawing upon the research and technology developed in the field of Information Retrieval. There is, however, an additional source of information available that can used to aid even the simplest task of rank ordering and sorting documents for inspection: behavior models associated with the documents can be used to identify and group sources in interesting new ways. This is demonstrated by the Email Mining Toolkit that applies a variety of data mining techniques for profiling and behavior modeling of email sources. The deployment of behavior-based techniques for intelligence investigation and tracking tasks represents a significant qualitative step in the counter-intelligence "arms race". Because there is no way to predict what data mining will discover over any given data set, "counter-escalation" is particularly difficult. Behavior-based misuse detection is more robust against standard knowledgebased techniques. Behavior-b...

That Ain’t You: Blocking Spearphishing Through Behavioral Modelling

“…Stolfo et al presented Email Mining Toolkit (EMT) [29,30]. This tool mines email logs to find cliques of users who frequently contact each other.…”

“…Techniques that leverage the similarities in the email templates or fingerprint the email lists to which bots send emails [24,32,43] work well in detecting spam and bot-infected machines, but fall short in detecting one-of-a-kind targeted email attacks, in which an attacker crafts an email tailored to the victim, and sends it only once. Even systems that look for changes of behavior in email accounts leverage the fact that accounts compromised by the same cybercriminals will show a similar behavior [10,29,30].…”

Internet Traffic Behavior Profiling for Network Security Monitoring

“…The studies in [28], [29] focus on communication patterns or profiles of applications instead of broader network traffic. Concurrent with our work, [30], [31] are most similar in spirit, and in a sense are complementary, to ours.…”

Achieving Team-Awareness in Scientific Grid Environments

“…The Email Mining Toolkit (EMT) [13] provides interaction metrics while [4] aims at extracting the underlying activity structure. However all of these lack the notion of applied resources (e.g.…”