Universal distances detect anomalies in big data

Abstract

The detection of anomalies in a set of data represents a ubiquitous challenge in computer science emerging in a wide range of fields such as financial transactions, particle physics and communication traffic. At the very heart of our method to attack this problem lies a representation of the data by key-value pairs with sets of associated identifiers. We compute distances between sets of key-value pairs with the help of a reference key and associated identifiers, and visualize these distances using heatmaps to bring out with few prior assumptions anomalies. Our method is universal since it works with any tabular or document-oriented data, independent of their content and form. We demonstrate the power of our technique by identifying three anomalies in the New York City Taxi and Limousine Commission Trip Record Data.