Data smashing: uncovering lurking order in data

Abstract

From automatic speech recognition to discovering unusual stars, underlying almost all automated discovery tasks is the ability to compare and contrast data streams with each other, to identify connections and spot outliers. Despite the prevalence of data, however, automated methods are not keeping pace. A key bottleneck is that most data comparison algorithms today rely on a human expert to specify what 'features' of the data are relevant for comparison. Here, we propose a new principle for estimating the similarity between the sources of arbitrary data streams, using neither domain knowledge nor learning. We demonstrate the application of this principle to the analysis of data from a number of real-world challenging problems, including the disambiguation of electro-encephalograph patterns pertaining to epileptic seizures, detection of anomalous cardiac activity from heart sound recordings and classification of astronomical objects from raw photometry. In all these cases and without access to any domain knowledge, we demonstrate performance on a par with the accuracy achieved by specialized algorithms and heuristics devised by domain experts. We suggest that data smashing principles may open the door to understanding increasingly complex observations, especially when experts do not know what to look for.

Vehicle classification using road side sensors and feature-free data smashing approach

“…It was recently proposed by Chattopadhyay and Lipson in [12], [13]. The main goal of the data smashing process is to determine whether two compared data streams were produced by the same source (i.e.…”

“…Due to the space limitations, this section gives only a highlevel overview and the algorithmic steps involved in data smashing. Readers interested in the theoretical foundation and proofs are referred to [12] for the detailed description. Note that the method does not attempt to explicitly reconstruct the probabilistic automata from the data streams.…”

“…Authors in [12], [13] presented results of using data smashing for several machine learning problems from varied domains where the raw data can be represented as a stream. In particular, the method was used to solve two typical problems: clustering (unsupervised learning) and classification (supervised learning).…”

“…Stream inversion in turn requires generation of independent stream copies from the quantized stream, s. The algorithmic procedure for the generation of the independent stream copy uses streams generated by flat white noise (FWN). According to [12] for the alphabet Σ, FWN generates the current symbol of a stream from the uniform distribution, i.e. the probability of appearance of each alphabet's symbol is 1/|Σ|.…”

“…The main contribution of this study is the use of the recently proposed data mining method called data smashing [12] to the vehicle classification problem. The advantage of this classification approach is that it, in contrast to the most of traditional machine learning algorithms, does not require the extraction of features from raw signals.…”