Discovering longest-lasting correlation in sequence databases

Abstract

Most existing work on sequence databases use correlation (e.g., Euclidean distance and Pearson correlation) as a core function for various analytical tasks. Typically, it requires users to set a length for the similarity queries. However, there is no steady way to define the proper length on different application needs. In this work we focus on discovering longest-lasting highly correlated subsequences in sequence databases, which is particularly useful in helping those analyses without prior knowledge about the query length. Surprisingly, there has been limited work on this problem. A baseline solution is to calculate the correlations for every possible subsequence combination. Obviously, the brute force solution is not scalable for large datasets. In this work we study a space-constrained index that gives a tight correlation bound for subsequences of similar length and offset by intra-object grouping and inter-object grouping techniques. To the best of our knowledge, this is the first index to support normalized distance metric of arbitrary length subsequences. Extensive experimental evaluation on both real and synthetic sequence datasets verifies the efficiency and effectiveness of our proposed methods.

SigCO: Mining significant correlations via a distributed real-time computation engine

“…Various indexing techniques for querying the correlations of static time-series data stored in a centralized system have been proposed in [11], [12], [20], [23]. Such techniques are not suitable for our dynamic environment, where the index maintenance cost incurs high processing latency.…”

“…GC: it is based on distributed group-based join [15], which optimizes the sliding window replication and enables incremental correlation computing [11]. GC computes pair-wise correlations and then performs significance tests.…”

Efficient Query Processing in Time Series

“…To reasonably compare the similarity of two time series, the sequence values should be Znormalized [15,26]. The Z-normalization is to transform a time series into its normalized from whose mean is approximately zero, and the standard deviation is in a range close to 1.…”

“…As an example, the normalized Euclidean distance of two longer subsequences is more likely larger than the normalized Euclidean distance of two shorter subsequences. Thus in longest-lasting correlation query, we use Pearson correlation as the underline similarity measure since it not only reveals the true similarity of time series by Z-normalization but also makes the similarity comparison fairer by length normalization [15]. For clarity, the definition of Pearson correlation is given as follows.…”

“…Request permissions from permissions@acm.org. on discovering longest-lasting correlated subsequence [15] and time series motif [16,20,24].…”