Fast window correlations over uncooperative time series

Cole, Richard; Shasha, Dennis; Zhao, Xiaojian

doi:10.1145/1081870.1081966

Cited by 57 publications

(57 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Their StatStream system can be used to detect pairs of financial time series with high correlation, among many available data streams. Cole, Shasha, and Zhao (2005) combine several techniques (random projections, grid structures, and others) in order to compute Pearson correlation coefficients between data streams. Other measures, such as dynamic time warping, have also been suggested Capitani and Ciaccia, 2005. Real-time feature selection can be complemented by feature extraction.…”

Section: Discussionmentioning

confidence: 99%

Flexible least squares for temporal data mining and statistical arbitrage

Montana

Triantafyllopoulos

Tsagaris

2009

Expert Systems with Applications

View full text Add to dashboard Cite

A number of recent emerging applications call for studying data streams, potentially infinite flows of information updated in realtime. When multiple co-evolving data streams are observed, an important task is to determine how these streams depend on each other, accounting for dynamic dependence patterns without imposing any restrictive probabilistic law governing this dependence. In this paper we argue that flexible least squares (FLS), a penalized version of ordinary least squares that accommodates for time-varying regression coefficients, can be deployed successfully in this context. Our motivating application is statistical arbitrage, an investment strategy that exploits patterns detected in financial data streams. We demonstrate that FLS is algebraically equivalent to the well-known Kalman filter equations, and take advantage of this equivalence to gain a better understanding of FLS and suggest a more efficient algorithm. Promising experimental results obtained from a FLS-based algorithmic trading system for the S&P 500 Futures Index are reported.

show abstract

Section: Discussionmentioning

confidence: 99%

Flexible least squares for temporal data mining and statistical arbitrage

Montana

Triantafyllopoulos

Tsagaris

2009

Expert Systems with Applications

View full text Add to dashboard Cite

show abstract

“…Such techniques are not suitable for our dynamic environment, where the index maintenance cost incurs high processing latency. Computing real-time correlations using a standalone machine has been a key focus of [6], [9], [18], however these techniques are ineffective in a distributed environment. The StatStream system [25] specializes in discovering correlations using a grid structure, but it incurs prohibitive communication cost in a distributed environment.…”

Section: Related Workmentioning

confidence: 99%

SigCO: Mining significant correlations via a distributed real-time computation engine

Guo

Calbimonte

Zhuang

et al. 2015

2015 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

Abstract-The dramatic rise of time-series data produced in a variety of contexts, such as stock markets, mobile sensing, sensor networks, data centre monitoring, etc., has fuelled the development of large-scale distributed real-time computation systems (e.g., Apache Storm, Samza, Spark Streaming, S4, etc.). However, it is still unclear how certain time series mining tasks could be performed using such new emerging systems. In this paper, we focus on the task of efficiently discovering statistically significant correlations among a large number of time series via a distributed realtime computation engine. We propose a framework referred to as SigCO. In SigCO, we put forward a novel partition-aware data shuffling, which is able to adaptively shuffle time series data only to the relevant nodes of the distributed real-time computation engine. On the other hand, in SigCO we design a δ-hypercube structure based correlation computation approach which is capable of pruning unnecessary correlation computations. Finally, our extensive experimental evaluations on real and synthetic datasets establish that SigCO outperforms the baseline approaches in terms of diverse performance metrics.

show abstract

“…Computing the Pearson's correlation coefficient using DFTbased techniques provides inaccurate results when the time series contain white [4] call such time series uncooperative and propose methods for discovering correlation amongst such signals. All these studies, however, typically only consider the correlation coefficient and do not propose an unified approach for computing and querying a wide variety of statistical measures, which includes the correlation coefficient.…”

Section: Related Workmentioning

confidence: 99%

“…Computing statistical measures for large databases of time series is a fundamental primitive for querying and mining time-series data [1][2][3][4][5][6]. This primitive is gaining importance with the increasing number and rapid growth of time series databases.…”

mentioning

confidence: 99%

AFFINITY: Efficiently querying statistical measures on time-series data

Sathe

Aberer

2013

2013 IEEE 29th International Conference on Data Engineering (ICDE)

View full text Add to dashboard Cite

Abstract. Computing statistical measures for large databases of time series is a fundamental primitive for querying and mining time-series data [1][2][3][4][5][6]. This primitive is gaining importance with the increasing number and rapid growth of time series databases. In this paper we introduce a framework for efficient computation of statistical measures by exploiting the concept of affine relationships. Affine relationships can be used to infer statistical measures for time series from other related time series instead of directly computing them; thus, reducing the overall computation cost significantly. The resulting methods show at least one order of magnitude improvement over the best known methods. To the best of our knowledge, this is the first work that presents an unified approach for computing and querying several statistical measures on time-series data. Our approach includes three key components, which exploit affine relationships. First, the AFCLST algorithm that clusters the time-series data such that high-quality affine relationships could be easily found. Second, the SYMEX algorithm that uses the clustered time series and efficiently computes the desired affine relationships. Third, the SCAPE index structure that produces a many-fold improvement in the performance of processing several statistical queries by seamlessly indexing the affine relationships. Finally, we establish the effectiveness of our approaches by performing comprehensive experimental evaluation using real datasets.

show abstract

Fast window correlations over uncooperative time series

Cited by 57 publications

References 31 publications

Flexible least squares for temporal data mining and statistical arbitrage

Flexible least squares for temporal data mining and statistical arbitrage

SigCO: Mining significant correlations via a distributed real-time computation engine

AFFINITY: Efficiently querying statistical measures on time-series data

Contact Info

Product

Resources

About