Analyzing very large time series using suffix arrays

Abstract

Suffix arrays form a powerful data structure for pattern detection and matching. In a previous work, we presented a novel algorithm (COV) which is the only algorithm that allows the detection of all repeated patterns in a time series by using the actual suffix array. However, the requirements for storing the actual suffix strings even on external media makes the use of suffix arrays impossible for very large time series. We have already proved that using the concept of Longest Expected Repeated Pattern (LERP) allows the actual suffices to be stored in linear capacity O(n) on external media. The repeated pattern detection using LERP has analogous time complexity, and thus makes the analysis of large time series feasible and limited only to the size Further, when empirical knowledge related to the distribution of repeated pattern's length is available, the proposed method (MLERP) can achieve better time performance compared to the standard LERP method and definitely much better than using any other pattern matching algorithm and applying brute force techniques which are unfeasible in logical (human) time frame. Thus, we may argue that MLERP is a very useful tool for detecting all repeated patterns in a time series regardless of its size and hardware limitations.

An efficient approach to mine flexible periodic patterns in time series databases

“…The protocol can be utilized in emerging cloud computing infrastructure where timestamp based privacy preserving data can be classified in a cloud among clients and server. In Xylogiannopoulos et al (2014), authors proposed COV method that utilizes suffix arrays, LERP method that allows actual suffixes stored in linear capacity of external media and MLERP methods that analyze very large time series by maximal utilization of available hardware.…”

A time series retrieval tool for sub-series matching

“…As regards perspective (i), the work in [40], for instance, introduces a tool for detecting all repeated patterns in a time series regardless of its size and hardware limitations. It focuses on the problem of repeated pattern detection in a single time series, and extends existing approaches to deal with larger data.…”

Early DDoS Detection Based on Data Mining Techniques

“…The method proposed in this paper is based on the Suffix Array data structure that is used to detect all repeated patterns in a sequence. More specifically, the ARPaD Algorithm [19] is used as it has been derived by COV Algorithm [15], [16], [17]. A Suffix Array is a data structure that contains an array of all suffixes of a string [20] and it is mainly used for pattern detection.…”

“…This is needed for the ARPaD Algorithm in order to perform the analysis as the strings have directly come from a Suffix Array data structure. This is the most time consuming part since it has complexity O(nlogn), while ARPaD Algorithm has been proven experimentally to have on average complexity O(n) [16], [18], [19]. Therefore, the total complexity of the method is on average O(nlogn) which allows a very fast analysis of the IP addresses data.…”

“…In this experiment, the total attack time per 500,000 hits is approximately 52 seconds, time considerably longer than the time ARPaD Algorithm needs to detect all repeated patterns. ARPaD Algo-rithm it has been proved to be linear on average [15], [16], [19] and that is why it pre-serves the ratio of 1/10 for the simple pattern detection and approximately 1/4 for the full pattern detection (including positions). Finally, we run one more experiment for the whole data set for the 1.5 million IPs ( Table 3).…”