The Swiss army knife of time series data mining: ten useful things you can do with the matrix profile and ten lines of code

Zhu, Yada; Gharghabi, Shaghayegh; Silva, Diego Furtado; Dau, Son Hoang; Yeh, Chin‐Chia Michael; Senobari, Nader Shakibay; Almaslukh, Abdulaziz; Kamgar, Kaveh; Zimmerman, Zachary; Funning, Gareth J.; Mueen, Abdullah; Keogh, Eamonn

doi:10.1007/s10618-019-00668-6

Cited by 23 publications

(10 citation statements)

References 42 publications

(73 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The matrix profile is a data structure that annotates a time series (Yeh et al, 2018 ; Zhu et al, 2020 ). It allows for exact, simple, and fast (Zhu et al, 2017b ) similarity search or discord discovery and is among the state-of-the-art techniques in the field of discrete time-series analysis (Zhu et al, 2017a ; Madrid et al, 2019 ).…”

Section: Methodsmentioning

confidence: 99%

“…It allows for exact, simple, and fast (Zhu et al, 2017b ) similarity search or discord discovery and is among the state-of-the-art techniques in the field of discrete time-series analysis (Zhu et al, 2017a ; Madrid et al, 2019 ). The matrix profile has been used in processing biological signals like EEG (Mueen et al, 2009 ), ECG, and gait cycles (Zhu et al, 2020 ). It was applied in this study to accurately identify recurring waveforms in the LDV-acceleration data.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Template Matching and Matrix Profile for Signal Quality Assessment of Carotid and Femoral Laser Doppler Vibrometer Signals

Seoni

Beeckman

et al. 2022

Front. Physiol.

View full text Add to dashboard Cite

Background: Laser-Doppler Vibrometry (LDV) is a laser-based technique that allows measuring the motion of moving targets with high spatial and temporal resolution. To demonstrate its use for the measurement of carotid-femoral pulse wave velocity, a prototype system was employed in a clinical feasibility study. Data were acquired for analysis without prior quality control. Real-time application, however, will require a real-time assessment of signal quality. In this study, we (1) use template matching and matrix profile for assessing the quality of these previously acquired signals; (2) analyze the nature and achievable quality of acquired signals at the carotid and femoral measuring site; (3) explore models for automated classification of signal quality.Methods: Laser-Doppler Vibrometry data were acquired in 100 subjects (50M/50F) and consisted of 4–5 sequences of 20-s recordings of skin displacement, differentiated two times to yield acceleration. Each recording consisted of data from 12 laser beams, yielding 410 carotid-femoral and 407 carotid-carotid recordings. Data quality was visually assessed on a 1–5 scale, and a subset of best quality data was used to construct an acceleration template for both measuring sites. The time-varying cross-correlation of the acceleration signals with the template was computed. A quality metric constructed on several features of this template matching was derived. Next, the matrix-profile technique was applied to identify recurring features in the measured time series and derived a similar quality metric. The statistical distribution of the metrics, and their correlates with basic clinical data were assessed. Finally, logistic-regression-based classifiers were developed and their ability to automatically classify LDV-signal quality was assessed.Results: Automated quality metrics correlated well with visual scores. Signal quality was negatively correlated with BMI for femoral recordings but not for carotid recordings. Logistic regression models based on both methods yielded an accuracy of minimally 80% for our carotid and femoral recording data, reaching 87% for the femoral data.Conclusion: Both template matching and matrix profile were found suitable methods for automated grading of LDV signal quality and were able to generate a quality metric that was on par with the signal quality assessment of the expert. The classifiers, developed with both quality metrics, showed their potential for future real-time implementation.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Template Matching and Matrix Profile for Signal Quality Assessment of Carotid and Femoral Laser Doppler Vibrometer Signals

Seoni

Beeckman

et al. 2022

Front. Physiol.

View full text Add to dashboard Cite

show abstract

“…Patternbased time series methods handle large scale time series well with excellent performance and interpretation. Recently, Matrix Profile (MP) [27,30] is proposed as an efficient and effective data representation on subsequence level and support most major fundamental tasks for downstream applications in a broad range of domains applications [27,30,28]. Existing work such as [31] typically use the raw data to compute MP as the first, feature generation step, and then design algorithms or models based on the computed MP.…”

Section: Related Workmentioning

confidence: 99%

PMP: Privacy-Aware Matrix Profile against Sensitive Pattern Inference for Time Series

Zhang¹,

Ding²,

Gao³

et al. 2023

Preprint

View full text Add to dashboard Cite

Recent rapid development of sensor technology has allowed massive fine-grained time series data to be collected and set foundation for the development of data-driven services and applications. During the process, data sharing is often involved to allow the third-party modelers to perform specific time series data mining tasks based on the need of data owner. The high resolution of time series data brings new challenges in protecting privacy. On one hand, meaningful information in high-resolution time series shifts from concrete point values to local shape-based segments. On the other hand, numerous research have found that long shape-based patterns could contain more sensitive information and may potentially be extracted and misused by a malicious third party. However, the privacy issue for time series patterns is surprisingly seldom explored in privacy-preserving literature. In this work, we consider a new privacy preserving problem: preventing malicious inference on long shape-based patterns while preserving short segment information for the utility task performance. To mitigate the challenge, we investigate an alternative approach by sharing Matrix Profile (MP), which is a nonlinear transformation of original data and a versatile data structure that supports many data mining tasks. We found that while MP can prevent the concrete shape leakage, the canonical correlation in MP index can still reveal the location of sensitive long pattern information. Based on this observation, we design two attacks named Location Attack and Entropy Attack to extract the pattern location from MP. To further protect MP from these two attacks, we propose a Privacy-Aware Matrix Profile (PMP) via perturbing the local correlation and breaking the canonical correlation in MP index vector. We evaluate our proposed PMP against baseline noise-adding methods through quantitative analysis and real-world case study to show the effectiveness of the proposed method. Our source code is available at https://github.com/lzhang18/PMP.

show abstract

“…Applications of the MP for time series data mining have already generated many insights [ 11 ]. One study discovered motifs using MP in retail product sales time series and used them to analyse the temporary sales correlations among products thus indicating that customers’ product preferences are not stable and change with time [ 12 ].…”

Section: Introductionmentioning

confidence: 99%

Calculating the matrix profile from noisy data

Hehir,

Smeaton

2023

PLoS ONE

View full text Add to dashboard Cite

The matrix profile (MP) is a data structure computed from a time series which encodes the data required to locate motifs and discords, corresponding to recurring patterns and outliers respectively. When the time series contains noisy data then the conventional approach is to pre-filter it in order to remove noise but this cannot apply in unsupervised settings where patterns and outliers are not annotated. The resilience of the algorithm used to generate the MP when faced with noisy data remains unknown. We measure the similarities between the MP from original time series data with MPs generated from the same data with noisy data added under a range of parameter settings including adding duplicates and adding irrelevant data. We use three real world data sets drawn from diverse domains for these experiments Based on dissimilarities between the MPs, our results suggest that MP generation is resilient to a small amount of noise being introduced into the data but as the amount of noise increases this reslience disappears.

show abstract

The Swiss army knife of time series data mining: ten useful things you can do with the matrix profile and ten lines of code

Cited by 23 publications

References 42 publications

Template Matching and Matrix Profile for Signal Quality Assessment of Carotid and Femoral Laser Doppler Vibrometer Signals

Template Matching and Matrix Profile for Signal Quality Assessment of Carotid and Femoral Laser Doppler Vibrometer Signals

PMP: Privacy-Aware Matrix Profile against Sensitive Pattern Inference for Time Series

Calculating the matrix profile from noisy data

Contact Info

Product

Resources

About