Evaluating Subjective Accuracy in Time Series Pattern-Matching Using Human-Annotated Rankings

Eichmann, Philipp; Zgraggen, Emanuel

doi:10.1145/2678025.2701379

Cited by 20 publications

(18 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Apart from the query interface, Keogh and Pazzani [36,37] show that the notion of similarity can heavily depend on the user's mental model and might not be captured well with an objective similarity measure. Eichmann and Zgraggen [17] extend this line of work and show that the objective similarity can differ markedly from the perceived similarity, even for simple patterns.…”

Section: Related Workmentioning

confidence: 70%

“…Using the query, the system retrieves regions that are most similar given some notion of similarity. But the search can fail when the analyst's subjectively perceived similarity does not match the system's similarity measure [17]. The larger the query region, the more likely it is that the query contains several distinct visual features, such as peaks, trends, or troughs, which can be hard to capture with current techniques (Figure 2).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Peax Interactive Visual Pattern Search in Sequential Data Using Unsupervised Deep Representation Learning

Lekschas

Peterson

Haehn

et al. 2019

Preprint

View full text Add to dashboard Cite

We present PEAX, a novel feature-based technique for interactive visual pattern search in sequential data, like time series or data mapped to a genome sequence. Visually searching for patterns by similarity is often challenging because of the large search space, the visual complexity of patterns, and the user's perception of similarity. For example, in genomics, researchers try to link patterns in multivariate sequential data to cellular or pathogenic processes, but a lack of ground truth and high variance makes automatic pattern detection unreliable. We have developed a convolutional autoencoder for unsupervised representation learning of regions in sequential data that can capture more visual details of complex patterns compared to existing similarity measures. Using this learned representation as features of the sequential data, our accompanying visual query system enables interactive feedback-driven adjustments of the pattern search to adapt to the users' perceived similarity. Using an active learning sampling strategy, PEAX collects user-generated binary relevance feedback. This feedback is used to train a model for binary classification, to ultimately find other regions that exhibit patterns similar to the search target. We demonstrate PEAX's features through a case study in genomics and report on a user study with eight domain experts to assess the usability and usefulness of PEAX. Moreover, we evaluate the effectiveness of the learned feature representation for visual similarity search in two additional user studies. We find that our models retrieve significantly more similar patterns than other commonly used techniques.

show abstract

Section: Related Workmentioning

confidence: 70%

Section: Introductionmentioning

confidence: 99%

Peax Interactive Visual Pattern Search in Sequential Data Using Unsupervised Deep Representation Learning

Lekschas

Peterson

Haehn

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…Future Work: The primary direction will be researching on similarity metrics that correlate with the trained human perception in order to better quantify the assessment of adversarial ECG examples. This is per the fact that qualitative classifications like ECG data are highly reliant on particular patterns and features the curved lines present [13,15], a property that conventional metrics such as distance cannot entirely capture. A secondary direction is the development of methods that can detect these adversarial ECG inputs, as the means of defending against the application of low-pass filters that improve the attack's perceptual smoothness and similarity [20,30] is an area still largely unexplored.…”

Section: Discussion and Future Workmentioning

confidence: 99%

Hard-Label Black-Box Adversarial Attack on Deep Electrocardiogram Classifier

Lam

Quan

et al. 2020

Proceedings of the 1st ACM International Workshop on Security and Safety for Intelligent Cyber-Physical Systems

View full text Add to dashboard Cite

Through aiding the process of diagnosing cardiovascular diseases (CVD) such as arrhythmia, electrocardiograms (ECGs) have progressively improved prospects for an automated diagnosis system in modern healthcare. Recent years have seen the promising applications of deep neural networks (DNNs) in analyzing ECG data, even outperforming cardiovascular experts in identifying certain rhythm irregularities. However, DNNs have shown to be susceptible to adversarial attacks, which intentionally compromise the models by adding perturbations to the inputs. This concept is also applicable to DNN-based ECG classifiers and the prior works generate these adversarial attacks in a white-box setting where the model details are exposed to the attackers. However, the black-box condition, where the classification model's architecture and parameters are unknown to the attackers, remains mostly unexplored. Thus, we aim to fool ECG classifiers in the black-box and hard-label setting where given an input, only the final predicted category is visible to the attacker. Our attack on the DNN classification model for the PhysioNet Computing in Cardiology Challenge 2017 [12] database produced ECG data sets mostly indistinguishable from the whitebox version of an adversarial attack on this same database. Our results demonstrate that we can effectively generate the adversarial ECG inputs in this black-box setting, which raises significant concerns regarding the potential applications of DNN-based ECG classifiers in security-critical systems.

show abstract

“…A user can select several data points and change the amplitude of the selected subsequence. Similar to TimeSketch [7], users can also select a subsequence in the Time Series Viewer and replace it with a hand-drawn sequence (Figure 2K). These actions cause Metro-Viz to re-apply all selected anomaly detectors.…”

Section: Metro-viz Client: User Interfacementioning

confidence: 99%

Visual Exploration of Time Series Anomalies with Metro-Viz

Eichmann

Solleza

Tatbul

et al. 2019

Proceedings of the 2019 International Conference on Management of Data

Self Cite

View full text Add to dashboard Cite

This demo presents a novel data visualization solution for exploring the results of time series anomaly detection systems. When anomalies are reported, there is a need to reason about the results. We introduce Metro-Viz -a visual tool to assist data scientists in performing this analysis. Metro-Viz offers a rich set of interaction features (e.g., comparative analysis, what-if testing) backed by data management strategies specifically tailored to the workload. We show our tool in action via multiple time series datasets and anomaly detectors. CCS CONCEPTS• Information systems → Data management systems; Temporal data; Main memory engines; Database web servers;• Human-centered computing → Interactive systems and tools; Graphical user interfaces; Visualization; • Computing methodologies → Machine learning.

show abstract

Evaluating Subjective Accuracy in Time Series Pattern-Matching Using Human-Annotated Rankings

Cited by 20 publications

References 31 publications

Peax Interactive Visual Pattern Search in Sequential Data Using Unsupervised Deep Representation Learning

Peax Interactive Visual Pattern Search in Sequential Data Using Unsupervised Deep Representation Learning

Hard-Label Black-Box Adversarial Attack on Deep Electrocardiogram Classifier

Visual Exploration of Time Series Anomalies with Metro-Viz

Contact Info

Product

Resources

About