Making Time Series Embeddings More Interpretable in Deep Learning

Abstract

With the success of language models in deep learning, multiple new time series embeddings have been proposed. However, the interpretability of those representations is often still lacking compared to word embeddings. This paper tackles this issue, aiming to present some criteria for making time series embeddings applied in deep learning models more interpretable using higher-level features in symbolic form. For that, we investigate two different approaches for extracting symbolic approximation representations regarding the frequency and the trend information, i.e. the Symbolic Fourier Approximation (SFA) and the Symbolic Aggregate approXimation (SAX). In particular, we analyze and discuss the impact of applying the different representation approaches. Furthermore, in our experimentation, we apply a state-of-the-art Transformer model to demonstrate the efficacy of the proposed approach regarding explainability in a comprehensive evaluation using a large set of time series datasets.

Identifying Informative Nodes in Attributed Spatial Sensor Networks Using Attention for Symbolic Abstraction in a GNN-based Modeling Approach

“…Therefore, we aim to use more symbolic embedding approaches, like Symbolic Aggregate approXimation (SAX) (Lin et al 2003) or Symbolic Fourier Approximation (SFA) (Schäfer and Högqvist 2012). SAX, in particular, has been successfully applied as a symbolic embedding for deep learning (Lavangnananda and Sawasdimongkol 2012;Schwenke and Atzmueller 2021c;2021b;Criado-Ramón, Ruiz, and Pegalajar 2022;Tabassum, Menon, and Jastrzebska 2022), via a more human related representation, cf. Atzmueller et al (2017); Ramirez, Wimmer, and Atzmueller (2019).…”

“…Transformers are especially interesting for time series data due to their ability to handle long-term dependencies (Li et al 2019), while also being able to act as data encoder, e. g., the original encoderdecoder application. Additionally, the introduction of attention with the MHA enables further explainable methods (Vig 2019;Škrlj et al 2020;Schwenke and Atzmueller 2021c;2021b). Here, we utilize the Transformer architecture as a foundation of our attention-based abstraction approach.…”

“…This is even more important because attention-based methods from other domains seem not to work well on time series data (Ismail et al 2020). Recently, e. g., Schwenke and Atzmueller (2021c) introduced an attention-based local abstraction method for time series data. In this paper, we adapted this method to our sensor graph context, including high-frequency time series assigned to the respective nodes of the network as attributes.…”

“…We refer to Schwenke and Atzmueller (2021a) for a detailed discussion about this aggregation process. Afterwards, locally for each sample, all vector entries below a certain threshold get masked; compared to Schwenke and Atzmueller (2021c) we use only one threshold due to the graph structured data. By re-training the model with the masked values, we verify that the information needed for the task is still included.…”

“…We observe that the LASA 1.0 model approximates the original model, for both datasets. This could however also be due to a suboptimal threshold, which is not always simple to fine-tune due to the sometimes noisy attention values (Schwenke and Atzmueller 2021c). In our experimentation, we only tested three thresholds as an approximation, where overall t1.0 performed best.…”