A Transformer-based Framework for Multivariate Time Series Representation Learning

Zerveas, George; Jayaraman, Srideepika; Patel, Dhaval; Bhamidipaty, Anuradha; Eickhoff, Carsten

doi:10.1145/3447548.3467401

Cited by 468 publications

(284 citation statements)

References 2 publications

Supporting

Mentioning

196

Contrasting

Unclassified

Order By: Relevance

“…Similarly, variational autoencoders (VAEs), RNNs [38] and generative adversarial networks (GANs) [35] have been successfully applied for synthetic offline handwriting generation, but not for the online case so far. For the time-series classification task, standard convolutional architectures [34,42,77,108,108,117], spatio-temporal methods [6,21,30,47,48] and variants [33,93,95,104] as well as Transformers [26,114] have been employed. While many approaches predict one class after the other, [20,63] predicted sequences similar to our approach.…”

Section: Methodsmentioning

confidence: 99%

Benchmarking Online Sequence-to-Sequence and Character-based Handwriting Recognition from IMU-Enhanced Pens

Ott¹,

Rügamer²,

Heublein³

et al. 2022

Preprint

View full text Add to dashboard Cite

Handwriting is one of the most frequently occurring patterns in everyday life and with it come challenging applications such as handwriting recognition (HWR), writer identification, and signature verification. In contrast to offline HWR that only uses spatial information (i.e., images), online HWR (OnHWR) uses richer spatio-temporal information (i.e., trajectory data or inertial data). While there exist many offline HWR datasets, there is only little data available for the development of OnHWR methods as it requires hardwareintegrated pens. This paper presents data and benchmark models for real-time sequence-to-sequence (seq2seq) learning and single character-based recognition. Our data is recorded by a sensor-enhanced ballpoint pen, yielding sensor data streams from triaxial accelerometers, a gyroscope, a magnetometer and a force sensor at 100 Hz. We propose a variety of datasets including equations and words for both the writer-dependent and writer-independent tasks. We provide an evaluation benchmark for seq2seq and single character-based HWR using recurrent and temporal convolutional networks and Transformers combined with a connectionist temporal classification (CTC) loss and cross entropy losses. Our methods do not resort to language or lexicon models.

show abstract

Section: Methodsmentioning

confidence: 99%

Benchmarking Online Sequence-to-Sequence and Character-based Handwriting Recognition from IMU-Enhanced Pens

Ott¹,

Rügamer²,

Heublein³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…While recent work in time series representation learning focused on various aspects of representation learning such how to sample contrastive pairs (Franceschi et al, 2020;Tonekaboni et al, 2021), taking a Transformer based approach (Zerveas et al, 2021), exploring complex contrastive learning tasks (Eldele et al, 2021), as well as constructing temporally hierarchical representations (Yue et al, 2021), none have touched upon learning representations composed of trend and seasonal features. Whereas existing work have focused exclusively on time series classification tasks, Yue et al (2021) first showed that time series representations learned via contrastive learning establishes a new stateof-the-art performance on deep forecasting benchmarks.…”

Section: Related Workmentioning

confidence: 99%

“…TST (Zerveas et al, 2021) TST is a Transformer based approach using a reconstruction loss. We use the open source implementation 9 as is.…”

Section: E Details On Baselinesmentioning

confidence: 99%

CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting

Woo¹,

Liu²,

Kumar³

et al. 2022

Preprint

View full text Add to dashboard Cite

Deep learning has been actively studied for time series forecasting, and the mainstream paradigm is based on the end-to-end training of neural network architectures, ranging from classical LSTM/RNNs to more recent TCNs and Transformers. Motivated by the recent success of representation learning in computer vision and natural language processing, we argue that a more promising paradigm for time series forecasting, is to first learn disentangled feature representations, followed by a simple regression fine-tuning step -we justify such a paradigm from a causal perspective. Following this principle, we propose a new time series representation learning framework for long sequence time series forecasting named CoST, which applies contrastive learning methods to learn disentangled seasonal-trend representations. CoST comprises both time domain and frequency domain contrastive losses to learn discriminative trend and seasonal representations, respectively. Extensive experiments on real-world datasets show that CoST consistently outperforms the state-of-the-art methods by a considerable margin, achieving a 21.3% improvement in MSE on multivariate benchmarks. It is also robust to various choices of backbone encoders, as well as downstream regressors.Code is available at https://github.com/salesforce/CoST.

show abstract

“…We adopt the time series classification task to evaluate the model performance for temporal sequences. We select 10 multivariate datasets from UEA Time Series Classification Archive (Bagnall et al, 2018) for experiments and follow the data pre-processing in (Zerveas et al, 2021). We use 2 layers for Transformer-based models with 512 hidden channels and 8 heads for the attention mechanism.…”

Section: Time Series Classificationmentioning

confidence: 99%

Flowformer: Linearizing Transformers with Conservation Flows

Wu¹,

Wu²,

Xu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Transformers based on the attention mechanism have achieved impressive success in various areas. However, the attention mechanism has a quadratic complexity, significantly impeding Transformers from dealing with numerous tokens and scaling up to bigger models. Previous methods mainly utilize the similarity decomposition and the associativity of matrix multiplication to devise linear-time attention mechanisms. They avoid degeneration of attention to a trivial distribution by reintroducing inductive biases such as the locality, thereby at the expense of model generality and expressiveness. In this paper, we linearize Transformers free from specific inductive biases based on the flow network theory. We cast attention as the information flow aggregated from the sources (values) to the sinks (results) through the learned flow capacities (attentions). Within this framework, we apply the property of flow conservation with attention and propose the Flow-Attention mechanism of linear complexity. By respectively conserving the incoming flow of sinks for source competition and the outgoing flow of sources for sink allocation, Flow-Attention inherently generates informative attentions without using specific inductive biases. Empowered by the Flow-Attention, Flowformer yields strong performance in linear time for wide areas, including long sequence, time series, vision, natural language, and reinforcement learning.

show abstract

A Transformer-based Framework for Multivariate Time Series Representation Learning

Cited by 468 publications

References 2 publications

Benchmarking Online Sequence-to-Sequence and Character-based Handwriting Recognition from IMU-Enhanced Pens

Benchmarking Online Sequence-to-Sequence and Character-based Handwriting Recognition from IMU-Enhanced Pens

CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting

Flowformer: Linearizing Transformers with Conservation Flows

Contact Info

Product

Resources

About