Comparing Hidden Markov Models and Long Short Term Memory Neural Networks for Learning Action Representations

Panzner, Maximilian; Cimiano, Philipp

doi:10.1007/978-3-319-51469-7_8

Cited by 40 publications

(16 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We use vanilla RNN units, as we experimentally observe that gated recurrent layers quickly lead to overfitting problems. We speculate that this is due to the small size of the dataset used here compared to datasets usually employed to train deep LSTM and GRU recurrent networks [20,13]. We use a one-dimensional global average pooling layer to summarise temporal patterns extracted by the recurrent layers.…”

Section: Convolutional Recurrent Neural Networkmentioning

confidence: 99%

Calibrating the Classifier: Siamese Neural Network Architecture for End-to-End Arousal Recognition from ECG

Patané

Kwiatkowska

2019

Machine Learning, Optimization, and Data Science

View full text Add to dashboard Cite

Affective analysis of physiological signals enables emotion recognition in mobile wearable devices. In this paper, we present a deep learning framework for arousal recognition from ECG (electrocardiogram) signals. Specifically, we design an end-to-end convolutional and recurrent neural network architecture to (i) extract features from ECG; (ii) analyse time-domain variation patterns; and (iii) non-linearly relate those to the user's arousal level. The key novelty is our use of a sharedparameter siamese architecture to implement user-specific feature calibration. At each forward and backward pass, we concatenate to the input a user-dependent template that is processed by an identical copy of the network. The siamese architecture makes feature calibration an integral part of the training process, allowing modelling of general dependencies between the user's ECG at rest and those during emotion elicitation. On leave-one-user-out cross validation, the proposed architecture obtains +21.5% score increase compared to state-of-the-art techniques. Comparison with alternative network architectures demonstrates the effectiveness of the siamese network in achieving user-specific feature calibration.

show abstract

Section: Convolutional Recurrent Neural Networkmentioning

confidence: 99%

Calibrating the Classifier: Siamese Neural Network Architecture for End-to-End Arousal Recognition from ECG

Patané

Kwiatkowska

2019

Machine Learning, Optimization, and Data Science

View full text Add to dashboard Cite

show abstract

“…This benchmark can be described as either a fast Markov model or a history search, and in essense is a conditional probability maximization method. When there are constraints on data or computational power, Hidden Markov Models can match the performance of LSTMs (Panzner & Cimiano, 2016) so we believe (Vaswani et al, 2017): The Transformer consists of an encoder and decoder each made up of N blocks. Input is a sequence of events, output is a sequence of predicted events.…”

Section: Benchmarkmentioning

confidence: 99%

Pay attention and you won’t lose it: a deep learning approach to sequence imputation

Sucholutsky

Narayan

Schonlau

et al. 2019

PeerJ Computer Science

View full text Add to dashboard Cite

In most areas of machine learning, it is assumed that data quality is fairly consistent between training and inference. Unfortunately, in real systems, data are plagued by noise, loss, and various other quality reducing factors. While a number of deep learning algorithms solve end-stage problems of prediction and classification, very few aim to solve the intermediate problems of data pre-processing, cleaning, and restoration. Long Short-Term Memory (LSTM) networks have previously been proposed as a solution for data restoration, but they suffer from a major bottleneck: a large number of sequential operations. We propose using attention mechanisms to entirely replace the recurrent components of these data-restoration networks. We demonstrate that such an approach leads to reduced model sizes by as many as two orders of magnitude, a 2-fold to 4-fold reduction in training times, and 95% accuracy for automotive data restoration. We also show in a case study that this approach improves the performance of downstream algorithms reliant on clean data.How to cite this article Sucholutsky I, Narayan A, Schonlau M, Fischmeister S. 2019. Pay attention and you won't lose it: a deep learning approach to sequence imputation. PeerJ Comput. Sci. 5:e210 http://doi.

show abstract

“…This benchmark can be described as either a fast Markov model or a history search, and in essense is a conditional probability maximization method. When there are constraints on data or computational power, Hidden Markov Models can match the performance of LSTMs [Panzner and Cimiano, 2016] so we believe this is a good benchmark that both the LSTM from Sucholutsky et al…”

Section: Benchmarkmentioning

confidence: 99%

Peer Review #3 of "Pay attention and you won’t lose it: a deep learning approach to sequence imputation (v0.1)"

2019

View full text Add to dashboard Cite

In most areas of machine learning, it is assumed that data quality is fairly consistent between training and inference. Unfortunately, in real systems, data are plagued by noise, loss, and various other quality reducing factors. While a number of deep learning algorithms solve end-stage problems of prediction and classification, very few aim to solve the intermediate problems of data pre-processing, cleaning, and restoration. Long Short-Term Memory (LSTM) networks have previously been proposed as a solution for data restoration, but they suffer from a major bottleneck: a large number of sequential operations. We propose using attention mechanisms to entirely replace the recurrent components of these data-restoration networks. We demonstrate that such an approach leads to reduced model sizes by as many as 2 orders of magnitude, a 2-fold to 4-fold reduction in training times, and 95% accuracy for automotive data restoration. We also show in a case study that this approach improves the performance of downstream algorithms reliant on clean data.

show abstract

Comparing Hidden Markov Models and Long Short Term Memory Neural Networks for Learning Action Representations

Cited by 40 publications

References 13 publications

Calibrating the Classifier: Siamese Neural Network Architecture for End-to-End Arousal Recognition from ECG

Calibrating the Classifier: Siamese Neural Network Architecture for End-to-End Arousal Recognition from ECG

Pay attention and you won’t lose it: a deep learning approach to sequence imputation

Peer Review #3 of "Pay attention and you won’t lose it: a deep learning approach to sequence imputation (v0.1)"

Contact Info

Product

Resources

About