Structured Inference for Recurrent Hidden Semi-markov Model

Abstract

Segmentation and labeling for high dimensional time series is an important yet challenging task in a number of applications, such as behavior understanding and medical diagnosis. Recent advances to model the nonlinear dynamics in such time series data, has suggested to involve recurrent neural networks into Hidden Markov Models. However, this involvement has caused the inference procedure much more complicated, often leading to intractable inference, especially for the discrete variables of segmentation and labeling. To achieve both flexibility and tractability in modeling nonlinear dynamics of discrete variables, we present a structured and stochastic sequential neural network (SSNN), which composes with a generative network and an inference network. In detail, the generative network aims to not only capture the long-term dependencies but also model the uncertainty of the segmentation labels via semi-Markov models. More importantly, for efficient and accurate inference, the proposed bi-directional inference network reparameterizes the categorical segmentation with the Gumbel-Softmax approximation and resorts to the Stochastic Gradient Variational Bayes. We evaluate the proposed model in a number of tasks, including speech modeling, automatic segmentation and labeling in behavior understanding, and sequential multi-objects recognition. Experimental results have demonstrated that our proposed model can achieve significant improvement over the state-of-the-art methods.

Neuronal Sequence Models for Bayesian Online Inference

“…The most widely-used models for discrete sequence generation are hidden Markov models (HMM) and their time-dependent generalisation, hidden semi-Markov models (HSMM) (Yu, 2015). In particular, HMMs and HSMMs are standard tools in a wide range of applications concerned with e.g., speech recognition (Liu et al, 2018;Zen et al, 2004;Deng et al, 2006) and activity recognition (Duong et al, 2005). Furthermore, they have often been used for the analysis of neuronal activity (Tokdar et al, 2010) and human behavior in general (Eldar et al, 2011).…”

A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learning

“…proposed by [21], [22], and the authors of [23] relax the discrete latents and durations of a recurrent hidden semi-Markov Model. Applications of (Gumbel-based) discrete latent variable models as described above include (among others) planning [24], syntactic parsing [25], text modelling [26], speech modelling [27], [28], paraphrase generation [29], recommender systems [30], drug-drug interaction modelling [31], and event modelling [32].…”

Switching Recurrent Kalman Networks

“…[Qiu et al, 2020] introduces a novel inference technique that accounts for multimodality. Other works employ the idea of switching regimes incorporated with deep learning, such as [Johnson et al, 2016, Farnoosh et al, 2020, Dai et al, 2016, Liu et al, 2018. These models assume the Markov assumption on the state Figure 1: The architecture of the Switching Recurrent Kalman Network.…”