Multi Channel Sequence Processing

Abstract

Abstract. This paper summarizes some of the current research challenges arising from multichannel sequence processing. Indeed, multiple real life applications involve simultaneous recording and analysis of multiple information sources, which may be asynchronous, have different frame rates, exhibit different stationarity properties, and carry complementary (or correlated) information. Some of these problems can already be tackled by one of the many statistical approaches towards sequence modeling. However, several challenging research issues are still open, such as taking into account asynchrony and correlation between several feature streams, or handling the underlying growing complexity. In this framework, we discuss here two novel approaches, which recently started to be investigated with success in the context of large multimodal problems. These include the asynchronous HMM, providing a principled approach towards the processing of multiple feature streams, and the layered HMM approach, providing a good formalism for decomposing large and complex (multi-stream) problems into layered architectures. As briefly reported here, combination of these two approaches yielded successful results on several multi-channel tasks, ranging from audio-visual speech recognition to automatic meeting analysis. 2 IDIAP-RR 05-04

Hierarchical group process representation in multi-agent activity recognition

“…If we compress all features into a single vector,the model will face the dimension curse of feature vector and lose interactive information between different streams. Researchers [2,27,28,3] have discussed that it is necessary to combine available streams in multi-channel settings. Asynchrony between streams.…”

“…To avoid feature space and state space exploding, it is necessary to solve modeling problems in multi-channel settings [2,27,28,3]. Parallel HMMs as shown in Fig.…”

Extracting information from multimedia meeting collections

“…This second step could be designed to integrate a variable number of individual high-level data. From a more abstract modeling level, other problems are still open, and several of them are discussed in [5]. For instance, assuming each individual behavior is represented by a separate stream, and a single group model is used to incorporate all these streams, current modeling techniques, based on Markovian assumptions, often need exponential resources with respect to the number of streams, which quickly becomes intractable.…”

Learning interactions among multi-channel sequences with dynamical influence models

“…Multi-channel methods for characterizing such an event or system have proven successful both in theory and in practice. Multi-channel methods are also named multi-modal, multi-band, multi-stream methods in different applications [2], such as biometric authentication, speech recognition and activity recognition [1][2][3][4][5].…”

“…With the development of electronic recoding devices, it becomes more and more feasible to simultaneously capture a same event with several devices, generating richer and more robust sets of feature streams [1] (thus resulting in multiple multiple observation streams). It is very helpful to utilize multiple complementary information such as in speech recognition.…”