Multimodal Gesture Recognition via Multiple Hypotheses Rescoring

Pitsikalis, Vassilis; Katsamanis, Athanasios; Theodorakis, Stavros; Maragos, Petros

doi:10.1007/978-3-319-57021-1_16

Cited by 18 publications

(21 citation statements)

References 66 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…As already mentioned, the performance of the proposed multimodal gesture scheme has been evaluated on the ChaLearn corpus [Pavlakos et al 2014, Pitsikalis et al 2015. In Table 11.3, recognition results are presented for each stream separately, as well as after single-or twostage fusion.…”

Section: Resultsmentioning

confidence: 99%

“…In particular, the first ChaLearn Challenge and corpus concerned multimodal gesture recognition [Escalera et al 2013b]. Most of the developed systems on these data were based on HMMs [Pitsikalis et al 2015], but other approaches such as neural networks, SVMs, and random forests were also applied. For the visual recognition of gestures, low-level features such as space-time interest points [Willems et al 2008] were used by Nandakumar et al [2013], and, to model large-scale temporal dependencies, recurrent neural networks (RNNs) were applied by Neverova et al [2013].…”

Section: Data Resources and Challenges For Gesture Recognitionmentioning

confidence: 99%

“…More elaborate statistical approaches for fusion at the lexical layer for multimodal gesture recognition are presented in [Miki et al 2014, Pitsikalis et al 2015. There, N-best lists…”

Section: Recognizing Speech and Gestures 17mentioning

confidence: 99%

“…In this section, we highlight the main ideas and methods of the authors' recent work on multimodal gesture recognition [Pitsikalis et al 2015]. The developed system comprises two main modules: feature extraction and modeling for each of three available unimodal data streams, as overviewed in Section 11.4.1, and multimodal fusion that is discussed in Section 11.4.2.…”

Section: A System In Detailmentioning

confidence: 99%

See 3 more Smart Citations

Multimodal gesture recognition

Katsamanis¹,

Pitsikalis²,

Theodorakis³

et al. 2017

The Handbook of Multimodal-Multisensor Interfaces: Foundations, User Modeling, and Common Modality Combinations - Volume 1

Self Cite

View full text Add to dashboard Cite

2 Chapter 11 Multimodal Gesture Recognition Figure 11.1 Screenshot from the "Put That There!" demonstration video by the Architecture Machine Group at MIT [Bolt 1980]. ers [Kinect 2016]. Such advancements have led to intensified efforts to integrate multimodal gesture interfaces in real-life applications. Indeed, the field of multimodal gesture recognition has been attracting increasing interest, being driven by novel HCI paradigms on a continuously expanding range of devices equipped with multimodal sensors and ever-increasing computational power, for example smartphones and smart television sets. Nevertheless, the capabilities of modern multimodal gesture systems remain limited. In particular, the set of gestures accounted for in typical setups is mostly constrained to pointing gestures, a number of emblematic ones like an open palm, and gestures corresponding to some sort of interaction with a physical object, e.g., pinching for zooming. At the same time, fusion with speech remains in most cases just an experimental feature. When compared to the abundance and variety of gestures and their interaction with speech in natural human communication, it clearly seems that there is still a long way to go for the corresponding HCI research and development [Kopp 2013]. Multimodal gesture recognition constitutes a wide multidisciplinary field. This chapter makes an effort to provide a comprehensive overview of it, both in theoretical and application terms. More specifically, basic concepts related to gesturing, the multifaceted interplay of gestures and speech, and the importance of gestures in HCI are discussed in Section 11.2. An overview of the current trends in the field of multimodal gesture recognition is provided in Section 11.3, separately focusing on gestures, speech, and multimodal fusion. Further, a stateof-the-art recognition setup developed by the authors is described in detail in Section 11.4, in order to facilitate a better understanding of all practical considerations involved in such a system. In closing, the future of multimodal gesture recognition and related challenges are discussed in Section 11.5. Finally, a set of Focus Questions to aid comprehension of the material is also provided. 11.2 Multimodal Communication and Gestures 3 Glossary Terminology for Understanding Multimodal Gesture Recognition Co-speech gestures are gestures produced while speaking. Their interplay with speech and their role in human interaction are discussed in Section 11.2.2. Gesture recognition models are primarily statistical constructs that can be trained to represent specific gestures based on corresponding data. In this chapter, traditional hidden Markov models (HMMs) with Gaussian mixture model (GMM) observation probabilities are considered, as part of the authors' gesture recognition system detailed in Section 11.4. Further, a number of models based on deep learning approaches are overviewed in Section 11.3.1. A more elaborate discussion of deep learning for multimodal interaction modeling can be found in the second vol...

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Data Resources and Challenges For Gesture Recognitionmentioning

confidence: 99%

“…More elaborate statistical approaches for fusion at the lexical layer for multimodal gesture recognition are presented in [Miki et al 2014, Pitsikalis et al 2015. There, N-best lists…”

Section: Recognizing Speech and Gestures 17mentioning

confidence: 99%

Section: A System In Detailmentioning

confidence: 99%

See 2 more Smart Citations

Multimodal gesture recognition

Katsamanis¹,

Pitsikalis²,

Theodorakis³

et al. 2017

The Handbook of Multimodal-Multisensor Interfaces: Foundations, User Modeling, and Common Modality Combinations - Volume 1

Self Cite

View full text Add to dashboard Cite

show abstract

“…A task that is closely related to SLR is gesture recognition. Deep neural networks have proven to be successful for this problem, given a small vocabulary (20 gestures) [22,30] and/or with a multi-modal approach [24,18].…”

Section: Introductionmentioning

confidence: 99%

Gesture and Sign Language Recognition with Temporal Residual Networks

Pigou

Herreweghe

Dambre

2017

2017 IEEE International Conference on Computer Vision Workshops (ICCVW)

View full text Add to dashboard Cite

Gesture and sign language recognition in a continuous video stream is a challenging task, especially with a large vocabulary. In this work, we approach this as a framewise classification problem. We tackle it using temporal convolutions and recent advances in the deep learning field like residual networks, batch normalization and exponential linear units (ELUs). The models are evaluated on three different

show abstract