HandSOM - neural clustering of hand motion for gesture recognition in real time

Abstract

Gesture recognition is an important task in Human-Robot Interaction (HRI) and the research effort towards robust and high-performance recognition algorithms is increasing. In this work, we present a neural network approach for learning an arbitrary number of labeled training gestures to be recognized in real time. The representation of gestures is hand-independent and gestures with both hands are also considered. We use depth information to extract salient motion features and encode gestures as sequences of motion patterns. Preprocessed sequences are then clustered by a hierarchical learning architecture based on self-organizing maps. We present experimental results on two different data sets: command-like gestures for HRI scenarios and communicative gestures that include cultural peculiarities, often excluded in gesture recognition research. For better recognition rates, noisy observations introduced by tracking errors are detected and removed from the training sets. Obtained results motivate further investigation of efficient neural network methodologies for gesture-based communication.

Self-organizing neural integration of pose-motion features for human action recognition

“…Finally, the reported results motivate the embedding of our learning system into mobile robot platforms to conduct further evaluations in more complex scenarios, where the robust recognition of actions plays a key role. For instance, the visual detection of dangerous events for assistive robotics such as fall events (Parisi and Wermter, 2013 , 2015 ), and the recognition of actions with learning robots in HRI scenarios (Soltoggio et al, 2013a , b ; Barros et al, 2014 ; Parisi et al, 2014a , b ).…”

“…Learning systems using depth information from low-cost sensors are increasingly popular in the research community encouraged by the combination of computational efficiency and robustness to light changes in indoor environments. In recent years, a large number of applications using 3D motion information has been proposed for human activity recognition such as classification of full-body actions (Faria et al, 2014 ; Shan and Akella, 2014 ; Parisi et al, 2014c ), fall detection (Rougier et al, 2011 ; Mastorakis and Makris, 2012 ; Parisi and Wermter, 2013 ), and recognition of hand gestures (Suarez and Murphy, 2012 ; Parisi et al, 2014a , b ; Yanik et al, 2014 ). A vast number of depth-based methods has used a 3D human skeleton model to extract relevant action features for the subsequent use of a classification algorithm.…”

Human Action Recognition and Assessment via Deep Neural Network Self-Organization

“…The proposed architectures can be considered a further step towards more flexible neural network models for learning robust visual representations on the basis of visual experience. Successful applications of deep neural network self-organization include human action recognition (Parisi, Weber & Wermter 2014, Elfaramawy et al 2017, gesture recognition (Parisi, Barros & Wermter 2014, Parisi, Jirak & Wermter 2014, body motion assessment (Parisi, von Stosch, Magg & Wermter 2015, Parisi, Magg & Wermter 2016, humanobject interaction (Mici et al 2017(Mici et al , 2018, continual learning (Parisi et al 2017, Parisi, Tani, Weber & Wermter 2018, and audio-visual integration (Parisi, Tani, Weber & Wermter 2016). Models of hierarchical action learning are typically feedforward.…”

Multi-modal Feedback for Affordance-driven Interactive Reinforcement Learning

“…For recognizing gestures, we used an extended version of neural network learning for gesture recognition [25] that extracts hand-independent gesture features from depth map sequences. The learning model consists of a set of two hierarchically arranged self-organizing networks that learn the spatiotemporal structure of the input sequences in terms of gesture features.…”