Echo State Networks and Long Short-Term Memory for Continuous Gesture Recognition: a Comparative Study

Abstract

Recent developments of sensors that allow tracking of human movements and gestures enable rapid progress of applications in domains like medical rehabilitation or robotic control. Especially the inertial measurement unit (IMU) is an excellent device for real-time scenarios as it rapidly delivers data input. Therefore, a computational model must be able to learn gesture sequences in a fast yet robust way. We recently introduced an echo state network (ESN) framework for continuous gesture recognition (Tietz et al., 2019) including novel approaches for gesture spotting, i.e., the automatic detection of the start and end phase of a gesture. Although our results showed good classification performance, we identified significant factors which also negatively impact the performance like subgestures and gesture variability. To address these issues, we include experiments with Long Short-Term Memory (LSTM) networks, which is a state-of-the-art model for sequence processing, to compare the obtained results with our framework and to evaluate their robustness regarding pitfalls in the recognition process. In this study, we analyze the two conceptually different approaches processing continuous, variable-length gesture sequences, which shows interesting results comparing the distinct gesture accomplishments. In addition, our results demonstrate that our ESN framework achieves comparably good performance as the LSTM network but has significantly lower training times. We conclude from the present work that ESNs are viable models for continuous gesture recognition delivering reasonable performance for applications requiring real-time performance as in robotic or rehabilitation tasks. From our discussion of this comparative study, we suggest prospective improvements on both the experimental and network architecture level.

A journey in ESN and LSTM visualisations on a language task

“…The results obtained were comparable but the ESN architecture was much less computationally demanding. That is also the conclusion of [13] in which classical ESNs and LSTMs were compared on a gesture classification task.…”

Heterogeneous data fusion for the improved non-destructive detection of steel-reinforcement defects using echo state networks

“…They were developed with an engineering perspective in mind, and have proved to be more effective than traditional ANN methods at predicting complex time-series data, while also having a much shorter training time. 17 For example, ESNs have been shown to provide good performance and significantly superior training times in comparison to state-of-the-art long short-term memory networks (LSTMs), 18,19 temporal expression tree classifiers and genetic programming ensembles. 20 One of the most useful aspects of ESNs is their ability to recall past inputs through the presence of a short-term memory, which can be influenced by a careful tuning of the network parameters.…”

Optimizing echo state networks for continuous gesture recognition in mobile devices: A comparative study

“…Past research into continuous gesture recognition [6] was limited by a lack of hyperparameter tuning. Before [6] was published, Tietz et al [7] used grid search; however, the reservoir size was fixed at 400.…”

“…Past research into continuous gesture recognition [6] was limited by a lack of hyperparameter tuning. Before [6] was published, Tietz et al [7] used grid search; however, the reservoir size was fixed at 400. Additionally, past research did not experiment with different variants of ESN reservoirs and readouts, which could result in better performance.…”