End-to-end multiple modals deep learning system for hand posture recognition

Abstract

Multi-modal or multi-view dataset that was captured from various resources (e.g. RGB and Depth) of a subject at the same time. Combination between different cues has still faced to many challenges as unique data and complementary in-formation. In adition, the proposed method for multiple modalities recognition consists of discrete blocks, such as: extract features for separative data flows, combine of features, and classify gestures. To address the challenges, we pro-posed two novel end-to-end hand posture recognition frameworks, which are integrated all steps into a convolution neuronal network (CNN) system from capturing various types of cues (RGB and Depth images) to classify hand ges-ture labels. Both frameworks use the Resnet50 backbone that was pretrained by ImageNet dataset. We proposed a novel end-to-end multi-modal frameworks, which are named attention convolution module (ACM) and gated concatenation module (GCM). Both of them are deployed, evaluated and compared on vari-ous multi-modalities hand posture datasets. Experimental results show that our proposed method outperforms with others state-of-the-art techniques (SOTA) methods.

An Efficient Strategy for Catastrophic Forgetting Reduction in Incremental Learning

“…The results of the first scenario were compared with those of iCaRL method to show the better performance of our memory reconstruction strategy in comparison with iCaRL. In the second scenario, the performance of our memory reconstruction strategy with loss combination was evaluated with the different cases of the α coefficient in Formula (14). Based on this, the best α value was chosen for further comparative evaluations with other SOTA methods.…”

“…In order for some networks to be effective, it is frequently necessary to reuse model parameters that were learned from training a sizable dataset [12]. The pre-trained models are then fine-tuned on specialized data to obtain higher efficiency [13][14][15]. Transfer learning can cause the old knowledge to be overwritten by the new one.…”

New blender-based augmentation method with quantitative evaluation of CNNs for hand gesture recognition

“…The blender-based datasets are captured as illustrated in blue flows of Figure 2. Doan et al [11], explained a hand glove was designed that composed five flex sensors and accelerator sensor to change the finger's curvatures and hand's movement. In this work, this electronic glove is continuously utilized to connect to a Blender program.…”

“…These surveys covered a large number of studies published in the research community. The deep learning-based approaches [10], [11] have been shown to outperform handcrafted feature-based approaches [5], [12]- [14] in most relevant tasks of hand such detection [15], pose estimation [5], [7], and gesture recognition [16], [17]. The convolution neuron network (CNN) architectures [18]- [20] require a very large dataset [21], [22] to train models while existing hand gesture datasets have not adapted for this demand.…”