Improving Real-Time Hand Gesture Recognition with Semantic Segmentation

Abstract

Hand gesture recognition (HGR) takes a central role in human–computer interaction, covering a wide range of applications in the automotive sector, consumer electronics, home automation, and others. In recent years, accurate and efficient deep learning models have been proposed for real-time applications. However, the most accurate approaches tend to employ multiple modalities derived from RGB input frames, such as optical flow. This practice limits real-time performance due to intense extra computational cost. In this paper, we avoid the optical flow computation by proposing a real-time hand gesture recognition method based on RGB frames combined with hand segmentation masks. We employ a light-weight semantic segmentation method (FASSD-Net) to boost the accuracy of two efficient HGR methods: Temporal Segment Networks (TSN) and Temporal Shift Modules (TSM). We demonstrate the efficiency of the proposal on our IPN Hand dataset, which includes thirteen different gestures focused on interaction with touchless screens. The experimental results show that our approach significantly overcomes the accuracy of the original TSN and TSM algorithms by keeping real-time performance.

No Interface, No Problem: Gesture Recognition on Physical Objects Using Radar Sensing

“…Regardless of popular technologies used for implementing gesture recognisers such as RGB [ 20 , 21 , 22 , 23 ] or infrared (IR) [ 24 , 25 , 26 , 27 , 28 ] cameras, Radio-frequency solutions including radar [ 29 , 30 ], Wi-Fi [ 31 , 32 , 33 ], GSM [ 34 ], and RFID [ 35 ] offer several advantages. Above all, RF sensing technologies are insensitive to light, which usually affects the camera and especially, IR-based solutions.…”

Hand gesture recognition algorithm combining hand-type adaptive algorithm and effective-area ratio for efficient edge computing

“…Furthermore, these algorithms are not sufficiently intuitive to represent the hand gestures formed by different hand types. Other algorithms consider fingers as the features and they are detected on the basis of ridge detection, [25][26][27][28][29][30] a circle drawn on the hand centroid, 31,32 or convex decomposition. 33 However, the method in Ref.…”

“…33 However, the method in Ref. 34 is time-consuming, while the others [28][29][30][31][32] are not effective to handle fingers with distortion. Subsequent classification algorithms 28,32 are learning-based, which require many training images for each class.…”

“…34 is time-consuming, while the others [28][29][30][31][32] are not effective to handle fingers with distortion. Subsequent classification algorithms 28,32 are learning-based, which require many training images for each class. Moreover, algorithms that use rule classifiers and zero training images [29][30][31] lack adaptability for certain gestures with distortion and varying postures.…”

“…In previous studies, [28][29][30] the Euclidean distance between a pixel and its nearest boundary in linear time was used to extract gesture features. Here, we propose other features as follows.…”