CNN Deep Learning with Wavelet Image Fusion of CCD RGB-IR and Depth-Grayscale Sensor Data for Hand Gesture Intention Recognition

Abstract

Pixel-based images captured by a charge-coupled device (CCD) with infrared (IR) LEDs around the image sensor are the well-known CCD Red–Green–Blue IR (the so-called CCD RGB-IR) data. The CCD RGB-IR data are generally acquired for video surveillance applications. Currently, CCD RGB-IR information has been further used to perform human gesture recognition on surveillance. Gesture recognition, including hand gesture intention recognition, is attracting great attention in the field of deep neural network (DNN) calculations. For further enhancing conventional CCD RGB-IR gesture recognition by DNN, this work proposes a deep learning framework for gesture recognition where a convolution neural network (CNN) incorporated with wavelet image fusion of CCD RGB-IR and additional depth-based depth-grayscale images (captured from depth sensors of the famous Microsoft Kinect device) is constructed for gesture intention recognition. In the proposed CNN with wavelet image fusion, a five-level discrete wavelet transformation (DWT) with three different wavelet decomposition merge strategies, namely, max-min, min-max and mean-mean, is employed; the visual geometry group (VGG)-16 CNN is used for deep learning and recognition of the wavelet fused gesture images. Experiments on the classifications of ten hand gesture intention actions (specified in a scenario of laboratory interactions) show that by additionally incorporating depth-grayscale data into CCD RGB-IR gesture recognition one will be able to further increase the averaged recognition accuracy to 83.88% for the VGG-16 CNN with min-max wavelet image fusion of the CCD RGB-IR and depth-grayscale data, which is obviously superior to the 75.33% of VGG-16 CNN with only CCD RGB-IR.

A Human Manipulator Collaboration-based Scheme for Object Inspection by Robust and Fast Recognition of Hand Pose Sensing Images of Numeric Symbols Combined with Non-numeric Expressions for Smart Factories

“…The recognition of operator hand poses in an HRC-based inspection system is fast and robust and can include the classification of hand pose sensing images of numeric symbols combined with non-numeric expressions. The utilization of hand gesture recognition to analyze and then classify a person's hand gestures or hand poses has been achieved in many fields, (11)(12)(13)(14)(15) such as the robotic handling of smart materials, (11) the recognition of the intention of hand gestures, (12) and the recognition of the identity and emotion associated with hand gestures. (13,14) Studies on the use of hand gesture recognition in object inspections are extremely rare.…”