Real-Time Human-UAV Interaction: New Dataset and Two Novel Gesture-Based Interacting Systems

Abstract

Two novel gesture-based Human-UAV Interaction (HUI) systems are proposed to launch and control a UAV in real-time utilizing a monocular camera and a ground computer. The first proposal is an endto-end static Gesture-Based Interaction (GBI) system based on classifying the interacted user poses while discarding the gesture interpreting component to boost the system performance up to 99 % with a speed of 28 fps. On the other hand, the second proposal is a dynamic one that adopts a simple model to detect three parts (face and two hands) of the interacted person and tracks them for a certain number of frames till a specific dynamic gesture is recognized. The proposed dynamic method is efficient, decreases the complexity, and speeds the interaction up to 27 fps comparing with the recent multimodel ones. Its backbone is a simplified Tiny-You Only Look Once (YOLO) network saves the resources and speeds the detection process up to 120 fps. Moreover, a comprehensive new gestures dataset was established to facilitate the learning process and aid the research work. A comparative study is carried out to show the performance and efficiency of the proposed dynamic HUI system in terms of detection accuracy and speed with the baseline detector on the human gesture public dataset. Finally, a non-expert volunteer examines the proposed HUIs by launching and driving a Bebop 2 micro UAV through a set of real flights. INDEX TERMS Gesture dataset, gesture recognition, human-machine interaction, object tracking.

Motion Estimation and Hand Gesture Recognition-Based Human–UAV Interaction Approach in Real Time

“…In this section, we conducted a test to confirm the classification performance of static gesture recognition. We used confusion matrix as metric to usefully express the classification performance of the static gestures by referring to the preliminary research on the HGR system [ 15 , 16 , 25 , 36 ]. Figure 12 shows the confusion matrix and normalized confusion matrix on the test set, and the diagonal value of the normalized confusion matrix shows high density.…”

“…In the case of Kasab, Mohamed A. [ 15 ], a system for recognizing dynamic gesture through directly developed simplified Tiny-YOLOv2 was proposed. It is performed with KF algorithm and takes 42.7786 ms. Liu, C. [ 6 ] proposed two dynamic gesture recognition systems using CNN-based detection, which takes 20 ms.…”

“…However, deep feature-based approach for gesture recognition has many aspects to be solved for dynamic hand motion recognition, and various studies are being conducted. In Ref [ 15 ], Kassab, M.A. proposed to detect three parts of the interacted person and tracking the detected parts using Kalman filter (KF).…”

“…In the case of static gesture recognition, it shows high performance through neural-network-centered approaches, but dynamic gesture recognition still has a challenging problem, in that it is difficult to estimate the pose of a multi-dimensional hand gesture on 2D images. To solve this problem, as in the case of tracking through simple KF of gestures detected through the proposed developed model of Tiny-YOLOv2 [ 15 ] and tracking through Deep SORT of skeleton extracted through OpenPose [ 16 ], tracking algorithms in addition to deep-learning networks are combined and consequently increase computational complexity. However, complex algorithms are not suitable for the goal of safe and intuitive design of the HUI system, a hard real-time system that values real-time performance.…”

SqueezeNet and Fusion Network-Based Accurate Fast Fully Convolutional Network for Hand Detection and Gesture Recognition

“…Human hand detection and recognition are regarded as a way for computers to understand human language, enabling people to communicate with machines and interact naturally without any mechanical equipment. Human hands and gestures have applications in many computer fields, such as human-computer interaction (HCI) [1], rehabilitation medicine [2], anomaly detection [3], sign language recognition [4], gesture interaction [5], virtual reality [6], etc. Hand detection is a problem that detects the locations of all hands in an image.…”