FollowMe: Person following and gesture recognition with a quadrocopter

Abstract

Abstract-In this paper, we present an approach that allows a quadrocopter to follow a person and to recognize simple gestures using an onboard depth camera. This enables novel applications such as hands-free filming and picture taking. The problem of tracking a person with an onboard camera however is highly challenging due to the self-motion of the platform. To overcome this problem, we stabilize the depth image by warping it to a virtual-static camera, using the estimated pose of the quadrocopter obtained from vision and inertial sensors using an Extended Kalman filter. We show that such a stabilized depth video is well suited to use with existing person trackers such as the OpenNI tracker. Using this approach, the quadrocopter not only obtains the position and orientation of the tracked person, but also the full body pose -which can then for example be used to recognize hand gestures to control the quadrocopter's behaviour. We implemented a small set of example commands ("follow me", "take picture", "land"), and generate corresponding motion commands. We demonstrate the practical performance of our approach in an extensive set of experiments with a quadrocopter. Although our current system is limited to indoor environments and small motions due to the restrictions of the used depth sensor, it indicates that there is large potential for such applications in the near future.

Personalizing vision-based gestural interfaces for HRI with UAVs: a transfer learning approach

“…Therefore detecting gaze and body movements is considered a central problem. In particular, vision-mediated solutions for recognizing gestures have been proposed in several previous works [1], [2], [3] and few of them have considered specifically HRI with UAVs. In [1] a system where a teams of UAVs receive a series of commands from an uninstrumented human is presented.…”

“…However, while in [1] the authors focused on a multi-robot scenario, we specifically address the problem of recognizing userspecific hand movements. In [2] a single robot/single user setup is considered and a vision based gestural interface is realized. However a depth sensor is used instead of a traditional RGB camera.…”

“…Aiming at emulating the way humans interact only through eye contact and simple gestures, non-verbal communication methods specifically targeted to HRI with UAVs have been proposed recently [1], [2], [3]. These approaches have the advantages of avoiding the need of instrumenting the human as the only adopted sensors are those on-board of the robot.…”

“…Previous works [1], [2] have considered the use of low cost cameras as on board sensors and visual based gesture interaction systems for UAVs have been proposed. Many challenges arise in this context.…”

HRI in the sky: Creating and commanding teams of UAVs with a vision-mediated gestural interface

“…Jones et al [25] performed a user study to investigate how different modalities can be used to control a swarm of simulated UAVs in a virtual reality environment. Naseer et al [26] developed an autonomous system that enables a single quadrocopter to follow a human and respond to hand gestures using active RGB-D sensor with vision-based ego-motion cancellation.…”