Human-swarm interaction through distributed cooperative gesture recognition

Abstract

VIDEO OVERVIEWThe video presents the first results of a Swiss-funded project focusing on symbiotic peer-to-peer interaction and cooperation between humans and robot swarms. As a first step, we considered human-swarm interaction, and selected the use of hand gestures to let a human communicate with a swarm of relatively simple mobile robots. In our scenario, a hand gesture encodes a command, that the swarm will execute. The robots that we used are the foot-bots, developed in the Swarmanoid project [1].Hand gestures are a powerful and intuitive way to communicate, and do not require the use of additional devices. However, real-time vision-based recognition of hand gestures is a challenging task for the single foot-bot, due to its limited processing power and field of view. We investigated how to exploit robot mobility, swarm spatial distribution, and multihop wireless communications, to let the robots in the swarm: (i) implement a distributed and cooperative sensing of hand gestures, and (ii) robustly reach a consensus about a gesture.The first step was to use 13 foot-bots to collect 70,000 hand gestures images representing finger counts (from 0 to 5) and five furniture-like shapes. With this data set we trained .

Human Interaction With Robot Swarms: A Survey

“…Most of the research on proximal swarm interactions has focused on enabling the interaction through gesture recognition [88]- [90] as well as face engagement and speech [91]. The distributed gesture recognition presented in [88] and [90] facilitates the communication of a wide range of instructions to all swarm robots within sight.…”

“…The distributed gesture recognition presented in [88] and [90] facilitates the communication of a wide range of instructions to all swarm robots within sight. The human wears an orange glove that is easily recognizable by the cameras on board the robots.…”

HRI in the sky: Creating and commanding teams of UAVs with a vision-mediated gestural interface

“…This has been done for a single robot system (e.g. [12], [34]); however, an interesting extension would be to coordinate multiple robots to cooperatively estimate the vector given the system's ability to simultaneously capture images of the user from multiple angles (in the spirit of [13]). …”

“…Earlier work by Kortenkamp et al presents a mobile robot that uses an active vision system to recognize static gestures by building a skeleton model of the human operator; a vector of the human's arm is used to direct the robot to a particular point [12]. Giusti et al [13] demonstrated how a swarm of mobile robots can cooperatively detect a static human gesture and act upon it.…”

Gesturing at Subswarms: Towards Direct Human Control of Robot Swarms

“…To the best of our knowledge, there is only one existing HSI approach that does not rely on an extra modelling layer: Giusti et al's [9] hand gesture-based interaction system. The goal of their work is to allow robots to decode hand gestures.…”