Adaptable Mixed-Reality Sensorimotor Interface for Human-Swarm Teaming: Person with Limb Loss Case Study and Field Experiments

Abstract

This paper presents the design, evaluation, and field experiment of the innovative Adapt- able Human-Swarm Teaming ( α -SWAT) interface developed to support military field operations. Human-swarm teaming requires collaboration between a team of humans and a team of robotic agents for strategic decision-making and task performance. α -SWAT allows multiple human team members with different roles, physical capabilities, or preferences to interact with the swarm via a configurable, multimodal user interface (UI). The system has an embedded task allocation algorithm for the rapid assignment of tasks created by the mission planner to the swarm. The multimodal UI supports swarm visualization via a mixed reality display or a conventional 2D display, human gesture inputs via a camera or an electromyography device, tactile feedback via a vibration motor or implanted peripheral nerve interface, and audio feedback. In particular, the UI system interfacing with the implanted electrodes through a neural interface enables gesture detection and tactile feedback for individuals with upper limb amputation to participate in human-swarm teaming. The multimodality of α -SWAT’s UI adapts to the needs of three different roles of the human team members: Swarm Planner, Swarm Tactician Rear, and Swarm Tactician Forward. A case study evaluated the functionality and usability of α -SWAT to enable a participant with limb loss and an implanted neural interface to assign three tasks to a simulated swarm of 150 robotic agents. α -SWAT was also used to visualize live telemetry from 40 veridical robotic agents for multiple simultaneous human participants at a field experiment.