Adaptive Human-Robot Interactions for Multiple Unmanned Aerial Vehicles

Abstract

Advances in unmanned aircraft systems (UAS) have paved the way for progressively higher levels of intelligence and autonomy, supporting new modes of operation, such as the one-to-many (OTM) concept, where a single human operator is responsible for monitoring and coordinating the tasks of multiple unmanned aerial vehicles (UAVs). This paper presents the development and evaluation of cognitive human-machine interfaces and interactions (CHMI2) supporting adaptive automation in OTM applications. A CHMI2 system comprises a network of neurophysiological sensors and machine-learning based models for inferring user cognitive states, as well as the adaptation engine containing a set of transition logics for control/display functions and discrete autonomy levels. Models of the user’s cognitive states are trained on past performance and neurophysiological data during an offline calibration phase, and subsequently used in the online adaptation phase for real-time inference of these cognitive states. To investigate adaptive automation in OTM applications, a scenario involving bushfire detection was developed where a single human operator is responsible for tasking multiple UAV platforms to search for and localize bushfires over a wide area. We present the architecture and design of the UAS simulation environment that was developed, together with various human-machine interface (HMI) formats and functions, to evaluate the CHMI2 system’s feasibility through human-in-the-loop (HITL) experiments. The CHMI2 module was subsequently integrated into the simulation environment, providing the sensing, inference, and adaptation capabilities needed to realise adaptive automation. HITL experiments were performed to verify the CHMI2 module’s functionalities in the offline calibration and online adaptation phases. In particular, results from the online adaptation phase showed that the system was able to support real-time inference and human-machine interface and interaction (HMI2) adaptation. However, the accuracy of the inferred workload was variable across the different participants (with a root mean squared error (RMSE) ranging from 0.2 to 0.6), partly due to the reduced number of neurophysiological features available as real-time inputs and also due to limited training stages in the offline calibration phase. To improve the performance of the system, future work will investigate the use of alternative machine learning techniques, additional neurophysiological input features, and a more extensive training stage.

Scheduling and Path-Planning for Operator Oversight of Multiple Robots

“…We want to continue exploring applications and extend this work to human studies to investigate the framework's effectiveness for complex teleoperation tasks. A domain of interest where our ideas can apply are one-tomany (OTM) scenarios where a human operator needs to monitor and coordinate multiple multiple autonomous vehicles [39].…”

“…Neglect is mainly due to the robots competing for operator attention [38]. In [39], the authors propose real-time measurements of neurophysiological parameters to estimate workload as a potential input to new forms of adaptive automation.…”

Online Multimodal Inference of Mental Workload for Cognitive Human Machine Systems

“…Hence, these studies failed to implement a conventional ANFIS paradigm to produce accurate results of the online inference of MWL. Whereas studies by Zhang et al and Wang et al [46,58,59] demonstrated large time intervals, Lim et al [52] lacked the ability to demonstrate an inference of MWL during an online validation. Moreover, recent studies have outlined the importance of investigating the features contributing to the performance of the respective model used [50,51].…”

“…In addition to this, some studies compare several classification models when performing MWL estimations [50,51]. The use of regression models, such as Neuro Fuzzy Systems (NFS) [46,52,53] and Gaussian process regression [44], provides a continuous estimation of MWL but is less reported. Many of the studies that perform multimodal data fusion implement ML techniques that perform the calibration and validation in offline processing.…”

“…Nevertheless, they also implemented a conventional ANFIS model for comparison, which showed poor results. Lim et al [52] also implemented an ANFIS model in their study, which was calibrated on data from cardiac features, eye activity features and features from an fNIR. The offline validation showed good results from calibrating the model on the normalized features, but the study failed to demonstrate good results during the online validation.…”