Team Situation Awareness in Human-Autonomy Teaming: A Systems Level Approach

Abstract

Project overview. The current study focuses on analyzing team flexibility by measuring entropy (where higher values correspond to system reorganization and lower values correspond to more stable system organization) across all-human teams and Human-Autonomy Teams (HAT). We analyzed teams in the context of a fully-fledged synthetic agent that acts as a pilot for a three-agent Remotely Piloted Aircraft System (RPAS) ground crew. The synthetic agent must be able to communicate and coordinate with human teammates in a constructive and timely manner to be effective. This study involved three heterogeneous team members who had to take photographs of target waypoints and communicate via a text-based communication system. The three team members’ roles were: 1) navigator provides information about flight plan with speed and altitude restrictions at each waypoint; 2) pilot adjusts altitude and airspeed to control the Remotely Piloted Aircraft (RPA), and negotiates with the photographer about the current altitude and airspeed to take good photos for the targets; and 3) photographer screens camera settings, and sends feedback to other team members regarding the target photograph status. The three conditions differed based on the manipulation of the pilot role: 1) Synthetic – the pilot was the synthetic agent, 2) Control – the pilot was a randomly assigned participant, and 3) Experimenter – the pilot was a well-trained experimenter who focused on sending and receiving information in a timely manner. The goal of this study is to examine how overall RPAS flexibility across HATs and all-human teams are associated with Team Situation Awareness (TSA). Method. There were 30 teams (10-teams per condition): control teams consisted of three participants randomly assigned to each role; synthetic and experimenter teams included two participants randomly assigned to the navigator and photographer roles. The experiment took place over five 40-minute missions, and the goal was to take as many “good” photos of ground targets as possible while avoiding alarms and rule violations. We obtained several measures, including mission and target level team performance scores, team process measures (situation awareness, process ratings, communication and coordination), and other measures (teamwork knowledge, workload, and demographics). We first estimated amount of system reorganization of the RPAS via an information entropy measure, i.e., the number of arrangements the system occupied over a given period of time (Shannon & Weaver, 1975). Based on information entropy, we defined four layers to represent the RPAS (Gorman, Demir, Cooke, & Grimm, In Review): 1) communications - the chat-based communication among team members; 2) vehicle - the RPA itself, e.g., speed, altitude; 3) control - interface between the RPA and the user; and system - the overall activity of the sub-layers. Then, we looked at the relationship between layers and TSA, which was based on successfully overcoming and completing ad hoc embedded target waypoints. Results and conclusion. Overall, the experimenter teams adapted to more roadblocks than the synthetic teams, who were equivalent to control teams (Demir, McNeese, & Cooke, 2016). The findings indicate that: 1) synthetic teams demonstrated rigid systems level activity, which consisted of less reorganization of communication, control and vehicle layers as conditions changed, which also resulted in less adaptation to roadblocks; 2) control teams demonstrated less communication reorganization, but more control and vehicle reorganization, which also resulted in less adaptation to roadblocks; and 3) experimenter teams demonstrated more reorganization across communication, control and vehicle layers, which resulted in better adaptation to roadblocks. Thus, the ability of a system to reorganize across human and technical layers as situations change is needed to adapt to novel conditions of team performance in a dynamic task

Novel Human-in-the-Loop (HIL) Simulation Method to Study Synthetic Agents and Standardize Human–Machine Teams (HMT)

“…However, most recently, HAT has been frequently used in association with highly intelligent agents based on AI, machine learning, and cognitive modeling that are as competent as humans. Such work has directed the implementation of human-autonomy collaboration from concept to practice [25][26][27][28][29].…”

Human–Autonomy Teaming: A Review and Analysis of the Empirical Literature

“…HAT has been described as at least one human working cooperatively with at least one autonomous agent (McNeese et al, 2018), where an autonomous agent is a computer entity with a partial or high degree of self-governance with respect to decision-making, adaptation, and communication (Demir et al, 2016; Mercado et al, 2016; Myers et al, 2019). As noted by Larson and DeChurch (2020, p. 10), “we are quickly approaching a time when digital technologies are as agentic as are human counterparts.” With continuous advancements in artificial intelligence (AI), autonomous agents can perform a greater number of dynamical functions in both teamwork and taskwork than ever before (Seeber et al, 2020), and they are beginning to be viewed as teammates rather than tools (Grimm et al, 2018a; Lyons et al, 2018). For example, autonomous agents can increasingly participate in teamwork activity involving coordination, task reallocation, and continuous interaction with humans and other autonomous agents (Chen et al, 2016; Johnson et al, 2012; Shannon et al, 2017).…”

“…In the 1990s, as the above quotes illustrate, discussions emerged with respect to autonomous agents playing roles as genuine team players. Therefore, the concept of HAT has been considered by academics for three decades, but it was not until more recently that the term HAT has emerged and been used frequently (e.g., Demir et al, 2019; Demir, Likens, et al, 2018; Demir, McNeese, et al 2018; Dubey et al, 2020, Fiore & Wiltshire, 2016, Grimm et al, 2018a, 2018b, Grimm et al, 2018c; McNeese et al, 2018, 2019; Shannon et al, 2017; Wohleber et al, 2017). We believe the emerging use of the term HAT is due to significant advances in AI, machine learning, and cognitive modeling.…”

Coordinating Human-Robot Teams with Dynamic and Stochastic Task Proficiencies

“…As the subjects are exposed to varied task assignments in exploration-based strategies, they are cognizant of which tasks they are most proficient in and thereby have greater situational awareness about how the robot is trying to optimize their schedule. Prior studies in automation have shown how loss of situational awareness can negatively impact trust in the system [28,31,37]. Hence, we hypothesize that a better understanding of the scheduling process by subjects in exploration-based strategies will lead to greater trust in the robot.…”