Evaluating sociotechnical dynamics in a simulated remotely-piloted aircraft system: a layered dynamics approach

Gorman, Jamie C.; Demir, Mustafa; Cooke, Nancy J.; Grimm, David

doi:10.1080/00140139.2018.1557750

Cited by 25 publications

(27 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Interactions with a clear sender and receiver (e.g., speech, chat messages, or any directed interaction). Gorman et al (2019); Amazeen (2018); Demir et al (2018) Approach: Emotional states…”

Section: Demonstrated Structural Differences Inmentioning

confidence: 99%

See 1 more Smart Citation

Approaches for assessing communication in human-autonomy teams

Baker

Fitzhugh

Huang

et al. 2021

Hum.-Intell. Syst. Integr.

Self Cite

View full text Add to dashboard Cite

Evaluation of team communication can provide critical insights into team dynamics, cohesion, trust, and performance on joint tasks. Although many communication-based measures have been tested and validated for human teams, this review article extends this research by identifying key approaches specific to human-autonomy teams. It is not possible to identify all approaches for all situations, though the following seem to generalize and support multi-size teams and a variety of military operations. Therefore, this article will outline several key approaches to assessing communication, associated data requirements, example applications, verification of methods through HAT use cases, and lessons learned, where applicable. Some approaches are based on the structure of team communication; others draw from dynamical systems theory to consider perspectives across different timescales; other approaches leverage features of team members’ voices or facial expressions to detect emotional states that can provide windows into other workings of the team; still others consider the content of communication to produce insights. Taken together, these approaches comprise a varied toolkit for deriving critical information about how team interactions affect, and are affected by, coordination, trust, cohesion, and performance outcomes. Future research directions describe four critical areas for further study of communication in human-autonomy teams.

show abstract

Section: Demonstrated Structural Differences Inmentioning

confidence: 99%

“…Note that the set of possible dimensions such as communication content may need to be condensed into more tractable units such as the information conveyed or themes. For a detailed example of this process applied to team communications data, see Gorman et al (2019).…”

Section: Quantifying Exploratory Communicationmentioning

confidence: 99%

Approaches for assessing communication in human-autonomy teams

Baker

Fitzhugh

Huang

et al. 2021

Hum.-Intell. Syst. Integr.

Self Cite

View full text Add to dashboard Cite

show abstract

“…This does not solve the complex adaptive systems problem, however, that team cognition is not reducible to its parts or factors. Due to the inherent complexities of these systems, changing one factor can impact other aspects of team cognition in unforeseen ways (Gorman, Demir, Cooke, & Grimm, 2019). Therefore, we must consider how team cognition emerges from the simultaneous organization of individual-level inputs, team-level constraints, and environmental contingencies.…”

Section: Introductionmentioning

confidence: 99%

Measuring Real-Time Team Cognition During Team Training

et al. 2019

Self Cite

View full text Add to dashboard Cite

Objective A method for detecting real-time changes in team cognition in the form of significant communication reorganizations is described. We demonstrate the method in the context of scenario-based simulation training. Background We present the dynamical view that individual- and team-level aspects of team cognition are temporally intertwined in a team’s real-time response to challenging events. We suggest that this real-time response represents a fundamental team cognitive skill regarding the rapidity and appropriateness of the response, and methods and metrics are needed to track this skill. Method Communication data from medical teams (Study 1) and submarine crews (Study 2) were analyzed for significant communication reorganization in response to training events. Mutual information between team members informed post hoc filtering to identify which team members contributed to reorganization. Results Significant communication reorganizations corresponding to challenging training events were detected for all teams. Less experienced teams tended to show delayed and sometimes ineffective responses that more experienced teams did not. Mutual information and post hoc filtering identified the individual-level inputs driving reorganization and potential mechanisms (e.g., leadership emergence, role restructuring) underlying reorganization. Conclusion The ability of teams to rapidly and effectively reorganize coordination patterns as the situation demands is a team cognitive skill that can be measured and tracked. Application Potential applications include team monitoring and assessment that would allow for visualization of a team’s real-time response and provide individualized feedback based on team member’s contributions to the team response.

show abstract

“…Synergistic relationships among a system's human and technological components provide the basis for emergent systems-level outcomes. Layered dynamics, a recent empirical modelling technique aimed at achieving this objective (Gorman et al, 2019), considers reorganization of the sociotechnical system across individual components and the overall system. The current study examined layered dynamics of HATs during automation and autonomy failures and addresses how members of HATs interact with each other and technological aspects of the system.…”

mentioning

confidence: 99%

“…The primary manipulation was the application of three degraded conditions: (1) automation failure -role-level display failures, (2) autonomy failure -autonomous agent's abnormal behavior, and (3) malicious cyber-attacks -the hijaking of the RPAS, with the synthetic agent providing false, detrimental information. We symbolically represented RPAS using layered dynamics, and calculated entropy measures for each (Gorman et al, 2019): (1) communications: team members interacting within the chat system; (2) vehicle: states of the RPA, including airspeed/altitude, turns, fuel, battery, remaining film, and termperature level; and (3) controls: the interface controls between the RPA and the team members. To measure team performance, we used a time and coordination based metric for each target in each mission.…”

mentioning

confidence: 99%

Layered Dynamics and System Effectiveness of Human-Autonomy Teams Under Degraded Conditions

Grimm

Demir

Gorman

et al. 2019

Proceedings of the Human Factors and Ergonomics Society Annual

Self Cite

View full text Add to dashboard Cite

Project overview Teamwork can be defined as dynamic team interaction between two or more interdependent members to achieve a shared goal. Many studies have examined how coordination dynamics are associated with team effectiveness in the context of all-human teams (Gorman, Amazeen, & Cooke, 2010), and later, in human-autonomy teams (HAT)s (Demir, Likens, Cooke, Amazeen, & McNeese, 2018). HATs must have autonomous agents that act as effective teammates and help enable HATs to function as collaborative systems. Synergistic relationships among a system’s human and technological components provide the basis for emergent systems-level outcomes. Layered dynamics, a recent empirical modelling technique aimed at achieving this objective (Gorman et al., 2019), considers reorganization of the sociotechnical system across individual components and the overall system. The current study examined layered dynamics of HATs during automation and autonomy failures and addresses how members of HATs interact with each other and technological aspects of the system. Design and Method We utilized a simulated Remotely Piloted Aircrtaft System (RPAS) Synthetic Task Environment with three heteregeneous and interdepedent roles: (1) a navigator, who created a dynamic flight plan and provided waypoint related information; (2) a pilot, who used this information to monitor and adjust settings. The pilot also communicated with the photographer to negotiate settings and enable proper conditions to obtain a good photograph; and (3) a photographer, who monitored and adjusted the camera to take good target photos, and provided feedback to the team. This study utilized a Wizard of Oz paradigm, in which the navigator and photographer were instructed that the pilot was a synthetic agent. However, the pilot was a highly-trained experimenter, in a separate room, who simulated an autonomous agent using limited vocabulary. There were 22 teams, and two participants were randomly assigned to the navigator and photographer roles. This task was comprised of ten 40-minute missions, and teams needed to take as many good photos as possible while avoiding alarms and rule violations. The primary manipulation was the application of three degraded conditions: (1) automation failure - role-level display failures, (2) autonomy failure - autonomous agent’s abnormal behavior, and (3) malicious cyber-attacks - the hijaking of the RPAS, with the synthetic agent providing false, detrimental information. We symbolically represented RPAS using layered dynamics, and calculated entropy measures for each (Gorman et al., 2019): (1) communications: team members interacting within the chat system; (2) vehicle: states of the RPA, including airspeed/altitude, turns, fuel, battery, remaining film, and termperature level; and (3) controls: the interface controls between the RPA and the team members. To measure team performance, we used a time and coordination based metric for each target in each mission. Results and Discussion Our main findings were: 1) vehicle and communication entropy were higher than control entropy and were associated with better adaptation to both failures, and 2) control entropy had a negative association with initial status on team performance, while vehicle entropy had a positive association. These findings describe the tendency of low performing teams to anticipate targets poorly. This was due to a failure to interact with the technology in a timely manner. This lagged effect can be attributed to teams taking too long to interact with the technology. These findings shed light on how the layered dynamics approach can help understand team behavior under degraded conditions. Acknowledgements This research is supported by ONR Award N000141712382 (Program Managers: Marc Steinberg, Micah Clark). We also acknowledge the assistance of Steven M. Shope, Sandia Research Corporation who integrated the synthetic agent and the testbed.

show abstract

Evaluating sociotechnical dynamics in a simulated remotely-piloted aircraft system: a layered dynamics approach

Cited by 25 publications

References 48 publications

Approaches for assessing communication in human-autonomy teams

Approaches for assessing communication in human-autonomy teams

Measuring Real-Time Team Cognition During Team Training

Layered Dynamics and System Effectiveness of Human-Autonomy Teams Under Degraded Conditions

Contact Info

Product

Resources

About