Novel Human-in-the-Loop (HIL) Simulation Method to Study Synthetic Agents and Standardize Human–Machine Teams (HMT)

Abstract

This work presents a multi-year study conducted at the University of Toledo, aimed at improving human–machine teaming (HMT) methods and technologies. With the advent of artificial intelligence (AI) in 21st-century machines, collaboration between humans and machines has become highly complicated for real-time applications. The penetration of intelligent and synthetic assistants (IA/SA) in virtually every field has opened up a path to the area of HMT. When it comes to crucial tasks such as patient treatment/care, industrial production, and defense, the use of non-standardized HMT technologies may pose a risk to human lives and cost billions of taxpayer dollars. A thorough literature survey revealed that there are not many established standards or benchmarks for HMT. In this paper, we propose a method to design an HMT based on a generalized architecture. This design includes the development of an intelligent collaborative system and the human team. Followed by the identification of processes and metrics to test and validate the proposed model, we present a novel human-in-the-loop (HIL) simulation method. The effectiveness of this method is demonstrated using two controlled HMT scenarios: Emergency care provider (ECP) training and patient treatment by an experienced medic. Both scenarios include humans processing visual data and performing actions that represent real-world applications while responding to a Voice-Based Synthetic Assistant (VBSA) as a collaborator that keeps track of actions. The impact of various machines, humans, and HMT parameters is presented from the perspective of performance, rules, roles, and operational limitations. The proposed HIL method was found to assist in standardization studies in the pursuit of HMT benchmarking for critical applications. Finally, we present guidelines for designing and benchmarking HMTs based on the case studies’ results analysis.

Joint human and autonomy teaming for defense: status, challenges, and perspectives

“…In accordance with different literature, such as, 38 , 39 and, 40 we highlight and identify different key components and relevant aspects of the integration of HAT in applied settings. Additionally, HAT components and aspects can also serve as a guide for human-autonomy interaction/teaming experiments to inform appropriate methodologies.…”

Human–Agent Team Dynamics: A Review and Future Research Opportunities

“…Li, et al [72] further substantiate this by showing that complementary policy adaptation enhances team performance. Damacharla, et al [73] have outlined guidelines for effective HAT composition, including role identification, rule establishment, and specialized training for human members, etc.…”

“…Common models often look at individual contributions, the overall team setting, and both individual and collective group performance [73]. Our review has identified as many as 28 different dimensions used to measure HAT performance, including commonly used measurements such as team effectiveness, task performance, mission performance, mission success, human workload, average efficiency, and performance perception.…”

“…This highlights the need for the development of standardized evaluation methods to facilitate effective learning and ongoing improvement in HATs. Damacharla, et al [73] suggest that a well-rounded evaluation of HATs should include three types of metrics: those that measure human performance, machine performance, and overall team performance.…”

The Development of a Service System for Facilitating Food Resource Allocation and Service Exchange

“…Recent achievements in natural language processing have allowed the design of human-machine interaction systems through natural language. Some research directions related to the human-inthe-hoop stream can be mentioned, such as improving collaboration between humans and machines [56] and the interaction between humans and agents in the context of a smart city [57].…”