Joint Activity Testing: Towards a Multi-Dimensional, High-Resolution Evaluation Method for Human-Machine Teaming

Abstract

Quantitative evaluations of human-machine teams (HMTs) are desperately needed to ensure technological implementations are helpful rather than harmful to overall system performance; however, as machines increasingly behave like active cognitive teammates, traditional evaluation strategies risk overestimating HMT capabilities. Areliable HMT evaluation method should include multiple high-resolution, continuous measures for both system performance and system challenges that can be implemented unobtrusively in real-time operations. In our prior work, we proposed joint activity testing (JAT) as acandidate evaluation framework to satisfy these requirements. Preliminary efforts with asingle dimension of performance and challenge have indicated that the method can identify the additive benefits of joint activity with aspecific technology. In this paper, we explore the operationalization of multi-dimensional JAT by synthesizing our work in two intelligence and two healthcare domains. The patterns observed between domains will guide future JAT, reveal paths towards real-time implementation, and spark future research evaluating resilience.

Computer-assisted approaches for measuring, segmenting, and analyzing functional upper extremity movement: a narrative review of the current state, limitations, and future directions

“…System-level user requirement studies can also inform the integration process ( 147 , 148 ). Additionally, cognitive systems engineering research could be an area that provides valuable quantitative evaluations on how computational tools integrate into domain expert workflows, such as the recently proposed joint activity testing framework ( 152 ). The need for more user-friendly kinematics measurement, segmentation, and analysis methods, as well as investigating how to integrate kinematic analyses into domain expert workflows (i.e., human factors), underscores the multidisciplinary approach required to meaningfully improve the quality and administration of UEFAs.…”

Incorporating resilience engineering into simulation for health care education and training

“…JAT has now also been used to evaluate and compare the performance of teams with and without AI or advanced analytics, and has proved to be valuable even in high-stakes, high-uncertainty settings with only a small number (10-15) of participants (Morey et. al., 2020;Morey et. al.…”