Predicting Graceful Extensibility of Human-Machine Systems: A New Analysis Method for Evaluating Extensibility Plots to Anticipate Distributed System Performance

Abstract

With all of the research and investment dedicated to artificial intelligence and other automation technologies, there is a paucity of evaluation methods for how these technologies integrate into effective joint human-machine teams. Current evaluation methods, which largely were designed to measure performance of discrete representative tasks, provide little information about how the system will perform when operating outside the bounds of the evaluation. We are exploring a method of generating Extensibility Plots, which predicts the ability of the human-machine system to respond to classes of challenges at intensities both within and outside of what was tested. In this paper we test and explore the method, using performance data collected from a healthcare setting in which a machine and nurse jointly detect signs of patient decompensation. We explore the validity and usefulness of these curves to predict the graceful extensibility of the system.

From Reactive to Proactive Safety: Joint Activity Monitoring for Infection Prevention

“…Motivate actionable and feasible interventions Joint Activity Monitoring (JAM) addresses each of these requirements to begin or sustain a proactive safety program. It extends our recent work on Joint Activity Testing (JAT), where periodic human-on-the-loop simulations are used to test system response (Morey et al, 2020), to be a continuous and low-cost process (i.e., requirements 1-2). JAM leverages continuous unobtrusive monitoring of daily operations to identify troubling trends or targeted vulnerabilities.…”

“…It does this both through instrumenting the work tools of relevant operators (Murphy et al, 2017) and collecting environmental data. Through similar mechanisms to Joint Activity Testing (JAT), JAM facilitates analyzing the relationships between these data to understand how the system is sustaining through current demands, trending over time, and potentially vulnerable to future scenarios (Morey et al, 2020). JAM extends JAT by tracking these critically important performance measures over time, paying particular attention to trends that predict sustained system performance and decompensation events.…”

Incorporating resilience engineering into simulation for health care education and training

“…The objectives were to explicitly determine (1) the performance contribution (positive or negative) of one solution's JAA relative to that of another, and (2) the predicted results of all observed solutions when subjected to challenges outside of the testing set. JAT has now also been used to evaluate and compare the performance of teams with and without AI or advanced analytics, and has proved to be valuable even in high-stakes, high-uncertainty settings with only a small number (10-15) of participants (Morey et. al., 2020;Morey et.…”

Joint Activity Testing: Towards a Multi-Dimensional, High-Resolution Evaluation Method for Human-Machine Teaming

“…As a proof of concept, Morey (2021) demonstrated the feasibility of this method, which we are calling joint activity testing (JAT), with a case study including one dimension of performance and one dimension of challenge. However, fully realizing JAT will require integrating multiple metrics of performance and multiple classes of challenges to better reflect the complexity of systems (Morey et al, 2020).…”

“…In this paper, we extend the work of Morey et al (2020) to move towards multidimensional JAT. Through a series of vignettes, we discuss how we have operationalized multiple continuous dimensions of performance and challenge in four domains (two in intelligence analysis and two in healthcare).…”