The effectiveness of uncertainty communication with varied automation reliability: Specificity is necessary but (potentially) not enough

Abstract

Automation reliance and functionality are ever increasing, especially in supervisory control environments like unmanned aerial vehicle (UAV) missions. Of particular relevance is understanding how automation transparency, i.e., explaining the capabilities and limitations of automation to the human in real-time, can improve human-automation performance across automated systems that vary in reliability. Two hundred seventy one Naval Aviation trainees completed a simulated multi-UAV supervisory control mission for 42 minutes with three automated systems that varied in reliability. Participants were never explicitly told the reliability varied, but halfway through the mission, they were alerted that the least reliable system may falter. Results indicated human-automation performance improved after the alert for this specific system, but not as a whole, as one system’s human-automation performance deteriorated. This work suggests uncertainty communication should not only include the specific, real-time capabilities of the automation, but also communicate unintentional consequences it may have on the whole environment.

The need for specificity in uncertainty communication is further supported: Preliminary evidence from eye tracking

“…So, for the systems not referenced by uncertainty communication, every metric in Table 1 would be predicted to decrease upon its delivery, as it does not doubt these systems' reliability (Bhaskara et al, 2021). However, we previously found performance did not improve for these other automated systems (Devlin et al, 2022), which may mean mental models were not better informed. Therefore, we predicted uncertainty communication to have no impact on the monitoring policy of the other reliable automated systems, i.e., there will be no change in any of the eye tracking metrics in Table 1.…”

“…As previously mentioned, the present work is based on the experiment detailed in Devlin et al (2022). However, it centers on the eye tracking analysis, which was not included in the previous work.…”

“…In line with the situation awareness-based agent transparency (SAT) model (Chen et al, 2014), we proposed that the system-wide approach may be further eliminated if the automation itself effectively communicates its real-time capabilities and limitations to the operator. As detailed in Devlin et al (2022), we assessed this by studying how human-automation performance changed upon the delivery of uncertainty communication , i.e., when “doubts” about the automation’s ability to complete its assigned task(s) are provided to the human in the environment (Kunze et al, 2019). Uncertainty communication is an example of automation transparency , which is defined as providing additional information about the automation’s output to the operator in a real-time, obvious, and interpretable format (Billings, 1997; Christoffersen & Woods, 2002).…”

“…In this work, we build on our most recent findings by studying the effect uncertainty communication had on the monitoring policy of automated systems differing in reliability and associating it with the previously observed human-automation performance outcomes. The ultimate goal is to better inform the details surrounding our final conclusion in that work, i.e., uncertainty communication needs to be specific to be effective (Devlin et al, 2022). Human-automation interaction research has previously studied a range of eye tracking metrics, but three metrics were of present interest given they have been particularly informative on participants’ ability to correct automation failures (Foroughi et al, 2023, p. 548).…”

“…Previous research has found performance improvements are associated with monitoring policies that focus in on the system referenced by uncertainty communication (Bruder & Hasse, 2020; Kunze et al, 2019). Given Devlin et al (2022) found that performance improved for the referenced system, we predicted every metric in Table 1 to increase for this system once uncertainty communication was delivered. Analogously, our second research question was: How does uncertainty communication about a specific automated system impact the monitoring policy of the other automated systems, and how does it further explain to the previously observed human-automation performance outcomes?…”