Experience of automation failures in training: effects on trust, automation bias, complacency and performance

Abstract

This work examined the effects of operators' exposure to various types of automation failures in training. Forty-five participants were trained for 3.5 h on a simulated process control environment. During training, participants either experienced a fully reliable, automatic fault repair facility (i.e. faults detected and correctly diagnosed), a misdiagnosis-prone one (i.e. faults detected but not correctly diagnosed) or a miss-prone one (i.e. faults not detected). One week after training, participants were tested for 3 h, experiencing two types of automation failures (misdiagnosis, miss). The results showed that automation bias was very high when operators trained on miss-prone automation encountered a failure of the diagnostic system. Operator errors resulting from automation bias were much higher when automation misdiagnosed a fault than when it missed one. Differences in trust levels that were instilled by the different training experiences disappeared during the testing session. Practitioner Summary: The experience of automation failures during training has some consequences. A greater potential for operator errors may be expected when an automatic system failed to diagnose a fault than when it failed to detect one.

Introduction matters: Manipulating trust in automation and reliance in automated driving

“…In this study, we aim to transfer the mentioned findings to conditional automated driving and investigate if prior information determines the interaction with an automated driving system. Given the mentioned findings, automation that raises positive expectations of its performance (Schaefer, Chen, Szalma, & Hancock, 2016), signals high reliability (Mayer et al, 2006) and is experienced as reliable (Sauer et al, 2015) should increase subjective trust in automation. Based on this, we expect the following relationship:…”

“…This effect is greater when participants actually experienced automation failures during training than for only being instructed that the automated system may fail (Bahner, Hüper, & Manzey, 2008). Sauer, Chavaillaz, and Wastell (2015) found that the level of system reliability experienced during training influenced the trust in an automated system and the degree of automation bias in the following test session. In this study, we aim to transfer the mentioned findings to conditional automated driving and investigate if prior information determines the interaction with an automated driving system.…”

New Information Technology and Implicit Bias

“…A number of empirical studies suggest that people view new information technology that performs its functions in a "human-like" manner to be more trustworthy than information technology that is less human in its performance (e.g., Lankton et al, 2015;Montague, Winchester, & Kleiner, 2010;Sauer, Chavaillaz, & Wastell, 2016;Schaefer et al, 2016;Verberne, Ham, & Midden, 2012). In these studies, humanness in performance might include allowing input by humans, taking turns with humans in decision making, expressing shared goals with human operators, or exhibiting politeness and friendliness in responding to human users.…”

Shared Team Experiences and Team Effectiveness: Unpacking the Contingent Effects of Entrained Rhythms and Task Characteristics

“…Consistent with Wiener (1981), we conceptualize complacency as a psychological state of automaticity with affective and cognitive components. The cognitive component contributes a sense of predictability due to the assumption that "all is well" (Billings, Lauber, Funkhouser, Lyman, & Huff, 1976;Parasuraman & Manzey, 2010) and the affective component contributes a sense of comfort and trust (Sauer, Chavaillaz, & Wastell, 2016). Together, these components manifest in similar behaviors, such as lower levels of system monitoring, lower levels of learning, and reduced vigilance-type behaviors (Johnson, 2012).…”

“…Similarly, at the individual level, the minimization of complacency may aid in team efficiency, but may have detrimental effects on care providers' wellbeing. For example, scholars found that when complacency was induced via computer automation systems, it reduced the individual's cognitive workload, a well-known job demand (Sauer et al, 2016). Furthermore, depending on the other demands on the individual, complacency may serve as a recovery mechanism that ultimately helps the individual contribute to the team more in the future (e.g., Cegarra & Hoc, 2008).…”