Automation bias with a conversational interface: User confirmation of misparsed information

Abstract

We investigate automation bias for confirming erroneous information with a conversational interface. Participants in our studies used a conversational interface to report information in a simulated intelligence, surveillance, and reconnaissance (ISR) task. In the task, for flexibility and ease of use, participants reported information to the conversational agent in natural language. Then, the conversational agent interpreted the user's reports in a human-and machine-readable language. Next, participants could accept or reject the agent's interpretation. Misparses occur when the agent incorrectly interprets the report and the user erroneously accepts it. We hypothesize that the misparses naturally occur in the experiment due to automation bias and complacency because the agent interpretation was generally correct (92%). These errors indicate some users were unable to maintain situation awareness using the conversational interface. Our results illustrate concerns for implementing a flexible conversational interface in safety critical environments (e.g., military, emergency operations).

The human side of human-chatbot interaction: A systematic literature review of ten years of research on text-based chatbots

“…These strategies were preferred because they explicitly acknowledge a potential breakdown, manifest initiative from the chatbot, and are actionable to recover from the breakdown. Zaroukian et al (2017), instead, identify an automation bias that may prevent the conversation with a chatbot to be successful. This phenomenon occurs when the chatbot misunderstands the user's request, providing her with an incorrect answer, and, nonetheless, the user exhibits a complacency behavior.…”

Don't Explain without Verifying Veracity: An Evaluation of Explainable AI with Video Activity Recognition

“…Another issues for consideration is whether users with Low Veracity Exp suffer from automation bias [8,13,78,84,99]. Automation bias is often times referred to as over-reliance on automated outputs [8], over-trusting output from automated systems even when it is wrong [47,84].…”

The future role of human operators in highly automated electric power systems

“…Experiences and developments on SA in the automotive and aviation sectors also hold important findings [73,74,75]. Case studies on SA in control rooms provide insight on workflows in the mentioned domains as documented in [76,20,24,77,69,17,18,35,78,79].…”