A Naturalistic Investigation of Trust, AI, and Intelligence Work

Abstract

Artificial Intelligence (AI) is often viewed as the means by which the intelligence community will cope with increasing amounts of data. There are challenges in adoption, however, as outputs of such systems may be difficult to trust, for a variety of factors. We conducted a naturalistic study using the Critical Incident Technique (CIT) to identify which factors were present in incidents where trust in an AI technology used in intelligence work (i.e., the collection, processing, analysis, and dissemination of intelligence) was gained or lost. We found that explainability and performance of the AI were the most prominent factors in responses; however, several other factors affected the development of trust. Further, most incidents involved two or more trust factors, demonstrating that trust is a multifaceted phenomenon. We also conducted a broader thematic analysis to identify other trends in the data. We found that trust in AI is often affected by the interaction of other people with the AI (i.e., people who develop it or use its outputs), and that involving end users in the development of the AI also affects trust. We provide an overview of key findings, practical implications for design, and possible future areas for research.

Self-Repairing and/or Buoyant Trust in Artificial Intelligence

“…A naturalistic approach was used to understand how intelligence professionals gained or lost trust in AI. We provide an overview of methods particularly relevant to this analysis; however, a more exhaustive description of the overall study can be found in Dorton & Harper (2022).…”

Where the Cognitive Engineering Rubber Meets the Systems Development Road

“…The difficulty of modern technology development is evidenced by high rates of failed completion and transition (e.g., Dorton & Harper, 2022). To mitigate the difficulties and improve the likelihood of a successful transition, SD emphasizes decomposition of technologies into their mostbasic elements, clear and concise bite-sized requirements and specifications, quantitative and objective criteria, and minimal changes in design specifications across the development timeline; in other words, the removal and avoidance of complexity.…”

“…Developers of AI often conflate algorithmic performance with trustworthiness, ignoring the complexities of the sociotechnical system in which the AI will reside. There are dozens of factors affecting trust in AI, where explainability and predictability play prominent roles (Dorton & Harper, 2022). To that end, we have had considerable success in employing naturalistic CE methods to design intelligent systems to be both explainable and predictable.…”

The TRUSTS Work System Resilience Framework: A Foundation for Resilience-Aware Development and Transition

“…We define successful transition as encompassing both a technology's integration into and positive contributions to a work system. Successful transition has been a challenge for decades (e.g., Dorton & Harper, 2022;Dwived, et al, 2015;Kheybari et al, 2020;Neville et al, 2008). Even when defined simply as completed technology that meets contractual requirements, the statistics are concerning.…”