Challenges of Explaining the Behavior of Black-Box AI Systems

Asatiani, Aleksandre; Malo, Pekka; Nagbøl, Per Rådberg; Penttinen, Esko; Rinta-Kahila, Tapani; Salovaara, Antti

doi:10.17705/2msqe.00037

Cited by 59 publications

(43 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The services of AI virtual assistants are executed based on AI algorithms, but users do not fully trust the information or services provided by AI virtual assistants due to the inherent blackbox problem of AI technology. Existing studies suggest that trust is an essential driver of technology acceptance and can positively influence users' acceptance of new technologies (Kaplan and Haenlein, 2019;van Pinxteren et al, 2019;Asatiani et al, 2020). This paper confirms through empirical research that trust is significantly and positively correlated with AI virtual assistant acceptance, and that its ability to reduce users' negative emotions toward AI virtual assistants plays a key role in improving AI virtual assistant acceptance.…”

Section: Discussionsupporting

confidence: 74%

“…Trust is defined as the user's confidence that the AI virtual assistant can reliably deliver a service (Wirtz et al, 2018). The services of AI virtual assistants are based on artificial intelligence algorithms, but due to the inherent black box problem of artificial intelligence technology (Asatiani et al, 2020), users will not fully trust the information or services provided by artificial intelligence virtual assistants (Kaplan and Haenlein, 2019). The existing research shows that only meeting the technical and social needs of users does not truly increase their loyalty to AI virtual assistants (Hassanein and Head, 2007).…”

Section: Research Framework and Hypothesis Developmentmentioning

confidence: 99%

See 1 more Smart Citation

Motivation, Social Emotion, and the Acceptance of Artificial Intelligence Virtual Assistants—Trust-Based Mediating Effects

et al. 2021

View full text Add to dashboard Cite

The complexity of the emotional presentation of users to Artificial Intelligence (AI) virtual assistants is mainly manifested in user motivation and social emotion, but the current research lacks an effective conversion path from emotion to acceptance. This paper innovatively cuts from the perspective of trust, establishes an AI virtual assistant acceptance model, conducts an empirical study based on the survey data from 240 questionnaires, and uses multilevel regression analysis and the bootstrap method to analyze the data. The results showed that functionality and social emotions had a significant effect on trust, where perceived humanity showed an inverted U relationship on trust, and trust mediated the relationship between both functionality and social emotions and acceptance. The findings explain the emotional complexity of users toward AI virtual assistants and extend the transformation path of technology acceptance from the trust perspective, which has implications for the development and design of AI applications.

show abstract

Section: Discussionsupporting

confidence: 74%

Section: Research Framework and Hypothesis Developmentmentioning

confidence: 99%

Motivation, Social Emotion, and the Acceptance of Artificial Intelligence Virtual Assistants—Trust-Based Mediating Effects

et al. 2021

View full text Add to dashboard Cite

show abstract

“…This trade-off raises questions regarding individuals' freedom of choice (Zuboff, 2015(Zuboff, , 2019. Second, some advanced algorithms operate as a black box, hiding their inner workings and decision-making processes from human decision-makers (Asatiani et al, 2020(Asatiani et al, , 2021Faraj et al, 2018). This opacity leads to a difficulty in establishing accountability, assessing the accuracy and robustness of output generated, and a lack of trust in such technologies (Goldenfein, 2019;de Laat, 2018).…”

Section: Algorithmic Decision-making In Organisationsmentioning

confidence: 99%

“…Further, many complex algorithmic models lack transparency, which makes their operating logic hard to understand (Faraj et al, 2018). Such opacity requires organisations to anticipate potential unintended effects and put various safety measures in place to prevent them (Asatiani et al, 2020(Asatiani et al, , 2021.…”

Section: Introductionmentioning

confidence: 99%

Algorithmic decision-making and system destructiveness: A case of automatic debt recovery

Rinta-Kahila

Someh

Gillespie

et al. 2021

European Journal of Information Systems

Self Cite

View full text Add to dashboard Cite

Governments are increasingly relying on algorithmic decision-making (ADM) to deliver public services. Recent information systems literature has raised concerns regarding ADM's negative unintended consequences, such as widespread discrimination, which in extreme cases can be destructive to society. The extant empirical literature, however, has not sufficiently examined the destructive effects of governmental ADM. In this paper, we report on a case study of the Australian government's "Robodebt" programme that was designed to automatically calculate and collect welfare overpayment debts from citizens but ended up causing severe distress to citizens and welfare agency staff. Employing perspectives from systems thinking and organisational limits, we develop a research model that explains how a socially destructive government ADM programme was initiated, sustained, and delegitimized. The model offers a set of generalisable mechanisms that can benefit investigations of ADM's consequences. Our findings contribute to the literature of unintended consequences of ADM and demonstrate to practitioners the importance of setting up robust governance infrastructures for ADM programmes.

show abstract

“…Another important trade-off arises in the form of accurate explanations versus comprehensible explanations; the more accurate the explanation, the more incomprehensible it will be for AI-illiterate stakeholders. Even though it may be possible to use xAI frameworks to accurately depict the model output or how the model produced the output, this description may still be incomprehensible to AIilliterate stakeholders [24], [51], [52]. While these dilemmas have been acknowledged by scholars, there is a limited understanding of how they are dealt with in practice.…”

Section: Emergence Of Xaimentioning

confidence: 99%

Explainable AI, But Explainable to Whom? An Exploratory Case Study of xAI in Healthcare

Gerlings

Jensen

Shollo

2021

Intelligent Systems Reference Library

View full text Add to dashboard Cite

Advances in AI technologies have resulted in superior levels of AI-based model performance. However, this has also led to a greater degree of model complexity, resulting in "black box" models. In response to the AI black box problem, the field of explainable AI (xAI) has emerged with the aim of providing explanations catered to human understanding, trust, and transparency. Yet, we still have a limited understanding of how xAI addresses the need for explainable AI in the context of healthcare. Our research explores the differing explanation needs amongst stakeholders during the development of an AI-system for classifying COVID-19 patients for the ICU. We demonstrate that there is a constellation of stakeholders who have different explanation needs, not just the "user." Further, the findings demonstrate how the need for xAI emerges through concerns associated with specific stakeholder groups i.e., the development team, subject matter experts, decision makers, and the audience. Our findings contribute to the expansion of xAI by highlighting that different stakeholders have different explanation needs. From a practical perspective, the study provides insights on how AI systems can be adjusted to support different stakeholders needs, ensuring better implementation and operation in a healthcare context.

show abstract

Challenges of Explaining the Behavior of Black-Box AI Systems

Cited by 59 publications

References 0 publications

Motivation, Social Emotion, and the Acceptance of Artificial Intelligence Virtual Assistants—Trust-Based Mediating Effects

Motivation, Social Emotion, and the Acceptance of Artificial Intelligence Virtual Assistants—Trust-Based Mediating Effects

Algorithmic decision-making and system destructiveness: A case of automatic debt recovery

Explainable AI, But Explainable to Whom? An Exploratory Case Study of xAI in Healthcare

Contact Info

Product

Resources

About