Sociotechnical Envelopment of Artificial Intelligence: An Approach to Organizational Deployment of Inscrutable Artificial Intelligence Systems

Abstract

The paper presents an approach for implementing inscrutable (i.e., nonexplainable) artificial intelligence (AI) such as neural networks in an accountable and safe manner in organizational settings. Drawing on an exploratory case study and the recently proposed concept of envelopment, it describes a case of an organization successfully “enveloping” its AI solutions to balance the performance benefits of flexible AI models with the risks that inscrutable models can entail. The authors present several envelopment methods—establishing clear boundaries within which the AI is to interact with its surroundings, choosing and curating the training data well, and appropriately managing input and output sources—alongside their influence on the choice of AI models within the organization. This work makes two key contributions: It introduces the concept of sociotechnical envelopment by demonstrating the ways in which an organization’s successful AI envelopment depends on the interaction of social and technical factors, thus extending the literature’s focus beyond mere technical issues. Secondly, the empirical examples illustrate how operationalizing a sociotechnical envelopment enables an organization to manage the trade-off between low explainability and high performance presented by inscrutable models. These contributions pave the way for more responsible, accountable AI implementations in organizations, whereby humans can gain better control of even inscrutable machine-learning models.

Organizational Capabilities for AI Implementation—Coping with Inscrutability and Data Dependency in AI

“…Learning refers to the AI system's ability to improve through data and experience (Ågerfalk, 2020;Janiesch et al, 2021). Inscrutability refers to the unintelligibility of AI systems to some audiences, given their complex inner workings and probabilistic outputs (Asatiani et al, 2021;Jöhnk et al, 2021). These characteristics are expected to even exacerbate as the field of AI moves forward and new techniques and approaches emerge.…”

“…We propose that the identified capabilities help organizations cope with two characteristics in AI: inscrutability and data dependency. Inscrutability refers to the unintelligibility of AI systems to some audiences due to their complex inner workings and probabilistic nature (Asatiani et al, 2021;Berente et al, 2021;Jöhnk et al, 2021). Data dependency refers to the high dependence of AI systems on the underlying data, as these systems are typically built by learning and generalizing from data (Ågerfalk, 2020;Berente et al, 2021;Janiesch et al, 2021).…”

“…On the other hand, business functions require transparency and explanations from the IT department regarding the inner functioning, performance, and limitations of the developed AI system (Watson, 2017). Business functions need to understand how an AI system works to deal with its potential limitations (Asatiani et al, 2021). Explaining the workings of AI systems can also help address potential user resistance caused by misconceptions about the developed AI system (Reis et al, 2020).…”

“…Second, the data needs to be appropriately curated and prepared for AI implementation (Asatiani et al, 2021). In other words, data needs to adhere to specific quality standards that need to be defined (Jöhnk et al, 2021;.…”

How to explain AI systems to end users: a systematic literature review and research agenda

“…Previous research has delineated several potential barriers to the explainability of AI systems, including technical challenges (Anjomshoae et al, 2019), limitations of human logic (Asatiani et al, 2020(Asatiani et al, , 2021 and even intentional secrecy (Burrell, 2016). However, even with full explanations, the issue of how to communicate about the AI system to end users remains a challenge (Brennen, 2020).…”