Towards Better Trust in Human-Machine Teaming through Explainable Dependability

Abstract

The human-machine teaming paradigm is increasingly widespread in critical domains, such as healthcare and domestic assistance. The paradigm goes beyond human-on-the-loop and human-in-the-loop systems by promoting tight teamwork between humans and autonomous machines that collaborate in the same physical space. These systems are expected to build a certain level of trust by enforcing dependability and exhibiting interpretable behavior. We present emerging results in this direction, with a novel framework aiming at achieving better trust in human-machine teaming leveraging formal analysis, as well as eXplainable AI. We illustrate our approach and the emerging results with an example from the healthcare domain.

A Conceptual Framework for Explainability Requirements in Software-Intensive Systems

“…In recent years, explainability, seen as the ability to provide a human with understandable explanations of the results produced by AI and ML algorithms, has become an essential aspect of designing tools based on these techniques [1], especially in critical areas such as healthcare [26]. Even if explainability is a term coined in the area of AI, interest in it is also growing in the software engineering and requirement engineering communities [9], [25]; researchers in these communities have proposed, for example, explainable analytical models for predictions and decision-making [25], explainable counterexamples [14], explainable quality attribute trade-offs in software architecture selection [4], the analysis of explainability as a non-functional requirement and its tradeoff with other quality attributes [9], [15] and in relation to human-machine teaming [3]. Work describing the theoretical basis of explainability, exploiting concepts from philosophy, psychology, and sociology can be found, for example, in [8], [21], [22], [24].…”

“…The framework is complemented by a set of meta-requirements and means to be engineered within a system to make it capable of producing explanations. We illustrate the applicability of the proposed framework by instantiating its main conceptual aspects in a Human-Machine-Teaming (HMT) [3], [19] application scenario where service robots assist patients and hospital staff during daily operations.…”

“…The proposed framework leverages our preliminary work [3], [7] evaluating the feasibility of a software system for producing human-interpretable explanations in specific application domains. Specifically, in [7], we introduce our idea of explainable self-adaptation for endowing a self-adaptive system capable of providing human-understandable explanations for successful and unsuccessful adaptations in critical scenarios.…”

“…Specifically, in [7], we introduce our idea of explainable self-adaptation for endowing a self-adaptive system capable of providing human-understandable explanations for successful and unsuccessful adaptations in critical scenarios. In [3], we present an emerging idea for generating dependability-related explanations and partially evaluate it on the estimation of success/failure of service robot missions in the HMT domain [2]. We also conducted preliminary feasibility experiments in selected domains.…”