In-Time Explainability in Multi-Agent Systems: Challenges, Opportunities, and Roadmap

Abstract

In the race for automation, distributed systems are required to perform increasingly complex reasoning to deal with dynamic tasks, often not controlled by humans. On the one hand, systems dealing with strict-timing constraints in safety-critical applications mainly focused on predictability, leaving little room for complex planning and decisionmaking processes. Indeed, real-time techniques are very efficient in predetermined, constrained, and controlled scenarios. Nevertheless, they lack the necessary flexibility to operate in evolving settings, where the software needs to adapt to the changes of the environment. On the other hand, Intelligent Systems (IS) increasingly adopted Machine Learning (ML) techniques (e.g., subsymbolic predictors such as Neural Networks). The seminal application of those IS started in zero-risk domains producing revolutionary results. However, the ever-increasing exploitation of ML-based approaches generated opaque systems, which are nowadays no longer socially acceptable-calling for eXplainable AI (XAI). Such a problem is exacerbated when IS tend to approach safety-critical scenarios. This paper highlights the need for on-time explainability. In particular, it proposes to embrace the Real-Time Beliefs Desires Intentions (RT-BDI) framework as an enabler of eXplainable Multi-Agent Systems (XMAS) in time-critical XAI.

Building XAI-Based Agents for IoT Systems

“…Some AI and ML application domains can be considered very sensitive and highrisk, closely associated with human lives, their wellness, and safety-industrial control, healthcare, military domain, or self-driving cars perfectly represent instances of such safetycritical domains. Most authors agree that the demand for XAI in safety-critical domains is inseparable from further developments, as they involve high-consequence decisions, and a failure of the AI system could result in significant harm to people or the environment [7][8][9].…”

“…Experiments have shown that XAI principles implemented through conversational and virtual agents contribute to increasing user trust in the XAI system [17]. The authors in [9] discuss the concept of real-time multi-agent systems and how they are designed to operate in highly dynamic environments, highlighting the importance of meeting both soft and hard deadlines. The paper also explores the use of BDI-based agents, which are suited for unpredictable scenarios requiring dynamic decision-making, and how they can be used to develop explainable and real-time compliant MAS.…”

“…The paper also explores the use of BDI-based agents, which are suited for unpredictable scenarios requiring dynamic decision-making, and how they can be used to develop explainable and real-time compliant MAS. Additionally, the research to date on explainable multi-agent systems comes under various definitions, namely "XMAS" [9], "XMASE" [19], "Explainable agents" [20], "Explainable Agency" [21], or goal-driven XAI [20].…”

IEEE P7001: A Proposed Standard on Transparency

“…Explaining AI applications, especially those involving Machine Learning (ML) ( Holzinger, 2018 ), and Deep Neural Networks (DNN) ( Angelov and Soares, 2020 ; Booth et al, 2021 ), is howbeit still an ongoing effort, due to the high complexity and sophistication of the processes in place (e.g., data handling, algorithm tuning, etc.) as well as the wide range of AI systems such as recommendation systems ( Zhang and Chen, 2020 ), human-agent systems ( Rosenfeld and Richardson, 2019 ), planning systems ( Chakraborti et al, 2020 ), multi-agent systems ( Alzetta et al, 2020 ), autonomous systems ( Langley et al, 2017 ), or robotic systems ( Anjomshoae et al, 2019 ; Rotsidis et al, 2019 ).…”

Explainable Artificial Intelligence for Developing Smart Cities Solutions

“…In the literature, the explainable AI presents some challenges in terms of it being used as a selective decision making that focuses on explanations and background knowledge [127], a large amount of information [128] and using case-specific decision making [129]. Moreover, we have faced other challenges as well in this work which might be useful to other researchers working on utilising Explainable AI for developing Smart Cities' solutions.…”