Black box algorithms and the rights of individuals: no easy solution to the “explainability” problem

Abstract

Over the last few years, the interpretability of classification models has been a very active area of research. Recently, the concept of interpretability was given a more specific legal context. In 2016, the EU adopted the General Data Protection Regulation (GDPR), containing the right to explanation for people subjected to automated decision-making (ADM). The regulation itself is very reticent about what such a right might imply. As a result, since the introduction of the GDPR there has been an ongoing discussion about not only the need to introduce such a right, but also about its scope and practical consequences in the digital world. While there is no doubt that the right to explanation may be very difficult to implement due to technical challenges, any difficulty in explaining how algorithms work cannot be considered a sufficient reason to completely abandon this legal safeguard. The aim of this article is twofold. First, to demonstrate that the interpretability of "black box" machine learning algorithms is a challenging technical problem for which no solutions have been found. Second, to demonstrate how the explanation task should instead be completed using well-known and well-trialled IT solutions, such as event logging or statistical analysis of the algorithm. Based on the evidence exposed in this paper, the authors find that the most effective solution would be to benchmark the automated decision-making algorithms using certification frameworks, thus balancing the need to ensure adequate protection of individuals' rights with the understandable expectations of AI technology providers to have their intellectual property rights protected.

Survey of explainable artificial intelligence techniques for biomedical imaging with deep neural networks

“…European Union (EU) General Data Protection Regulation (GDPR) is legislating a "right to explanation" for EU citizens impacted by automated algorithmic decisions [49]. The EU legislation is likely to increase the focus of computer scientists towards developing algorithms and systems that will enable explanations and avoid discrimination [50].…”

“…Similarly, formal mathematical notations and techniques can unambiguously provide explanations for domain experts but are not targeted to the end users [63]. One of the ways forward could be based on the statistical correctness and fairness of the model [49].…”

“…Certification frameworks for a formal validation of models could testify the model operation statistically and ensure that the model predictions meet the required regulatory standards such as ISO/IEC CD 23053 (2020) for establishing a framework for use of AI systems [49]. An Independent, Accredited Certification Program for Responsible AI RAII Certification uses six dimensions of responsible AI (System Operations, Explainability and Interpretability, Accountability, Consumer Protection, Bias and Fairness, and Robustness) to represent trust that an AI system has been designed, built, and deployed in line with the five Organization for Economic Co-operation and Development (OECD) principles [65] on AI [66].…”

“…This has gained an increased importance due to the recent emphasis on the patients' role for data ownership and the need for providing a personalized healthcare. The patients require simplified explanations for the clinical decision as the DNN models are very complex and cannot be explained to a lay user [49]. Patients require assurance that the XAI methods can be trusted, and are providing a valid and consistent prediction.…”

“…Thus, more efforts should be placed on the models and datasets that are aligned for the intended purpose [57]. Certification of DNN models by agencies external to the developer or user could help to eliminate risk of unauthorized changes to how the system works and can prove beneficial in case of litigation to prove the correctness of the system [49]. The challenge against an automated DNN decision could thus not require specialized technical knowledge of the AI system [49].…”