Machine Learning Interpretability: A Survey on Methods and Metrics

Abstract

Machine learning systems are becoming increasingly ubiquitous. These systems’s adoption has been expanding, accelerating the shift towards a more algorithmic society, meaning that algorithmically informed decisions have greater potential for significant social impact. However, most of these accurate decision support systems remain complex black boxes, meaning their internal logic and inner workings are hidden to the user and even experts cannot fully understand the rationale behind their predictions. Moreover, new regulations and highly regulated domains have made the audit and verifiability of decisions mandatory, increasing the demand for the ability to question, understand, and trust machine learning systems, for which interpretability is indispensable. The research community has recognized this interpretability problem and focused on developing both interpretable models and explanation methods over the past few years. However, the emergence of these methods shows there is no consensus on how to assess the explanation quality. Which are the most suitable metrics to assess the quality of an explanation? The aim of this article is to provide a review of the current state of the research field on machine learning interpretability while focusing on the societal impact and on the developed methods and metrics. Furthermore, a complete literature review is presented in order to identify future directions of work on this field.

Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making

“…Explainable Artificial Intelligence is a rapidly growing research discipline [1,4,11,22]. The quest for explainability has its roots in the growing adoption of high-performance "black-box" AI models, which spurs public concerns about the safety and ethical usage of AI.…”

“…To improve people's distrust in ML models, many considered the importance of transparency by providing explanations for the ML model [4,9,28]. In particular, local explanations that explain the rationale for a single prediction (in contrast to global explanations describing the overall logic of the model) are recommended to help people judge whether to trust a model on a case-by-case basis [28].…”

XGBoost-Based Framework for Smoking-Induced Noncommunicable Disease Prediction

“…Meanwhile, predictive model interpretability concerns the understanding of model decisions by humans. Interpretability methods can be categorized into three types: explain data, build an inherently interpretable model (in modeling), and allow to explain it after building the models [ 43 ]. In practice, there have been some needs for using machine learning models to ensure which factors are used to make key decisions with boosted trees [ 44 ].…”

Combining deep learning with token selection for patient phenotyping from electronic health records

“…In our case, this means using all words and sentences in discharge summary notes provides us with a mechanism for categorizing disorders, however, without pointing to individual words as the main discriminators. In order to compensate for this shortcoming we performed token selection (for explainable AI this is a simple mechanism to obtain better interpretations for models 60 ). Interestingly, we found that, independent of the disorder, a substantial amount of tokens can be removed without a deteriorating classification performance (see Fig.…”