Interpretable machine learning: definitions, methods, and applications

Abstract

Machine-learning models have demonstrated great success in learning complex patterns that enable them to make predictions about unobserved data. In addition to using models for prediction, the ability to interpret what a model has learned is receiving an increasing amount of attention. However, this increased focus has led to considerable confusion about the notion of interpretability. In particular, it is unclear how the wide array of proposed interpretation methods are related, and what common concepts can be used to evaluate them.We aim to address these concerns by defining interpretability in the context of machine learning and introducing the Predictive, Descriptive, Relevant (PDR) framework for discussing interpretations. The PDR framework provides three overarching desiderata for evaluation: predictive accuracy, descriptive accuracy and relevancy, with relevancy judged relative to a human audience. Moreover, to help manage the deluge of interpretation methods, we introduce a categorization of existing techniques into model-based and post-hoc categories, with sub-groups including sparsity, modularity and simulatability. To demonstrate how practitioners can use the PDR framework to evaluate and understand interpretations, we provide numerous real-world examples. These examples highlight the often under-appreciated role played by human audiences in discussions of interpretability. Finally, based on our framework, we discuss limitations of existing methods and directions for future work. We hope that this work will provide a common vocabulary that will make it easier for both practitioners and researchers to discuss and choose from the full range of interpretation methods. * For clarity, throughout the paper we use the term to refer to both machine-learning models and algorithms.accuracy, and relevancy, where relevancy is judged by a human audience. Using these terms, we categorize a broad range of existing methods, all grounded in real-world examples † . In doing so, we provide a common vocabulary for researchers and practitioners to use in evaluating and selecting interpretation methods. We then show how our work enables a clearer discussion of open problems for future research.A. Defining interpretable machine learning. On its own, interpretability is a broad, poorly defined concept. Taken to its full generality, to interpret data means to extract information (of some form) from it. The set of methods falling under this umbrella spans everything from designing an initial experiment to visualizing final results. In this overly general form, interpretability is not substantially different from the established concepts of data science and applied statistics.Instead of general interpretability, we focus on the use of interpretations in the context of ML as part of the larger datascience life cycle. We define interpretable machine learning as the use of machine-learning models for the extraction of relevant knowledge about domain relationships contained in data. Here, we view knowledge as being relevant if it pr...

Shallow decision trees for explainable $k$-means clustering

“…Explainability and interpretability are topics of growing interest in the machine learning community [Ribeiro et al, 2016, Lundberg and Lee, 2017, Adadi and Berrada, 2018, Rudin, 2019, Murdoch et al, 2019, Molnar, 2020. While there has been some focus on what Dasgupta et al [2020] calls postmodeling explainability, or the ability to explain the output of a black-box model [Ribeiro et al, 2016, Lundberg and Lee, 2017, Kauffmann et al, 2019, the practice has also been criticized in contrast with pre-modelling explainability, or the use of interpretable models to begin with [Rudin, 2019].…”

Towards Trustworthy Deception Detection: Benchmarking Model Robustness across Domains, Modalities, and Languages

“…Deep learning systems have been adopted in many areas from medicine to autonomous driving (Ahmad et al, 2018;Claybrook and Kildare, 2018) and as these algorithms are incorporated, the need for explainable and transparent models becomes more urgent. One approach researchers use to overcome the inherent ambiguity of these black-box methods is to develop additional models to learn and explain the decisions of existing models, analyze when these models fail, and introduce a human-in-the-loop component to improve performance (Ribeiro et al, 2016;Murdoch et al, 2019;Poursabzi-Sangdeh et al, 2018). Another way to tackle this challenge is to design models with a goal of interpretability in place when development starts (Ridgeway et al, 1998;Rudin, 2018;Gilpin et al, 2018;Lahav et al, 2018;Hooker et al, 2019).…”

How to Find a Good Explanation for Clustering?

“…While post-modeling explainability focuses on giving reasoning behind decisions made by black box models, pre-modeling explainability deals with ML systems that are inherently understandable or perceivable by humans. One of the canonical approaches to pre-modelling explainability builds on decision trees [35,37]. In fact, a significant amount of work on explainable clustering is based on unsupervised decision trees [3,17,20,21,29,36].…”