Explainable Machine Learning in Deployment

Abstract

Explainable machine learning seeks to provide various stakeholders with insights into model behavior via feature importance scores, counterfactual explanations, and influential samples, among other techniques. Recent advances in this line of work, however, have gone without surveys of how organizations are using these techniques in practice. This study explores how organizations view and use explainability for stakeholder consumption. We find that the majority of deployments are not for end users affected by the model but for machine learning engineers, who use explainability to debug the model itself. There is a gap between explainability in practice and the goal of public transparency, since explanations primarily serve internal stakeholders rather than external ones. Our study synthesizes the limitations with current explainability techniques that hamper their use for end users. To facilitate end user interaction, we develop a framework for establishing clear goals for explainability, including a focus on normative desiderata.

How to choose an Explainability Method? Towards a Methodical Implementation of XAI in Practice

“…The question of practically using machine learning explainability has been poorly covered in the existing literature. One notable exception is Bhatt et al [3], which study the use of explanation methods in practice and thus show that, currently, they are mainly used by machine learning engineers in an ad hoc way as sanity checks for the models they build and deploy. One reason identified by the authors is that organizations lack frameworks for making decisions regarding explainability, leaving these methods only understandable by people with a background in machine learning and obscure to others.…”

“…Typical categorizations of stakeholders are based on their role in an organization [3,9,12,16], their machine learning experience [18] or a combination of the two [15]. Also for the categorization of the stakeholder needs regarding explainability, different propositions are made in the literature.…”

“…Some authors mention possible high-level goals of explainability, such as model debugging, monitoring etc. [3] or revealing (un)known (un)knowns [12]. Langer et al [9] provide a list of more detailed needs such as privacy, fairness, legal compliance, etc.…”

“…There is thus a risk to fail to achieve a transparent use of interpretability, which would be in contradiction with the initial objective of enlightening and empowering model's stakeholders. This issue has been noticed in previous works [3,9], but a concrete methodology is lacking in the existing body of literature. The goal of this paper is to make an argument for the need of prescriptive guidance to match stakeholder needs with an appropriate explanation method.…”

Unpacking the Black Box: Regulating Algorithmic Decisions

“…We add to a growing literature in computer science that studies algorithmic audits and derives specific explainability techniques from axioms about their deployment-agnostic properties (e.g. Bhatt et al, 2020;Carvalho et al, 2019;Chen et al, 2018;Doshi-Velez and Kim, 2017;Guidotti et al, 2018;Hashemi and Fathi, 2020;Lundberg and Lee, 2017;Murdoch et al, 2019;Ribeiro et al, 2016a). In particular, Lakkaraju and Bastani (2020), Slack et al (2020), andLakkaraju et al (2019) study the limitations of post-hoc explanation tools in providing useful and accurate descriptions of the underlying models, and show that simple explanations can be inadequate in distinguishing relevant model behavior.…”