Auditing black-box models for indirect influence

Abstract

Abstract-Data-trained predictive models see widespread use, but for the most part they are used as black boxes which output a prediction or score. It is therefore hard to acquire a deeper understanding of model behavior, and in particular how different features influence the model prediction. This is important when interpreting the behavior of complex models, or asserting that certain problematic attributes (like race or gender) are not unduly influencing decisions.In this paper, we present a technique for auditing black-box models, which lets us study the extent to which existing models take advantage of particular features in the dataset, without knowing how the models work. Our work focuses on the problem of indirect influence: how some features might indirectly influence outcomes via other, related features. As a result, we can find attribute influences even in cases where, upon further direct examination of the model, the attribute is not referred to by the model at all.Our approach does not require the black-box model to be retrained. This is important if (for example) the model is only accessible via an API, and contrasts our work with other methods that investigate feature influence like feature selection. We present experimental evidence for the effectiveness of our procedure using a variety of publicly available datasets and models. We also validate our procedure using techniques from interpretable learning and feature selection, as well as against other black-box auditing procedures.

Fairness in Machine Learning

“…Some methods transform the constrained optimization problem via the method of Lagrange multipliers [3,15,34,37,38,59,97,135,183,185] or add penalties to the objective [5,14,22,44,46,56,58,62,73,74,75,79,82,87,88,89,93,94,95,98,104,115,119,123,133,134,135,138,139,154,165,176,179,180,182], others use adversary techniques to maximize the system ability to predict the target while minimizing the ability to predict the sensitive attribute [189]. Post-Processing methods consist in transforming the model outputs in order to make them fair [2,7,…”

“…Over the last few years, researchers have introduced a rich set of definitions formalizing different fairness desiderata that can be used for evaluating and designing ML systems [2,7,14,15,21,22,23,25,28,29,31,34,44,45,46,52,53,54,55,56,59,66,69,71,72,73,74,75,82,84,87,89,90,94,95,97,98,99,103,104,106,114,115,116,121,123,126,138,139,141,148,…”

“…where " • " is the inner product between two vectors in a Hilbert space . We can then learn the parameter vector by 2…”

Explaining Supervised Learning Models: A Preliminary Study on Binary Classifiers

“…Another related post hoc technique called black box auditing (Adler et al, 2016) can be used to decide the extent to which a specific feature contributes to the accuracy (percentage of correct predictions) of a trained model. To quantify the direct effect of a feature, we can replace the feature by random noise and see how much the model accuracy drops.…”

Are you sure? Prediction revision in automated decision‐making

“…Such techniques use an interpretable model and apply it to the predictions returned by the black box (Hall, Phan, & Ambati, 2017). Currently many different explanation approaches exist, for example, providing logical statements (Lakkaraju, Bach, & Leskovec, 2016; Su, Wei, Varshney, & Malioutov, 2015; Wang et al, 2015; Wang & Rudin, 2014), local models (Rüping, 2005; Turner, 2016) or feature importance (Adler et al, 2018; Datta, Sen, & Zick, 2016; Goldstein, Kapelner, Bleich, & Pitkin, 2015; Puolamäki & Ukkonen, 2017).…”