Auditing Black-Box Models for Indirect Influence

Adler, Philip D. F.; Falk, Casey; Friedler, Sorelle A.; Rybeck, Gabriel; Scheidegger, Carlos; Smith, Brandon; Venkatasubramanian, Suresh

doi:10.1109/icdm.2016.0011

Cited by 115 publications

(140 citation statements)

References 23 publications

Supporting

Mentioning

139

Contrasting

Order By: Relevance

“…Auditing ML models. Much recent work aims to understand the behavior of ML models with black-box access [2,16]. These approaches improve the interpretability of the model by showing how features or training data points influence the model's predictions.…”

Section: Related Workmentioning

confidence: 99%

Auditing Data Provenance in Text-Generation Models

Song

Shmatikov

2019

Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery &Amp; Data Mining

133

110

View full text Add to dashboard Cite

To help enforce data-protection regulations such as GDPR and detect unauthorized uses of personal data, we develop a new model auditing technique that helps users check if their data was used to train a machine learning model. We focus on auditing deeplearning models that generate natural-language text, including word prediction and dialog generation. These models are at the core of popular online services and are often trained on personal data such as users' messages, searches, chats, and comments.We design and evaluate a black-box auditing method that can detect, with very few queries to a model, if a particular user's texts were used to train it (among thousands of other users). We empirically show that our method can successfully audit well-generalized models that are not overfitted to the training data. We also analyze how text-generation models memorize word sequences and explain why this memorization makes them amenable to auditing. CCS CONCEPTS• Computing methodologies → Machine learning; • Security and privacy → Software and application security.

show abstract

Section: Related Workmentioning

confidence: 99%

Auditing Data Provenance in Text-Generation Models

Song

Shmatikov

2019

Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery &Amp; Data Mining

133

110

View full text Add to dashboard Cite

show abstract

“…, and extended chance ( 1− p 1 1− p ) [29]. Another similar conception is called statistical parity, which means that the demographics of the set of individuals receiving positive (or negative) decisions are identical to the demographics of the population as a whole.…”

Section: Discrimination Discoverymentioning

confidence: 99%

“…Data preprocessing methods [1,12,13,17,21,37,43] modify the historic data to remove discriminatory effect according to some discrimination measure before learning a predictive model. For example, in [17] several methods for modifying data were proposed.…”

Section: Discrimination Preventionmentioning

confidence: 99%

“…These approaches classify discrimination into different types such as group discrimination, individual discrimination, direct and indirect discrimination. Based on that, methods for discrimination prevention have been proposed [1,7,9,[12][13][14][15][17][18][19]21,35,37,43] which either use data preprocessing or algorithm tweaking. However, these works are mainly based on correlation or association-based measures which cannot be used to estimate the causal effect of the protected attributes on the decision.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Anti-discrimination learning: a causal modeling-based framework

Zhang

2017

Int J Data Sci Anal

View full text Add to dashboard Cite

Anti-discrimination learning is an increasingly important task in data mining. Discrimination discovery is the problem of unveiling discriminatory practices by analyzing a dataset of historical decision records, and discrimination prevention aims to remove discrimination by modifying the biased data and/or the predictive algorithms. Discrimination is causal, which means that to prove discrimination one needs to derive a causal relationship rather than an association relationship. Although it is well known that association does not mean causation, the gap between association and causation is not paid enough attention by many researchers. In this paper, we introduce a causal modeling-based framework for anti-discrimination learning. Discrimination is categorized according to two dimensions: direct/indirect and system/group/individual level. Within the causal framework, we introduce a work for discovering and preventing both direct and indirect system-level discrimination in the training data, and a work for extending the non-discrimination result from the training data to prediction. We then introduce two works for group-level direct discrimination and individual-level direct discrimination respectively. The aim of this paper is to deepen the understanding of discrimination in data mining from the causal modeling perspective, and suggest several potential future research directions.

show abstract

“…al. present a technique for auditing black-box models, which can reveal the extent to which the models take advantage of particular features in the dataset [42], without knowing how the models work [43]. While there may be technical challenges [44] in allowing public auditing while protecting proprietary information, private auditing may be the right option, for which methods need to be developed [43], [45], [46].…”

Section: A Fairness and Accountabilitymentioning

confidence: 99%

Interpretability of deep learning models: A survey of results

Chakraborty

Tomsett

Raghavendra

et al. 2017

2017 IEEE SmartWorld, Ubiquitous Intelligence &Amp; Computing, Advanced &Amp; Trusted Computed, Scalable Computing &Amp; Commun

278

189

View full text Add to dashboard Cite

Abstract-Deep neural networks have achieved near-human accuracy levels in various types of classification and prediction tasks including images, text, speech, and video data. However, the networks continue to be treated mostly as black-box function approximators, mapping a given input to a classification output. The next step in this human-machine evolutionary processincorporating these networks into mission critical processes such as medical diagnosis, planning and control -requires a level of trust association with the machine output.Typically, statistical metrics are used to quantify the uncertainty of an output. However, the notion of trust also depends on the visibility that a human has into the working of the machine. In other words, the neural network should provide humanunderstandable justifications for its output leading to insights about the inner workings. We call such models as interpretable deep networks.Interpretability is not a monolithic notion. In fact, the subjectivity of an interpretation, due to different levels of human understanding, implies that there must be a multitude of dimensions that together constitute interpretability. In addition, the interpretation itself can be provided either in terms of the lowlevel network parameters, or in terms of input features used by the model. In this paper, we outline some of the dimensions that are useful for model interpretability, and categorize prior work along those dimensions. In the process, we perform a gap analysis of what needs to be done to improve model interpretability.

show abstract

Auditing Black-Box Models for Indirect Influence

Cited by 115 publications

References 23 publications

Auditing Data Provenance in Text-Generation Models

Auditing Data Provenance in Text-Generation Models

Anti-discrimination learning: a causal modeling-based framework

Interpretability of deep learning models: A survey of results

Contact Info

Product

Resources

About