“Influence sketching”: Finding influential samples in large-scale regressions

Abstract

Abstract-There is an especially strong need in modern largescale data analysis to prioritize samples for manual inspection. For example, the inspection could target important mislabeled samples or key vulnerabilities exploitable by an adversarial attack. In order to solve the "needle in the haystack" problem of which samples to inspect, we develop a new scalable version of Cook's distance, a classical statistical technique for identifying samples which unusually strongly impact the fit of a regression model (and its downstream predictions). In order to scale this technique up to very large and high-dimensional datasets, we introduce a new algorithm which we call "influence sketching." Influence sketching embeds random projections within the influence computation; in particular, the influence score is calculated using the randomly projected pseudo-dataset from the post-convergence Generalized Linear Model (GLM). We validate that influence sketching can reliably and successfully discover influential samples by applying the technique to a malware detection dataset of over 2 million executable files, each represented with almost 100,000 features. For example, we find that randomly deleting approximately 10% of training samples reduces predictive accuracy only slightly from 99.47% to 99.45%, whereas deleting the same number of samples with high influence sketch scores reduces predictive accuracy all the way down to 90.24%. Moreover, we find that influential samples are especially likely to be mislabeled. In the case study, we manually inspect the most influential samples, and find that influence sketching pointed us to new, previously unidentified pieces of malware.

Scalable Explanation of Inferences on Large Graphs

“…To explain differentiable parametric predictive models, such as deep networks [19] and linear models [26,47], the gradients of the output with respect to the parameters and input data [39] can signify key factors that explain the output. However, graphical models aim to model long range and more complicated types of interaction among variables.…”

Exact and Consistent Interpretation of Piecewise Linear Models Hidden behind APIs: A Closed Form Solution

“…First, the instance attribution methods find the training instances that significantly influence the prediction on an instance to be interpreted. Wojnowicz et al [45] used influence sketching to identify the training instances that strongly affect the fit of a regression model by efficiently estimating Cook's distance [9]. Koh et al [22] proposed influence functions to trace the prediction of a model and identify the training instances that are the most responsible for the prediction.…”

Lazy Stochastic Principal Component Analysis

“…where Ω ∈ R n×k is a matrix of random numbers and c is a scalar for norm preservation that depends upon the random projection method used. 1,2 Random projection is a computationally cheap technique that approximately preserves pairwise distances between samples with high probability (with error depending on k and m). 1 Where possible, vectors in R m are denoted by u and vectors in R n are denoted by v. 2 Note that a random projection is technically not actually a projection (an endomorphism X P -X that satisfies P 2 = P ).…”

“…Stochastic DR may be applied as a preprocessing step before feeding the data into a computationally expensive classifier, e.g. a neural network [1], or it may be directly embedded within algorithms to improve scalability [2].…”