Projecting "Better Than Randomly": How to Reduce the Dimensionality of Very Large Datasets in a Way That Outperforms Random Projections

Wojnowicz, Michael; Zhang, Di; Chisholm, Glenn; Zhao, Xuan; Wolff, Matt

doi:10.1109/dsaa.2016.26

Cited by 8 publications

(8 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This dataset [15] consists of 4,608,517 portable executable files and determined to be either malicious or clean. Each file is represented as 98,450 features, mostly binary, with mean density 0.0244.…”

Section: A Data and Methodsmentioning

confidence: 99%

“…To see this, we will express X = f f T X using constructions 6 in ( 6) and ( 8) and show: a) For points in R m , the operation f f T (which maps X to X) will leave points in I unchanged, and will map all points in I ⊥ to 0 [see ( 14), (15)]. Since I ≈ im(X) by Step 1, most points u ∈ im(X) will be well-approximated by component u a , where u = u a + u b and u a ∈ I, u b ∈ I ⊥ .…”

Section: A Overviewmentioning

confidence: 99%

See 1 more Smart Citation

Lazy Stochastic Principal Component Analysis

Wojnowicz

Nguyen²,

et al. 2017

2017 IEEE International Conference on Data Mining Workshops (ICDMW)

Self Cite

View full text Add to dashboard Cite

Abstract-Stochastic principal component analysis (SPCA) has become a popular dimensionality reduction strategy for large, high-dimensional datasets. We derive a simplified algorithm, called Lazy SPCA, which has reduced computational complexity and is better suited for large-scale distributed computation. We prove that SPCA and Lazy SPCA find the same approximations to the principal subspace, and that the pairwise distances between samples in the lower-dimensional space is invariant to whether SPCA is executed lazily or not. Empirical studies find downstream predictive performance to be identical for both methods, and superior to random projections, across a range of predictive models (linear regression, logistic lasso, and random forests). In our largest experiment with 4.6 million samples, Lazy SPCA reduced 43.7 hours of computation to 9.9 hours. Overall, Lazy SPCA relies exclusively on matrix multiplications, besides an operation on a small square matrix whose size depends only on the target dimensionality.

show abstract

Section: A Data and Methodsmentioning

confidence: 99%

Section: A Overviewmentioning

confidence: 99%

Lazy Stochastic Principal Component Analysis

Wojnowicz

Nguyen²,

et al. 2017

2017 IEEE International Conference on Data Mining Workshops (ICDMW)

Self Cite

View full text Add to dashboard Cite

show abstract

“…As recommended by [7], the density of the random projection matrix was set to 1/ √ P ≈ .003. We set K = 5000 based on the results of [14]. Obtaining influence scores for all 2.3 million samples took about an hour and a half using non-optimized scripts written in the Julia programming language and executed on a single r3.8x instance (with 32 workers for parallel processing) in Amazon's EC2 system.…”

Section: Methodsmentioning

confidence: 99%

“Influence sketching”: Finding influential samples in large-scale regressions

Wojnowicz¹,

Cruz²,

Zhao³

et al. 2016

2016 IEEE International Conference on Big Data (Big Data)

Self Cite

View full text Add to dashboard Cite

Abstract-There is an especially strong need in modern largescale data analysis to prioritize samples for manual inspection. For example, the inspection could target important mislabeled samples or key vulnerabilities exploitable by an adversarial attack. In order to solve the "needle in the haystack" problem of which samples to inspect, we develop a new scalable version of Cook's distance, a classical statistical technique for identifying samples which unusually strongly impact the fit of a regression model (and its downstream predictions). In order to scale this technique up to very large and high-dimensional datasets, we introduce a new algorithm which we call "influence sketching." Influence sketching embeds random projections within the influence computation; in particular, the influence score is calculated using the randomly projected pseudo-dataset from the post-convergence Generalized Linear Model (GLM). We validate that influence sketching can reliably and successfully discover influential samples by applying the technique to a malware detection dataset of over 2 million executable files, each represented with almost 100,000 features. For example, we find that randomly deleting approximately 10% of training samples reduces predictive accuracy only slightly from 99.47% to 99.45%, whereas deleting the same number of samples with high influence sketch scores reduces predictive accuracy all the way down to 90.24%. Moreover, we find that influential samples are especially likely to be mislabeled. In the case study, we manually inspect the most influential samples, and find that influence sketching pointed us to new, previously unidentified pieces of malware.

show abstract

“…The above paper uses random projections, as introduced by Dahl et al [7], to reduce the dimensionality of input features in an unsupervised fashion (projecting 50,000 features into 4,000). Wojnowicz et al [20] improved on this approach by introducing randomized principal component analysis. To our knowledge, their data set (11.7M malware samples) was the largest to date used in malware classification or clustering research.…”

Section: Related Workmentioning

confidence: 99%