Parallel Framework for Dimensionality Reduction of Large-Scale Datasets

Abstract

Dimensionality reduction refers to a set of mathematical techniques used to reduce complexity of the original high-dimensional data, while preserving its selected properties. Improvements in simulation strategies and experimental data collection methods are resulting in a deluge of heterogeneous and high-dimensional data, which often makes dimensionality reduction the only viable way to gain qualitative and quantitative understanding of the data. However, existing dimensionality reduction software often does not scale to datasets arising in real-life applications, which may consist of thousands of points with millions of dimensions. In this paper, we propose a parallel framework for dimensionality reduction of large-scale data. We identify key components underlying the spectral dimensionality reduction techniques, and propose their efficient parallel implementation. We show that the resulting framework can be used to process datasets consisting of millions of points when executed on a 16,000-core cluster, which is beyond the reach of currently available methods. To further demonstrate applicability of our framework we perform dimensionality reduction of 75,000 images representing morphology evolution during manufacturing of organic solar cells in order to identify how processing parameters affect morphology evolution.

Scalable Manifold Learning for Big Data with Apache Spark

“…To alleviate these complexities, several approximate methods have been proposed [8], [9]. However, these approaches do not provide exactness guarantees, or are tailored for large-scale HPC systems [10].…”

Data Dimensionality Reduction in Materials Science

“…This is mainly attributed to the difference in the governing mathematics of the two techniques, where PCA is essentially a linear technique and Isomap is a nonlinear technique. We have further interpreted the hidden information captured by the Isomap classification map by focusing on the three regions separately in a forthcoming paper (Samudrala et al, 2013). …”

“…We showcase the results obtained by applying (a parallel version of) the SETDiR framework (Samudrala et al, 2013) on this pressing scientific problem. 1 We particularly focus on using dimensionality reduction to understand the effects of substrate patterning (patterning frequency and intensity) on morphology evolution.…”

“…1 We developed an in-house parallel solver PaDRe to apply DR to data sets as large as n 5 10 5 and D 5 10 6 . We detail the parallel algorithms and associated scaling studies in a separate paper (Samudrala et al, 2013). 2 Nano-tip photolithography patterning of the substrate has shown significant potential to guide morphology evolution (Coffey and Ginger, 2005).…”

Error Metrics for Learning Reliable Manifolds from Streaming Data

“…In our approach, we focus on Isomap because of its broad adoption in scientific computing [15] and bio-medical research, including fMRI analysis [18], clustering of oncology data [12], genes and proteins expression profiling [9], modeling of spatio-temporal relationships in data [6], and many others. Isomap provides a simple and elegant way to estimate the intrinsic geometry of the data manifold based on a rough estimate of each data points neighbors on the manifold.…”