Sketching Datasets for Large-Scale Learning (long version)

Abstract

This article considers "sketched learning," or "compressive learning," an approach to large-scale machine learning where datasets are massively compressed before learning (e.g., clustering, classification, or regression) is performed. In particular, a "sketch" is first constructed by computing carefully chosen nonlinear random features (e.g., random Fourier features) and averaging them over the whole dataset. Parameters are then learned from the sketch, without access to the original dataset. This article surveys the current state-ofthe-art in sketched learning, including the main concepts and algorithms, their connections with established signal-processing methods, existing theoretical guarantees-on both information preservation and privacy preservation, and important open problems.Big data can be a blessing: with very large training datasets it becomes possible to perform complex learning tasks with unprecedented accuracy. Yet, this improved performance comes at the price of enormous computational challenges. Thus, one may wonder: Is it possible to leverage the information content of huge datasets while keeping computational resources under control? Can this also help solve some of the privacy issues raised by large-scale learning? This is the ambition of sketched learning-or compressive learning-where the data is massively compressed before learning. Here, a "sketch" is first constructed by computing carefully chosen nonlinear random features (e.g., random Fourier features) and averaging them over the whole dataset. Parameters are then learned from the sketch, without access to the original dataset. This article surveys the current state-of-the-art in sketched learning, including the main concepts and algorithms; their connections with established signal-processing methods; existing theoretical guarantees, on both information preservation and privacy preservation; and important open problems.

Compressive Learning for Patch-Based Image Denoising

“…The excess risk of the GMM learning task is then controlled by the sum of an empirical error term and a modeling error term. This guarantees that the estimated GMM approximates well the distribution of the data [19].…”

“…Leveraging ideas from compressive sensing [15] and streaming algorithms [9], R. Gribonval et al propose a sketching method [25,19,20,18,17] to compress the training database.…”

Fitting large mixture models using stochastic component selection

“…The weighted variants of the variational (Feldman, Faulkner, and Krause 2011;Zhang et al 2016;Campbell andBeronov 2019) andsampling-based (McGrory et al 2014) methods then process the coresets. Reducing D relies on the compression of data into smaller representations via random projections (Siblini, Kuntz, and Meyer 2019;Ayesha, Hanif, and Talib 2020), which is achieved in two ways: (i) each data item is projected into an individual representation (Dasgupta 1999); (ii) all data items are projected into an overall representation, commonly referred to as sketch (Keriven et al 2018;Gribonval et al 2020).…”