Compressive Learning for Semi-Parametric Models

Sheehan, Michael P.; Gonon, Antoine; Davies, Mike E.

doi:10.48550/arxiv.1910.10024

Cited by 3 publications

(12 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When the RF feature map Φ(•) is used to compute the sketch z and the component densities p θ are Gaussian or α-stable, there exist analytic expressions for A(p θ ) and for the gradient of A(p θ ) with respect to the mixture parameters in θ [10]. These expressions are convenient when numerically optimizing (18).…”

Section: Learning From a Sketchmentioning

confidence: 99%

“…These approaches sequentially estimate and subtract, from z, each of the k components α A(p θ ) that best align with it, where "best" is measured via the correlation between z and A(p θ ). As another example, an iterative approach [16] was proposed that exploits the log-likelihood interpretation of C(θ| z) in (18) and the i.i.d. random nature of the linear transform W in the RF map (3).…”

Section: Learning From a Sketchmentioning

confidence: 99%

“…In [10], the authors compress 1000 hours of speech data (50 gigabytes) into a sketch of a few kilobytes on a single laptop, using an RF feature map Φ, and perform subsequent GMM estimation using a greedy algorithm to tackle the optimization problem (18). They compare this sketched mixture learning approach to the EM algorithm, which on the same machine can only be trained on 5 hours of speech given the available RAM.…”

Section: Learning From a Sketchmentioning

confidence: 99%

“…They observe that, by enabling the use of more data within a fixed memory budget, sketching produces better results despite the tremendous compression factor. Thus, with these justifications, the cost function (18) for sketched GMM would change to…”

Section: Learning From a Sketchmentioning

confidence: 99%

See 3 more Smart Citations

Sketching Datasets for Large-Scale Learning (long version)

Gribonval,

Chatalic,

Keriven

et al. 2020

Preprint

View full text Add to dashboard Cite

This article considers "sketched learning," or "compressive learning," an approach to large-scale machine learning where datasets are massively compressed before learning (e.g., clustering, classification, or regression) is performed. In particular, a "sketch" is first constructed by computing carefully chosen nonlinear random features (e.g., random Fourier features) and averaging them over the whole dataset. Parameters are then learned from the sketch, without access to the original dataset. This article surveys the current state-ofthe-art in sketched learning, including the main concepts and algorithms, their connections with established signal-processing methods, existing theoretical guarantees-on both information preservation and privacy preservation, and important open problems.Big data can be a blessing: with very large training datasets it becomes possible to perform complex learning tasks with unprecedented accuracy. Yet, this improved performance comes at the price of enormous computational challenges. Thus, one may wonder: Is it possible to leverage the information content of huge datasets while keeping computational resources under control? Can this also help solve some of the privacy issues raised by large-scale learning? This is the ambition of sketched learning-or compressive learning-where the data is massively compressed before learning. Here, a "sketch" is first constructed by computing carefully chosen nonlinear random features (e.g., random Fourier features) and averaging them over the whole dataset. Parameters are then learned from the sketch, without access to the original dataset. This article surveys the current state-of-the-art in sketched learning, including the main concepts and algorithms; their connections with established signal-processing methods; existing theoretical guarantees, on both information preservation and privacy preservation; and important open problems.

show abstract

Section: Learning From a Sketchmentioning

confidence: 99%

Section: Learning From a Sketchmentioning

confidence: 99%

Section: Learning From a Sketchmentioning

confidence: 99%

Section: Learning From a Sketchmentioning

confidence: 99%

See 2 more Smart Citations

Sketching Datasets for Large-Scale Learning (long version)

Gribonval,

Chatalic,

Keriven

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…PCA, ICA). In general, distribution free CL poses distinct challenges and advantages from the typical parametric CL framework [26]. Challenges arise when choosing an intermediary statistic space S, for instance (1) what set of intermediate statistics can we use?…”

Section: Compressive Principal Component Analysismentioning

confidence: 99%

Compressive Independent Component Analysis

Sheehan

Kotzagiannidis

Davies

2019

2019 27th European Signal Processing Conference (EUSIPCO)

Self Cite

View full text Add to dashboard Cite

Compressive learning forms the exciting intersection between compressed sensing and statistical learning where one exploits forms of sparsity and structure to reduce the memory and/or computational complexity of the learning task. In this paper, we look at the independent component analysis (ICA) model through the compressive learning lens. In particular, we show that solutions to the cumulant based ICA model have particular structure that induces a low dimensional model set that resides in the cumulant tensor space. By showing a restricted isometry property holds for random cumulants e.g. Gaussian ensembles, we prove the existence of a compressive ICA scheme. Thereafter, we propose two algorithms of the form of an iterative projection gradient (IPG) and an alternating steepest descent (ASD) algorithm for compressive ICA, where the order of compression asserted from the restricted isometry property is realised through empirical results. We provide analysis of the CICA algorithms including the effects of finite samples. The effects of compression are characterised by a trade-off between the sketch size and the statistical efficiency of the ICA estimates. By considering synthetic and real datasets, we show the substantial memory gains achieved over wellknown ICA algorithms by using one of the proposed CICA algorithms. Finally, we conclude the paper with open problems including interesting challenges from the emerging field of compressive learning.

show abstract

Compressive independent component analysis: theory and algorithms

Sheehan

Davies

2022

Information and Inference: A Journal of the IMA

View full text Add to dashboard Cite

Compressive learning forms the exciting intersection between compressed sensing and statistical learning where one exploits sparsity of the learning model to reduce the memory and/or computational complexity of the algorithms used to solve the learning task. In this paper, we look at the independent component analysis (ICA) model through the compressive learning lens. In particular, we show that solutions to the cumulant-based ICA model have a particular structure that induces a low-dimensional model set that resides in the cumulant tensor space. By showing that a restricted isometry property holds for random cumulants e.g. Gaussian ensembles, we prove the existence of a compressive ICA scheme. Thereafter, we propose two algorithms of the form of an iterative projection gradient and an alternating steepest descent algorithm for compressive ICA, where the order of compression asserted from the restricted isometry property is realized through empirical results. We provide analysis of the CICA algorithms including the effects of finite samples. The effects of compression are characterized by a trade-off between the sketch size and the statistical efficiency of the ICA estimates. By considering synthetic and real datasets, we show the substantial memory gains achieved over well-known ICA algorithms by using one of the proposed CICA algorithms.

show abstract

Compressive Learning for Semi-Parametric Models

Cited by 3 publications

References 0 publications

Sketching Datasets for Large-Scale Learning (long version)

Sketching Datasets for Large-Scale Learning (long version)

Compressive Independent Component Analysis

Compressive independent component analysis: theory and algorithms

Contact Info

Product

Resources

About