Matrix Multiplication on High-Density Multi-GPU Architectures: Theoretical and Experimental Investigations

Zhang, Peng; Gao, Yuxiang

doi:10.1007/978-3-319-20119-1_2

Cited by 29 publications

(7 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It theoretically scales in O (nmN ), but can be done in a distributed manner by splitting the dataset over several computing units and averaging the obtained sketches, such that the full data need never be stored in one single location. One can also exploit GPU computing for very large-scale matrix multiplication [21]. The proposed sketch can also be maintained online, which is another crucial property of typical dataset sketches [22].…”

Section: Complexity Of the Methodsmentioning

confidence: 99%

Compressive K-means

Keriven

Tremblay

Traonmilin

et al. 2017

2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

The Lloyd-Max algorithm is a classical approach to perform Kmeans clustering. Unfortunately, its cost becomes prohibitive as the training dataset grows large. We propose a compressive version of K-means (CKM), that estimates cluster centers from a sketch, i.e. from a drastically compressed representation of the training dataset.We demonstrate empirically that CKM performs similarly to Lloyd-Max, for a sketch size proportional to the number of centroids times the ambient dimension, and independent of the size of the original dataset. Given the sketch, the computational complexity of CKM is also independent of the size of the dataset. Unlike Lloyd-Max which requires several replicates, we further demonstrate that CKM is almost insensitive to initialization. For a large dataset of 10 7 data points, we show that CKM can run two orders of magnitude faster than five replicates of Lloyd-Max, with similar clustering performance on artificial data. Finally, CKM achieves lower classification errors on handwritten digits classification.

show abstract

Section: Complexity Of the Methodsmentioning

confidence: 99%

Compressive K-means

Keriven

Tremblay

Traonmilin

et al. 2017

2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…As m n, the computational cost of our algorithm is lower than that of the EM. Moreover, it is possible to exploit the advantages of GPU computing, the matrix multiplication can be performed by using multiple GPUs in parallel [63]. This could result in a speed-up, especially for the "final adjustment" step.…”

Section: Complexity Of Lr-ompmentioning

confidence: 99%

Compressive Learning for Patch-Based Image Denoising

Shi¹,

Traonmilin²,

Aujol³

2022

SIAM J. Imaging Sci.

View full text Add to dashboard Cite

The Expected Patch Log-Likelihood algorithm (EPLL) and its extensions have shown good performances for image denoising. The prior model used by EPLL is usually a Gaussian Mixture Model (GMM) estimated from a database of image patches. Classical mixture model estimation methods face computational issues as the high dimensionality of the problem requires training on large datasets. In this work, we adapt a compressive statistical learning framework to carry out the GMM estimation. With this method, called sketching, we estimate models from a compressive representation (the sketch) of the training patches. The cost of estimating the prior from the sketch no longer depends on the number of items in the original large database. To accelerate further the estimation, we add another dimension reduction technique (low-rank modeling of the covariance matrices) to the compressing learning framework. To demonstrate the advantages of our method, we test it on real large-scale data. We show that we can produce denoising performances similar to performances obtained with models estimated from the original training database using GMM priors learned from the sketch with improved execution times.

show abstract

“…A classical way to drastically reduce its cost consists in using GPU computing [104]. Recent architectures can even leverage giant matrix multiplication using multiple GPUs in parallel [109].…”

Section: Complexity Of Sketchingmentioning

confidence: 99%

Sketching for large-scale learning of mixture models

Keriven

Bourrier

Gribonval

et al. 2017

Information and Inference: A Journal of the IMA

148

View full text Add to dashboard Cite

Learning parameters from voluminous data can be prohibitive in terms of memory and computational requirements. We propose a "compressive learning" framework where we estimate model parameters from a sketch of the training data. This sketch is a collection of generalized moments of the underlying probability distribution of the data. It can be computed in a single pass on the training set, and is easily computable on streams or distributed datasets. The proposed framework shares similarities with compressive sensing, which aims at drastically reducing the dimension of high-dimensional signals while preserving the ability to reconstruct them. To perform the estimation task, we derive an iterative algorithm analogous to sparse reconstruction algorithms in the context of linear inverse problems. We exemplify our framework with the compressive estimation of a Gaussian Mixture Model (GMM), providing heuristics on the choice of the sketching procedure and theoretical guarantees of reconstruction. We experimentally show on synthetic data that the proposed algorithm yields results comparable to the classical Expectation-Maximization (EM) technique while requiring significantly less memory and fewer computations when the number of database elements is large. We further demonstrate the potential of the approach on real large-scale data (over 10 8 training samples) for the task of model-based speaker verification.Finally, we draw some connections between the proposed framework and approximate Hilbert space embedding of probability distributions using random features. We show that the proposed sketching operator can be seen as an innovative method to design translation-invariant kernels adapted to the analysis of GMMs. We also use this theoretical framework to derive information preservation guarantees, in the spirit of infinite-dimensional compressive sensing.

show abstract

Matrix Multiplication on High-Density Multi-GPU Architectures: Theoretical and Experimental Investigations

Cited by 29 publications

References 23 publications

Compressive K-means

Compressive K-means

Compressive Learning for Patch-Based Image Denoising

Sketching for large-scale learning of mixture models

Contact Info

Product

Resources

About