Determinantal Point Processes in Randomized Numerical Linear Algebra

Dereziński, Michał; Mahoney, Michael W.

doi:10.1090/noti2202

Cited by 41 publications

(39 citation statements)

References 105 publications

(187 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Determinantal point processes (DPPs) are also popular probabilistic techniques for selecting a diverse set of landmarks in the Nyström method [25]. Given full access to the the kernel matrix K, a DPP is a stochastic point process such that the probability of observing a subset I ⊂ {1, .…”

Section: Preliminaries and Landmark Selection Techniquesmentioning

confidence: 99%

“…To address this problem, several lines of work aim to develop datadependent sampling distributions for selecting landmarks, including leverage score sampling [23] and Determinantal Point Processes (DPPs) [24]. For example, λ-ridge leverage scores can be computed as the diagonal entries of the matrix K(K + λI n ) −1 , and DPPs are characterized by subdeterminants of the kernel matrix K [25]. A major limitation of these methods is the need to construct the entire kernel matrix or its high-quality approximation before utilizing the Nyström method, leading to time and space complexities that are often superlinear in the data size.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Kernel Matrix Approximation on Class-Imbalanced Data With an Application to Scientific Simulation

2021

View full text Add to dashboard Cite

Generating low-rank approximations of kernel matrices that arise in nonlinear machine learning techniques holds the potential to significantly alleviate the memory and computational burdens. A compelling approach centers on finding a concise set of exemplars or landmarks to reduce the number of similarity measure evaluations from quadratic to linear concerning the data size. However, a key challenge is to regulate tradeoffs between the quality of landmarks and resource consumption. Despite the volume of research in this area, current understanding is limited regarding the performance of landmark selection techniques in the presence of class-imbalanced data sets that are becoming increasingly prevalent in many applications. Hence, this paper provides a comprehensive empirical investigation using several realworld imbalanced data sets, including scientific data, by evaluating the quality of approximate low-rank decompositions and examining their influence on the accuracy of downstream tasks. Furthermore, we present a new landmark selection technique called Distance-based Importance Sampling and Clustering (DISC), in which the relative importance scores are computed for improving accuracy-efficiency tradeoffs compared to existing works that range from probabilistic sampling to clustering methods. The proposed landmark selection method follows a coarse-to-fine strategy to capture the intrinsic structure of complex data sets, allowing us to substantially reduce the computational complexity and memory footprint with minimal loss in accuracy.

show abstract

Section: Preliminaries and Landmark Selection Techniquesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Kernel Matrix Approximation on Class-Imbalanced Data With an Application to Scientific Simulation

2021

View full text Add to dashboard Cite

show abstract

“…As distributions over subsets of a large ground set that favour diversity, DPPs are intuitively good candidates at subsampling tasks, and one of their primary applications in ML has been as summary extractors [7]. Since then, DPPs or mixtures thereof have been used, e.g., to generate experimental designs for linear regression, leading to strong theoretical guarantees [11,12,13]; see also [14] for a survey of DPPs in randomized numerical algebra, or [15] for feature selection in linear regression with DPPs.…”

Section: Background and Relevant Literaturementioning

confidence: 99%

“…Our desired regularity properties may therefore be understood in terms of closeness to this limit, which itself has similar regularity. At the price of these assumptions, we can use analytic tools to derive fluctuations for Ξ A,DPP by working on the limit in (14). For the fluctuation analysis of the smoothed estimator Ξ A,s , we similarly assume that (q(w)…”

Section: Some Regularity Phenomenamentioning

confidence: 99%

Determinantal point processes based on orthogonal polynomials for sampling minibatches in SGD

Bardenet¹,

Ghosh²,

Lin³

2021

Preprint

View full text Add to dashboard Cite

Stochastic gradient descent (SGD) is a cornerstone of machine learning. When the number N of data items is large, SGD relies on constructing an unbiased estimator of the gradient of the empirical risk using a small subset of the original dataset, called a minibatch. Default minibatch construction involves uniformly sampling a subset of the desired size, but alternatives have been explored for variance reduction. In particular, experimental evidence suggests drawing minibatches from determinantal point processes (DPPs), tractable distributions over minibatches that favour diversity among selected items. However, like in recent work on DPPs for coresets, providing a systematic and principled understanding of how and why DPPs help has been difficult. In this work, we contribute an orthogonal polynomial-based determinantal point process paradigm for performing minibatch sampling in SGD. Our approach leverages the specific data distribution at hand, which endows it with greater sensitivity and power over existing data-agnostic methods. We substantiate our method via a detailed theoretical analysis of its convergence properties, interweaving between the discrete data set and the underlying continuous domain. In particular, we show how specific DPPs and a string of controlled approximations can lead to gradient estimators with a variance that decays faster with the batchsize than under uniform sampling. Coupled with existing finite-time guarantees for SGD on convex objectives, this entails that, for a large enough batchsize and a fixed budget of item-level gradients to evaluate, DPP minibatches lead to a smaller bound on the mean square approximation error than uniform minibatches. Moreover, our estimators are amenable to a recent algorithm that directly samples linear statistics of DPPs (i.e., the gradient estimator) without sampling the underlying DPP (i.e., the minibatch), thereby reducing computational overhead. We provide detailed synthetic as well as real data experiments to substantiate our theoretical claims.

show abstract

“…Determinantal Point Processes (DPPs) were first introduced in the context of quantum mechanics (Macchi, 1975) and have subsequently been extensively studied with applications in several areas of pure and applied mathematics like graph theory, combinatorics, random matrix theory (Hough et al, 2006;Borodin, 2009), and randomized numerical linear algebra (Derezinski & Mahoney, 2021). Discrete DPPs have gained widespread adoption in machine learning following the seminal work of Kulesza & Taskar (2012) and there has been a recent explosion of interest in DPPs in the machine learning community.…”

Section: Introductionmentioning

confidence: 99%

Online MAP Inference and Learning for Nonsymmetric Determinantal Point Processes

Reddy¹,

Rossi²,

Song³

et al. 2021

Preprint

View full text Add to dashboard Cite

In this paper, we introduce the online and streaming MAP inference and learning problems for Non-symmetric Determinantal Point Processes (NDPPs) where data points arrive in an arbitrary order and the algorithms are constrained to use a single-pass over the data as well as sub-linear memory. The online setting has an additional requirement of maintaining a valid solution at any point in time. For solving these new problems, we propose algorithms with theoretical guarantees, evaluate them on several real-world datasets, and show that they give comparable performance to state-of-the-art offline algorithms that store the entire data in memory and take multiple passes over it.

show abstract

Determinantal Point Processes in Randomized Numerical Linear Algebra

Cited by 41 publications

References 105 publications

Kernel Matrix Approximation on Class-Imbalanced Data With an Application to Scientific Simulation

Kernel Matrix Approximation on Class-Imbalanced Data With an Application to Scientific Simulation

Determinantal point processes based on orthogonal polynomials for sampling minibatches in SGD

Online MAP Inference and Learning for Nonsymmetric Determinantal Point Processes

Contact Info

Product

Resources

About