DPPNet: Approximating Determinantal Point Processes with Deep Networks

Abstract

Determinantal Point Processes (DPPs) provide an elegant and versatile way to sample sets of items that balance the point-wise quality with the set-wise diversity of selected items. For this reason, they have gained prominence in many machine learning applications that rely on subset selection. However, sampling from a DPP over a ground set of size N is a costly operation, requiring in general an O(N 3 ) preprocessing cost and an O(N k 3 ) sampling cost for subsets of size k. We approach this problem by introducing DPPNETs: generative deep models that produce DPP-like samples for arbitrary ground sets. We develop an inhibitive attention mechanism based on transformer networks that captures a notion of dissimilarity between feature vectors. We show theoretically that such an approximation is sensible as it maintains the guarantees of inhibition or dissimilarity that makes DPPs so powerful and unique. Empirically, we demonstrate that samples from our model receive high likelihood under the more expensive DPP alternative. * Work done while at Google Brain. 2 Here, we use diversity to mean useful coverage across dissimilar examples in a meaningful feature space, rather than other definitions diversity that may appear in ML fairness literature. 3 We are adopting here the L-Ensemble construction (Borodin & Rains, 2005) of a DPP.

Flexible Modeling of Diversity with Strongly Log-Concave Distributions

“…Particular attention has been paid to determinantal point processes due to the intuitive way they capture negative dependence, and the fact that they are parameterized by a single positive semi-definite kernel matrix. Convenient parameterization has allowed an abundance of fast algorithms for learning the kernel matrix [23,26,43,47], and sampling [2,40,46]. SR distributions are a fascinating and elegant probabilistic family whose applicability in machine learning is still an emerging topic [17,34,41,45].…”