Learning Functions over Sets via Permutation Adversarial Networks

Abstract

In this paper, we consider the problem of learning functions over sets, i.e., functions that are invariant to permutations of input set items. Recent approaches of pooling individual element embeddings [34] can necessitate extremely large embedding sizes for challenging functions. We address this challenge by allowing standard neural networks like LSTMs to succinctly capture the function over the set. However, to ensure invariance with respect to permutations of set elements, we propose a novel architecture called SPAN that simultaneously learns the function as well as adversarial or worst-case permutations for each input set. The learning problem reduces to a min-max optimization problem that is solved via a simple alternating block coordinate descent technique. We conduct extensive experiments on a variety of set-learning tasks and demonstrate that SPAN learns nearly permutation-invariant functions while still ensuring accuracy on test data. On a variety of tasks sampled from the domains of statistics, graph functions and linear algebra, we show that our method can significantly outperform state-of-the-art methods such as DeepSets [34] and Janossy Pooling [23]. Finally, we present a case study of how learning set-functions can help extract powerful features for recommendation systems, and show that such a method can be as much as 2% more accurate than carefully hand-tuned features on a real-world recommendation system.

Adversarial Permutation Guided Node Representations for Link Prediction

“…In theory (Zaheer et al 2017, Theorem 2), if COMB 1,2 are given 'sufficient' hidden units, this set representation is universal. In practice, however, commutative-associative aggregation suffers from limited expressiveness (Pabbaraju and Jain 2019;Wagstaff et al 2019;Garg, Jegelka, and Jaakkola 2020;Cohen-Karlik, David, and Globerson 2020), which degrades the quality of x u and s(•, •), as described below. Specifically, their expressiveness is constrained from two perspectives.…”

Equivariant Energy Flow Networks for Jet Tagging

“…This has motivated the development and application of neural network architectures which are insensitive to the ordering of the jet constituents, where jets are viewed as point clouds: a set of data points in space. The representation of functions on sets from which neural network architectures may be modeled has been much studied recently in the ML community [40,[42][43][44][45]. Collider physics studies in this vein include the use of architectures based on Graph Neural Networks such as Ref.…”

Adversarial Permutation Guided Node Representations for Link Prediction

“…In theory [52,Theorem 2], if COMB 1,2 are given 'sufficient' hidden units, this set representation is universal. In practice, however, commutative-associative aggregation suffers from limited expressiveness [9,30,44], which degrades the quality of x u and s(•, •), as described below. Specifically, their expressiveness is constrained from two perspectives.…”

“…Cuturi [8] exploited this to solve transportation problems approximately. It was soon realized [26,30] that row and column scaling transform an arbitrary matrix to a near-permutation matrix, while allowing backpropagation. After the seminal deep sets work of Zaheer et al [52], several efforts [5,20,38,40,41] were made to capture dependencies between set elements while retaining order invariance by design [27].…”