Learning Submodular Functions

Abstract

Abstract. Submodular functions are discrete functions that model laws of diminishing returns and enjoy numerous algorithmic applications that have been used in many areas, including combinatorial optimization, machine learning, and economics. In this work we use a learning theoretic angle for studying submodular functions. We provide algorithms for learning submodular functions, as well as lower bounds on their learnability. In doing so, we uncover several novel structural results revealing both extremal properties as well as regularities of submodular functions, of interest to many areas.Submodular functions are a discrete analog of convex functions that enjoy numerous applications and have structural properties that can be exploited algorithmically. They arise naturally in the study of graphs, matroids, covering problems, facility location problems, etc., and they have been extensively studied in operations research and combinatorial optimization for many years [8]. More recently submodular functions have become key concepts both in the machine learning and algorithmic game theory communities. For example, submodular functions have been used to model bidders' valuation functions in combinatorial auctions [12,6,3,14], and for solving feature selection problems in graphical models [11] or for solving various clustering problems [13]. In fact, submodularity has been the topic of several tutorials and workshops at recent major conferences in machine learning [1,9,10,2].Despite the increased interest on submodularity in machine learning, little is known about the topic from a learning theory perspective. In this work, we provide a statistical and computational theory of learning submodular functions in a distributional learning setting.Our study has multiple motivations. From a foundational perspective, submodular functions are a powerful, broad class of important functions, so studying their learnability allows us to understand their structure in a new way. To draw a parallel to the Boolean-valued case, a class of comparable breadth would be the class of monotone Boolean functions; the learnability of such functions has been intensively studied [4,5]. From an applications perspective, algorithms for learning submodular functions may be useful in some of the applications where these functions arise. For example, in the context of algorithmic game theory This note summarizes several results in the paper "Learning Submodular Functions", by Maria Florina Balcan and Nicholas Harvey, which appeared The 43rd ACM Symposium on Theory of Computing (STOC) 2011.

Learning with Submodular Functions: A Convex Optimization Perspective

“…It is of clear interest to learn the submodular functions directly from data. See, e.g., [13,77,197] for several approaches.…”

Gross substitutability: An algorithmic survey

“…However, the valuation v = r a + 2 · r b + 3 · r c does not satisfy gross substitutability. In order to see that, observe that for the price vector p = [4,5,4], D(v; p) = {{a}, {c}, {a, c}, {b, c}}. If the price of item c increases to ∞, i.e., p = [4, 5, ∞], then demand set becomes D(v; p ) = {{a}}.…”

“…Given a gross substitute valuation function v, a price vector p and a demanded set S ∈ D(v; p), then there is a tie breaking rule 4 for the greedy algorithm that produces the set S. Formally, if x 1 , x 2 , . .…”

“…Balcan and Harvey [4] show that even if we settle for an approximation of the valuation function v, we still can't improve the exponential dependency on n. We say thatv is an an α-approximation of v if for every S ⊆ [n], the relative error is at most α, i.e, v(S) ≤v(S) ≤ α · v(S). They show in [4] that if α < n 1/3 16 log(n) , then an exponential dependency on n is still necessary for the size of the representation.…”

“…They show in [4] that if α < n 1/3 16 log(n) , then an exponential dependency on n is still necessary for the size of the representation. Their is based on the following matroid construction: Theorem 11.1 (Balcan and Harvey [4]). For every n and > 0, there is a subset X ⊆ 2…”