Convex Analysis and Optimization with Submodular Functions: a Tutorial

Bach, Francis

doi:10.48550/arxiv.1010.4207

Cited by 5 publications

(14 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We are especially interested in the linear model described in (1), and in recovering w ⋆ consistently (i.e. recover w ⋆ exactly as n → ∞).…”

Section: Group Iterative Hard Thresholding For Overlapping Groupsmentioning

confidence: 99%

“…Recall that σ max (σ min ) are the maximum (minimum) singular value of Σ, and κ := σ max /σ min is the condition number of Σ. Theorem 3.2. Let the observations y follow the model in (1). Suppose w * is k * -group sparse and let f (w) := 1 2n Xw − y 2 2 .…”

Section: Linear Regression Guaranteesmentioning

confidence: 99%

“…Algorithm 2, which details the pseudocode for performing approximate projections, exactly corresponds to the greedy algorithm for submodular optimization [1], and this gives us a means to assess the quality of our projections.…”

Section: A Using Submodularity To Perform Projectionsmentioning

confidence: 99%

“…Specifically, our result holds for arbitrarily large κ, and arbitrary group structures. In particular, using IHT with greedy projections, we show that n = Ω (s log 1 ǫ + κ 2 k * log M ) log 1 ǫ samples suffice to recover ǫ-approximatation to w * when λ = 0. On the other hand, IHT for standard sparse regression [11] requires n = Ω(κ 2 s log p).…”

Section: Introductionmentioning

confidence: 99%

“…The main contribution of this work is to carefully use the greedy projection based procedure along with hard thresholding iterates to guarantee the convergence to the global optima as long as enough i.i.d. data points are generated from model (1).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Structured Sparse Regression via Greedy Hard-Thresholding

Jain¹,

Rao²,

Dhillon³

2016

Preprint

View full text Add to dashboard Cite

Several learning applications require solving high-dimensional regression problems where the relevant features belong to a small number of (overlapping) groups. For very large datasets and under standard sparsity constraints, hard thresholding methods have proven to be extremely efficient, but such methods require NP hard projections when dealing with overlapping groups. In this paper, we show that such NPhard projections can not only be avoided by appealing to submodular optimization, but such methods come with strong theoretical guarantees even in the presence of poorly conditioned data (i.e. say when two features have correlation ≥ 0.99), which existing analyses cannot handle. These methods exhibit an interesting computation-accuracy trade-off and can be extended to significantly harder problems such as sparse overlapping groups. Experiments on both real and synthetic data validate our claims and demonstrate that the proposed methods are orders of magnitude faster than other greedy and convex relaxation techniques for learning with group-structured sparsity.

show abstract

“…We are especially interested in the linear model described in (1), and in recovering w ⋆ consistently (i.e. recover w ⋆ exactly as n → ∞).…”

Section: Group Iterative Hard Thresholding For Overlapping Groupsmentioning

confidence: 99%

Section: Linear Regression Guaranteesmentioning

confidence: 99%

Section: A Using Submodularity To Perform Projectionsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Structured Sparse Regression via Greedy Hard-Thresholding

Jain¹,

Rao²,

Dhillon³

2016

Preprint

View full text Add to dashboard Cite

show abstract

Core-based criterion for extreme supermodular functions

Studený

Kroupa

2016

Discrete Applied Mathematics

View full text Add to dashboard Cite

We give a necessary and sufficient condition for extremality of a supermodular function based on its min-representation by means of (vertices of) the corresponding core polytope. The condition leads to solving a certain simple linear equation system determined by the combinatorial core structure. This result allows us to characterize indecomposability in the class of generalized permutohedra. We provide an in-depth comparison between our result and the description of extremality in the supermodular/submodular cone achieved by other researchers.

show abstract

An Algorithmic Theory of Dependent Regularizers, Part 1: Submodular Structure

Koepke,

Meila

2013

Preprint

View full text Add to dashboard Cite

We present an exploration of the rich theoretical connections between several classes of regularized models, network flows, and recent results in submodular function theory. This work unifies key aspects of these problems under a common theory, leading to novel methods for working with several important models of interest in statistics, machine learning and computer vision.In Part 1, we review the concepts of network flows and submodular function optimization theory foundational to our results. We then examine the connections between network flows and the minimum-norm algorithm from submodular optimization, extending and improving several current results. This leads to a concise representation of the structure of a large class of pairwise regularized models important in machine learning, statistics and computer vision.In Part 2, we describe the full regularization path of a class of penalized regression problems with dependent variables that includes the graph-guided LASSO and total variation constrained models. This description also motivates a practical algorithm. This allows us to efficiently find the regularization path of the discretized version of TV penalized models. Ultimately, our new algorithms scale up to high-dimensional problems with millions of variables.

show abstract

Convex Analysis and Optimization with Submodular Functions: a Tutorial

Abstract: all details of algorithms). In Section 9, we specialize some of our results to non-decreasing submodular functions. Finally, in Section 10, we present classical examples of submodular functions.

Cited by 5 publications

References 25 publications

Structured Sparse Regression via Greedy Hard-Thresholding

Structured Sparse Regression via Greedy Hard-Thresholding

Core-based criterion for extreme supermodular functions

An Algorithmic Theory of Dependent Regularizers, Part 1: Submodular Structure

Contact Info

Product

Resources

About