The condition number of a function relative to a set

Gutman, David H.; Peña, Javier

doi:10.1007/s10107-020-01510-4

Cited by 14 publications

(13 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The linear convergence results for strongly convex objectives are extended to compositions of strongly convex objectives with affine transformations in Beck and Shtern (2017), Lacoste-Julien and Jaggi (2015), Peña and Rodriguez (2018). In Gutman and Pena (2021), the linear convergence results for the AFW and the FW method with minimum in the interior are extended with respect to a generalized condition number L f ,C,D /μ f ,C,D , with D a distance function on C.…”

Section: Linear Convergence Under An Angle Conditionmentioning

confidence: 99%

Frank–Wolfe and friends: a journey into projection-free first-order optimization methods

2021

View full text Add to dashboard Cite

Invented some 65 years ago in a seminal paper by Marguerite Straus-Frank and Philip Wolfe, the Frank–Wolfe method recently enjoys a remarkable revival, fuelled by the need of fast and reliable first-order optimization methods in Data Science and other relevant application areas. This review tries to explain the success of this approach by illustrating versatility and applicability in a wide range of contexts, combined with an account on recent progress in variants, improving on both the speed and efficiency of this surprisingly simple principle of first-order optimization.

show abstract

Section: Linear Convergence Under An Angle Conditionmentioning

confidence: 99%

Frank–Wolfe and friends: a journey into projection-free first-order optimization methods

2021

View full text Add to dashboard Cite

show abstract

“…Since w ∈ W , it follows that x := prox f ( w) ∈ argmin{f (x) + g(x)} and ū := prox f * ( w) ∈ argmax{−f * (u) − g * (−u)}. Also, as the proximal mapping is Lipschitz continuous with parameter one (10)…”

Section: Algorithm 1 Douglas-rachfordmentioning

confidence: 99%

“…More generally, Lemma 4 in Section 5 below shows that if f is of the form f = h • A where h is strongly convex and A is a linear mapping, then f is strongly convex relative to A −1 (X) ⊆ dom(∂f ) for any nonempty closed convex set X ⊆ dom(∂h). The above concept of relative strong convexity is closely related to and inspired by recent developments in [10].…”

Section: Strong Convexity and Smoothnessmentioning

confidence: 99%

Linear convergence of the Douglas-Rachford algorithm via a generic error bound condition

Peña¹,

Vera²,

Zuluaga³

2021

Preprint

View full text Add to dashboard Cite

We provide new insight into the convergence properties of the Douglas-Rachford algorithm for the problem min x {f (x) + g(x)}, where f and g are convex functions. Our approach relies on and highlights the natural primal-dual symmetry between the above problem and its Fenchel dual min u {f * (u) + g * (u)} where g * (u) := g * (−u). Our main development is to show the linear convergence of the algorithm when a natural error bound condition on the Douglas-Rachford operator holds. We leverage our error bound condition approach to show and estimate the algorithm's linear rate of convergence for three special classes of problems. The first one is when f or g and f * or g * are strongly convex relative to the primal and dual optimal sets respectively. The second one is when f and g are piecewise linear-quadratic functions. The third one is when f and g are the indicator functions of closed convex cones. In all three cases the rate of convergence is determined by a suitable measure of well-posedness of the problem. In the conic case, if the two closed convex cones are a linear subspace L and R n + , we establish the following stronger finite termination result: the Douglas-Rachford algorithm identifies the maximum support sets for L ∩ R n + and L ⊥ ∩ R n + in finitely many steps. Our developments have straightforward extensions to the more general linearly constrained problem min x,y {f (x)+g(y) : Ax+By = b} thereby highlighting a direct and straightforward relationship between the Douglas-Rachford algorithm and the alternating direction method of multipliers (ADMM).

show abstract

“…Proof: The condition number is defined as ζ = ℓ/c in literatures [47], [48]. Recalling (17) and (21), we obtain…”

Section: Under the High Gradient Circumstance (D) Consider That The P...mentioning

confidence: 99%

Activated Gradients for Deep Neural Networks

Liu¹,

Chen²,

Du³

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep neural networks often suffer from poor performance or even training failure due to the ill-conditioned problem, the vanishing/exploding gradient problem, and the saddle point problem. In this paper, a novel method by acting the gradient activation function (GAF) on the gradient is proposed to handle these challenges. Intuitively, the GAF enlarges the tiny gradients and restricts the large gradient. Theoretically, this paper gives conditions that the GAF needs to meet, and on this basis, proves that the GAF alleviates the problems mentioned above. In addition, this paper proves that the convergence rate of SGD with the GAF is faster than that without the GAF under some assumptions. Furthermore, experiments on CIFAR, ImageNet, and PASCAL visual object classes confirm the GAF's effectiveness. The experimental results also demonstrate that the proposed method is able to be adopted in various deep neural networks to improve their performance. The source code is publicly available at https://github.com/LongJin-lab/Activated-Gradients-for-Deep-Neural-Networks.

show abstract

The condition number of a function relative to a set

Cited by 14 publications

References 25 publications

Frank–Wolfe and friends: a journey into projection-free first-order optimization methods

Frank–Wolfe and friends: a journey into projection-free first-order optimization methods

Linear convergence of the Douglas-Rachford algorithm via a generic error bound condition

Activated Gradients for Deep Neural Networks

Contact Info

Product

Resources

About