Dual subgradient algorithms for large-scale nonsmooth learning problems

Abstract

International audience"Classical" First Order (FO) algorithms of convex optimization, such as Mirror Descent algorithm or Nesterov's optimal algorithm of smooth convex optimization, are well known to have optimal (theoretical) complexity estimates which do not depend on the problem dimension. However, to attain the optimality, the domain of the problem should admit a "good proximal setup". The latter essentially means that (1) the problem domain should satisfy certain geometric conditions of "favorable geometry", and (2) the practical use of these methods is conditioned by our ability to compute at a moderate cost proximal transformation at each iteration. More often than not these two conditions are satisfied in optimization problems arising in computational learning, what explains why proximal type FO methods recently became methods of choice when solving various learning problems. Yet, they meet their limits in several important problems such as multi-task learning with large number of tasks, where the problem domain does not exhibit favorable geometry, and learning and matrix completion problems with nuclear norm constraint, when the numerical cost of computing proximal transformation becomes prohibitive in large-scale problems. We propose a novel approach to solving nonsmooth optimization problems arising in learning applications where Fenchel-type representation of the objective function is available. The approach is based on applying FO algorithms to the dual problem and using the accuracy certificates supplied by the method to recover the primal solution. While suboptimal in terms of accuracy guaranties, the proposed approach does not rely upon "good proximal setup" for the primal problem but requires the problem domain to admit a Linear Optimization oracle--the ability to efficiently maximize a linear form on the domain of the primal problem

Semi-Inner-Products for Convex Functionals and Their Use in Image Decomposition

“…and therefore the spectral filtering proposed above (Eqs. (16), (19), (20) ,which hold for the general one-homogeneous case) decomposes f correctly.…”

“…1 an example of spectral TV processing is shown with the response of the four filters defined above in Eqs. (19) through (22).…”

“…This measure has been widely used in the theoretical analysis of classification, clustering and convex optimization algorithms, see e.g. [7,17,30,19]. Specifically for image processing, a significant branch of studies has presented iterative variational solutions, new evolution formulations and numerical solvers based on the Bregman distance, especially in relation to total-variation and other one-homogeneous regularizing functionals [35,15,29,41,33], see a recent review of the topic in [13].…”

Hessian Barrier Algorithms for Linearly Constrained Optimization Problems

“…First introduced by Nemirovski and Yudin [39] for non-smooth problems, the mirror descent algorithm and its variants have met with prolific success in convex programming [9], online and stochastic optimization [46], variational inequalities [40], non-cooperative games [19,38], and many other fields of optimization theory and its applications. Nevertheless, despite the appealing convergence properties of (MD), it is often difficult to calculate the update step from x to x + when the problem's feasible region X is not "prox-friendly" -i.e., when there is no efficient oracle for solving the convex optimization problem in (MD) [24]. With this in mind, our main goal in this paper is to provide a convergent, forward discretization of (HRGD) which does not require solving a convex optimization problem at each update step.…”

New results on subgradient methods for strongly convex optimization problems with a unified analysis

“…The problems of minimizing continuously differentiable convex functions with Lipschitz continuous gradients.These two classes of convex problems can also be reformulated as structured convex problems, which have been receiving much attention in terms of both theoretical and application aspects. In particular, studies of (sub)gradient-based methods for the class of "smoothable" functions [1,6,9,27,35,36], the class of composite problems [1,5,8,17,18,19,26,38,42,43], and the class of weakly smooth problems [11,12,39,40] are notably important.In this paper, we particularly focus on the following two kinds of (sub)gradient methods: the Proximal (sub)Gradient Method (PGM) and the Conditional Gradient Method (CGM). Both methods may require easy-to-solve subproblems at each iteration.The PGM is executed using a prox-function to define a reasonable proximal operator.…”