Convergence Analysis of Deflected Conditional Approximate Subgradient Methods

Abstract

Abstract. Subgradient methods for nondifferentiable optimization benefit from deflection, i.e., defining the search direction as a combination of the previous direction and the current subgradient. In the constrained case they also benefit from projection of the search direction onto the feasible set prior to computing the steplength, that is, from the use of conditional subgradient techniques. However, combining the two techniques is not straightforward, especially if an inexact oracle is available which can only compute approximate function values and subgradients. We present a convergence analysis of several different variants, both conceptual and implementable, of approximate conditional deflected subgradient methods. Our analysis extends the available results in the literature by using the main stepsize rules presented so far, while allowing deflection in a more flexible way. Furthermore, to allow for (diminishing/square summable) rules where the stepsize is tightly controlled a priori, we propose a new class of deflection-restricted approaches where it is the deflection parameter, rather than the stepsize, which is dynamically adjusted using the "target value" of the optimization sequence. For both Polyak-type and diminishing/square summable stepsizes, we propose a "correction" of the standard formula which shows that, in the inexact case, knowledge about the error computed by the oracle (which is available in several practical applications) can be exploited in order to strengthen the convergence properties of the method. The analysis allows for several variants of the algorithm; at least one of them is likely to show numerical performances similar to these of "heavy ball" subgradient methods, popular within backpropagation approaches to train neural networks, while possessing stronger convergence properties.

A storm of feasibility pumps for nonconvex MINLP

“…Thus, SPMs can allow weighted simultaneous projection and relaxation; we mention in passing that these algorithms bear more than a casual resemblance with subgradient methods [18], as discussed in [5, §7]. The scheme (3)-(4) clearly corresponds to α i h = 1 ("unrelaxed") and λ i (i mod 2) = 1 ("cyclic control"), so that only one among the two projections actually need to be computed at any iteration (z i = v 2i−1 and w i = v 2i ).…”

Dynamic smoothness parameter for fast gradient methods

“…The parameters α = 1.2 andî = 10 are easily derived from the (average) convergence plot for f lb = f * , and used uniformly for all instances (being the convergence plots almost identical). Figures 2 and 3 show that the new dynamic strategy (17), albeit not as efficient as (6) with the accurate estimate of f * , is still consistently superior to the static strategy (5). Furthermore, it is resilient to rather inaccurate estimates of f * ; indeed, it is by far the preferable option in Figures 4 and 5.…”

“…While this suggests that the FG may benefit from some tuning, exploring this issue is out of the scope of the present paper. Therefore, in Figures 6 and 7, we mainly report the results of the three rules when using B = 1, denoted by (5), (6) and (17), while only plotting in Figure 6, the results of the original rule (6) to show how much worse the performances are (those of the other rules are similarly degraded). All in all, the results closely mirror those of the KR.…”

“…This is true in particular when the LR decomposes into several smaller subproblems (e.g., [8,9] and the references therein). The LD is typically a non-smooth problem, and it is usually solved by algorithms of two different families: subgradient methods (SM) [6,9,14] and bundle methods (BM) [7,8,10]. The former are easier to implement and their iteration cost is dominated by the function computation, whereas the latter are more complex and require the solution of a (potentially, costly) subproblem at each iteration; however, they have better convergence in practice.…”

A stabilized structured Dantzig–Wolfe decomposition method

“…A weaker version (that is sufficient in practice) showing that v(29) → 0 even if infinitely many aggregations are performed is also possible: one could then resort to employing the "poorman cutting-plane model" fx (cf. §4.1) at all steps, which basically makes the algorithm a subgradient-type approach with deflection [4,13] and results in much slower convergence [10,23]. Thus, this kind of development seems to be of little interest in our case.…”