Avoiding bad steps in Frank Wolfe variants

Abstract

The analysis of Frank Wolfe (FW) variants is often complicated by the presence of different kinds of "good" and "bad" steps. In this article we aim to simplify the convergence analysis of some of these variants by getting rid of such a distinction between steps, and to improve existing rates by ensuring a sizable decrease of the objective at each iteration. In order to do this, we define the Short Step Chain (SSC) procedure, which skips gradient computations in consecutive short steps until proper stopping conditions are satisfied. This technique allows us to give a unified analysis and converge rates in the general smooth non convex setting, as well as a linear convergence rate under a Kurdyka-Łojasiewicz (KL) property. While this setting has been widely studied for proximal gradient type methods, to our knowledge, it has not been analyzed before for the Frank Wolfe variants under study.An angle condition, ensuring that the directions selected by the methods have the steepest slope possible up to a constant, is used to carry out our analysis. We prove that this condition is satisfied on polytopes by the away step Frank-Wolfe (AFW), the pairwise Frank-Wolfe (PFW), and the Frank-Wolfe method with in face directions (FDFW).

Fast cluster detection in networks by first-order optimization

“…We used the stepsize α k = ᾱk with ᾱk given by (S1) for c = 2, corresponding to an estimate of 0.5 for the Lipschitz constant L of ∇h G . A gradient recycling scheme was adopted to use first order information more efficiently (see [27] for details). The code was written in MATLAB and the tests were performed on an Intel Core i7-10750H CPU 2.60GHz, 16GB RAM.…”

Frank-Wolfe and friends: a journey into projection-free first-order optimization methods

“…For the AFW, the PFW and the FDFW, linear rates with no bad steps (γ(k) = k) are given in [76] for non-convex objectives satisfying a Kurdyka-Lojasiewicz inequality. In [77], condition (61) was proved for the FW direction and orthographic retractions on some convex sets with smooth boundary.…”