OpReg-Boost: Learning to Accelerate Online Algorithms with Operator Regression

Abstract

This paper presents a new regularization approach -termed OpReg-Boost -to boost the convergence and lessen the asymptotic error of online optimization and learning algorithms. In particular, the paper considers online algorithms for optimization problems with a time-varying (weakly) convex composite cost. For a given online algorithm, OpReg-Boost learns the closest algorithmic map that yields linear convergence; to this end, the learning procedure hinges on the concept of operator regression. We show how to formalize the operator regression problem and propose a computationally-efficient Peaceman-Rachford solver that exploits a closed-form solution of simple quadratically-constrained quadratic programs (QCQPs). Simulation results showcase the superior properties of OpReg-Boost w.r.t. the more classical forward-backward algorithm, FISTA, and Anderson acceleration, and with respect to its close relative convex-regression-boost (CvxReg-Boost) which is also novel but less performing.

Neural Fixed-Point Acceleration for Convex Optimization

“…(Li and Malik, 2016;Finn et al, 2017;Wichrowska et al, 2017;Andrychowicz et al, 2016;Metz et al, 2019Metz et al, , 2021Gregor and LeCun, 2010), focuses on learning better solutions to parameter learning problems that arise for machine learning tasks. Bastianello et al (2021) approximates the fixed-point iteration with the closest contractive fixed-point iteration.…”