Complexity of First-Order Methods for Differentiable Convex Optimization

Abstract

This is a short tutorial on complexity studies for differentiable convex optimization. A complexity study is made for a class of problems, an "oracle" that obtains information about the problem at a given point, and a stopping rule for algorithms. These three items compose a scheme, for which we study the performance of algorithms and problem complexity. Our problem classes will be quadratic minimization and convex minimization in R n . The oracle will always be first order. We study the performance of steepest descent and Krylov space methods for quadratic function minimization and Nesterov's approach to the minimization of differentiable convex functions.

Fooling the Big Picture in Classification Tasks

“…Our formulations entail solving convex programs with N variables and linear constraints. The complexity of an iterative solver to said programs is measured by the complexity of the initialization procedure, the worst case complexity per iteration for a given target precision [ 24 ], and the convergence rate. Given our rADMM, the initialization process consists of calculating the matrix and computing the matrix , which has computational complexity for Algorithm 1 and the NOC approximation, and for the REC method.…”

Cell-Free Massive MIMO: Joint Maximum-Ratio and Zero-Forcing Precoder With Power Control

“…However, the worst case of computation complexity depends on the number of iterations of Algorithm 1, which is related to the convergence performance. The complexity of differentiable convex optimization has been reported in [31], however, the computation saving for non-convex problems has not been reported to the best of the authors' knowledge.…”

The Use of Quadratic Regularization with a Cubic Descent Condition for Unconstrained Optimization

“…See, for example, [2,4,5,6,8,11,14,19,21]. A review of complexity results for the convex case, in addition to novel techniques, can be found in [12].…”