CSG: A stochastic gradient method for a wide class of optimization problems appearing in a machine learning or data-driven context

Abstract

A recent article introduced the continuous stochastic gradient method (CSG) for the efficient solution of a class of stochastic optimization problems. While the applicability of known stochastic gradient type methods is typically limited to expected risk functions, no such limitation exists for CSG. This advantage stems from the computation of design dependent integration weights, allowing for optimal usage of available information and therefore stronger convergence properties. However, the nature of the formula used for these integration weights essentially limited the practical applicability of this method to problems in which stochasticity enters via a low-dimensional and sufficiently simple probability distribution. In this paper we significantly extend the scope of the CSG method by presenting alternative ways to calculate the integration weights. A full convergence analysis for this new variant of the CSG method is presented and its efficiency is demonstrated in comparison to more classical stochastic gradient methods by means of a number of problem classes relevant to stochastic optimization and machine learning.

Optimizing Color of Particulate Products

“…The initial design was chosen at random. Furthermore, the integration weights were obtained by the inexact hybrid method proposed in [14], while the step size was determined by the Armijo-type line search introduced in [15]. In this setting, it appears that the CSG approximations almost immediately give very close approximations to the full objective function, see Figure 3b.…”

“…The approximations to the objective function and the full gradient generated by the CSG method are asymptotically exact, as the next proposition shows. Proposition 3.1 Denote by Ĝn and Ĵn the CSG approximations to ∇J(u n ) and J(u n ) respectively, where the integration weights were calculated by one of the methods listed in [14]. Then, it holds…”

“…The weights (α i ) i=1,...,n appearing in this convex combination, also called integration weights, can be obtained in a number of different fashions, based on the concrete setting. An overview is given in [14]. The main idea behind the integration weights is visualized in Figure 2.…”

“…Instead of actually calculating the Voronoi tessalation to obtain the integration weights, one can use efficient approximations, e.g. the methods listed in [14]…”

“…It was shown in [13][14][15] that the approximation error tends to zero during the course of the iteration. Therefore, CSG in the limit behaves like a full gradient scheme and inherits similar convergence properties, see Theorem 3.2.…”