Adaptive sequential optimization with applications to machine learning

Abstract

A framework is introduced for solving a sequence of slowly changing optimization problems, including those arising in regression and classification applications, using optimization algorithms such as stochastic gradient descent (SGD). The optimization problems change slowly in the sense that the minimizers change at either a fixed or bounded rate. A method based on estimates of the change in the minimizers and properties of the optimization algorithm is introduced for adaptively selecting the number of samples needed from the distributions underlying each problem in order to ensure that the excess risk, i.e., the expected gap between the loss achieved by the approximate minimizer produced by the optimization algorithm and the exact minimizer, does not exceed a target level. Experiments with synthetic and real data are used to confirm that this approach performs well.

Adaptive sequential machine learning

“…Since {ρ i } are random variables, if we combine them by taking their maximum, any particular one step estimateρ i that is large will pull up the overall estimateρ n . This would driveρ n → diam(X ), as n → ∞, resulting in aρ n that is trivial in the limit of large n. 1 Note that a choice of t n that is no greater than 1/ √ n − 1 works here.…”

“…To apply cross-validation to our framework, we run C parallel versions of our approach and at time n we generate C different choices for the number of samples K (i) n . We then choose K n = max{K (1) n , . .…”

“…Consider solving a sequence of machine learning problems by minimizing the risk, i.e., expected value of a fixed loss function (w, z) at each time n: min w∈X f n (w) E z n ∼p n [ (w, z n )] ∀n ≥ 1 (1) where p n denotes the underlying (unknown) probabilistic model for the data z n at time n. For regression, z n = {x n , y n } corresponds to the {predictors, response} pair at time n and w parameterizes the regression model. For classification, z n = {x n , y n } corresponds to the {features, label} pair at time n, and w parameterizes the classifier.…”

“…Under this model, we find approximate minimizers w n of each function f n (w) by drawing K n samples {z n (k)} K n k=1 iid ∼ p n at time n. We do not make any assumptions about the Research reported in the paper was supported by the NSF under award CCF 11-11342, and by the Army Research Laboratory under Cooperative Agreement W911NF-17-2-0196, through the University of Illinois at Urbana-Champaign. Part of this work was presented in ICASSP 2016 [1] and Asilomar Conference 2016 [2]. Craig Wilson is now at Google.…”

“…We also consider extensions relevant to the machine learning setting, including a cost-based approach to select the number of samples with a cost budget over a fixed horizon, and an approach to applying cross-validation for model selection. Some of the results in this paper have reported in conference publications [1] and [2], which do not contain proofs of the key results due to space limitations. Moreover, we provide substantially more detailed numerical results and simulations in this paper than those given in [1] and [2].…”