Scalable statistical inference for averaged implicit stochastic gradient descent

Abstract

Stochastic gradient descent (SGD) provides a scalable way to compute parameter estimates in applications involving large‐scale data or streaming data. As an alternative version, averaged implicit SGD (AI‐SGD) has been shown to be more stable and more efficient. Although the asymptotic properties of AI‐SGD have been well established, statistical inferences based on it such as interval estimation remain unexplored. The bootstrap method is not computationally feasible because it requires to repeatedly resample from the entire data set. In addition, the plug‐in method is not applicable when there is no explicit covariance matrix formula. In this paper, we propose a scalable statistical inference procedure, which can be used for conducting inferences based on the AI‐SGD estimator. The proposed procedure updates the AI‐SGD estimate as well as many randomly perturbed AI‐SGD estimates, upon the arrival of each observation. We derive some large‐sample theoretical properties of the proposed procedure and examine its performance via simulation studies.

Renewable Estimation and Incremental Inference in Generalized Linear Models with Streaming Data Sets

“…Toulis et al (2014) proposed an AISGD algorithm that was shown to be more stable than the explicit SGD algorithm. Later, Fang (2019)…”

“…() proposed an AISGD algorithm that was shown to be more stable than the explicit SGD algorithm. Later, Fang () extended AISGD by adding a random weight to the gradient, resulting in the following implicit SGD procedure:…”

“…For the AISGD method, we use the R package sgd with one‐dimensional learning rate (Xu, ) and hyperparameters set at α =1, γ 0 =1 and . Following Fang (), we set S =200 bootstrap samples. Computational efficiency is also assessed by (e)computation time CTime and (f)running time RTime. …”

“…Consequently, standard errors are not properly estimated by Fang's perturbation resampling method, resulting in 0% coverage probability. According to Fang (2019), the resampling method may not be able to deal with high dimensional large-scale data.…”

“…Although some relevant analytic expressions for the asymptotic variances have been derived in both explicit and implicit SGD (Toulis and Airold, 2017), the work of developing on-line confidence intervals remains unexplored because of the lack of suitable asymptotic results that may be directly applied to establish on-line inference. Recently Fang (2019) proposed a perturbation-based resampling method to construct confidence intervals for AISGD. Even though this on-line bootstrap procedure can be parallelized to improve computational efficiency, as shown in the simulation studies later in the paper, it does not achieve desirable statistical efficiency and may produce misleading inference in the case of large regression parameters.…”