Statistical methods and computing for big data

Wang, Chun; Chen, Ming‐Hui; Schifano, Elizabeth D.; Wu, Jing; Yan, Jun

doi:10.4310/sii.2016.v9.n4.a1

Cited by 98 publications

(73 citation statements)

References 54 publications

(90 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Online updating is a useful strategy for analyzing large‐scale data and streaming data, and recently, stochastic gradient decent has become a popular method for doing online updating (Wang, Chen, Schifano, Wu, & Yan, ). Furthermore, it has been shown that implicit SGD is more stable than explicit SGD.…”

Section: Discussionmentioning

confidence: 99%

Scalable statistical inference for averaged implicit stochastic gradient descent

Fang

2019

Scandinavian J Statistics

View full text Add to dashboard Cite

Stochastic gradient descent (SGD) provides a scalable way to compute parameter estimates in applications involving large‐scale data or streaming data. As an alternative version, averaged implicit SGD (AI‐SGD) has been shown to be more stable and more efficient. Although the asymptotic properties of AI‐SGD have been well established, statistical inferences based on it such as interval estimation remain unexplored. The bootstrap method is not computationally feasible because it requires to repeatedly resample from the entire data set. In addition, the plug‐in method is not applicable when there is no explicit covariance matrix formula. In this paper, we propose a scalable statistical inference procedure, which can be used for conducting inferences based on the AI‐SGD estimator. The proposed procedure updates the AI‐SGD estimate as well as many randomly perturbed AI‐SGD estimates, upon the arrival of each observation. We derive some large‐sample theoretical properties of the proposed procedure and examine its performance via simulation studies.

show abstract

Section: Discussionmentioning

confidence: 99%

Scalable statistical inference for averaged implicit stochastic gradient descent

Fang

2019

Scandinavian J Statistics

View full text Add to dashboard Cite

show abstract

“…In applications where the data are too large to analyze all at once, it has been proposed to use divide and recombine approaches, in which we divide the data into a number of smaller samples, analyze these subsamples individually, and then combine the results of these individual analyses …”

Section: Methodsmentioning

confidence: 99%

“…Recently, new statistical and/or computational methodologies have been proposed that scale problems to a reasonable size . These include the “divide and conquer” approach, where the data are divided into subsamples, the subsamples are analyzed in parallel and the results are then combined across subsamples .…”

Section: Introductionmentioning

confidence: 99%

Computationally efficient methods for fitting mixed models to electronic health records data

Rhodes

Turner

Payne

et al. 2018

Statistics in Medicine

View full text Add to dashboard Cite

Motivated by two case studies using primary care records from the Clinical Practice Research Datalink, we describe statistical methods that facilitate the analysis of tall data, with very large numbers of observations. Our focus is on investigating the association between patient characteristics and an outcome of interest, while allowing for variation among general practices. We explore ways to fit mixed-effects models to tall data, including predictors of interest and confounding factors as covariates, and including random intercepts to allow for heterogeneity in outcome among practices. We introduce (1) weighted regression and (2) meta-analysis of estimated regression coefficients from each practice. Both methods reduce the size of the dataset, thus decreasing the time required for statistical analysis. We compare the methods to an existing subsampling approach. All methods give similar point estimates, and weighted regression and meta-analysis give similar standard errors for point estimates to analysis of the entire dataset, but the subsampling method gives larger standard errors. Where all data are discrete, weighted regression is equivalent to fitting the mixed model to the entire dataset. In the presence of a continuous covariate, meta-analysis is useful. Both methods are easy to implement in standard statistical software.

show abstract

“…To work around this computational barrier, there are 3 immediate options: (1) increasing computing power, ie, a high-performance computing cluster, (2) modifying the fitting algorithm to improve computational performance, and (3) subsampling large data to a manageable and workable size. 1 Each option has both advantages and disadvantages, and depending on the modelling application, a combination of all 3 options might be the best approach. In this work we focus on option 3, namely, subsampling a large dataset into a manageable size.…”

Section: Introductionmentioning

confidence: 99%

A novel case‐control subsampling approach for rapid model exploration of large clustered binary data

Wright

Ryan

Pham

2017

Statistics in Medicine

View full text Add to dashboard Cite

In many settings, an analysis goal is the identification of a factor, or set of factors associated with an event or outcome. Often, these associations are then used for inference and prediction. Unfortunately, in the big data era, the model building and exploration phases of analysis can be time-consuming, especially if constrained by computing power (ie, a typical corporate workstation). To speed up this model development, we propose a novel subsampling scheme to enable rapid model exploration of clustered binary data using flexible yet complex model set-ups (GLMMs with additive smoothing splines). By reframing the binary response prospective cohort study into a case-control-type design, and using our knowledge of sampling fractions, we show one can approximate the model estimates as would be calculated from a full cohort analysis. This idea is extended to derive cluster-specific sampling fractions and thereby incorporate cluster variation into an analysis. Importantly, we demonstrate that previously computationally prohibitive analyses can be conducted in a timely manner on a typical workstation. The approach is applied to analysing risk factors associated with adverse reactions relating to blood donation.

show abstract

Statistical methods and computing for big data

Cited by 98 publications

References 54 publications

Scalable statistical inference for averaged implicit stochastic gradient descent

Scalable statistical inference for averaged implicit stochastic gradient descent

Computationally efficient methods for fitting mixed models to electronic health records data

A novel case‐control subsampling approach for rapid model exploration of large clustered binary data

Contact Info

Product

Resources

About