Renewable Estimation and Incremental Inference in Generalized Linear Models with Streaming Data Sets

Luo, Lan; Song, Peter X.‐K.

doi:10.1111/rssb.12352

Cited by 65 publications

(74 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although there have been some recent articles expressing concern about the online updating method for analysis of datastreams (see Schifano et al (2016); Luo and Song (2020) and the references therein), the issues of developing effective methodologies and theories for statistical modeling and inference of massive datastreams still remain. As most of the existing procedures and formulae were mainly developed based on the assumption that the observations come from the same model across time and sources.…”

Section: Model Setup and Our Contributionmentioning

confidence: 99%

Dynamic statistical inference in massive datastreams

Wang¹,

Du²,

Zou³

et al. 2021

Preprint

View full text Add to dashboard Cite

Modern technological advances have expanded the scope of applications requiring analysis of large-scale datastreams that comprise multiple indefinitely long time series. There is an acute need for statistical methodologies that perform online inference and continuously revise the model to reflect the current status of the underlying process. In this manuscript, we propose a dynamic statistical inference framework-named dynamic tracking and screening (DTS)-that is not only able to provide accurate estimates of the underlying parameters in a dynamic statistical model, but also capable of rapidly identifying irregular individual streams whose behavioral patterns deviate from the majority. Concretely, by fully exploiting the sequential feature of datastreams, we develop a robust estimation approach under a framework of varying coefficient model. The procedure naturally accommodates unequally-spaced design points and updates the coefficient estimates as new data arrive without the need to store historical data. A data-driven choice of an optimal smoothing parameter is accordingly proposed. Furthermore, we suggest a new multiple testing procedure tailored to the streaming environment. The resulting DTS scheme is able to adapt time-varying structures appropriately, track changes in the underlying models, and hence maintain high accuracy in detecting time periods during which individual streams exhibit irregular behavior. Moreover, we derive rigorous statistical guarantees of the procedure and investigate its finite-sample performance through simulation studies. We demonstrate the proposed methods through a mobile health example to estimate the timings when subjects' sleep and physical activities have unusual influence upon their mood.

show abstract

Section: Model Setup and Our Contributionmentioning

confidence: 99%

Dynamic statistical inference in massive datastreams

Wang¹,

Du²,

Zou³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Ongoing efforts have been made to develop modern distributed computing frameworks such as Hadoop, Spark and Storm (Ghemawat et al, 2003;Dean and Ghemawat, 2004;Bifet et al, 2015;Chintapalli et al, 2016). Aligned with these architectures, various statistical methods and algorithms for online estimation and inference have been proposed, including aggregated estimating equation (Lin and Xi, 2011), stochastic gradient descent (Robbins and Monro, 1951;Sakrison, 1965;Toulis and Airoldi, 2015), cumulative estimating equation (Schifano et al, 2016), as well as renewable estimator (Luo and Song, 2020).…”

Section: Introductionmentioning

confidence: 99%

Online Causal Inference with Application to Near Real-Time Post-Market Vaccine Safety Surveillance

Shi¹,

Luo²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Streaming data routinely generated by mobile phones, social networks, e-commerce, and electronic health records present new opportunities for near real-time surveillance of the impact of an intervention on an outcome of interest via causal inference methods. However, as data grow rapidly in volume and velocity, storing and combing data become increasingly challenging. The amount of time and effort spent to update analyses can grow exponentially, which defeats the purpose of instantaneous surveillance. Data sharing barriers in multi-center studies bring additional challenges to rapid signal detection and update. It is thus time to turn static causal inference to online causal learning that can incorporate new information as it becomes available without revisiting prior observations. In this paper, we present a framework for online estimation and inference of treatment effects leveraging a series of datasets that arrive sequentially without storing or re-accessing individual-level raw data. We establish estimation consistency and asymptotic normality of the proposed framework for online causal inference. In particular, our framework is robust to biased data batches in the sense that the proposed online estimator is asymptotically unbiased as long as the pooled data is a random sample of the target population regardless of whether each data batch is. We also provide an R package for analyzing streaming observational data that enjoys great computation efficiency compared to existing software packages for offline analyses. Our proposed methods are illustrated with extensive simulations and an application to sequential monitoring of adverse events post COVID-19 vaccine.

show abstract

“…An approach with more and smaller blocks sacrifices statistical efficiency for lower computational cost, whereas an approach with fewer and larger blocks retains desirable statistical efficiency at the price of increased computational cost. For the latter approach with large data blocks, using a computationally inexpensive and statistically efficient estimation procedure in each large data block, such as renewable learning Luo and Song (2020), would mitigate the trade-off between computational speed and statistical efficiency.…”

Section: Introductionmentioning

confidence: 99%

“…The renewable learning proposed by Luo and Song (2020) is an online estimation and inference method that uses the Rho architecture to maintain low computing cost with no loss of statistical efficiency. This streaming approach divides data into smaller data batches and updates parameter estimates sequentially from the first to the last data batch using only summary statistics and without re-accessing individual-level raw data from previous data batches.…”

Section: Introductionmentioning

confidence: 99%

Parallel-and-stream accelerator for computationally fast supervised learning

Hector¹,

Li²,

Song³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Two dominant distributed computing strategies have emerged to overcome the computational bottleneck of supervised learning with big data: parallel data processing in the MapReduce paradigm and serial data processing in the online streaming paradigm. Despite the two strategies' common divide-and-combine approach, they differ in how they aggregate information, leading to different trade-offs between statistical and computational performance. In this paper, we propose a new hybrid paradigm, termed a Parallel-and-Stream Accelerator (PASA), that uses the strengths of both strategies for computationally fast and statistically efficient supervised learning. PASA's architecture nests online streaming processing into each distributed and parallelized data process in a MapReduce framework. PASA leverages the advantages and mitigates the disadvantages of both the MapReduce and online streaming approaches to deliver a more flexible paradigm satisfying practical computing needs. We study the analytic properties and computational complexity of PASA, and detail its implementation for two key statistical learning tasks. We illustrate its performance through simulations and a large-scale data example building a prediction model for online purchases from advertising data.

show abstract

Renewable Estimation and Incremental Inference in Generalized Linear Models with Streaming Data Sets

Cited by 65 publications

References 55 publications

Dynamic statistical inference in massive datastreams

Dynamic statistical inference in massive datastreams

Online Causal Inference with Application to Near Real-Time Post-Market Vaccine Safety Surveillance

Parallel-and-stream accelerator for computationally fast supervised learning

Contact Info

Product

Resources

About