Online Censoring for Large-Scale Regressions with Application to Streaming Big Data

Abstract

On par with data-intensive applications, the sheer size of modern linear regression problems creates an ever-growing demand for efficient solvers. Fortunately, a significant percentage of the data accrued can be omitted while maintaining a certain quality of statistical inference with an affordable computational budget. This work introduces means of identifying and omitting less informative observations in an online and data-adaptive fashion. Given streaming data, the related maximum-likelihood estimator is sequentially found using first- and second-order stochastic approximation algorithms. These schemes are well suited when data are inherently censored or when the aim is to save communication overhead in decentralized learning setups. In a different operational scenario, the task of joint censoring and estimation is put forth to solve large-scale linear regressions in a centralized setup. Novel online algorithms are developed enjoying simple closed-form updates and provable (non)asymptotic convergence guarantees. To attain desired censoring patterns and levels of dimensionality reduction, thresholding rules are investigated too. Numerical tests on real and synthetic datasets corroborate the efficacy of the proposed data-adaptive methods compared to data-agnostic random projection-based alternatives.

A survey of machine learning for big data processing

“…The SP and online learning techniques for big data analytics described in [110] provides a good research direction for future work. Based on this, in [117], the authors developed online algorithms for large-scale regressions with application to streaming big data. In addition, Slavakis and Giannakis further used accelerated stochastic approximation method with online and modular learning algorithms to deal with a large class of nonconvex data models [118].…”

Decentralized RLS With Data-Adaptive Censoring for Regressions Over Large-Scale Networks

“…As time evolves, all local estimates consent on the centralized RLS solution. This paper builds on both [4] and [16] by developing censoring-based decentralized RLS algorithms, thus catering to efficient online linear regression over largescale networks.…”

Quickest convergence of online algorithms via data selection

“…Dividing numerator and denominator byN , (6) A threshold γ satisfying this necessary condition can be found using the Robbins-Monro iteration [14], which iteratively sets γ n+1 = γ n + µ n max(∆n − γ n , 0) − γ n Tc Tu (10) where µ n > 0 represents the step size. if∆ n|k ≥ γ n then 6: k ← k + 1 7:ŵ k = Fw(dn,ŵ k−1 , S k−1 ) 8: …”

“…For example, in [6,7] observations with low innovations are discarded to reduce computational complexity of estimating linear regression coefficients. Similar principles are exploited in [6][7][8][9][10], where non-informative observations are censored to minimize the communication overhead in bandwidth-constrained sensor networks.…”

“…For example, in [6,7] observations with low innovations are discarded to reduce computational complexity of estimating linear regression coefficients. Similar principles are exploited in [6][7][8][9][10], where non-informative observations are censored to minimize the communication overhead in bandwidth-constrained sensor networks. The related randomized Kaczmarz algorithm [11][12][13] accomplishes data selection according to the norm of the prediction vectors, but cannot be implemented in a purely online fashion since the selection probabilities depend on the norm of all other vectors.…”