Robust, Sparse and Scalable Inference Using Bootstrap and Variable Selection Fusion

Abstract

This paper introduces a new regularized version of the robust τ -regression estimator for analyzing high-dimensional data sets subject to gross contamination in the response variables and covariates. We call the resulting estimator adaptive τ -Lasso that is robust to outliers and high-leverage points and simultaneously employs adaptive 1-norm penalty term to reduce the bias associated with large true regression coefficients. More specifically, this adaptive 1-norm penalty term assigns a weight to each regression coefficient. For a fixed number of predictors p, we show that the adaptive τ -Lasso has the oracle property with respect to variable-selection consistency and asymptotic normality for the regression vector corresponding to the true support, assuming knowledge of the true regression vector support. We then characterize its robustness via the finite-sample breakdown point and the influence function. We carry-out extensive simulations to compare the performance of the adaptive τ -Lasso estimator with that of other competing regularized estimators in terms of prediction and variable selection accuracy in the presence of contamination within the response vector/regression matrix and additive heavy-tailed noise. We observe from our simulations that the class of τ -Lasso estimators exhibits robustness and reliable performance in both contaminated and uncontaminated data settings, achieving the best or close-to-best for many scenarios, except for oracle estimators. However, it is worth noting that no particular estimator uniformly dominates others. We also validate our findings on robustness properties through simulation experiments.

Distributed inference for two‐sample U‐statistics in massive data analysis

“…Finally, the BLB estimator is obtained by averaging over each subset. Several follow‐up works includes: the subsampled double bootstrap (SDB) proposed by Sengupta et al (2016), which combines the idea of the BLB and a fast double bootstrap by only using one inflated resample for each subset; the bag of little fast and robust bootstrap (BLFRB) by Basiri et al (2015), which incorporates the BLB and a fast fixed‐point estimation technique from the fast and robust bootstrap method for each subset and thus can be applied to any estimator representable as smooth fixed‐point equations; a two‐stage algorithm from Mozafari‐Majd and Koivunen (2019) for datasets with sparsity and outlying observations, where variable selection is performed for each disjoint subset in the first stage and the actual inference is performed in the second stage by employing a robust MM‐estimator based extension of the BLB method.…”

“…However, for two‐sample U ‐statistics, the computational advantage of the BLB and SDB may not be available. Moreover, the bootstrap algorithms proposed in Basiri et al (2015) and Mozafari‐Majd and Koivunen (2019) cannot be applied to the two‐sample U ‐statistic directly. Therefore, we pursue other versions of bootstrap algorithms.…”

“…Another major contribution of this paper is to propose bootstrap algorithms to approximate the distributions of the distributed and blockwise linear two‐sample U ‐statistic, under both nondegenerate and degenerate cases. Bootstrap implementation for massive data has been studied by Kleiner et al (2014), Sengupta et al (2016), Basiri et al (2015), Mozafari‐Majd and Koivunen (2019), Yu et al (2020), and Chen and Peng (2021). For the distributed two‐sample U ‐statistic, a distributed bootstrap (DB) and a distributed weighted bootstrap (DWB) are introduced to estimate its distribution function.…”