Distributed Quantile Regression with Non-Convex Sparse Penalties

Abstract

The surge in data generated by IoT sensors has increased the need for scalable and efficient data analysis methods, particularly for robust algorithms like quantile regression, which can be tailored to meet a variety of situations, including nonlinear relationships, distributions with heavy tails, and outliers. This paper presents a sub-gradient-based algorithm for distributed quantile regression with non-convex, and non-smooth sparse penalties such as the Minimax Concave Penalty (MCP) and Smoothly Clipped Absolute Deviation (SCAD). These penalties selectively shrink non-active coefficients towards zero, addressing the limitations of traditional penalties like the l1-penalty in sparse models. Existing quantile regression algorithms with nonconvex penalties are designed for centralized cases, whereas our proposed method can be applied to distributed quantile regression using non-convex penalties, thereby improving estimation accuracy. We provide a convergence proof for our proposed algorithm and demonstrate through numerical simulations that it outperforms state-of-the-art algorithms in sparse and moderately sparse scenarios.

Smoothing ADMM for Sparse-Penalized Quantile Regression With Non-Convex Penalties

“…This section presents a comprehensive simulation study to evaluate the performance of the proposed smoothing timeincreasing ADMM (SIAD) algorithm in the context of sparse quantile regression. We compare the SIAD algorithm with existing state-of-the-art approaches, including QICD [27], LPA [18], LSCD [18], also sub-gradient method (SUB) [30]. The performance of these algorithms is assessed in terms of convergence rate, efficiency in terms of mean square error (MSE), and accuracy in recognizing active and non-active coefficients.…”

“…A recently proposed sub-gradient algorithm, designed for weakly convex functions, achieves a convergence rate such that, after K iterations, it converges to a O(K − 1 4 )-stationary point, based on the derivative of the Moreau-envelope function [29]. This algorithm can be adapted for quantile regression penalized with MCP or SCAD [30], but the result depends on the step size, and the convergence speed might not be efficient. Thus, there is ongoing research to find more robust and efficient solutions for non-smooth, non-convex optimization problems.…”

ADMM for Sparse-Penalized Quantile Regression with Non-Convex Penalties

“…In this section, we present a comprehensive simulation study to evaluate the performance of the proposed smoothing timeincreasing penalty ADMM (SIAD) algorithm in the context of sparse quantile regression. We compare the SIAD algorithm with existing state-of-the-art approaches, including QICD [27], LPA [18], LSCD [18], also sub-gradient method [30]. The performance of these algorithms is assessed in terms of convergence rate, efficiency in terms of mean square error (MSE), and accuracy in recognizing active and non-active coefficients.…”

“…A recently proposed sub-gradient algorithm can handle weakly convex functions, achieving a convergence rate of O(K − 1 4 ) to the 1 K -stationary point based on the derivative of Moreau-envelope function [29]. This algorithm can be adapted for quantile regression penalized with MCP or SCAD [30], but the result depends on the step size, and the convergence speed might not be efficient. Thus, there is ongoing research to find more robust and efficient solutions for non-smooth, non-convex optimization problems.…”