Penalized linear regression approaches that include an L1 term have become an important tool in day-to-day statistical data analysis. One prominent example is the least absolute shrinkage and selection operator (Lasso), though the class of L1 penalized regression operators also includes the fused and graphical Lasso, the elastic net, etc. Although the L1 penalty makes their objective function convex, it is not differentiable everywhere, motivating the development of proximal gradient algorithms such as Fista, the current gold standard in the literature. In this work, we take a different approach based on smoothing. The methodological contribution of our article is threefold: (1) We introduce a unified framework to compute closed-form smooth surrogates of a whole class of L1 penalized regression problems using Nesterov smoothing. The surrogates preserve the convexity of the original (unsmoothed) objective functions, are uniformly close to them, and have closed-form derivatives everywhere for efficient minimization via gradient descent; (2) We prove that the estimates obtained with the smooth surrogates can be made arbitrarily close to the ones of the original (unsmoothed) objective functions, and provide explicitly computable bounds on the accuracy of our estimates; (3) We propose an iterative algorithm to progressively smooth the L1 penalty which increases accuracy and is virtually free of tuning parameters. The proposed methodology is applicable to a large class of L1 penalized regression operators, including all the operators mentioned above. Using simulation studies, we compare our framework to current gold standards such as Fista, glmnet, gLasso, etc. Our simulation results suggest that our proposed smoothing framework provides estimates of equal or higher accuracy than the gold standards while keeping the aforementioned theoretical guarantees and having roughly the same asymptotic runtime scaling.
We consider solving a high dimensional linear regression problem, using LASSO to account for sparsity. Though the LASSO objective function is convex, it is not differentiable everywhere, making the use of gradient descent methods for minimization not straightforward. To avoid this technical issue, we suggest to use Nesterov smoothing of the LASSO objective function which enables us to compute closed form derivatives for efficient and fast minimization. The contribution of this work is threefold: (1) We propose an algorithm to efficiently compute estimates of the LASSO regression parameters; (2) we prove explicit bounds on the accuracy of the obtained estimates which show that the estimates obtained through the smoothed problem can be made arbitrary close to the ones of the original (unsmoothed) LASSO problem; (3) we propose an iterative procedure of the proposed approach to progressively smooth the objective function, which facilitates minimization and increases accuracy. Using simulation studies for polygenic risk scores based on genetic data from a genome-wide association study (GWAS) for COPD, we evaluate accuracy and runtime of the proposed approaches and compare them with the current standard in the literature. The results of the simulation studies suggest that the proposed methodology, progressive smoothing, provides estimates with higher accuracy than standard approaches that rely on GLMNET and that the differences are of practical relevance, while the computation time for progressive smoothing increases only be a constant factor, compared to GLMNET.
High dimensional linear regression problems are often fitted using Lasso approaches. Although the Lasso objective function is convex, it is not differentiable everywhere, making the use of gradient descent methods for minimization not straightforward. To avoid this technical issue, we apply Nesterov smoothing to the original (unsmoothed) Lasso objective function. We introduce a closed-form smoothed Lasso which preserves the convexity of the Lasso function, is uniformly close to the unsmoothed Lasso, and allows us to obtain closed-form derivatives everywhere for efficient and fast minimization via gradient descent. Our simulation studies are focused on polygenic risk scores using genetic data from a genome-wide association study (GWAS) for chronic obstructive pulmonary disease (COPD). We compare accuracy and runtime of our approach to the current gold standard in the literature, the FISTA algorithm. Our results suggest that the proposed methodology provides estimates with equal or higher accuracy than the the FISTA algorithm while having the same asymptotic runtime scaling.
We revisit the problem of estimating the center of symmetry θ of an unknown symmetric density f . Although Stone (1975), Van Eeden (1970), andSacks (1975) constructed adaptive estimators of θ in this model, their estimators depend on external tuning parameters. In an effort to reduce the burden of tuning parameters, we impose an additional restriction of log-concavity on f . We construct truncated one-step estimators which are adaptive under the log-concavity assumption. Our simulations indicate that the untruncated version of the one step estimator, which is tuning parameter free, is also asymptotically efficient. We also study the maximum likelihood estimator (MLE) of θ in the shape-restricted model.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.