Distributed coordinate descent for generalized linear models with regularization

Trofimov, Ilya; Genkin, Alexander

doi:10.1134/s1054661817020122

Cited by 7 publications

(8 citation statements)

References 23 publications

(41 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Also, they do not allow the re-use of arbitrary local solvers. On the theoretical side, the convergence rate results provided by Mahajan et al (2017); Trofimov and Genkin (2016); and Yuan et al (2012) are not explicit convergence rates but only asymptotic, as the quadratic upper bounds are not explicitly controlled for safety as with our σ .…”

Section: Related Workmentioning

confidence: 82%

“…If hypothetically each of our quadratic subproblems G σ k (∆α [k] ) as defined in (10) were to be minimized exactly, the resulting steps could be interpreted as block-wise Newton-type steps on each coordinate block k, where the Newton-subproblem is modified to also contain the L 1 -regularizer (Mahajan et al, 2017;Yuan et al, 2012;Qu et al, 2016). While Mahajan et al (2017) allows a fixed accuracy for these subproblems, but not arbitrary approximation quality Θ as in our framework, the works of Trofimov and Genkin (2016); Yuan et al (2012);and Yen et al (2015) assume that the quadratic subproblems are solved exactly. Therefore, these methods are not able to freely trade off communication and computation.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

CoCoA: A General Framework for Communication-Efficient Distributed Optimization

Smith,

Forte,

et al. 2016

Preprint

View full text Add to dashboard Cite

The scale of modern datasets necessitates the development of efficient distributed optimization methods for machine learning. We present a general-purpose framework for distributed computing environments, CoCoA, that has an efficient communication scheme and is applicable to a wide variety of problems in machine learning and signal processing. We extend the framework to cover general non-strongly-convex regularizers, including L1-regularized problems like lasso, sparse logistic regression, and elastic net regularization, and show how earlier work can be derived as a special case. We provide convergence guarantees for the class of convex regularized loss minimization objectives, leveraging a novel approach in handling non-strongly-convex regularizers and non-smooth loss functions. The resulting framework has markedly improved performance over state-of-the-art methods, as we illustrate with an extensive set of experiments on real distributed datasets.

show abstract

Section: Related Workmentioning

confidence: 82%

Section: Related Workmentioning

confidence: 99%

CoCoA: A General Framework for Communication-Efficient Distributed Optimization

Smith,

Forte,

et al. 2016

Preprint

View full text Add to dashboard Cite

show abstract

“…The GLM fits generalized linear models to the data by maximizing the log-likelihood. The GLM is a flexible, robust and highly interpretable model provide useful approach to modeling classification [79].…”

Section: Generalized Linear Model (Glm)mentioning

confidence: 99%

Intelligent Hybrid Feature Selection for Textual Sentiment Classification

Khan¹,

Alam

Lee

2021

IEEE Access

View full text Add to dashboard Cite

Sentiment Analysis (SA) aims to extract useful information from online Unstructured User-Generated Contents (UUGC) and classify them into positive and negative classes. State-of-the-art techniques for SA suffer a high dimensional feature space because of noisy and irrelevant features from the UUGC. Researchers have also proposed feature extraction and selection techniques to reduce high dimensional feature space, but they fall short in extracting and selecting the most effective sentiment features for sentiment model learning. Effective feature extraction and selection are significant for the SA because they can boost the learning algorithm's predictive performance while reducing the high-dimensional feature space. To address these concerns, we propose an Intelligent Hybrid Feature Selection for Sentiment Analysis (IHFSSA) based on ensemble learning methods. IHFSSA first identifies sentiment features in the review text utilizing Penn Treebank part-of-speech tagset and integrated Wide Coverage Sentiment Lexicons (WCSL).The sentiment features subset is then selected employing a fast and simple rank-based ensemble of multiple filters feature selection method. The selected sentiment features are further refined by applying a wrapperbased backward feature selection method. Finally, for textual sentiment classification, the well-known classification algorithms Support Vector Machine (SVM), Naive Bayes (NB), Generalized Linear Model (GLM) are trained in the ensemble model on the refined sentiment feature set. The in-depth evaluation using heterogeneous domain benchmark datasets demonstrates that IHFSSA outperforms existing SA techniques.

show abstract

“…Line-search vs Trust-region. Line-search techniques are a popular way to guarantee convergence and they have recently been explored in distributed settings, e.g., (Hsieh et al, 2016;Lee & Chang, 2017;Trofimov & Genkin, 2017;Mahajan et al, 2017;Lee et al, 2018). Our trust-region approach has clear advantages compared to line-search methods: i) a line-search method assumes a fixed auxiliary model -which may be an arbitrarily bad approximation of the true objective-that is used to find an acceptable step size.…”

Section: Related Workmentioning

confidence: 99%

A Distributed Second-Order Algorithm You Can Trust

Dünner,

Lucchi,

Gargiani

et al. 2018

Preprint

View full text Add to dashboard Cite

Due to the rapid growth of data and computational resources, distributed optimization has become an active research area in recent years. While firstorder methods seem to dominate the field, secondorder methods are nevertheless attractive as they potentially require fewer communication rounds to converge. However, there are significant drawbacks that impede their wide adoption, such as the computation and the communication of a large Hessian matrix. In this paper we present a new algorithm for distributed training of generalized linear models that only requires the computation of diagonal blocks of the Hessian matrix on the individual workers. To deal with this approximate information we propose an adaptive approach thatakin to trust-region methods -dynamically adapts the auxiliary model to compensate for modeling errors. We provide theoretical rates of convergence for a wide class of problems including L 1regularized objectives. We also demonstrate that our approach achieves state-of-the-art results on multiple large benchmark datasets.

show abstract

Distributed coordinate descent for generalized linear models with regularization

Abstract: Generalized linear model with L 1 and L 2 regularization is a widely used technique for solving classification, class probability estimation and regression problems.

Cited by 7 publications

References 23 publications

CoCoA: A General Framework for Communication-Efficient Distributed Optimization

CoCoA: A General Framework for Communication-Efficient Distributed Optimization

Intelligent Hybrid Feature Selection for Textual Sentiment Classification

A Distributed Second-Order Algorithm You Can Trust

Contact Info

Product

Resources

About