Bundle CDN: A Highly Parallelized Approach for Large-Scale ℓ1-Regularized Logistic Regression

Bian, Yatao; Li, Xiong; Cao, Mingqi; Liu, Yuncai

doi:10.1007/978-3-642-40994-3_6

Cited by 4 publications

(5 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Experimental results show that the proposed PCDN algorithm performs favorably against the other methods. As a feature-based parallel algorithm, the proposed PCDN solver performs well for sparse datasets with more features as shown by the results on the rcv1 and news20 datasets, which are very sparse (training data sparsity, defined by the ratio of zero elements in design matrix X and explained in Table 2, is 99.85% and 99.97%, respectively) with a large number of features (47,236 and 1,355,191). In such cases, the PCDN algorithm performs well against the TRON method.…”

Section: Methodsmentioning

confidence: 90%

“…Proof. (1) We first prove that E B t [ λ(B t )] is monotonically increasing with respect to P and E B t [ λ(B t )] is constant with respect to P , if λ i is constant or…”

Section: Discussionmentioning

confidence: 99%

“…where τ (s) ≡ 1 1+e −s is the derivative of the logistic loss function log(1 + e s ). Because 0 < τ (s) < 1, we have 0 < ∇ 2 jj L(w) ≤ 1 4 c s i=1 x 2 ij = 1 4 c(X X) jj (the equal sign holds when τ (s) = 1 2 ), and thus (12) holds when θ = 1 4 for logistic regression. As λ(N ) is the maximum element of (X X) jj where j ∈ N , ∇ 2 jj L(w) ≤ h = θc λ(N ) in ( 13) also holds.…”

Section: Discussionmentioning

confidence: 99%

“…In [1] Bian et al present a high dimensional line search algorithm to ensure global convergence while performing parallel coordinate updates for L 1 -regularized logistic regression problem. While this method performs well, no analysis of convergence rate is presented.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Parallel Coordinate Descent Newton Method for Efficient $L_{1}$ -Regularized Loss Minimization

Bian

Liu

et al. 2019

IEEE Trans. Neural Netw. Learning Syst.

Self Cite

View full text Add to dashboard Cite

The recent years have witnessed advances in parallel algorithms for large scale optimization problems. Notwithstanding the demonstrated success, existing algorithms that parallelize over features are usually limited by divergence issues under high parallelism or require data preprocessing to alleviate these problems. In this work, we propose a Parallel Coordinate Descent algorithm using approximate Newton steps (PCDN) that is guaranteed to converge globally without data preprocessing. The key component of the PCDN algorithm is the high-dimensional line search, which guarantees the global convergence with high parallelism. The PCDN algorithm randomly partitions the feature set into b subsets/bundles of size P , and sequentially processes each bundle by first computing the descent directions for each feature in parallel and then conducting P -dimensional line search to compute the step size. We show that (i) the PCDN algorithm is guaranteed to converge globally despite increasing parallelism; (ii) the PCDN algorithm converges to the specified accuracy within the limited iteration number of T , and T decreases with increasing parallelism. In addition, the data transfer and synchronization cost of the P -dimensional line search can be minimized by maintaining intermediate quantities. For concreteness, the proposed PCDN algorithm is applied to L 1 -regularized logistic regression and L 1 -regularized L 2 -loss SVM problems. Experimental evaluations on seven benchmark datasets show that the proposed PCDN algorithm exploits parallelism well and outperforms the state-of-the-art methods. * Source code is available at https

show abstract

Section: Methodsmentioning

confidence: 90%

“…Proof. (1) We first prove that E B t [ λ(B t )] is monotonically increasing with respect to P and E B t [ λ(B t )] is constant with respect to P , if λ i is constant or…”

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Parallel Coordinate Descent Newton Method for Efficient $L_{1}$ -Regularized Loss Minimization

Bian

Liu

et al. 2019

IEEE Trans. Neural Netw. Learning Syst.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Some past works have considered using diagonal blocks as the approximation of the Hessian. For logistic regression, Bian et al (2013) consider diagonal elements of the Hessian to solve several one-variable sub-problems in parallel. Mahajan et al (2017) study a more general setting in which using diagonal blocks is a special case.…”

Section: Diagonal Gauss-newton Matrix Approximationmentioning

confidence: 99%

Distributed Newton Methods for Deep Neural Networks

et al. 2018

View full text Add to dashboard Cite

Deep learning involves a difficult nonconvex optimization problem with a large number of weights between any two adjacent layers of a deep structure. To handle large data sets or complicated networks, distributed training is needed, but the calculation of function, gradient, and Hessian is expensive. In particular, the communication and the synchronization cost may become a bottleneck. In this letter, we focus on situations where the model is distributedly stored and propose a novel distributed Newton method for training deep neural networks. By variable and feature-wise data partitions and some careful designs, we are able to explicitly use the Jacobian matrix for matrix-vector products in the Newton method. Some techniques are incorporated to reduce the running time as well as memory consumption. First, to reduce the communication cost, we propose a diagonalization method such that an approximate Newton direction can be obtained without communication between machines. Second, we consider subsampled Gauss-Newton matrices for reducing the running time as well as the communication cost. Third, to reduce the synchronization cost, we terminate the process of finding an approximate Newton direction even though some nodes have not finished their tasks. Details of some implementation issues in distributed environments are thoroughly investigated. Experiments demonstrate that the proposed method is effective for the distributed training of deep neural networks. Compared with stochastic gradient methods, it is more robust and may give better test accuracy.

show abstract