On the rate of convergence of the Difference-of-Convex Algorithm (DCA)

Abbaszadehpeivasti, Hadi; Klerk, Etienne de; Zamani, Moslem

doi:10.48550/arxiv.2109.13566

Cited by 6 publications

(11 citation statements)

References 38 publications

(57 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The actively studied numerical methods for solving a DC program include DCA An, 1998, 1997;An and Tao, 2005;Souza et al, 2016), which is also known as the concave-convex procedure (Yuille and Rangarajan, 2003;Sriperumbudur and Lanckriet, 2009;Lipp and Boyd, 2016), the proximal DCA (Sun et al, 2003;Moudafi and Maingé, 2006;Moudafi, 2008;An and Nam, 2017), and the direct gradient methods (Khamaru and Wainwright, 2018). However, when the two convex compo-nents are both non-smooth, the existing methods have only asymptotic convergence results except the method by Abbaszadehpeivasti et al (2021), who considered a stopping criterion different from ours. When at least one component is smooth, non-asymptotic convergence rates have been established with and without the Kurdyka-Lojasiewicz (KL) condition (Souza et al, 2016;Artacho et al, 2018;Wen et al, 2018;An and Nam, 2017;Khamaru and Wainwright, 2018).…”

Section: Related Workmentioning

confidence: 94%

Large-scale Optimization of Partial AUC in a Range of False Positive Rates

Yao¹,

Lin²,

Yang³

2022

Preprint

View full text Add to dashboard Cite

The area under the ROC curve (AUC) is one of the most widely used performance measures for classification models in machine learning. However, it summarizes the true positive rates (TPRs) over all false positive rates (FPRs) in the ROC space, which may include the FPRs with no practical relevance in some applications. The partial AUC, as a generalization of the AUC, summarizes only the TPRs over a specific range of the FPRs and is thus a more suitable performance measure in many real-world situations. Although partial AUC optimization in a range of FPRs had been studied, existing algorithms are not scalable to big data and not applicable to deep learning. To address this challenge, we cast the problem into a non-smooth difference-of-convex (DC) program for any smooth predictive functions (e.g., deep neural networks), which allowed us to develop an efficient approximated gradient descent method based on the Moreau envelope smoothing technique, inspired by recent advances in non-smooth DC optimization. To increase the efficiency of large data processing, we used an efficient stochastic block coordinate update in our algorithm. Our proposed algorithm can also be used to minimize the sum of ranked range loss, which also lacks efficient solvers. We established a complexity of Õ(1/ǫ 6 ) for finding a nearly ǫ-critical solution. Finally, we numerically demonstrated the effectiveness of our proposed algorithms for both partial AUC maximization and sum of ranked range loss minimization.

show abstract

Section: Related Workmentioning

confidence: 94%

Large-scale Optimization of Partial AUC in a Range of False Positive Rates

Yao¹,

Lin²,

Yang³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Note that our objective function is non-convex, since the within-group penalty p w in ( 7) is not convex and the loss function in (3) involves minimization. To tackle this challenge, we decompose our objective function as a difference of two convex functions and solve the optimization problem based on the difference of convex (DC) algorithm (Le Thi Hoai and Tao, 1997; Shen et al, 2012), which is shown to converge to a stationary point under regularity conditions (Abbaszadehpeivasti et al, 2021). Also, we use a smooth approximation of the between-group fused Lasso penalty.…”

Section: Methodsmentioning

confidence: 99%

Heterogeneous Mediation Analysis on Epigenomic PTSD and Traumatic Stress in a Predominantly African American Cohort

Xue

Tang

Kim

et al. 2020

Preprint

View full text Add to dashboard Cite

Mediation analysis is widely used to understand mediating mechanisms of variables in causal inference. However, existing approaches do not consider heterogeneity in mediation effects. Mediators in different sub-populations could have opposite effects on the outcome, and could be difficult to identify under the homogeneous model framework. In this paper, we propose a new mediator selection method, which can identify sub-populations and select mediators in each sub-population for heterogeneous data simultaneously. We perform a multi-directional clustering analysis to determine sub-group mediators and the corresponding subjects. Specifically, to select mediators, we propose a new joint penalty which penalizes the effect of independent variable on a mediator and the effect of a mediator on the response jointly. The proposed algorithm is implemented through the convex-smooth gradient descent. Our numerical studies show that the proposed method outperforms the existing methods for heterogeneous data. We also apply the proposed mediation method to estimate mediation effects of DNA methylation variations in glucocorticoid receptor regulatory network genes for post-traumatic stress disorder (PTSD) among African-Americans. Based on data from the Detroit Neighborhood Health Study, we have found heterogeneous mediators, which are indeed associated with PTSD or traumatic experiences according to literature but were not selected by existing homogeneous mediation selection methods.

show abstract

“…We assume that strong duality holds for problem (1), that is max λ∈R r D(λ) = min Ax+Bz=b f (x) + g(z).…”

Section: Algorithm 1 Admmmentioning

confidence: 99%

“…Problem (1) appears naturally (or after variable splitting) in many applications in statistics, machine learning and image processing to name but a few [5,19,24,34]. The most common method for solving problem (1) is the alternating direction method of multipliers (ADMM).…”

Section: Introductionmentioning

confidence: 99%

The exact worst-case convergence rate of the alternating direction method of multipliers

Abbaszadehpeivasti¹,

Klerk²,

Zamani³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Recently, semidefinite programming performance estimation has been employed as a strong tool for the worst-case performance analysis of first order methods. In this paper, we derive new non-ergodic convergence rates for the alternating direction method of multipliers (ADMM) by using performance estimation. We give some examples which show the exactness of the given bounds. We establish that ADMM enjoys a global linear convergence rate if and only if the dual objective satisfies the Polyak-Lojasiewicz (P L) inequality in the presence of strong convexity. In addition, we give an explicit formula for the linear convergence rate factor.

show abstract

On the rate of convergence of the Difference-of-Convex Algorithm (DCA)

Cited by 6 publications

References 38 publications

Large-scale Optimization of Partial AUC in a Range of False Positive Rates

Large-scale Optimization of Partial AUC in a Range of False Positive Rates

Heterogeneous Mediation Analysis on Epigenomic PTSD and Traumatic Stress in a Predominantly African American Cohort

The exact worst-case convergence rate of the alternating direction method of multipliers

Contact Info

Product

Resources

About