Vector-output ReLU Neural Network Problems are Copositive Programs: Convex Analysis of Two Layer Networks and Polynomial-time Algorithms

Sahiner, A.; Ergen, Tolga; Pauly, John M.; Pilancı, Mert

doi:10.48550/arxiv.2012.13329

Cited by 5 publications

(11 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…+ V for any orthogonal matrix L, where X = UΣV is the SVD of X. While (5) does not appear convex, it has been shown that its solution is equivalent to a convex program [26,34], which for convex sets K i is expressed as…”

Section: Overview Of Main Resultsmentioning

confidence: 99%

“…The optimal solution to (6) can be found in polynomial-time in all problem dimensions when Z is fixed-rank, and can construct the optimal generator weights W * 1 , W * 2 [26].…”

Section: Overview Of Main Resultsmentioning

confidence: 99%

“…The work in [24] studies convex duality of divergence measures, where the insights motivate regularizing the discriminator's Lipschitz constant for improved GAN performance. For supervised two-layer networks, a recent of line of work has established zero-duality gap and thus equivalent convex networks with ReLU activation that can be solved in polynomial time for global optimality; see e.g., [25][26][27][28][29][30]. These works focus on single-player networks for supervised learning.…”

Section: Related Workmentioning

confidence: 99%

“…In this section, we re-iterate the results of [26] for demonstrating an equivalent convex formulation to the generator problem (5):…”

Section: B1 Convexity and Polynomial-time Trainability Of Two-layer R...mentioning

confidence: 99%

“…For the output of the network (ZW 1 ) + W 2 , the fitting term is a convex loss function. From [26], we know that this is equivalent to the following convex optimization problem…”

Section: B33 Relu-activation Generatormentioning

confidence: 99%

See 4 more Smart Citations

Hidden Convexity of Wasserstein GANs: Interpretable Generative Models with Closed-Form Solutions

Sahiner¹,

Ergen²,

Ozturkler³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Generative Adversarial Networks (GANs) are commonly used for modeling complex distributions of data. Both the generators and discriminators of GANs are often modeled by neural networks, posing a non-transparent optimization problem which is non-convex and non-concave over the generator and discriminator, respectively. Such networks are often heuristically optimized with gradient descent-ascent (GDA), but it is unclear whether the optimization problem contains any saddle points, or whether heuristic methods can find them in practice. In this work, we analyze the training of Wasserstein GANs with two-layer neural network discriminators through the lens of convex duality, and for a variety of generators expose the conditions under which Wasserstein GANs can be solved exactly with convex optimization approaches, or can be represented as convex-concave games. Using this convex duality interpretation, we further demonstrate the impact of different activation functions of the discriminator. Our observations are verified with numerical results demonstrating the power of the convex interpretation, with applications in progressive training of convex architectures corresponding to linear generators and quadratic-activation discriminators for CelebA image generation. The code for our experiments is available at https://github.com/ardasahiner/ProCoGAN. * Equal contribution Preprint. Under review.

show abstract

Section: Overview Of Main Resultsmentioning

confidence: 99%

“…The optimal solution to (6) can be found in polynomial-time in all problem dimensions when Z is fixed-rank, and can construct the optimal generator weights W * 1 , W * 2 [26].…”

Section: Overview Of Main Resultsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

“…In this section, we re-iterate the results of [26] for demonstrating an equivalent convex formulation to the generator problem (5):…”

Section: B1 Convexity and Polynomial-time Trainability Of Two-layer R...mentioning

confidence: 99%

“…For the output of the network (ZW 1 ) + W 2 , the fitting term is a convex loss function. From [26], we know that this is equivalent to the following convex optimization problem…”

Section: B33 Relu-activation Generatormentioning

confidence: 99%

See 3 more Smart Citations

Hidden Convexity of Wasserstein GANs: Interpretable Generative Models with Closed-Form Solutions

Sahiner¹,

Ergen²,

Ozturkler³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Rigorous Analysis of Data Orthogonalization for Self-Organizing Maps in Machine Learning Cyber Intrusion Detection for LoRa Sensors

Nair

Cappello

Dang

et al. 2023

IEEE Trans. Microwave Theory Techn.

View full text Add to dashboard Cite

Neural Spectrahedra and Semidefinite Lifts: Global Convex Optimization of Polynomial Activation Neural Networks in Fully Polynomial-Time

Bartan¹,

Pilancı²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

The training of two-layer neural networks with nonlinear activation functions is an important non-convex optimization problem with numerous applications and promising performance in layerwise deep learning. In this paper, we develop exact convex optimization formulations for two-layer neural networks with second degree polynomial activations based on semidefinite programming. Remarkably, we show that semidefinite lifting is always exact and therefore computational complexity for global optimization is polynomial in the input dimension and sample size for all input data. The developed convex formulations are proven to achieve the same global optimal solution set as their non-convex counterparts. More specifically, the globally optimal two-layer neural network with polynomial activations can be found by solving a semidefinite program (SDP) and decomposing the solution using a procedure we call Neural Decomposition. Moreover, the choice of regularizers plays a crucial role in the computational tractability of neural network training. We show that the standard weight decay regularization formulation is NP-hard, whereas other simple convex penalties render the problem tractable in polynomial time via convex programming. We extend the results beyond the fully connected architecture to different neural network architectures including networks with vector outputs and convolutional architectures with pooling. We provide extensive numerical simulations showing that the standard backpropagation approach often fails to achieve the global optimum of the training loss. The proposed approach is significantly faster to obtain better test accuracy compared to the standard backpropagation procedure.

show abstract

Vector-output ReLU Neural Network Problems are Copositive Programs: Convex Analysis of Two Layer Networks and Polynomial-time Algorithms

Cited by 5 publications

References 28 publications

Hidden Convexity of Wasserstein GANs: Interpretable Generative Models with Closed-Form Solutions

Hidden Convexity of Wasserstein GANs: Interpretable Generative Models with Closed-Form Solutions

Rigorous Analysis of Data Orthogonalization for Self-Organizing Maps in Machine Learning Cyber Intrusion Detection for LoRa Sensors

Neural Spectrahedra and Semidefinite Lifts: Global Convex Optimization of Polynomial Activation Neural Networks in Fully Polynomial-Time

Contact Info

Product

Resources

About