Neural Network Training as an Optimal Control Problem : — An Augmented Lagrangian Approach —

Evens, Brecht; Latafat, Puya; Themelis, Andreas; Suykens, Johan A. K.; Patrinos, Panagiotis

doi:10.1109/cdc45484.2021.9682842

Cited by 4 publications

(13 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1] and leveraging the idea behind [15,Ex. 4.12], the convergence properties of [26,Alg. 1] hinge on the upper boundedness of the augmented Lagrangian along the iterates ensured by the initialization at a feasible point.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Constrained composite optimization and augmented Lagrangian methods

et al. 2023

View full text Add to dashboard Cite

We investigate finite-dimensional constrained structured optimization problems, featuring composite objective functions and set-membership constraints. Offering an expressive yet simple language, this problem class provides a modeling framework for a variety of applications. We study stationarity and regularity concepts, and propose a flexible augmented Lagrangian scheme. We provide a theoretical characterization of the algorithm and its asymptotic properties, deriving convergence results for fully nonconvex problems. It is demonstrated how the inner subproblems can be solved by off-the-shelf proximal methods, notwithstanding the possibility to adopt any solvers, insofar as they return approximate stationary points. Finally, we describe our matrix-free implementation of the proposed algorithm and test it numerically. Illustrative examples show the versatility of constrained composite programs as a modeling tool and expose difficulties arising in this vast problem class.

show abstract

Section: Related Workmentioning

confidence: 99%

“…We deviate in this respect, seeking instead a method able to start from any x 0 ∈ R n . Nonetheless, if a feasible point is readily available for (P), one can adopt [26,Alg. 1] in its original form, replacing the augmented Lagrangian function and inner solver accordingly.…”

Section: Related Workmentioning

confidence: 99%

Constrained composite optimization and augmented Lagrangian methods

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Augmented Lagrangian schemes have been extensively investigated [15,7,11,47,26], also in the infinite-dimensional setting [31,3]. Merely lower semicontinuous cost functions have been considered in [23]. Inspired by [25,Alg.…”

Section: Related Workmentioning

confidence: 99%

“…1] and leveraging the idea behind [11,Ex. 4.12], the convergence properties of [23,Alg. 1] hinge on the upper boundedness of the augmented Lagrangian along the iterates ensured by the initialization at a feasible point.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Constrained composite optimization and augmented Lagrangian methods

Marchi¹,

Jia²,

Kanzow³

et al. 2022

Preprint

View full text Add to dashboard Cite

We investigate and develop numerical methods for finite dimensional constrained structured optimization problems. Offering a comprehensive yet simple and expressive language, this problem class provides a modeling framework for a variety of applications. A general and flexible algorithm is proposed that interlaces proximal methods and safeguarded augmented Lagrangian schemes. We provide a theoretical characterization of the algorithm and its asymptotic properties, deriving convergence results for fully nonconvex problems. Adopting a proximal gradient method with an oracle as a formal tool, it is demonstrated how the inner subproblems can be solved by off-the-shelf methods for composite optimization, without introducing slack variables and despite the possibly set-valued projections. Finally, we describe our open-source matrix-free implementation ALPS of the proposed algorithm and test it numerically. Illustrative examples show the versatility of constrained structured programs as a modeling tool, expose difficulties arising in this vast problem class and highlight benefits of the implicit approach developed.

show abstract

Recurrent Neural Network Training With Convex Loss and Regularization Functions by Extended Kalman Filtering

Bemporad

2023

IEEE Trans. Automat. Contr.

View full text Add to dashboard Cite

Neural Network Training as an Optimal Control Problem : — An Augmented Lagrangian Approach —

Abstract: C14/18/068; Fonds de la Recherche Scientifique -FNRS and the Fonds Wetenschappelijk Onderzoek -Vlaanderen under EOS project no 30468160 (SeLMA). Johan Suykens and Panagiotis Patrinos are affiliated to Leuven.AI -

Cited by 4 publications

References 9 publications

Constrained composite optimization and augmented Lagrangian methods

Constrained composite optimization and augmented Lagrangian methods

Constrained composite optimization and augmented Lagrangian methods

Recurrent Neural Network Training With Convex Loss and Regularization Functions by Extended Kalman Filtering

Contact Info

Product

Resources

About