Convergence of Anisotropic Consensus-Based Optimization in Mean-Field Law

Fornasier, Massimo; Klock, Timo; Riedl, Konstantin

doi:10.48550/arxiv.2111.08136

Cited by 2 publications

(8 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Indeed, while in the isotropic case every dimension is equally explored, in the anisotropic process the particles explore each dimension at a different rate. The componentwise exploration better suits high dimensional problems, as the particles convergence rate is independent of the dimension d [13,24]. While we will focus the convergence analysis on the isotropic CBO method, we will discuss extensions to the anisotropic case and compare the isotropic and anisotropic exploration processes on numerical examples in Section 4.3.…”

Section: Constrained Cbo Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Constrained consensus-based optimization

Borghi¹,

Herty²,

Pareschi³

2021

Preprint

View full text Add to dashboard Cite

In this work we are interested in the construction of numerical methods for high dimensional constrained nonlinear optimization problems by particle-based gradient-free techniques. A consensus-based optimization (CBO) approach combined with suitable penalization techniques is introduced for this purpose. The method relies on a reformulation of the constrained minimization problem in an unconstrained problem for a penalty function and extends to the constrained settings the class of CBO methods. Exact penalization is employed and, since the optimal penalty parameter is unknown, an iterative strategy is proposed that successively updates the parameter based on the constrained violation. Using a mean-field description of the the many particle limit of the arising CBO dynamics, we are able to show convergence of the proposed method to the minimum for general nonlinear constrained problems. Properties of the new algorithm are analyzed. Several numerical examples, also in high dimensions, illustrate the theoretical findings and the good performance of the new numerical method.

show abstract

Section: Constrained Cbo Methodsmentioning

confidence: 99%

“…For any fixed parameter β, we need to solve an unconstrained optimization problem for which the behavior of the CBO method has been broadly analyzed, e.g. in [10,13,23,24,32,47].…”

Section: The Update Strategy For the Penalty Parametermentioning

confidence: 99%

Constrained consensus-based optimization

Borghi¹,

Herty²,

Pareschi³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…It is also worth noting that Equation (2.1) bears a certain resemblance to CBO [36,5,6,14,15]. Indeed, as made rigorous in [7], CBO methods can be derived from PSO in the small inertia limit m → 0.…”

Section: Pso Without Memory Effectsmentioning

confidence: 98%

“…While this might have a locality flavor, the condition is generally fulfilled in practical applications. Moreover, for CBO methods there is recent work where such assumption about the initial datum is reduced to the absolute minimum [14,15].…”

Section: Convergence Of the Pso Model Without Memory Effects To A Glo...mentioning

confidence: 99%

“…To feature the latter, the two remaining terms in (1.2c), each associated with a drift term, are diffusion terms injecting randomness into the dynamics through independent standard Brownian motions (B 1,i t ) t≥0 i=1,...,N and (B 2,i t ) t≥0 i=1,...,N . The two commonly studied diffusion types for similar methods are isotropic [36,5,14] and anisotropic [6,15] diffusion with…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

On the Global Convergence of Particle Swarm Optimization Methods

Huang¹,

Qiu²,

Riedl³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper we provide a rigorous convergence analysis for the renowned Particle Swarm Optimization method using tools from stochastic calculus and the analysis of partial differential equations. Based on a time-continuous formulation of the particle dynamics as a system of stochastic differential equations, we establish the convergence to a global minimizer in two steps. First, we prove the consensus formation of the dynamics by analyzing the time-evolution of the variance of the particle distribution. Consecutively, we show that this consensus is close to a global minimizer by employing the asymptotic Laplace principle and a tractability condition on the energy landscape of the objective function. Our results allow for the usage of memory mechanisms, and hold for a rich class of objectives provided certain conditions of well-preparation of the hyperparameters and the initial datum are satisfied. To demonstrate the applicability of the method we propose an efficient and parallelizable implementation, which is tested in particular on a competitive and well-understood high-dimensional benchmark problem in machine learning.

show abstract

Convergence of Anisotropic Consensus-Based Optimization in Mean-Field Law

Cited by 2 publications

References 10 publications

Constrained consensus-based optimization

Constrained consensus-based optimization

On the Global Convergence of Particle Swarm Optimization Methods

Contact Info

Product

Resources

About