Mean-field particle swarm optimization

Grassi, Sara; Huang, Hui; Pareschi, Lorenzo; Qiu, Jinniao

doi:10.48550/arxiv.2108.00393

Cited by 5 publications

(10 citation statements)

References 67 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A different realization of (1.2b) might result in a higher cost. Implementational aspects: A discretization of the SDE (1.2) in line 11 can be obtained for instance from a simple Euler-Maruyama or semi-implicit scheme [37,19], see, e.g., [17,Equation (6.3)]. In our numerical experiments below Equation (1.3) is used for updating the local best position, which corresponds to κ = 1/(2∆t), θ = 0, and β = ∞.…”

Section: Implementation Of Pso and Numerical Resultsmentioning

confidence: 99%

“…All of the formerly mentioned results though are obtained through the analysis of the particles' trajectories generated by a time-discretized algorithm as in [17,Equation (6.3)]. The present paper takes a different point of view by studying the time-continuous description of the PSO model (1.2) through the lens of the mean-field approximation (1.7).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

On the Global Convergence of Particle Swarm Optimization Methods

Huang¹,

Qiu²,

Riedl³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper we provide a rigorous convergence analysis for the renowned Particle Swarm Optimization method using tools from stochastic calculus and the analysis of partial differential equations. Based on a time-continuous formulation of the particle dynamics as a system of stochastic differential equations, we establish the convergence to a global minimizer in two steps. First, we prove the consensus formation of the dynamics by analyzing the time-evolution of the variance of the particle distribution. Consecutively, we show that this consensus is close to a global minimizer by employing the asymptotic Laplace principle and a tractability condition on the energy landscape of the objective function. Our results allow for the usage of memory mechanisms, and hold for a rich class of objectives provided certain conditions of well-preparation of the hyperparameters and the initial datum are satisfied. To demonstrate the applicability of the method we propose an efficient and parallelizable implementation, which is tested in particular on a competitive and well-understood high-dimensional benchmark problem in machine learning.

show abstract

Section: Implementation Of Pso and Numerical Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

On the Global Convergence of Particle Swarm Optimization Methods

Huang¹,

Qiu²,

Riedl³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…In this case, not only the objective function is bounded but also particles are evolving on a compact set. Under the assumptions on the objective function as in our paper, in the diffusion case weak convergence of the empirical measure of a particle system to the law of the corresponding mean field SDEs has been proved in [GHPQ21,HQ21] exploiting Prokhorov's theorem. Here we prove convergence of the particle system to the mean-field SDEs in the mean-square sense for a quadratically growing locally-Lipschitz objective function defined on R .…”

Section: Introductionmentioning

confidence: 91%

“…This combination of simplicity and effectiveness has fuelled the application of metaheuristic in complex engineering problems such as shape optimization, scheduling problems, and hyperparameter tuning in machine learning models. However, it is often the case that metaheuristics lack rigorous convergence results, a question which has become an active area of research [GP21,GHPQ21].…”

Section: Introductionmentioning

confidence: 99%

“…This iterated interaction generates trajectories which flock towards a consensus point which correspond to a global minimizer of ( ), hence the name Consensus Based Optimization (CBO). We refer to [Tot22,GHPQ21] for two recent surveys on the topic. The dynamics of existing CBO models are governed by stochastic differential equations with Wiener noise [PTTM17,CCTT18,CJLZ21].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Consensus based optimization via jump-diffusion stochastic differential equations

Kalise¹,

Sharma²,

Tret'yakov³

2022

Preprint

View full text Add to dashboard Cite

We introduce a new consensus based optimization (CBO) method where interacting particle system is driven by jump-diffusion stochastic differential equations. We study well-posedness of the particle system as well as of its mean-field limit. The major contributions of this paper are proofs of convergence of the interacting particle system towards the mean-field limit and convergence of a discretized particle system towards the continuous-time dynamics in the mean-square sense. We also prove convergence of the mean-field jump-diffusion SDEs towards global minimizer for a large class of objective functions. We demonstrate improved performance of the proposed CBO method over earlier CBO methods in numerical simulations on benchmark objective functions.

show abstract

On the Global Convergence of Particle Swarm Optimization Methods

2023

View full text Add to dashboard Cite

In this paper we provide a rigorous convergence analysis for the renowned particle swarm optimization method by using tools from stochastic calculus and the analysis of partial differential equations. Based on a continuous-time formulation of the particle dynamics as a system of stochastic differential equations, we establish convergence to a global minimizer of a possibly nonconvex and nonsmooth objective function in two steps. First, we prove consensus formation of an associated mean-field dynamics by analyzing the time-evolution of the variance of the particle distribution, which acts as Lyapunov function of the dynamics. We then show that this consensus is close to a global minimizer by employing the asymptotic Laplace principle and a tractability condition on the energy landscape of the objective function. These results allow for the usage of memory mechanisms, and hold for a rich class of objectives provided certain conditions of well-preparation of the hyperparameters and the initial datum. In a second step, at least for the case without memory effects, we provide a quantitative result about the mean-field approximation of particle swarm optimization, which specifies the convergence of the interacting particle system to the associated mean-field limit. Combining these two results allows for global convergence guarantees of the numerical particle swarm optimization method with provable polynomial complexity. To demonstrate the applicability of the method we propose an efficient and parallelizable implementation, which is tested in particular on a competitive and well-understood high-dimensional benchmark problem in machine learning.

show abstract

Mean-field particle swarm optimization

Cited by 5 publications

References 67 publications

On the Global Convergence of Particle Swarm Optimization Methods

On the Global Convergence of Particle Swarm Optimization Methods

Consensus based optimization via jump-diffusion stochastic differential equations

On the Global Convergence of Particle Swarm Optimization Methods

Contact Info

Product

Resources

About