Annealing stochastic approximation Monte Carlo algorithm for neural network training

Abstract

We propose a general-purpose stochastic optimization algorithm, the so-called annealing stochastic approximation Monte Carlo (ASAMC) algorithm, for neural network training. ASAMC can be regarded as a space annealing version of the stochastic approximation Monte Carlo (SAMC) algorithm. Under mild conditions, we show that ASAMC can converge weakly at a rate of Ω(1/ √ t) toward a neighboring set (in the space of energy) of the global minimizers. ASAMC is compared with simulated annealing, SAMC, and the BFGS algorithm for training MLPs on a number of examples. The numerical results indicate that ASAMC outperforms the other algorithms in both training and test errors. Like other stochastic algorithms, ASAMC requires longer training time than do the gradient-based algorithms. It provides, however, an efficient approach to train MLPs for which the energy landscape is rugged.

Annealing Stochastic Approximation Monte Carlo for Global Optimization

“…For example, for a continuous problem, q(x, y) can be chosen as a random walk Gaussian proposal y ~ N(x, σ 2 ) with σ 2 being calibrated to have a desired acceptance rate. Issues on implementation of the algorithm, such as how to partition the sample space, how to choose the gain factor sequence, and how to set the number of iterations, have been discussed at length in Liang et al (2007). SAMC falls into the category of stochastic approximation algorithms (Benveniste et al,1990;Andrieu et al, 2005).…”

“…Otherwise, set t ←t + 1 and go to step (b). It has been shown in Liang (2007) that if the gain factor sequence satisfie (3) and the proposal distribution satisfies the minorisation condition (5), ASAMC can converge weakly toward a neighboring set of the global minima of U(x) in the space of energy. More precisely, the sample x (t) converges in distribution to a random variable with the density function (9) where u min is the global minimum value of U(x),…”

“…Note that in all runs of the three algorithms, the total numbers of iterations were the same, and they cost about the same CPU times because the CPU time cost by each iteration is dominated by the part used for energy evaluation. This is especially true for more complicated problems, e.g., the neural network training problems studied in Liang (2007). …”

“…In this article, we consider applications of SAMC in optimization. Two modified versions of SAMC, annealing SAMC (Liang, 2007;Zhu et al, 2007) and annealing evolutionary SAMC (Liang, 2008), are discussed. Both algorithms have inherited self-adjusting ability from the SAMC algorithm.…”

Parallel and interacting stochastic approximation annealing algorithms for global optimisation

“…Several stochastic optimisation algorithms have been proposed in the literature, e.g. simulated annealing (SA) (Kirkpatrick et al, 1983;Metropolis et al, 1953), genetic algorithm (Goldberg, 1989;Holland, 1975), annealing stochastic approximation Monte Carlo (ASAMC ) (Liang, 2007), annealing evolutionary stochastic approximation Monte Carlo (AESAMC) (Liang, 2011), stochastic approximation annealing (SAA) . Albeit their success, they encounter various difficulties in converging to the global minimum, an issue that becomes more severe when U p¨q is highly rugged or high dimensional.…”