Training high-dimensional neural networks with cooperative particle swarm optimiser

Rakitianskaia, Anna; Engelbrecht, Andries P.

doi:10.1109/ijcnn.2014.6889933

Cited by 9 publications

(2 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although a number of studies have proposed the use of populationbased algorithms for training neural networks [9]- [14], they are seldom used in practice. One of the challenges is that the search space of neural network weights is unbounded and algorithms such as particle swarm optimisation may fail due to high weight magnitudes leading to hidden unit saturation [17], [18]. The one domain where population-based metaheuristics have shown competitive results compared to gradient-based methods is in deep reinforcement learning tasks [19].…”

Section: Introductionmentioning

confidence: 99%

A Continuous Optimisation Benchmark Suite from Neural Network Regression

Malan,

Cleghorn

2021

Preprint

View full text Add to dashboard Cite

For the design of optimisation algorithms that perform well in general, it is necessary to experiment with and benchmark on a range of problems with diverse characteristics. The training of neural networks is an optimisation task that has gained prominence with the recent successes of deep learning. Although evolutionary algorithms have been used for training neural networks, gradient descent variants are by far the most common choice with their trusted good performance on largescale machine learning tasks. With this paper we contribute CORNN (Continuous Optimisation of Regression tasks using Neural Networks), a large suite that can easily be used to benchmark the performance of any continuous black-box algorithm on neural network training problems. By employing different base regression functions and neural network architectures, problem instances with different dimensions and levels of difficulty can be created. We demonstrate the use of the CORNN Suite by comparing the performance of three evolutionary and swarmbased algorithms on a set of over 300 problem instances. With the exception of random search, we provide evidence of performance complementarity between the algorithms. As a baseline, results are also provided to contrast the performance of the best population-based algorithm against a gradient-based approach (Adam). The suite is shared as a public web repository to facilitate easy integration with existing benchmarking platforms.

show abstract

Section: Introductionmentioning

confidence: 99%

A Continuous Optimisation Benchmark Suite from Neural Network Regression

Malan,

Cleghorn

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Therefore, PSO is not susceptible to the gradient problem [31]. It seems that the PSO method is a potentially good deep learning approach, but research has indicated that the pure PSO training method struggles to train relatively large neural networks [32]. Reference [33] presented an integration framework of the PSO and GD for the LSTM when identifying handwriting.…”

Section: Introductionmentioning

confidence: 99%

An Enhanced LSTM for Trend Following of Time Series

Sun

Nie

et al. 2019

IEEE Access

View full text Add to dashboard Cite

Mining and analysis of time series data (TSD) have drawn a great concern, especially in the TSD clustering, classification, and forecast. In the industrial field, e.g., the work condition monitoring and the environmental safety, it is crucial to follow the trend of the corresponding TSD for a safety forecast, and few studies have been devoted to such a trend following. Motivated by this, we propose a trend following the strategy of TSD by using a long short-term memory (LSTM) network for safety forecast, in which the training method aggregates the particle swarm optimization (PSO) algorithm with gradient descent (GD) to obtain more competitive model parameters. Three kinds of trend representations of TSD are first defined based on the corresponding research in stock option. Then, the LSTM optimized with the PSO-GD is developed to perform the trend following. From the viewpoint of safety forecast, the trends varied in different time length are further predicted and analyzed. The superiority of the proposed algorithm is experimentally demonstrated by applying it to the electromagnetic radiation intensity TSD sampled from an actual coal mine and PM2.5 in UCI repository.

show abstract

A Continuous Optimisation Benchmark Suite from Neural Network Regression

Malan

Cleghorn

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Training high-dimensional neural networks with cooperative particle swarm optimiser

Cited by 9 publications

References 18 publications

A Continuous Optimisation Benchmark Suite from Neural Network Regression

A Continuous Optimisation Benchmark Suite from Neural Network Regression

An Enhanced LSTM for Trend Following of Time Series

A Continuous Optimisation Benchmark Suite from Neural Network Regression

Contact Info

Product

Resources

About