Parameters optimization of deep learning models using Particle swarm optimization

Abstract

Deep learning has been successfully applied in several fields such as machine translation, manufacturing, and pattern recognition. However, successful application of deep learning depends upon appropriately setting its parameters to achieve high-quality results. The number of hidden layers and the number of neurons in each layer of a deep machine learning network are two key parameters, which have main influence on the performance of the algorithm. Manual parameter setting and grid search approaches somewhat ease the users' tasks in setting these important parameters. Nonetheless, these two techniques can be very time-consuming. In this paper, we show that the Particle swarm optimization (PSO) technique holds great potential to optimize parameter settings and thus saves valuable computational resources during the tuning process of deep learning models. Specifically, we use a dataset collected from a Wi-Fi campus network to train deep learning models to predict the number of occupants and their locations. Our preliminary experimentsindicate that PSO provides an efficient approach for tuning the optimal number of hidden layers and the number of neurons in each layer of the deep learning algorithm when compared to the grid search method. Our experiments illustrate that the exploration process of the landscape of configurations to find the optimal parameters is decreased by 77 % -85%. In fact, the PSO yields even better accuracy results.

A Novel Approach in Determining Neural Networks Architecture to Classify Data With Large Number of Attributes

“…Former literature solely discusses on how to determine the number of hidden neurons (assuming by using one hidden layer), but does rarely discuss the way to determine the optimal number of hidden layers. This fact is due to the assumption that a network with only one hidden layer is sufficient to universally approve almost all functions [3]- [7]. However, several studies probed that the application of two hidden layers provides better performance compare to one hidden layer in some cases [3]- [7].…”

“…Since the increasing ability of computers, the application of neural networks with more than one hidden layer has become one of the attractions for researchers, especially since the use of deep neural networks (DNN) to solve problems in various fields. The utilization of DNN is defined as a technique applying neural networks by using numerous hidden layers between the input and output layers [3], [10]; or in other words it is considered as machine learning with DNN. One of the challenges in the successful implementation of deep neural networks lies on the determination of the network architecture, which is closely related to the number of hidden layers and hidden neurons.…”

Optimizing Convolutional Neural Network Hyperparameters by Enhanced Swarm Intelligence Metaheuristics

“…Please refer to Table 2. [16,2048] With the objectives of reducing the amount of parameters (weights), the computational burden, and to control overfitting, for each convolutional layer, we used a pooling (downsampling) layer of a fixed size of 2 × 2 along with the max pooling option. Further, we utilized the ReLU (rectified linear unit) activation function and employed the Adam optimizer with a fixed learning rate of θ = 0.0001 and the mean squared error (MSE) loss function.…”

Optimization of culture conditions for differentiation of melon based on artificial neural network and genetic algorithm

“…GA have many successful applications in the domains of deep learning and CNNs 17,18 . Better results can be obtained by applying meta-heuristic methods such as genetic algorithm (GA) 17 and swarm intelligence 27 to the process of CNN hyperparameter optimization.…”