Hyperparameter Optimization Techniques for Designing Software Sensors Based on Artificial Neural Networks

Abstract

Software sensors are playing an increasingly important role in current vehicle development. Such soft sensors can be based on both physical modeling and data-based modeling. Data-driven modeling is based on building a model purely on captured data which means that no system knowledge is required for the application. At the same time, hyperparameters have a particularly large influence on the quality of the model. These parameters influence the architecture and the training process of the machine learning algorithm. This paper deals with the comparison of different hyperparameter optimization methods for the design of a roll angle estimator based on an artificial neural network. The comparison is drawn based on a pre-generated simulation data set created with ISO standard driving maneuvers. Four different optimization methods are used for the comparison. Random Search and Hyperband are two similar methods based purely on randomness, whereas Bayesian Optimization and the genetic algorithm are knowledge-based methods, i.e., they process information from previous iterations. The objective function for all optimization methods consists of the root mean square error of the training process and the reference data generated in the simulation. To guarantee a meaningful result, k-fold cross-validation is integrated for the training process. Finally, all methods are applied to the predefined parameter space. It is shown that the knowledge-based methods lead to better results. In particular, the Genetic Algorithm leads to promising solutions in this application.

Preliminary Classification of Selected Farmland Habitats in Ireland Using Deep Neural Networks

“…Each model was trained with different optimizers by varying the learning rates and the best-fit hyperparameters were chosen as the ones with the highest validation accuracy. Based on previous work in image classification, the four most suitable optimizers, namely, Adam, RMSProp, SGD, and Nadam were chosen for tuning each model [ 70 , 71 , 72 ]. The learning rate was varied from 10 −2 to 10 −5 to determine the best fit.…”

Modeling Multioutput Response Uses Ridge Regression and MLP Neural Network with Tuning Hyperparameter through Cross Validation

“…Meanwhile, van de Wiel et al [30] proposed fast hyperparameter tuning, and Meanti et al [31] proposed Efficient Hyperparameter tuning in the kernel of ridge regression based on cross-validation of data. Tuning hyperparameters on a neural network model using crossvalidation data, among others, was carried out by Blume et al [32], and also by Lainder et al [33]. Although there is controversy over the advantages and disadvantages of applying the cross-validation method to set up model hyperparameters, this method is systematic and fair.…”

Artificial Neural Network Hyperparameters Optimization: A Survey

“…Those two techniques are effective in prediction and classification. The application in [37], [68], [74], [80], [81], [83] used GS and RS as hyperparameter selection methods. The GS explored all hyperparameters combinations in the search space, but it is expensive and not efficient if the search space has high dimensions, while RS from its name selected hyperparameters randomly and considered as the most efficient way of searching the hyperparameters configurations [5], [96], [100].…”

“…The second problem accurse is filling in local minimum the most active hyperparameter in this case is momentum coefficient just like problem that showed in [73]. In [4], [78], [81], [83] the batch size hyper parameter was improved since it represents the number of the samples that have been processed prior to updating the model and the number of the complete passes through the whole training dataset in cases of a large dataset. Furthermore, the learning algorithm's dynamics are influenced by an important hyperparameter.…”