Hyperparameter Tuning for Deep Neural Networks Based Optimization Algorithm

Abstract

For training the present Neural Network (NN) models, the standard technique is to utilize decaying Learning Rates (LR). While the majority of these techniques commence with a large LR, they will decay multiple times over time. Decaying has been proved to enhance generalization as well as optimization.Other parameters, such as the network's size, the number of hidden layers, dropouts to avoid overfitting, batch size, and so on, are solely based on heuristics. This work has proposed Adaptive Teaching Learning Based (ATLB) Heuristic to identify the optimal hyperparameters for diverse networks. Here we consider three architectures Recurrent Neural Networks (RNN), Long Short Term Memory (LSTM), Bidirectional Long Short Term Memory (BiLSTM) of Deep Neural Networks for classification. The evaluation of the proposed ATLB is done through the various learning rate schedulers Cyclical Learning Rate (CLR), Hyperbolic Tangent Decay (HTD), and Toggle between Hyperbolic Tangent Decay and Triangular mode with Restarts (T-HTR) techniques. Experimental results have shown the performance improvement on the 20Newsgroup, Reuters Newswire and IMDB dataset.

An Assessment of Contemporary Methods and Data-Enabled Approaches for Early Cataract Detection

“…Absence of External Validation: Some studies relied solely on the same dataset for training and testing without utilising external datasets for model validation. 36 This lack of external validation can lead to an overestimation of the model's performance and restrict the applicability of the findings. Inadequate Comparative Analysis: While Cheng et al 97 conducted a comparison between deep learning and traditional machine learning algorithms, many publications did not evaluate their proposed approaches against established procedures or alternative algorithms.…”

“…Improving the algorithm's performance entails meticulous adjustments through hyperparameter tuning and careful exploration of the hyperparameter space using approaches such as grid search, random search, and Bayesian optimisation. 35,36 Data collection and preparation, designing an appropriate model architecture, separating data for training and validation, optimising hyperparameters, and conducting model training and validation are all critical aspects for developing improved cataract detection models. 35,37 These elements are critical for improving the effectiveness of machine-learning models.…”

Credit Card Fraud Detection Using Improved Deep Learning Models

“…Random search performs a randomized search over hyperparameters from certain distributions over possible hyperparameter values. The search process continues until the desired metric is reached or until the predetermined computational budget is exhausted [36,37].…”