An efficient modified Hyperband and trust-region-based mode-pursuing sampling hybrid method for hyperparameter optimization

Lin, Jingliang; Li, Haiyan; Huang, Yunbao; Chen, Jinghuan; Huang, Pengcheng; Huang, Zeying

doi:10.1080/0305215x.2020.1862823

Cited by 24 publications

(28 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The tuning was performed using the Keras Tuner ( https://keras.io/keras_tuner ), which provides a framework to apply different search algorithms. Both the Hyperband ( Li et al , 2016 ) and the Bayesian Optimization ( Snoek et al , 2012 ) tuning algorithms were used. Utilizing the two methods in combination has been shown to outperform using them separately as, ‘bandit-based approaches’ (like Hyperband) ‘lack guidance’, whereas Bayesian optimization across the entire search space can be ‘computationally infeasible’ ( Falkner et al , 2018 ).…”

Section: Methodsmentioning

confidence: 99%

LMPred: predicting antimicrobial peptides using pre-trained language models and deep learning

Dee

2022

Bioinformatics Advances

View full text Add to dashboard Cite

Motivation Antimicrobial peptides (AMPs) are increasingly being used in the development of new therapeutic drugs in areas such as cancer therapy and hypertension. Additionally, they are seen as an alternative to antibiotics due to the increasing occurrence of bacterial resistance. Wet-laboratory experimental identification, however, is both time consuming and costly, so in-silico models are now commonly used in order to screen new AMP candidates. Results This paper proposes a novel approach for creating model inputs; using pre-trained language models to produce contextualized embeddings, representing the amino acids within each peptide sequence, before a convolutional neural network is trained as the classifier. The results were validated on two datasets - one previously used in AMP prediction research, and a larger independent dataset created by this paper. Predictive accuracies of 93.33% and 88.26% were achieved respectively, outperforming previous state-of-the-art classification models. Availability and implementation All code is available and can be accessed here: https://github.com/williamdee1/LMPred_AMP_Prediction. Supplementary information Supplementary data are available at Bioinformatics online.

show abstract

Section: Methodsmentioning

confidence: 99%

LMPred: predicting antimicrobial peptides using pre-trained language models and deep learning

Dee

2022

Bioinformatics Advances

View full text Add to dashboard Cite

show abstract

“…, S is the number of trials The problem of Eq. ( 8), also known as Hyper-parameter optimization of deep learning algorithm [50], [51], [61], has been well studied in the machine learning community, but may not be well known in the engineering optimization community. To facilitate engineering application, a general framework for deep model training is given in this paper, as shown in Fig.…”

Section: General Framework For Deep Model Trainingmentioning

confidence: 99%

“…When the dimension is high or each dimension needs to be carefully divided, S is a very large number, and some brute-force search algorithms [62] such as Grid search and Random search will be difficult to cope with. In this case, heuristic hyper-parameter optimization algorithms such as Hyperband [52], Bayesian optimization [51], [63], or hybrid methods [50], [64] should be used. Whichever algorithm is chosen, it can be easily incorporated into the framework of Fig.…”

Section: General Framework For Deep Model Trainingmentioning

confidence: 99%

“…The training algorithm of deep neural networks has been widely studied in recent years. From different perspectives, the gradient descent algorithm [46], [47], the multi-task learning algorithm [48], [49], and the hyper-parameter optimization algorithm [50], [51], [52] are often involved, but there is a lack of a general framework to apply them to engineering practice. Compared with training algorithms, little attention is paid to data planning for transfer learning, and the commonly used random sampling has been considered insufficient to effectively promote the adaptability of the deep model [41], [43].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Multi-Objective Surrogate Modeling Through Transfer Learning for Telescopic Boom Forklift

Lin

Huang

et al. 2023

IEEE Access

View full text Add to dashboard Cite

Simulation and optimization methods have been widely used in forklift design due to their cost-effectiveness. However, this type of method involves challenges such as the accuracy of the simulation model and the simulation solution time. These challenges reduce the stability and precision of the surrogate model and hence generate further optimization errors. In this paper, a multi-objective surrogate modeling (MSM) method for telescopic boom forklifts based on closed-loop transfer learning is proposed in order to solve these challenges. The MSM consists of the following two steps: to pre-train an initial deep neural network model (deep model) with a large amount of existing simulation data from the same type of forklift and to transfer the model with a small amount of measurement data collected on the current forklift. A general framework for deep neural network (DNN) training is introduced to improve the approximation ability of the initial model. Moreover, a novel uncertainty-analysis-based sampling method is suggested for measurement data development, and combined with transfer learning to form a closed-loop mode to improve the stability of the final model. The superiority of MSM is demonstrated through comparative studies with the fine-tuning method on a telescopic boom forklift with two objectives. The experimental results show that the Correlation coefficient (R) of the deep model can reach 0.9971 by using only 80 sets of training data. In addition, it can also achieve an improvement of at least a 13.25% reduction in Root Mean Squared Error (RMSE) and a 9.19% reduction on average in Maximum Absolute Error (MAE), as well as stronger robustness compared to the benchmarks. Furthermore, it will provide a valuable reference for the simulation optimization of complex electromechanical products.INDEX TERMS Telescopic boom forklift, surrogate model, transfer learning, uncertainty analysis, hyperparameter optimization.

show abstract

“…They showed that their algorithm outperformed genetic CNN by 11.74% on an amyloid brain image dataset used for Alzheimer's disease diagnosis. Lin et al [24] proposed a hybrid approach by combining a modified hyperband, mode-pursuing sampling in the trust-region, and repeating the selection and sampling process until a termination criterion is met for hyperparameter optimization. Shaziya et al [25]  ISSN: 2088-8708 Int J Elec & Comp Eng, Vol.…”

Section: Related Workmentioning

confidence: 99%

Hyperparameter optimization using custom genetic algorithm for classification of benign and malicious traffic on internet of things–23 dataset

Thavasimani¹,

Srinath²

2022

IJECE

View full text Add to dashboard Cite

Hyperparameter optimization is one of the main challenges in deep learning despite its successful exploration in many areas such as image classification, speech recognition, natural language processing, and fraud detections. Hyperparameters are critical as they control the learning rate of a model and should be tuned to improve performance. Tuning the hyperparameters manually with default values is a challenging and time-intensive task. Though the time and efforts spent on tuning the hyperparameters are decreasing, it is always a burden when it comes to a new dataset or solving a new task or improving the existing model. In our paper, we propose a custom genetic algorithm to auto-tune the hyperparameters of the deep learning sequential model to classify benign and malicious traffic from internet of things-23 dataset captured by Czech Technical University, Czech Republic. The dataset is a collection of 30.85 million records of malicious and benign traffic. The experimental results show a promising outcome of 98.9% accuracy.

show abstract

An efficient modified Hyperband and trust-region-based mode-pursuing sampling hybrid method for hyperparameter optimization

Cited by 24 publications

References 16 publications

LMPred: predicting antimicrobial peptides using pre-trained language models and deep learning

LMPred: predicting antimicrobial peptides using pre-trained language models and deep learning

Multi-Objective Surrogate Modeling Through Transfer Learning for Telescopic Boom Forklift

Hyperparameter optimization using custom genetic algorithm for classification of benign and malicious traffic on internet of things–23 dataset

Contact Info

Product

Resources

About