Calibrating the adaptive learning rate to improve convergence of ADAM

Tong, Qianqian; Liang, Guannan; Bi, Jinbo

doi:10.1016/j.neucom.2022.01.014

Cited by 36 publications

(16 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…input layer and hidden layers) based on a given dropout probability in every iteration. The ‘learning rate’ of a DNN represents the amount of changes needed at every weight update of the model based on the estimated error [ 80 ]. The convergence of the model for an optimal solution depends on the learning rate.…”

Section: Resultsmentioning

confidence: 99%

Interactive framework for Covid-19 detection and segmentation with feedback facility for dynamically improved accuracy and trust

et al. 2022

View full text Add to dashboard Cite

Due to the severity and speed of spread of the ongoing Covid-19 pandemic, fast but accurate diagnosis of Covid-19 patients has become a crucial task. Achievements in this respect might enlighten future efforts for the containment of other possible pandemics. Researchers from various fields have been trying to provide novel ideas for models or systems to identify Covid-19 patients from different medical and non-medical data. AI-based researchers have also been trying to contribute to this area by mostly providing novel approaches of automated systems using convolutional neural network (CNN) and deep neural network (DNN) for Covid-19 detection and diagnosis. Due to the efficiency of deep learning (DL) and transfer learning (TL) models in classification and segmentation tasks, most of the recent AI-based researches proposed various DL and TL models for Covid-19 detection and infected region segmentation from chest medical images like X-rays or CT images. This paper describes a web-based application framework for Covid-19 lung infection detection and segmentation. The proposed framework is characterized by a feedback mechanism for self learning and tuning. It uses variations of three popular DL models, namely Mask R-CNN, U-Net, and U-Net++. The models were trained, evaluated and tested using CT images of Covid patients which were collected from two different sources. The web application provide a simple user friendly interface to process the CT images from various resources using the chosen models, thresholds and other parameters to generate the decisions on detection and segmentation. The models achieve high performance scores for Dice similarity, Jaccard similarity, accuracy, loss, and precision values. The U-Net model outperformed the other models with more than 98% accuracy.

show abstract

Section: Resultsmentioning

confidence: 99%

Interactive framework for Covid-19 detection and segmentation with feedback facility for dynamically improved accuracy and trust

et al. 2022

View full text Add to dashboard Cite

show abstract

“…As a hyper-parameter, the learning rate of SGD is often difficult to tune because the magnitudes of multiple parameters change greatly, and adjustment is required during the training process. Several adaptive gradient descent variants have been created to address this problem, including Adaptive Moment Estimation (Adam) [ 115 ], RMSprop [ 116 ], Ranger [ 117 ], Momentum [ 118 ], and Nesterov [ 119 ]. These algorithms automatically adapt the learning rate to different parameters, based on the statistics of gradient leading to faster convergence, simplifying learning strategies, and have been seen in many neural networks applied to CEA applications, as demonstrated in Figure 11 .…”

Section: Discussionmentioning

confidence: 99%

Deep Learning in Controlled Environment Agriculture: A Review of Recent Advancements, Challenges and Prospects

Ojo

Zahid

2022

Sensors

View full text Add to dashboard Cite

Controlled environment agriculture (CEA) is an unconventional production system that is resource efficient, uses less space, and produces higher yields. Deep learning (DL) has recently been introduced in CEA for different applications including crop monitoring, detecting biotic and abiotic stresses, irrigation, microclimate prediction, energy efficient controls, and crop growth prediction. However, no review study assess DL’s state of the art to solve diverse problems in CEA. To fill this gap, we systematically reviewed DL methods applied to CEA. The review framework was established by following a series of inclusion and exclusion criteria. After extensive screening, we reviewed a total of 72 studies to extract the useful information. The key contributions of this article are the following: an overview of DL applications in different CEA facilities, including greenhouse, plant factory, and vertical farm, is presented. We found that majority of the studies are focused on DL applications in greenhouses (82%), with the primary application as yield estimation (31%) and growth monitoring (21%). We also analyzed commonly used DL models, evaluation parameters, and optimizers in CEA production. From the analysis, we found that convolutional neural network (CNN) is the most widely used DL model (79%), Adaptive Moment Estimation (Adam) is the widely used optimizer (53%), and accuracy is the widely used evaluation parameter (21%). Interestingly, all studies focused on DL for the microclimate of CEA used RMSE as a model evaluation parameter. In the end, we also discussed the current challenges and future research directions in this domain.

show abstract

“…We took the mean over pings for all loss terms. We took the mean over the batch dimension; for outputs conditioned on the orientation of the echosounder, we masked out irrelevant samples The model was optimized using the RangerVA optimizer (Wright, 2019), which combines RAdam, Lookahead, and gradient centralization (Zhang et al, 2019;Liu et al, 2020;Yong et al, 2020;Tong et al, 2022), with a weight decay of 1 × 10 −5 . We used a batch size of 12 samples, and stratified the batches to contain the same ratio of downfacing and upfacing samples as available in the aggregated training set.…”

Section: Model Trainingmentioning

confidence: 99%

Echofilter: A Deep Learning Segmention Model Improves the Automation, Standardization, and Timeliness for Post-Processing Echosounder Data in Tidal Energy Streams

Lowe

McGarry²,

Douglas³

et al. 2022

Front. Mar. Sci.

View full text Add to dashboard Cite

Understanding the abundance and distribution of fish in tidal energy streams is important for assessing the risks presented by the introduction of tidal energy devices into the habitat. However, tidal current flows suitable for tidal energy development are often highly turbulent and entrain air into the water, complicating the interpretation of echosounder data. The portion of the water column contaminated by returns from entrained air must be excluded from data used for biological analyses. Application of a single algorithm to identify the depth-of-penetration of entrained air is insufficient for a boundary that is discontinuous, depth-dynamic, porous, and varies with tidal flow speed.Using a case study at a tidal energy demonstration site in the Bay of Fundy, we describe the development and application of deep machine learning models with a U-Net based architecture that produce a pronounced and substantial improvement in the automated detection of the extent to which entrained air has penetrated the water column.Our model, Echofilter, was found to be highly responsive to the dynamic range of turbulence conditions and sensitive to the fine-scale nuances in the boundary position, producing an entrained-air boundary line with an average error of 0.33 m on mobile downfacing and 0.5–1.0 m on stationary upfacing data, less than half that of existing algorithmic solutions. The model’s overall annotations had a high level of agreement with the human segmentation, with an intersection-over-union score of 99% for mobile downfacing recordings and 92–95% for stationary upfacing recordings. This resulted in a 50% reduction in the time required for manual edits when compared to the time required to manually edit the line placement produced by the currently available algorithms. Because of the improved initial automated placement, the implementation of the models permits an increase in the standardization and repeatability of line placement.

show abstract

Calibrating the adaptive learning rate to improve convergence of ADAM

Cited by 36 publications

References 2 publications

Interactive framework for Covid-19 detection and segmentation with feedback facility for dynamically improved accuracy and trust

Interactive framework for Covid-19 detection and segmentation with feedback facility for dynamically improved accuracy and trust

Deep Learning in Controlled Environment Agriculture: A Review of Recent Advancements, Challenges and Prospects

Echofilter: A Deep Learning Segmention Model Improves the Automation, Standardization, and Timeliness for Post-Processing Echosounder Data in Tidal Energy Streams

Contact Info

Product

Resources

About