A survey of regularization strategies for deep models

Moradi, Reza; Berangi, Reza; Minaei, Behrouz

doi:10.1007/s10462-019-09784-7

Cited by 117 publications

(61 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…the mean of outputs of ensemble method) is likely to be accurate. In realising the ensemble method, we utilise the dropout technique [21], which is widely used in deep learning models. The dropout technique is commonly used to solve the overfitting problem, typically observed in training a deep learning model.…”

Section: Dropout-based Ensemble Methodsmentioning

confidence: 99%

“…if a standard deviation of the distribution is small, an uncertainty for that forecast is low and that forecast is likely to be accurate). We realise the ensemble method by using the dropout technique [21], which is widely used in deep learning forecasting models. Unlike a typical deep learning process that applies the dropout technique only for training a model, we adopt the dropout technique also for a test and that approach allows us to have multiple models and corresponding outputs for the same input.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Forecast uncertainty‐based performance degradation diagnosis of solar PV systems

Lee

2020

IET Renewable Power Generation

View full text Add to dashboard Cite

In this study, the authors are interested in estimating how much a PV system underperforms than expected by exploiting forecast uncertainty. For this, they first study a forecast accuracy-related forecast uncertainty metric using the ensemble method based on the dropout technique, which is widely used in deep learning forecasting models. Given the forecast accuracy-related uncertainty metric, the rationale of the authors' approach is that forecast accuracy is likely to decrease compared to the normal case of similar uncertainty metric values if any performance degradation happens. It is because similar uncertainty metric values are likely to show similar forecast accuracy. Therefore, they generate a standard table by simulating possible performance degradation cases and conduct the performance degradation diagnosis by looking up the standard table based on the uncertainty metric. From the experiments, in the case of persistent degradation, they show that their approach estimates the performance degradation with the estimation error of around 1% while an uncertainty-unaware approach shows the estimation error of up to 5%. In the case of temporal degradation, their approach shows the estimation error of around 3%, while the uncertainty-unaware approach does not show meaningful result.

show abstract

Section: Dropout-based Ensemble Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Forecast uncertainty‐based performance degradation diagnosis of solar PV systems

Lee

2020

IET Renewable Power Generation

View full text Add to dashboard Cite

show abstract

“…Since excessive increase in model complexity may also result in overfitting, several regularization techniques can be used to improve model generalizability, such as L 1 and L 2 regularization, batchnormalization, dropout, early stopping, and data augmentation techniques. These techniques can be combined to take advantage of the complementary effects of different approaches, as detailed in a comprehensive overview [53] of the most frequently adopted regularization techniques and of their effects on DL model performance.…”

Section: Deep Learning Modelsmentioning

confidence: 99%

AI applications to medical images: From machine learning to deep learning

Rundo²,

et al. 2021

View full text Add to dashboard Cite

Artificial intelligence (AI) models are playing an increasing role in biomedical research and healthcare services. This review focuses on challenges points to be clarified about how to develop AI applications as clinical decision support systems in the real-world context. Methods: A narrative review has been performed including a critical assessment of articles published between 1989 and 2021 that guided challenging sections. Results: We first illustrate the architectural characteristics of machine learning (ML)/radiomics and deep learning (DL) approaches. For ML/radiomics, the phases of feature selection and of training, validation, and testing are described. DL models are presented as multi-layered artificial/convolutional neural networks, allowing us to directly process images. The data curation section includes technical steps such as image labelling, image annotation (with segmentation as a crucial step in radiomics), data harmonization (enabling compensation for differences in imaging protocols that typically generate noise in non-AI imaging studies) and federated learning. Thereafter, we dedicate specific sections to: sample size calculation, considering multiple testing in AI approaches; procedures for data augmentation to work with limited and unbalanced datasets; and the interpretability of AI models (the so-called black box issue). Pros and cons for choosing ML versus DL to implement AI applications to medical imaging are finally presented in a synoptic way. Conclusions: Biomedicine and healthcare systems are one of the most important fields for AI applications and medical imaging is probably the most suitable and promising domain. Clarification of specific challenging points facilitates the development of such systems and their translation to clinical practice.

show abstract

“…When more training leads to an improvement in performance on the training dataset but an otherwise worsening of the performance on the validation dataset, this is a sign that overfitting is occurring which can be typically visualised by plotting so-called loss curves over training time. Overfitting may be prevented by increasing the training dataset’s diversity using, for instance, data augmentation 44 , 45 or using strategies such as reducing the model complexity, adding regularisation (L1, L2) or early stopping during training 46 . DL tools dedicated to training would enormously benefit from these features as these simplify the assessment and potential improvement on model optimisation for the user.…”

Section: Choosing a DL Toolmentioning

confidence: 99%

Avoiding a replication crisis in deep-learning-based bioimage analysis

et al. 2021

View full text Add to dashboard Cite

Avoiding a replication crisis in deep-learningbased bioimage analysisDeep learning algorithms are powerful tools for analyzing, restoring and transforming bioimaging data. One promise of deep learning is parameter-free one-click image analysis with expert-level performance in a fraction of the time previously required. However, as with most emerging technologies, the potential for inappropriate use is raising concerns among the research community. In this comment, we discuss key concepts that we believe are important for researchers to consider when using deep learning for their microscopy studies. We describe how results obtained using deep learning can be validated and propose what should, in our view, be considered when choosing a suitable tool. We also suggest what aspects of a deep learning analysis should be reported in publications to ensure reproducibility. We hope this perspective will foster further discussion among developers, image analysis specialists, users and journal editors to define adequate guidelines and ensure the appropriate use of this transformative technology.

show abstract

A survey of regularization strategies for deep models

Cited by 117 publications

References 28 publications

Forecast uncertainty‐based performance degradation diagnosis of solar PV systems

Forecast uncertainty‐based performance degradation diagnosis of solar PV systems

AI applications to medical images: From machine learning to deep learning

Avoiding a replication crisis in deep-learning-based bioimage analysis

Contact Info

Product

Resources

About