Robust Stochastic Gradient Descent With Student-t Distribution Based First-Order Momentum

Ilboudo, Wendyam Eric Lionel; Kobayashi, Taisuke; Sugimoto, Kenji

doi:10.1109/tnnls.2020.3041755

Cited by 46 publications

(32 citation statements)

References 24 publications

(34 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, GD [19], SGD [16], RMSProp [36], and Adam [18] are typical first-order methods. Furthermore, Yogi [21], Fromage [22], diffGrad [23], and TAdam [24] also were designed based on the first-order methods.…”

Section: Overview Of Optimization Methods For Machine Learningmentioning

confidence: 99%

“…Actually, there exist many methods to implement Equation (11). Among them, we compute Equation (11) according to the methods shown in [24] as…”

Section: Adaptive Coefficient Computation Methods For the Robust First Momentummentioning

confidence: 99%

“…Fromage [22] proposed a method to utilize the Euclidean distance to compute a difference between two gradients when computing the next search direction based on them. Meanwhile, TAdam [24] showed how to detect an outlier of gradients and estimate its coefficient adaptively when computing the first-momentum of gradients. For this, they derived the formulae to compute the firstmomentum from a Student-t distribution using its maximum-likelihood estimation and utilized them to derive the first-momentum.…”

Section: Algorithm 1: a Pseudocode Of Adam Optimization Algorithmmentioning

confidence: 99%

“…Accordingly, recent many studies have focused on improving the performance of Adam optimizer further or combining it with other optimization methods [19]. As the most representative cases, Adagrad [20], Yogi [21], Fromage [22], diffGrad [23], and TAdam [24] have successfully been proposed recently. However, they still have several defects in training latest CNNs.…”

Section: Introductionmentioning

confidence: 99%

“…First, they suffer from determining an optimal search velocity at each training step, which often causes the overfitting problem or worsens their training and test accuracies [25,26]. Second, the existing momentum used in Adam optimizer can easily be skewed toward an inaccurate search direction made by outlier gradients [24]. Third, they often fail to find the approximate optimal weights because they have difficulty identifying the current state of the optimization terrain in the solution space spanned by the weights.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

HyAdamC: A New Adam-Based Hybrid Optimization Algorithm for Convolution Neural Networks

Kim

Choi

2021

Sensors

View full text Add to dashboard Cite

As the performance of devices that conduct large-scale computations has been rapidly improved, various deep learning models have been successfully utilized in various applications. Particularly, convolution neural networks (CNN) have shown remarkable performance in image processing tasks such as image classification and segmentation. Accordingly, more stable and robust optimization methods are required to effectively train them. However, the traditional optimizers used in deep learning still have unsatisfactory training performance for the models with many layers and weights. Accordingly, in this paper, we propose a new Adam-based hybrid optimization method called HyAdamC for training CNNs effectively. HyAdamC uses three new velocity control functions to adjust its search strength carefully in term of initial, short, and long-term velocities. Moreover, HyAdamC utilizes an adaptive coefficient computation method to prevent that a search direction determined by the first momentum is distorted by any outlier gradients. Then, these are combined into one hybrid method. In our experiments, HyAdamC showed not only notable test accuracies but also significantly stable and robust optimization abilities when training various CNN models. Furthermore, we also found that HyAdamC could be applied into not only image classification and image segmentation tasks.

show abstract

Section: Overview Of Optimization Methods For Machine Learningmentioning

confidence: 99%

“…Actually, there exist many methods to implement Equation (11). Among them, we compute Equation (11) according to the methods shown in [24] as…”

Section: Adaptive Coefficient Computation Methods For the Robust First Momentummentioning

confidence: 99%

Section: Algorithm 1: a Pseudocode Of Adam Optimization Algorithmmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

HyAdamC: A New Adam-Based Hybrid Optimization Algorithm for Convolution Neural Networks

Kim

Choi

2021

Sensors

View full text Add to dashboard Cite

show abstract

Nonlinear spline prioritization optimization generalized hyperbolic secant adaptive filtering against alpha-stable noise

Guo

Zhi

2023

Nonlinear Dyn

View full text Add to dashboard Cite

SIF-DWTRL: solar irradiation forecasting using discrete wavelet transform and regression learning

Baghel,

Gosh,

Patidar

et al. 2023

Energy Syst

View full text Add to dashboard Cite

Robust Stochastic Gradient Descent With Student-t Distribution Based First-Order Momentum

Cited by 46 publications

References 24 publications

HyAdamC: A New Adam-Based Hybrid Optimization Algorithm for Convolution Neural Networks

HyAdamC: A New Adam-Based Hybrid Optimization Algorithm for Convolution Neural Networks

Nonlinear spline prioritization optimization generalized hyperbolic secant adaptive filtering against alpha-stable noise

SIF-DWTRL: solar irradiation forecasting using discrete wavelet transform and regression learning

Contact Info

Product

Resources

About